I’m looking for some guidance on an issue I am having with my emonPi/emonSD - I’m not sure where the problem actually is though!
Recently (maybe every month or so), my emonPi just hangs, stops capturing any feeds or shipping data to EmonCMS. I don’t always notice this for a day or two…invariably I find out it’s happened again when I see the EmonCMS app show me some nonsensical graph.
Once it gets into this state, I am always able to ping the device and access the emonSD admin screen etc. If I reboot the device, when it comes back there is still no monitoring of anything…and I actually have to shutdown the device, disconnect the power to do a full cold reboot of the device…at which point everything then comes back working normally and monitoring starts again.
I’m not sure where to start hunting down the issue here…any guidance or thoughts?
Sadly, I don’t seen anything of note in any of the log files. The last ‘hang’ took place sometime on the morning July 2nd…and there’s nothing jumping out at me in any of the logs around that timeframe to say there’s been an issue.
Sorry for being so imprecise. There are a variety of logs in the /var/log.old directory, but in the emonhub directory, there are only more recent files. In the emoncms log directory there are loads of entries (see image) - but when uncompressed the files all seem empty.
The EmonPi is used with a current clamp, and four wireless temperature sensors. When one of these ‘hang events’ occurs, I can open the emoncms admin page and everything seems to be running…yet no data is being collected from any sensors. Reboot and everything still seems to run, yet still no sensor data is collected. Power down and the start up again, everything comes back successful and data is once again collected. Everything is running OK now…and I’ll probably have to just wait a few weeks until it hangs again to gather some more information.
That sounds like the issue is with the front-end electronics, as rebooting does not reset the
328 microcontroller, but when the power is cycled, both the RPi and the 328 µC get reset.
That’s possible. First thoughts are a power supply issue. It has been known for a brown-out to lock up the RFM69CW, which will in turn hang everything else.
Unfortunately, the reset line on the RFM69CW is not used, so it’s not possible to reset it in software.
I’ve also had this problem where the data logging inexplicably stops and a power cycle is the only fix.
I have learned to live with it, since I was never able to figure out why it does it, and it only occurs two or three times each year,
However, something caught my eye a while back when I was looking at the EmonPi Schematic and I have wondered about it ever since.
I noticed that the ATMega328 microprocessor was running from a 3.3V supply, but has a 16MHz crystal. I’m sure I read somewhere that 16MHz was only supported when running on a 5V supply, If that’s true then I wondered if this was the cause of the spurious behaviour.
I think my emonPi has got into this state maybe 3 times this year so far. It seems unpredictable (as in I can’t correlate any other factors right now) but I’m going to keep a close eye on it from now on.
I could probably change the PSU for one of a number of 5V PSUs I have lying around, if that was a potential causal factor. I could even switch it to an open frame Bel Power linear 5V 3A PSU (that I had planned for my software defined radio gear) but would need to do some work to enclose it first.
It’s the PSU. It’s always the wretched PSU.
Don’t change it for some random 5V PSU, buy a decent RPi PSU. They’re usually a little higher than 5V, to allow for a bit of voltage drop when things get busy.
Had something similar happen to me because the Pi was running out of disk space.
Something in the EmonCMS UI was triggering the web server to write a lot of errors to its log file which would then automatically get cleaned up destroying the evidence of filling up the disk with error logs.
Maybe check it with “df” and make sure you have plenty of space.
Otherwise might need to change some linux logging settings which is what I did to get it to stop logging errors to disk.
Wish I could remember what it was that was causing all those errors. Maybe some kind of graph object or dashboard object that was referencing a feed that I had deleted or renamed or something like that. I remember having a lot of errors about something like that a while ago.
Mine is just plugged into a USB Socket as part of a 13A wall socket It’s rock solid.
Just your luck I think.
This has been a problem in the past, but larger cards mean it is rarely an issue. You can check the available space from the Admin page (which is just a df call IIRC).
On logs, again, a load of work was done to make sure these rotate correctly and saved from the RAMLOG to disk. However, that looks as if it might be broken now.
That is odd. @TrystanLea - this probably needs investigating as it should not be the case…
I fully agree. My Red Pitayas (which I run as software defined radios) have similar problems with flaky 5V PSUs. When the FPGAs are running full tilt, any weak 5v PSU fails pretty quickly (i.e. within a few minutes). To reduce the SMPS induced rfi for my radios, I am switching to Bel Power HB5-3/OVP-AG linear 5V/3A supplies for the Red Pitayas. That might be overkill for the emonpi though
I will secure another good quality RPi PSU in case it’s the PSU. The emonpi has been running for 8 days since the last crash…ad seems OK at the moment.
Have a look at this thread - it describes a similar situation.
I implemented the reset script and set it to trigger once a day from crontab. The problem seems to have gone.
It’s rock solid.
