Random Hangs of emonPi/emonSD

Sidicas · 2022-07-08T16:33:20.966Z

Had something similar happen to me because the Pi was running out of disk space.
Something in the EmonCMS UI was triggering the web server to write a lot of errors to its log file which would then automatically get cleaned up destroying the evidence of filling up the disk with error logs.
Maybe check it with “df” and make sure you have plenty of space.
Otherwise might need to change some linux logging settings which is what I did to get it to stop logging errors to disk.

Wish I could remember what it was that was causing all those errors. Maybe some kind of graph object or dashboard object that was referencing a feed that I had deleted or renamed or something like that. I remember having a lot of errors about something like that a while ago.

borpin · 2022-07-12T11:56:35.886Z

Mine is just plugged into a USB Socket as part of a 13A wall socket It’s rock solid.

Just your luck I think.

Sidicas:

Had something similar happen to me because the Pi was running out of disk space.

This has been a problem in the past, but larger cards mean it is rarely an issue. You can check the available space from the Admin page (which is just a df call IIRC).

On logs, again, a load of work was done to make sure these rotate correctly and saved from the RAMLOG to disk. However, that looks as if it might be broken now.

rdavies6:

In the emoncms log directory there are loads of entries (see image) - but when uncompressed the files all seem empty.

That is odd. @TrystanLea - this probably needs investigating as it should not be the case…

rdavies6 · 2022-07-12T16:51:39.210Z

It looks like there is plenty of diskspace. The emonpi has been up for 8 days since the last hang…I’m keeping an eye on it to see when it happens next

Screenshot 2022-07-12 at 17.47.441032×444 77.2 KB

rdavies6 · 2022-07-12T16:57:58.331Z

I fully agree. My Red Pitayas (which I run as software defined radios) have similar problems with flaky 5V PSUs. When the FPGAs are running full tilt, any weak 5v PSU fails pretty quickly (i.e. within a few minutes). To reduce the SMPS induced rfi for my radios, I am switching to Bel Power HB5-3/OVP-AG linear 5V/3A supplies for the Red Pitayas. That might be overkill for the emonpi though

I will secure another good quality RPi PSU in case it’s the PSU. The emonpi has been running for 8 days since the last crash…ad seems OK at the moment.

mdagli1 · 2022-07-13T20:50:31.110Z

Anyway to log low voltage events? My little USB UPS reduces PSU stresses:

Auto shutdown (when USB supply voltage drops) emonSD

OK, the hardware bits are coming in soon, I can begin assembly and conducting checks today. This is the basic circuit but will need to make refinements as I go along: Feedback welcome! (I’m using KiCad) [Screenshot 2022-03-15 163600] (edit: improved image clarity) (edit2: updated layout)

Guff666 · 2022-07-15T12:23:04.510Z

Have a look at this thread - it describes a similar situation.
I implemented the reset script and set it to trigger once a day from crontab. The problem seems to have gone.

Hanging hardware, requiring a power cycle Hardware

OK, thanks, Brian. Next time it happens, I’ll try the reset script mentioned in the other thread. If this is successful, I can build an auto-responder that kicks in when data stops flowing.

fluppie007 · 2022-07-31T09:05:12.809Z

I experience more or less the same freezes which causes wrong kwh/d processing.
I do use a Raspberry Pi Zero 2 W with emonscripts to install emonCMS.

SysInfo1128×602 54.7 KB

Today my emonCMS was very slow/laggy but I managed to login and check the admin page
After 4 dats of uptime:

Uptime4days1127×877 31.7 KB

After a reboot:

AfterReboot1130×875 30.5 KB

Could it be that the full swap file is causing this? Or the almost maxed out 512MB RAM from the RPi Z2W?

fluppie007 · 2022-07-31T12:44:47.859Z

Does it make sense to increase the swap file?

PiCockpit | Monitor and Control your Raspberry Pi: free for up to 5 Pis! – 28 Oct 21

Everything about Raspberry Pi Zero 2 W — PiCockpit | Monitor and Control your...

This is an in-depth review & information resource for the Raspberry Pi Zero 2 W. Specs, Tips, Workarounds, FAQs - everything Zero 2 W related.

Est. reading time: 55 minutes

Pi My Life Up – 27 Jul 20

Increasing the swap size on your Raspberry Pi

The swap file is used as virtual memory

Est. reading time: 3 minutes

What would happen if you disable the swap file on a 512MB RAM device? I heard/read that the swap file could reduce the SD lifetime. I use a genuine Sandisk Extreme with A2 rating for app performance.

borpin · 2022-07-31T20:55:57.166Z

@TrystanLea Looks like there is a memory leak somewhere.

This is an old PiZ running 10.2.6

image1128×613 19 KB

@fluppie007 That shouldn’t be happening.Have you installed anything in addition to emoncms on the Pi?

Could you post an htop output sorted on MEM%

borpin · 2022-07-31T20:57:07.375Z

fluppie007:

Does it make sense to increase the swap file?

No. I think the emonSD setup is such as to ensure the SWAP is only in RAM so as to preserve the SDCard.

fluppie007 · 2022-08-01T10:29:50.120Z

htop

afbeelding1120×911 84.9 KB

TrystanLea · 2022-08-01T10:34:55.626Z

borpin:

Looks like there is a memory leak somewhere.

Interesting re full swap, not something I’ve seen before.

Thanks for the htop @fluppie007 looks like something is going wrong there with mysql , don’t think there should be so many processes, perhaps just processes piling up due to laggy disk access caused by the swap issue?

What does it look like after a reboot?

TrystanLea · 2022-08-01T10:36:38.163Z

Do you have any other software running on the pi zero apart from that installed by emonscripts?

fluppie007 · 2022-08-01T13:49:26.678Z

Nope it’s a dedicated emonCMS zero 2 w pi. 64GB Sandisk Extreme A2 micro SD.
16h uptime:

16h_uptime1122×784 70.4 KB

After reboot:

AfterReboot_htop1133×912 85.8 KB

TrystanLea · 2022-08-02T11:14:26.081Z

Thanks! I think Im only seeing a single mysql process on mine,
What do you see with:

ps aux | grep mysql

This is what I see (just a single entry):

mysql      665  0.1  6.5 726140 61760 ?        Ssl  Jul22  29:00 /usr/sbin/mysqld

fluppie007 · 2022-08-02T11:59:39.902Z

After 22h uptime:

22h_uptime1122×829 77.6 KB

pi@emonrpiz2w:~ $ ps aux | grep mysql
mysql 599 0.1 13.9 725048 61052 ? Ssl Aug01 2:09 /usr/sbin/mysqld
pi 24745 0.0 0.1 7344 580 pts/0 S+ 12:56 0:00 grep --color=auto mysql
pi@emonrpiz2w:~ $

I do have three RPi4’s with HomeAssistant sending some energy information every 5 minutes to emonCMS with the Emoncms History - Home Assistant integration.

But the RPi Z2W does not have anything else than emonCMS running.

fluppie007 · 2022-08-03T16:50:49.860Z

2days_3hours_uptime1118×888 84 KB

Swapfile starts to fill, I expect lock-up in 1-2 days, maybe 3 days.

borpin · 2022-08-03T17:01:09.882Z

TrystanLea:

I think Im only seeing a single mysql process on mine,

Note in htop it is the same PID so ‘ps’ will only list one. I see multiple
htop Entries

[edit]
htop shows threads not processes (I think).

fluppie007 · 2022-08-04T09:27:28.425Z

Think we are close to a freeze, system gets slower and swap file is filling up.

2days_19hours_uptime1170×974 91.9 KB

rdavies6 · 2022-08-04T13:46:54.872Z

On the original question I raised…my emonPi has been running without any issue for a month now. Still using the same PSU, and things like swap space look OK. Still mystified as to what causes the intermittent hangs

Screenshot 2022-08-04 at 14.41.172176×1478 251 KB