Random Hangs of emonPi/emonSD

Does it make sense to increase the swap file?

What would happen if you disable the swap file on a 512MB RAM device? I heard/read that the swap file could reduce the SD lifetime. I use a genuine Sandisk Extreme with A2 rating for app performance.

@TrystanLea Looks like there is a memory leak somewhere.

This is an old PiZ running 10.2.6

@fluppie007 That shouldn’t be happening.Have you installed anything in addition to emoncms on the Pi?

Could you post an htop output sorted on MEM%

No. I think the emonSD setup is such as to ensure the SWAP is only in RAM so as to preserve the SDCard.

htop

Interesting re full swap, not something I’ve seen before.

Thanks for the htop @fluppie007 looks like something is going wrong there with mysql , don’t think there should be so many processes, perhaps just processes piling up due to laggy disk access caused by the swap issue?

What does it look like after a reboot?

Do you have any other software running on the pi zero apart from that installed by emonscripts?

Nope it’s a dedicated emonCMS zero 2 w pi. 64GB Sandisk Extreme A2 micro SD.
16h uptime:

After reboot:

Thanks! I think Im only seeing a single mysql process on mine,
What do you see with:

ps aux | grep mysql

This is what I see (just a single entry):

mysql      665  0.1  6.5 726140 61760 ?        Ssl  Jul22  29:00 /usr/sbin/mysqld

After 22h uptime:

pi@emonrpiz2w:~ $ ps aux | grep mysql
mysql 599 0.1 13.9 725048 61052 ? Ssl Aug01 2:09 /usr/sbin/mysqld
pi 24745 0.0 0.1 7344 580 pts/0 S+ 12:56 0:00 grep --color=auto mysql
pi@emonrpiz2w:~ $

I do have three RPi4’s with HomeAssistant sending some energy information every 5 minutes to emonCMS with the Emoncms History - Home Assistant integration.

But the RPi Z2W does not have anything else than emonCMS running.


Swapfile starts to fill, I expect lock-up in 1-2 days, maybe 3 days.

Note in htop it is the same PID so ‘ps’ will only list one. I see multiple
htop Entries

[edit]
htop shows threads not processes (I think).

Think we are close to a freeze, system gets slower and swap file is filling up.

On the original question I raised…my emonPi has been running without any issue for a month now. Still using the same PSU, and things like swap space look OK. Still mystified as to what causes the intermittent hangs :slight_smile:

Good to hear, myself I think I’ll have a lockup overnight. Swap is full and RAM usage is increasing.


After my last post, it took 8 hours for the Rpi Zero 2 W to hang/get stuck. When I took the module to reset, it was very hot, even though it’s in a alu case for cooling. Also the green LED was blinking instead of continous/solid green.
So I’m pretty sure it’s related to the swap file filling up to 100MB?

Doesn’t explian why though.

A quick google and I found this. https://www.cyberciti.biz/faq/linux-which-process-is-using-swap/

Try this command - it lists what is using Swap (i think)

for file in /proc/*/status ; do awk '/VmSwap|Name/{printf $2 " " $3}END{ print ""}' $file; done | sort -k 2 -n -r | less
1 Like

OK, let me wait till swap starts to fill, probably 2-3 days :slight_smile:

1 Like

I wonder if the problem is being caused by mysql or if the backing up mysql threads is a symptom of something else locking disk IO or similar…

Perhaps trying a mysql restart would be worthwhile?

sudo systemctl restart mysql

Might be worth enabling the slow query log:
https://mariadb.com/kb/en/slow-query-log-overview/

Are you using any of the mysql timeseries engines? e.g MYSQL TimeSeries or MYSQL Memory?

1 Like

Hi, been a while. Holidays and so. And also really weird, no hangs anymore…
But something is happening, uptime is never more than a day. And calculating from the uptime. It looks like it’s doing a daily reboot/reset?
Atm 13h23 here and the uptime is 12:22:40 up 1 min, 0 users, load average: 0.31, 0.16, 0.06
Can I check some logs why/if it does daily reboots at ± 1 o’clock at night?

Ok lol, it’s rebooting at exactly 1 am… Any ideas?
Uptime: 11:24:00 up 4:32, 0 users, load average: 0.01, 0.03, 0.01