Raspberry Pi 4 "Hanging" For No Apparent Reason

Emoncms Installation
1

Hi
I recently upgraded my Emoncms / HomeAssistant server hardware to an Argon ONE M.2 Case, Raspberry Pi 4 and SSD. Software installed includes Raspberry Pi 32bit Lite, Node-RED, Emoncms, Docker + HomeAssistant and Remote.it.

The SSD image was installed using the Raspberry Pi Imager and then connected to the Pi to let it sort out disk size, etc. Emoncms DIY BUILD was installed using the installation script at EmonCMS Install Scripts .
NO emonhub so I changed…

hostname=Pegasus
install_emonhub=false
install_emoncms_emonpi_modules=false
install_firmware=false
install_emonpilcd=false
install_emonsd=false
install_wifiap=false

and commented out:

declare -A emoncms_emonpi_modules (i.e. #declare -A emoncms_emonpi_modules)

I imported a backup from the previous installtion, Emoncms reported low-write 11.3.20 and all seemed to run correctly.

Initially the changeover went well but after ~2 months of working perfectly the system started intermittently “hanging” for no apparent reason (~10 events over the last three of weeks). Each time there was no access to Emoncms or HomeAssistant web pages, no MQTT broker and no SSH. Oddly it still responded to “ping” and the Pi board showed some processor activity (flashing green LED). Upon power cycling (generally the only way out!) it would run correctly again.

The previous system, based on Raspberry Pi 3 + HDD, had worked flawlessly for a few years. I was inclined to think the problem was a hardware issue and am pursuing that with the supplier of the parts.

However, during a rebuild, I noticed errors in the Emoncms installation script output. The attached file shows the issues (just a screen grab as it flew past). Re-installing everything again from scratch gave the same result. I have not noticed these messages before - but maybe I missed them.

ERRORS in emoncms installation script.txt (3.7 KB)

syslog shows a variety of issues but it is difficult to determine cause from effect.

Questions:

  1. Am I doing something wrong in my “NO emonhub” installation?
  2. Is it possible that the installation errors could cause the system to hang?
  3. Is there an installation log file which captures the whole of the installation process?

Thanks in advance for any ideas / help!

2

Hello @Dunum I wonder if this is related to Emoncms_mqtt.php very high ram use - #59 by TrystanLea is memory use by the emoncms_mqtt process increasing up to the point that the system hangs on your system?

3

The recommendation is to not have other software on an emoncms build - it can cause issues and in this case, there is no way of knowing which installed system is causing the issue.

As Trystan said, look at ram use (htop).

Also look at log files as the rotation might not be working correctly.

1 Like
4

Derek, I bought an Argon case and SSD to run an instance of Logitech Media Server. I ran into problems with the HDMI interface and after some investigation, found that some of the HDMI boards were faulty. After a bit of googling, I found who to contact and once I’d sent through the version number of the HDMI board, they sent a replacement.

I contacted → [email protected]. A guy called Kenneth was on support.

Have you tried running the system without the case. You can connect the SSD to the Pi with a USB cable. Maybe there is some issue with the case and the interface boards?

Simon

5

Hello Brian / Trystan

Following your suggestions I’ve done a few rebuilds and made changes to both hardware and software.

I have ended up with Emoncms hosted on different hardware (another Raspberry Pi 4 with a HDD rather than SDD) where Emoncms is the only software on the machine.

Having uses htop during this process, I noted that memory (RAM) usage continually increases, after reboot to the point, presumably, when there is insufficient memory left for the system to run. It then crashes or self reboots.

The only thing which has made any difference was a reduction of mqtt traffic. I changed sensor transmission, from other systems to Emoncms, from mqtt to http; although there remains significant traffic from a Zigbee2mqtt instance and Home-Assistant, sourced from other systems. The reduction slowed down the rate of increase in memory use but didn’t stop it.

The offending process seems to be usr/bin/php at /var/www/emoncms/scripts/services/emoncms_mqtt/emoncms_mqtt.php

MEM% has increased from ~0.6% (after reboot) to 5.8%, of the 3.7GB available, in the last 3 hours.

My suspicion is that the warnings which I noticed in the Emoncms installation script output during rebuilds (as described in my initial post) are something to do with this problem. They have been present every time I have rebuilt the system.

emoncms install script errors 1.txt (16.3 KB)
emoncms install script errors 2.txt (2.3 KB)

Any ideas?

6

Hello @Dunum could you try the following via command line:

cd /var/www/emoncms
git checkout fix_emoncms_mqtt_memory_leak

See Emoncms_mqtt.php very high ram use - #59 by TrystanLea for more details

7

Hi Trystan
Thanks for the swift reply.

8

Seems my problem may be down to “very high ram use”. I’ll run the fix for a few days and keep my fingers crossed!
Thanks for the heads up and contact info re. HDMI boards. Not an issue right now but I’ll check the HDMIs out just to make sure that I don’t have one of the faulty ones.
Derek

1 Like
9

The fix solved my problem. Memory usage stable for over a week now.
Many thanks

2 Likes