Random Hangs of emonPi/emonSD

Still working and still daily reboot around 1 at night (Belgium). Any ideas? Some script that’s asking for a daily reboot?

I assume 1 am is 00:00 UTC?

Your router, DHCP Lease time? Highly unlikely, and I can’t say I’ve ever heard anyone complain about the same thing before. I’m not on the very latest version, I have a 1-day DHCP lease and my emonPi has been running for 136 days.

Or something external that happens at this time, which causes the Pi to restart - like a big voltage dip?

Is the install a clean EmonSD install from the image, or is anything else installed on the ZPi2W?

It’s a clean system with image from RPi foundation and then EmonScripts
SSH Terminal says: Linux emonrpiz2w 5.10.103-v7+ #1529 SMP Tue Mar 8 12:21:37 GMT 2022 armv7l

I’ll have a spare Zero 2 W and will try with the latest image of Sept 6th 2022.

Have you checked memory use again? Is it the same problem with Swap?

Swap is empty since the many reboots. Today the reset was around 22h50 yesterday.
But on full load a RPi Zero 2 W is only drawing 0.5A or so. So I don’t think that’s the issue here.
afbeelding

Just went back over the thread - what exactly is your setup?

Pi Zero attached to other HW or just running Emoncms?

Ok but do you see it filling up?

Have you added any inputs/feeds?

Nope it’s a Zero 2 W that has a clean Buster image and only emonCMS installed with emonScripts
Now for some reason it’s 3 days up… What a non-consistent behaviour…

Second Zero 2 W with Bullseye, new install since this weekend.

1 Like

So just running emoncms with no harware attatched (e.g. using emonhub)?

Have to say I am at a loss. Clearly there seems to be an issue with Bullseye as the SWAP is filling up on that one.

You could try installing Netdata and see if that shines any light on what is going on under the hood.

@TrystanLea @glyn.hudson

1 Like

Cant think of anything to add, I havent tried building a new install using EmonScripts for a while so cant comment on reliability of a freshly build install either. The last official build (emonSD pre built SD card Download & Change Log · openenergymonitor/emonpi Wiki · GitHub) has been very stable for me, did you try that? … It is high time we did another release, my focus has been elsewhere on hardware but we should do another!

Problem seems to be on the newer Linux release.

Yes and it’s a Zero 2 W, which needs minimum an image from end of 2021.
But for some weird reason it’s now one week uptime :man_shrugging:

afbeelding

And swap at zero. The issue is definitely swap filling up, but why, remains a mystery.

[edit]
Also note this is the linux 5.10 version (which looked fine previously), it was the 5.15 version that had the swap problem above.

2 Likes

image

Swap filling up?
Ummm, doesn’t look like it according to the data in his latest post.
I noticed his swap file was at ~46% in one of his earlier posts, but wasn’t that
before he said it straightened up and decided to start flying right?

used is sitting at zero

free is at 100 MB

Was a testing kernel package installed? 5.15 isn’t in a released version of debian. Edit to add: it looks like Pi OS uses that testing kernel instead of the 5.19 released with debian 11. Not sure why.

1 Like

Yes, look at the previous screenshots. Random Hangs of emonPi/emonSD - #48 by fluppie007

My understanding was that an install on 5.15 kept hanging - screenshot above shows swap filling up. But 5.10 is fine (no use of Swap).

Update, getting 21 days of uptime with an empty swap on the v5.10 system
afbeelding

So this thing has kind of solved itself?

Is there a way to recalculate kWh/day accumulator of the previous months with a lot of hangs? Base feed is de reading from a kWh meter. 5 min intervals.

@TrystanLea just to note this only works on the older Release, so the issue isn’t solved.

So I just noticed my EmonPi stopped exporting feeds 4 days ago.

Before I power cycle the device…is there anything I can check that might be helpful to debugging this issue? It’s probably been about 4 months since the last hang. As usual everything looks OK (no excessive disk usage, services all running, nothing abnormal in log files)…I’m pretty sure a power cycle is required to get the feeds working again

Server Information

Server Information

Services

  • emonhub :- Active Running

  • emoncms_mqtt :- Active Running

  • feedwriter :- Active Running - sleep 300s 0 feed points pending write

  • service-runner :- Active Running

  • emonPiLCD :- Active Running

  • redis-server :- Active Running

  • mosquitto :- Active Running

  • demandshaper :- Active Running

Emoncms

Server

  • CPU :- 1 Threads(s) | 4 Core(s) | 1 Sockets(s) | Cortex-A53 | 89.60MIPS |
  • OS :- Linux 5.4.51-v7+
  • Host :- emonpi | emonpi | (192.168.2.10)
  • Date :- 2022-10-30 10:50:20 UTC
  • Uptime :- 10:50:20 up 25 days, 2:38, 1 user, load average: 0.06, 0.05, 0.00

Memory

  • RAM :- Used: 20.61%
    • Total :- 925.85 MB
    • Used :- 190.77 MB
    • Free :- 735.08 MB
  • Swap :- Used: 0.00%
    • Total :- 100 MB
    • Used :- 0 B
    • Free :- 100 MB

Disk

  • **** :- - / :- Used: 49.64%
    • Total :- 4.06 GB
    • Used :- 2.02 GB
    • Free :- 1.85 GB
    • Read Load :- 25.3 B/s
    • Write Load :- 149.83 B/s
    • Load Time :- 25 days 2 hours 34 mins
  • /var/opt/emoncms :- Used: 0.59%
    • Total :- 24.86 GB
    • Used :- 149.23 MB
    • Free :- 23.45 GB
    • Read Load :- 1.66 B/s
    • Write Load :- 60.89 B/s
    • Load Time :- 25 days 2 hours 34 mins
  • /boot :- Used: 21.52%
    • Total :- 252.05 MB
    • Used :- 54.23 MB
    • Free :- 197.81 MB
    • Read Load :- 0 B/s
    • Write Load :- 0 B/s
    • Load Time :- 25 days 2 hours 34 mins
  • /var/log :- Used: 1.45%
    • Total :- 50 MB
    • Used :- 740 KB
    • Free :- 49.28 MB
    • Read Load :- n/a
    • Write Load :- n/a
    • Load Time :- n/a

HTTP

  • Server :- Apache/2.4.38 (Raspbian) HTTP/1.1 CGI/1.1 80

MySQL

  • Version :- 5.5.5-10.3.23-MariaDB-0+deb10u1
  • Host :- 127.0.0.1 (127.0.0.1)
  • Date :- 2022-10-30 10:50:20 (UTC 00:00‌​)
  • Stats :- Uptime: 2169477 Threads: 9 Questions: 43718 Slow queries: 0 Opens: 46 Flush tables: 1 Open tables: 40 Queries per second avg: 0.020

Redis

  • Version :-
    • Redis Server :- 5.0.3
    • PHP Redis :- 5.3.1
  • Host :- localhost:6379
  • Size :- 578 keys (915.42K)
  • Uptime :- 25 days

MQTT Server

  • Version :- Mosquitto 1.5.7
  • Host :- localhost:1883 (127.0.0.1)

PHP

  • Version :- 7.3.19-1~deb10u1 (Zend Version 3.3.19)
  • Run user :- User: www-data Group: www-data video Script Owner: pi
  • Modules :- apache2handlercalendar Core ctype curl date dom v20031129exif fileinfo filter ftp gd gettext hash iconv json v1.7.0libxml mbstring mosquitto v0.4.0mysqli mysqlnd vmysqlnd 5.0.12-dev - 20150407 - $Id: 7cc7cc96e675f6d72e5cf0f267f48e167c2abb23 $openssl pcre PDO pdo_mysql Phar posix readline redis v5.3.1Reflection session shmop SimpleXML sockets sodium SPL standard sysvmsg sysvsem sysvshm tokenizer wddx xml xmlreader xmlwriter xsl Zend OPcache zlib

Pi

  • Model :- Raspberry Pi 3 Model B+ Rev 1.3 - 1GB (Sony UK)

  • Serial num. :- CEEE5A02

  • CPU Temperature :- 47.24°C

  • GPU Temperature :- 47.2°C

  • emonpiRelease :- emonSD-24Jul20

  • File-system :- read-write

Client Information

Client Information

HTTP

  • Browser :- Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/106.0.0.0 Safari/537.36
  • Language :- en-GB,en-US;q=0.9,en;q=0.8

Window

  • Size :- 1600 x 886

Screen

  • Resolution :- 1440 x 900

What does the Inputs page say? Are all the inputs Red?

1 Like