Random Hangs of emonPi/emonSD

fluppie007 · 2022-09-24T14:03:00.714Z

Swap is empty since the many reboots. Today the reset was around 22h50 yesterday.
But on full load a RPi Zero 2 W is only drawing 0.5A or so. So I don’t think that’s the issue here.
afbeelding

borpin · 2022-09-25T08:47:08.126Z

Just went back over the thread - what exactly is your setup?

Pi Zero attached to other HW or just running Emoncms?

fluppie007:

Swap is empty since the many reboots.

Ok but do you see it filling up?

Have you added any inputs/feeds?

fluppie007 · 2022-09-26T21:25:03.904Z

Nope it’s a Zero 2 W that has a clean Buster image and only emonCMS installed with emonScripts
Now for some reason it’s 3 days up… What a non-consistent behaviour…

afbeelding1122×484 17.9 KB

Second Zero 2 W with Bullseye, new install since this weekend.

afbeelding1127×498 18.5 KB

borpin · 2022-09-27T06:47:28.356Z

borpin:

what exactly is your setup?

fluppie007:

Nope it’s a Zero 2 W

So just running emoncms with no harware attatched (e.g. using emonhub)?

Have to say I am at a loss. Clearly there seems to be an issue with Bullseye as the SWAP is filling up on that one.

You could try installing Netdata and see if that shines any light on what is going on under the hood.

@TrystanLea @glyn.hudson

TrystanLea · 2022-09-27T12:35:43.377Z

Cant think of anything to add, I havent tried building a new install using EmonScripts for a while so cant comment on reliability of a freshly build install either. The last official build (emonSD pre built SD card Download & Change Log · openenergymonitor/emonpi Wiki · GitHub) has been very stable for me, did you try that? … It is high time we did another release, my focus has been elsewhere on hardware but we should do another!

borpin · 2022-09-27T12:52:29.567Z

Problem seems to be on the newer Linux release.

fluppie007 · 2022-10-01T20:49:03.110Z

Yes and it’s a Zero 2 W, which needs minimum an image from end of 2021.
But for some weird reason it’s now one week uptime

afbeelding

borpin · 2022-10-02T11:47:36.755Z

And swap at zero. The issue is definitely swap filling up, but why, remains a mystery.

[edit]
Also note this is the linux 5.10 version (which looked fine previously), it was the 5.15 version that had the swap problem above.

Bill.Thomson · 2022-10-02T21:04:51.588Z

Swap filling up?
Ummm, doesn’t look like it according to the data in his latest post.
I noticed his swap file was at ~46% in one of his earlier posts, but wasn’t that
before he said it straightened up and decided to start flying right?

used is sitting at zero

free is at 100 MB

mjr · 2022-10-02T21:50:44.768Z

borpin:

Also note this is the linux 5.10 version (which looked fine previously), it was the 5.15 version that had the swap problem above.

Was a testing kernel package installed? 5.15 isn’t in a released version of debian. Edit to add: it looks like Pi OS uses that testing kernel instead of the 5.19 released with debian 11. Not sure why.

borpin · 2022-10-03T06:21:02.535Z

Bill.Thomson:

Swap filling up?

Yes, look at the previous screenshots. Random Hangs of emonPi/emonSD - #48 by fluppie007

My understanding was that an install on 5.15 kept hanging - screenshot above shows swap filling up. But 5.10 is fine (no use of Swap).

fluppie007 · 2022-10-15T20:38:56.958Z

Update, getting 21 days of uptime with an empty swap on the v5.10 system
afbeelding

So this thing has kind of solved itself?

Is there a way to recalculate kWh/day accumulator of the previous months with a lot of hangs? Base feed is de reading from a kWh meter. 5 min intervals.

borpin · 2022-10-24T06:09:37.785Z

fluppie007:

Update, getting 21 days of uptime with an empty swap on the v5.10 system

@TrystanLea just to note this only works on the older Release, so the issue isn’t solved.

rdavies6 · 2022-10-30T11:02:04.207Z

So I just noticed my EmonPi stopped exporting feeds 4 days ago.

Screenshot 2022-10-30 at 10.49.281920×871 122 KB

Before I power cycle the device…is there anything I can check that might be helpful to debugging this issue? It’s probably been about 4 months since the last hang. As usual everything looks OK (no excessive disk usage, services all running, nothing abnormal in log files)…I’m pretty sure a power cycle is required to get the feeds working again

Server Information

Services

emonhub :- Active Running
emoncms_mqtt :- Active Running
feedwriter :- Active Running - sleep 300s 0 feed points pending write
service-runner :- Active Running
emonPiLCD :- Active Running
redis-server :- Active Running
mosquitto :- Active Running
demandshaper :- Active Running

Emoncms

Version :- low-write 11.0.9
Git :-
- URL :- GitHub - emoncms/emoncms: Web-app for processing, logging and visualising energy, temperature and other environmental data
- Branch :- * stable
- Describe :- 11.0.9
Components :- Emoncms Core v11.0.9 | App v2.5.2 | EmonHub Config v2.1.1 | Dashboard v2.2.4 | Device v2.1.8 | Graph v2.1.6 | Network Setup v1.0.2 | WiFi v2.1.1 | Backup v2.3.2 | DemandShaper v2.2.2 | Postprocess v2.2.4 | Sync v2.1.3 | Usefulscripts v2.3.9 | EmonScripts v1.4.1 | RFM2Pi v1.4.1 | Emonhub v2.3.3 | EmonPi v2.9.5

Server

CPU :- 1 Threads(s) | 4 Core(s) | 1 Sockets(s) | Cortex-A53 | 89.60MIPS |
OS :- Linux 5.4.51-v7+
Host :- emonpi | emonpi | (192.168.2.10)
Date :- 2022-10-30 10:50:20 UTC
Uptime :- 10:50:20 up 25 days, 2:38, 1 user, load average: 0.06, 0.05, 0.00

Memory

RAM :- Used: 20.61%
- Total :- 925.85 MB
- Used :- 190.77 MB
- Free :- 735.08 MB
Swap :- Used: 0.00%
- Total :- 100 MB
- Used :- 0 B
- Free :- 100 MB

Disk

**** :- - / :- Used: 49.64%
- Total :- 4.06 GB
- Used :- 2.02 GB
- Free :- 1.85 GB
- Read Load :- 25.3 B/s
- Write Load :- 149.83 B/s
- Load Time :- 25 days 2 hours 34 mins
/var/opt/emoncms :- Used: 0.59%
- Total :- 24.86 GB
- Used :- 149.23 MB
- Free :- 23.45 GB
- Read Load :- 1.66 B/s
- Write Load :- 60.89 B/s
- Load Time :- 25 days 2 hours 34 mins
/boot :- Used: 21.52%
- Total :- 252.05 MB
- Used :- 54.23 MB
- Free :- 197.81 MB
- Read Load :- 0 B/s
- Write Load :- 0 B/s
- Load Time :- 25 days 2 hours 34 mins
/var/log :- Used: 1.45%
- Total :- 50 MB
- Used :- 740 KB
- Free :- 49.28 MB
- Read Load :- n/a
- Write Load :- n/a
- Load Time :- n/a

HTTP

Server :- Apache/2.4.38 (Raspbian) HTTP/1.1 CGI/1.1 80

MySQL

Version :- 5.5.5-10.3.23-MariaDB-0+deb10u1
Host :- 127.0.0.1 (127.0.0.1)
Date :- 2022-10-30 10:50:20 (UTC 00:00‌)
Stats :- Uptime: 2169477 Threads: 9 Questions: 43718 Slow queries: 0 Opens: 46 Flush tables: 1 Open tables: 40 Queries per second avg: 0.020

Redis

Version :-
- Redis Server :- 5.0.3
- PHP Redis :- 5.3.1
Host :- localhost:6379
Size :- 578 keys (915.42K)
Uptime :- 25 days

MQTT Server

Version :- Mosquitto 1.5.7
Host :- localhost:1883 (127.0.0.1)

PHP

Version :- 7.3.19-1~deb10u1 (Zend Version 3.3.19)
Run user :- User: www-data Group: www-data video Script Owner: pi
Modules :- apache2handlercalendar Core ctype curl date dom v20031129exif fileinfo filter ftp gd gettext hash iconv json v1.7.0libxml mbstring mosquitto v0.4.0mysqli mysqlnd vmysqlnd 5.0.12-dev - 20150407 - $Id: 7cc7cc96e675f6d72e5cf0f267f48e167c2abb23 $openssl pcre PDO pdo_mysql Phar posix readline redis v5.3.1Reflection session shmop SimpleXML sockets sodium SPL standard sysvmsg sysvsem sysvshm tokenizer wddx xml xmlreader xmlwriter xsl Zend OPcache zlib

Pi

Model :- Raspberry Pi 3 Model B+ Rev 1.3 - 1GB (Sony UK)
Serial num. :- CEEE5A02
CPU Temperature :- 47.24°C
GPU Temperature :- 47.2°C
emonpiRelease :- emonSD-24Jul20
File-system :- read-write

Client Information

HTTP

Browser :- Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/106.0.0.0 Safari/537.36
Language :- en-GB,en-US;q=0.9,en;q=0.8

Window

Size :- 1600 x 886

Screen

Resolution :- 1440 x 900

borpin · 2022-10-30T13:51:58.222Z

What does the Inputs page say? Are all the inputs Red?

rdavies6 · 2022-10-31T07:52:42.876Z

Hi Brian

All the inputs are red (from both emonpi and temperature/humidity sensors). if this follows the normal pattern of behaviour, a reboot of the emonpi will not fix the feeds; a full power cycle is required.

Screenshot 2022-10-31 at 07.50.072844×1480 294 KB

borpin · 2022-10-31T19:15:21.997Z

Odd. Debugging is a bit of a PITA. You would need to check the emonhub logs first to see if data is being received and sent on, and then look and see if there is an issue at the emoncms end.

TBH I’d be inclined to flash a new SD Card and import your data.

rdavies6 · 2022-11-01T07:54:39.067Z

Thanks Brian, I might have a go at that…very little to loose, and it would be interesting to see if it brings stability

TrystanLea · 2022-11-01T08:24:23.785Z

@rdavies6 do you see any data coming in via emonhub.log?

My guess given that this requires a full power cycle is that the emonpi firmware itself has locked up. Probably due to an issue with communication with the RFM69 radio. If that’s right you should not see any data coming in in emonhub.log and if you use the Admin > Update > Firmware tool to update the emonPi firmware it wont necessarily free it either? It would be interested to hear if that’s the case?

rdavies6 · 2022-11-01T15:26:25.730Z

Hi Trystan

There don’t appear to be any records in emonhub.log (assuming I’m looking at the right file).

Screenshot 2022-11-01 at 15.14.37860×132 17.6 KB

I ran a firmware update…and it did restart the feeds!!

Thanks you!
firmwareupdate.txt (6.5 KB)

Thank you