Ok it will need a
sudo systemctl stop emonhub.service
sudo systemctl disable emonhub.service
sudo systemctl mask emonhub.service
To make sure there cannot be 2 running. (I’m not sure if disable is required before a mask but belt and braces).
Not really, not just for a test run it doesn’t. hence why I said a reboot will revert things back to normal.
Yes if you want to make it more complex, permanent or more robust you could, but I do not think it is essential for a test. I have run this myself on an emonpi image this afternoon. But by all means if you want belt and braces be my guest but IMO it’s a bit OTT for a quick test. This is not a proposed fix by any means.
Just for clarity if you do disable and mask. obviously those also have to be reverted to restore things to stock after the test.
There wasn’t any extension…I just had to add the .bin for the upload process 
Here’s what happened… I had to shorten the length of the message as there were so many repetitions of the lines (oh…and there was no logfile created!)
pi@emonpi:/var/log/emonhub $ sudo -u emonhub /usr/share/emonhub/emonhub.py --config-file=/home/pi/data/enonhub.conf --logfile /var/log/emonhub/emonhub.log&
[1] 1497
pi@emonpi:/var/log/emonhub $ XXX too high nonterminal number!
XXX too high nonterminal number!
XXX too high nonterminal number!
XXX too high nonterminal number!
XXX too high nonterminal number!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too high nonterminal number!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too high nonterminal number!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too high nonterminal number!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too high nonterminal number!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX ambiguity!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX ambiguity!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX ambiguity!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX ambiguity!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too high nonterminal number!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too high nonterminal number!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too high nonterminal number!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too many states!
XXX too high nonterminal number!
^C
[1]+ Segmentation fault sudo -u emonhub /usr/share/emonhub/emonhub.py --config-file=/home/pi/data/enonhub.conf --logfile /var/log/emonhub/emonhub.log
pi@emonpi:/var/log/emonhub $
That’s not something I recognise!
A quick search on those 2 error messages seem to point to Python in particular a acceler.c file used in multiple compiler parsers. I really have no idea how to debug further at this point.
This is intended to run on Python 2.7, which is still the default for the 2 Pi’s I have just checked, can you check yours?
pi@raspberrypi:~ $ python -V
Python 2.7.13
This isn’t actually explicitly defined in emonhub so it might be worth trying to either start with a modified shebang, in /home/pi/emonhub/src/emonhub.py edit the first line from #!/usr/bin/env python to #!/usr/bin/env python2.7 or possibly easier to just adapt the command line to
sudo -u emonhub python2.7 /usr/share/emonhub/emonhub.py --config-file=/home/pi/data/emonhub.conf --logfile /var/log/emonhub/emonhub.log&
bit of a long shot, but it’s all i got right now.
Yeah, that sounds right. emonhub creates the logfile when it is running, I don’t think were even getting passed the compile stage here.
What does this command give you
ls -la /home/pi/emonhub/src
just wondering if there are any .pyc files and what the dates are if there are any.
[edit]
Just to clarify, have you had emonhub run at all? I know the title of the thread is “emonhub fails to start” but one of your emonhub status’s above had Active: failed (Result: core-dump) since Fri 2019-05-03 10:27:00 UTC; 48min ago in it, which I took to be “48 mins running” but now I wonder if that is the case or if it has just been in a failed state for 48mins, can you confirm if emonhub runs at all? Does it fail to start at all or fail to run for any period of time?
1 Like
I was a bit surprised at the results 
Here’s what it says on my box
emonSD-30Oct18
pi@emonpi:~ $ python -V
Python 2.7.13
pi@emonpi:~ $
Looks like the right version…and i certainly haven’t done anything to change what was installed on the box.
Here’s what is says
pi@emonpi:~ $ ls -la /home/pi/emonhub/src
total 80
drwxr-xr-x 4 pi pi 4096 Jan 22 14:11 .
drwxr-xr-x 7 pi pi 4096 Mar 11 08:45 ..
-rw-r--r-- 1 pi pi 1142 Aug 2 2018 Cargo.py
-rw-r--r-- 1 pi pi 2834 Aug 2 2018 emonhub_buffer.py
-rw-r--r-- 1 pi pi 1148 Aug 2 2018 emonhub_coder.py
-rw-r--r-- 1 pi pi 24844 Aug 2 2018 emonhub_interfacer.py
-rwxr-xr-x 1 pi pi 12341 Jan 22 14:11 emonhub.py
-rw-r--r-- 1 pi pi 5219 Aug 2 2018 emonhub_setup.py
drwxr-xr-x 3 pi pi 4096 Jan 22 14:11 interfacers
drwxr-xr-x 2 pi pi 4096 Aug 2 2018 smalibrary
This EmonPi has been running since January…and was working perfectly, with data exported to emoncms.org…so I assume emonhub was working perfectly. However data stopped being exported on the 30th April…and emonhub hasn’t worked since 
The ls shows there are no .pyc files so there is a compiler/interpreter level issue by the looks of it.
Why or how that would suddenly manifest itself without a trigger eg an update, I have no idea.
I’m struggling for idea’s and I’m not around much at the moment, I would be useful if I could reproduce the issue but that is unlikely.
Just thinking out loud, I wonder if updating would help?
Am I right in thinking you purchased a pre-built sdcard image?
And you have only updated via the “update emonpi” button, but not immediately before the fail?
There are packages upgraded/installed for the emonpi that result in different images getting/not getting some upgrades, I wonder if this might be the root of the difference?
Earlier in the year anyone commissioning a new image or updating an existing image using the “update emopnpi” button would recieve these particular updates
but currently, as far as I can make out (and I’m not sure from exactly when) only users that have an LCD installed (and working at the time of update) will get these revised packages
And of course the day the update is run, will dictate what is pulled in by these, it’s a bit of a long shot but I’m really scratching around for differences and potential triggers.
I wonder if a dist-upgrade might be worth trying to, but at the very least I think you should try a “update emonpi” and post the emonpiupdate.log here.
It’s also interesting that service-runner and emonPiLCD are starting ok (can you confirm that with a sudo systemctl status emonPiLCD service-runner please).
The nuclear option is to export your data (via backup), get a new SD card, flash the SDImage, boot it up (and wait while it does quite a lot of updating) and then import your data and see what happens!
Although a new card may not be necessary, you may have one of the very few occurrences of this issue that might help debug. So if you do go the “nuclear” option, please keep the old image so we might revisit. Remember this didn’t occur initially, so even the “nuclear” option might wobble down the line. Until we know what the cause is, we cannot be sure.
It also offers a roll back in case the export is corrupted. Flashing existing card is the super-nuclear option 
.
Agreed.
Indeed, super-nuclear is potentially a extinction-level event (as far as historic data is concerned)
1 Like
Firstly I offer my gratitude and appreciation for all the support you and @borpin have been offering here 
The Emonpi was purchased with the pre-flashed SD card. It has been running near perfectly since January (I had one instance when the feeds stopped updating, but a reboot resolved that). There have been some less than graceful power outages (power cut, and contractors working at my house who have cut power). However it had been working pretty much flawless (and untouched) up until this issue started at the end of April. I have only ever updated the software via the “Update Emonpi” button… (I may have done this every month or so since January…)
Here’s the output:
emonSD-30Oct18
pi@emonpi:~ $ sudo systemctl status emonPiLCD service-runner
● emonPiLCD.service - LSB: emonPi LCD controller
Loaded: loaded (/etc/init.d/emonPiLCD; generated; vendor preset: enabled)
Active: active (exited) since Sun 2019-05-05 08:37:43 UTC; 24h ago
Docs: man:systemd-sysv-generator(8)
Process: 934 ExecStop=/etc/init.d/emonPiLCD stop (code=exited, status=0/SUCCESS)
Process: 940 ExecStart=/etc/init.d/emonPiLCD start (code=exited, status=0/SUCCESS)
CGroup: /system.slice/emonPiLCD.service
Warning: Journal has been rotated since unit was started. Log output is incomplete or unavailable.
● service-runner.service - Emoncms service-runner Input Script
Loaded: loaded (/var/www/emoncms/scripts/services/service-runner/service-runner.service; enabled; vendor preset: enabled)
Active: activating (auto-restart) (Result: signal) since Mon 2019-05-06 09:04:48 UTC; 5s ago
Docs: https://github.com/emoncms/emoncms/blob/master/scripts/services/install-service-runner-update.md
Process: 3382 ExecStart=/usr/bin/python /var/www/emoncms/scripts/services/service-runner/service-runner.py (code=killed, signal
Main PID: 3382 (code=killed, signal=SEGV)
May 06 09:04:48 emonpi systemd[1]: service-runner.service: Main process exited, code=killed, status=11/SEGV
May 06 09:04:48 emonpi systemd[1]: service-runner.service: Unit entered failed state.
May 06 09:04:48 emonpi systemd[1]: service-runner.service: Failed with result 'signal'.
The basic pathology that I see is:
- The LCD sits at the ‘booting…’ prompt, even though the emonpi is up and running ( hadn’t notice this until yesterday!)
- No data being collected from the sensors since 1pm on April 29th
- Emonhub fails to start
This is what i see on the status page -
I purchased a new SD card yesterday, and flashed a new image on it. It works perfectly, and I could continue with that…but I haven’t taken that step yet. The historical data on the old image would be a nice to have, but not essential to transfer over!
I also mounted the old image to an Ubuntu VM and fsck’ed the disk to correct the inode errors. However that doesn’t appear to have resolved the problem (although it was a nice trick to learn for the future).
Yes - absolutely! I will preserve the old SD card if we decide to just give up with debugging at the moment, and I can easily continue with the new card if necessary.
Thanks
Assuming you do not have any feeds etc set up on the new image, you should be able to mount the old image to the new emonpi image and drag over the historical data and mysql table, this isn’t the “approved” method so it’s not documented nor thoroughly tried and tested. I would recommend you stop all the services on the new image whilst you do that to prevent any clashes, over writing or race conditions whilst you are manipulating the data files. the main ones to stop are feedwriter, mysql (mariadb) redis-server and apache2 i think.
Since it is only the python services that are not starting you might be able to do a manual backup/export of your old data. @borpin might be able to tell you how. The fact that service-runner is having issues might mean you cannot do a backup from within the emoncms webapp, but it might be ok from the commandline.
I wouldn’t say we’ve given up exactly, but I’m not sure how to proceed at the moment and you are losing monitoring data. So it’s your call.
As much I would prefer to keep the old image as is until we locate the issue, I still wonder if a dist-upgrade might help? I think if you do decide to get the other image up and running with your old data, you could maybe dist-upgrade the old image once you have your data securely in the new image.
I don’t suppose you have another Pi kicking around to experiment on without losing use of the emonpi do you? No worries if you don’t. As nice as it would be to debug this fully, getting you back up and running again is the main priority.
You can run the backup script manually if you are running the old SD card.
ls -la /var/www/emoncms/Modules/
will give you the path to the backup module (via a symlink). Navigate to the folder and then go up one folder level and you will find a script emoncms-export.sh. I’m not sure of the exact path as I’m not running the SD Card image right now .
Just run that script and it will generate a backup zip file.
./emoncms-export.sh
HTH
well… I went the nuclear option from when I last posted here and imported back the images etc. System ran for a wekk then failed again 2 days ago with same error msgs as before. 
I will try a new SD card but im doubtfull
Oh dear!!!
As a matter of interest did you reinstall the image to the same sdcard or another one? If another, was it new?
When the system stopped, did you do any checks before rebooting or restarting, or is the same fault reported after it fails to reboot?
Just trying to understand the transition from working to won’t start. Is that the natural order of events or is it just failing . . . And then subsequently refusing to restart following intervention?
Installed on same SD the first time. When stopped this time around, I did the same snooping around and tried to restart via ssh etc and all exact same errors as previous. I just dont see the faulty or corrupt SD as a viable possibility as it the actual service failing to restart repeatedly. Not random errors anywhere in the file system. You would think that with the file system growing with time and a corrupt filesystem, the errors would not occur in the exact same place on disk. (I say disk, but you know what I mean)
Who are the contributors to the code on this project?
I will try another brand new SD tomorrow but in my scenerio with now 8 emonTX’s reporting to the emonPI with often 2-300 feeds waiting to be written, im thinking it just cant handle it and corrupts that part of the filesystem? You would think it would corrupt feedwriter thought. hmmm
Log in a tad different this time around after reboot
-- Logs begin at Thu 2016-11-03 17:16:43 UTC, end at Sat 2019-05-18 19:04:09 UTC. --
May 17 23:25:22 emonpi systemd[1]: emonhub.service: Main process exited, code=exited, status=1/FAILURE
May 17 23:25:22 emonpi systemd[1]: emonhub.service: Unit entered failed state.
May 17 23:25:22 emonpi systemd[1]: emonhub.service: Failed with result 'exit-code'.
May 17 23:25:22 emonpi systemd[1]: emonhub.service: Service hold-off time over, scheduling restart.
May 17 23:25:22 emonpi systemd[1]: Stopped emonHub service description.
May 17 23:25:22 emonpi systemd[1]: Started emonHub service description.
May 17 23:25:22 emonpi emonhub.py[1011]: SystemError: ../Objects/codeobject.c:112: bad argument to internal function
May 17 23:25:22 emonpi systemd[1]: emonhub.service: Main process exited, code=exited, status=1/FAILURE
May 17 23:25:22 emonpi systemd[1]: emonhub.service: Unit entered failed state.
May 17 23:25:22 emonpi systemd[1]: emonhub.service: Failed with result 'exit-code'.
May 17 23:25:22 emonpi systemd[1]: emonhub.service: Service hold-off time over, scheduling restart.
May 17 23:25:22 emonpi systemd[1]: Stopped emonHub service description.
May 17 23:25:22 emonpi systemd[1]: Started emonHub service description.
May 17 23:25:22 emonpi emonhub.py[1017]: SystemError: ../Objects/codeobject.c:112: bad argument to internal function
May 17 23:25:22 emonpi systemd[1]: emonhub.service: Main process exited, code=exited, status=1/FAILURE
May 17 23:25:22 emonpi systemd[1]: emonhub.service: Unit entered failed state.
Whilst I think a failed sdcard is unlikely, I’m not certain enough to rule it out as a possibility, installing to another sdcard (new or otherwise) would have gone a long way to put that idea to bed.
Sorry but I’m still a little vague on exactly what you’ve done. Same errors as last time prior to attempting stuff? or after attempting stuff? Your last post seems to suggest the error are NOT the same.
What we need to move forward is precise information about the status when it first fails and before any attempts to restart or reboot.
Yes, that is the issue you are eventually seeing, but emonhub does start perfectly ok and run without issue for some time (from a fresh image). Then something happens! emonhub doesn’t suddenly need to restart for no apparent reason. There is something that is causing emonhub to NEED to restart first, eg you are updating, you have rebooted, there has been a power outage or a crash due to an error induced within emonhub, external to emonhub or simply just due to the log partition filling up etc. We need to know why it needs to restart.
If you are flying down a motorway in your car and it suddenly goes kuput (and then won’t restart again) the issue is that it cut out whilst driving, not that it’s a non-starter, that is a subsequent symptom, it’s not the same as waking up one morning and the car doesn’t start. The fact you were doing 70mph and there was a loud bang and loads of smoke, is far more useful info than describing what now happens when it will no longer start.
Yes 8 emonTx’s is quite a load, but I have had the original emonhub running with 9 emonTx’s reporting at 5sec intervals along side 12 emonTH’s. There are differences in that my emonTx’s were connected via usb, but the traffic within emonhub (albeit a different version) would have been well over double what you have, so it is not a obvious concern (yet) but again I cannot rule it out due to the differences.
If there were going to be issues with so many emonTx’s, I would have expected that to have been in the RF domain, not within emonhub, eg rf collisions due to 8 unsync’d 10s transmissions. However, if you feel strongly about it being a traffic thing, you could try running with fewer devices for a trial period, or alter the reporting period so they report half as much (eg a 20s interval) to see if that has an effect.
The problem I’m having with getting my head around this is that the issue sticks, as if it is damaging something that doesn’t get rectified during a reboot, but does get rectified by redoing the image. This suggests something is being written to a file that is used in subsequent reboots.
As well as the known issue with ram filling up, I have since learnt about some additional issues with the logrotation. I would be keen to walk you through some changes to revert emonhub to log directly to /var/log and to check your log rotation settings too.
The trouble we have with starting with a new image is that it pulls in all the latest updates on firstboot, so each time we do that the playing field changes, we do not know if the current version will fail or not until it does, likewise, if you start afresh and we make some changes, if it’s successful, it’s not clear if it’s the changes or the latest update that has resolved things.
Actually, another good test might be to block the firstboot updates (on a fresh image) to see if that runs for a longer period of time (without pulling in any updates).
Which ever method you opt for, when/if it fails, that is the condition we want to know about, do not try any restarts or reboots. Currently a reboot will wipe all the relevant log data, we now know that emonhub will not restart after it fails, so there is nothing to gain by trying to reboot/restart, all that does is change the debug data to something possibly quite different to the initial failure.
The main maintainers are @TrystanLea and @glyn.hudson, but I wrote the “experimental” PoC version of emonhub that was taken and modified quite substantially to produce the emonpi variant. So I have a good understanding of what emonhub used to do or what it should do, but it is not my code that is deployed. Unfortunately, I do not use the emonpi variant or the emonSD myself, so the bulk of my “emonpi” experience is derived solely from helping debug issues such as yours.
Very well. I will reinstall on a new SD today and once it fails again I will pull all the logs and post here and we take it from there. No reboots. Thx for your time
Regards,