Emonbase started hanging randomly

After almost 2 years of continuous logging, my emonBase has developed an intermittent fault and stops logging data. The interval is now typically a few days, and I need to reboot to get it back up. The Pi itself is still alive, as I can ssh to it, and all seems good. I’ve checked the emonhub logs but the log file doesn’t have enough history to get back to the time when it stopped logging (say 12 hr).
I did check the syslog though and found thiis when it stopped logging:

Oct 12 23:20:41 emonpi emonhub[509]: Exception in thread RFM2Pi:
Oct 12 23:20:41 emonpi emonhub[509]: Traceback (most recent call last):
Oct 12 23:20:41 emonpi emonhub[509]: File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner
Oct 12 23:20:41 emonpi emonhub[509]: self.run()
Oct 12 23:20:41 emonpi emonhub[509]: File "/home/pi/emonhub/src/interfacers/emonhub_interfacer.py", line 73, in run
Oct 12 23:20:41 emonpi emonhub[509]: rxc = self.read()
Oct 12 23:20:41 emonpi emonhub[509]: File "/home/pi/emonhub/src/interfacers/EmonHubJeeInterfacer.py", line 137, in read
Oct 12 23:20:41 emonpi emonhub[509]: c.nodeid = int(f[0]) + int(self._settings['nodeoffset'])
Oct 12 23:20:41 emonpi emonhub[509]: IndexError: list index out of range
Oct 12 23:20:43 emonpi Node-RED[498]: 12 Oct 23:20:43 - [info] [debug:View all 'emon/#' MQTT traffic]
Oct 12 23:20:43 emonpi Node-RED[498]: { 'emon/emontx3/vrms': '0', 'emon/12/3': 239.28 }
Oct 12 23:20:57 emonpi Node-RED[498]: 12 Oct 23:20:57 - [info] [mqtt-broker:23cea9f8.21ad86] Connection failed to broker: mqtt://10.0.0.113:1883
Oct 12 23:21:01 emonpi CRON[11861]: (pi) CMD (/home/pi/emonpi/service-runner >> /var/log/service-runner.log 2>&1)
Oct 12 23:21:15 emonpi Node-RED[498]: 12 Oct 23:21:15 - [info] [mqtt-broker:23cea9f8.21ad86] Connection failed to broker: mqtt://10.0.0.113:1883
Oct 12 23:21:33 emonpi Node-RED[498]: 12 Oct 23:21:33 - [info] [mqtt-broker:23cea9f8.21ad86] Connection failed to broker: mqtt://10.0.0.113:1883
Oct 12 23:21:51 emonpi Node-RED[498]: 12 Oct 23:21:51 - [info] [mqtt-broker:23cea9f8.21ad86] Connection failed to broker: mqtt://10.0.0.113:1883

prior to this MQTT traffic was good - after this connection failed to broker…
Can anyone help with finding a cause? The log entry suggests it arises from emonhub_interfacer.py
Thanks

Unfortunately there is not enough info there to establish anything concrete.

It actually points the finger at EmonHubJeeInterfacer.py with a IndexError: list index out of range error, if a list doesn’t have an index 0, it’s an empty list, so it should not be trying to process an empty list (frame) but we do not have any accompanying data to check what’s what, was there a incomplete frame? Was the frame somehow lost? is the error being reported faithfully? etc etc

What version of emonhub are you running? There were many issues with the emonpi variant of emonhub up to a year or so ago. The main issue then with debugging was that all the threads were being started within the RFM2Pi thread and therefore almost ALL issues appeared to be the EmonHubJeeInterfacer.py.

If you reboot, all logs are lost due to the fact they are held in RAM. As you have some log in syslog for that period I assum that is not the issue. The emonhub log files (unfortunately) get rotated by an aggressive logrotate setting and may only go back a couple of hours. You could check to see if there are any rotated emonhub.log files that go back further than the current file

ls -la /var/log/emonhub/

otherwise you will have to wait until it occurs again and hopefully get to the logs a little sooner.

Thanks - how do I find out what version of emonhub is running - is there a command on the Pi or through emonCMS?

Hello @icenov

cd /home/pi/emonhub
git rev-parse HEAD

will give you the last commit , I can then check which version that relates too. Could you also check the branch:

git status

If you want to try the latest version of emonhub without updating the rest of the system the command is:

git pull origin emon-pi

assuming your still in the /home/pi/emonhub folder. Then you need to restart emonhub:

sudo service emonhub restart

Looks like I’ way behind…here are the results:

This is the last commit:
7a3b2eb385472e6a60f319dbdf6cd32c6e375070

git status is:

On branch emon-pi
Your branch is behind 'origin/emon-pi' by 184 commits, and can be fast-forwarded.
  (use "git pull" to update your local branch)
Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git checkout -- <file>..." to discard changes in working directory)

        modified:   README.md

no changes added to commit (use "git add" and/or "git commit -a")

but can’t try the latest version:

pi@emonpi(rw):~/emonhub$ git pull origin emon-pi
From https://github.com/openenergymonitor/emonhub
 * branch            emon-pi    -> FETCH_HEAD
Updating 7a3b2eb..899e526
error: Your local changes to the following files would be overwritten by merge:
        README.md
Please, commit your changes or stash them before you can merge.
Aborting

Thanks @icenov
Could you try:

git checkout .
git pull origin emon-pi

Thanks Trystan - that worked (had to use sudo). Will this be permanent?

Great to hear, yes should be permanent.