After almost 2 years of continuous logging, my emonBase has developed an intermittent fault and stops logging data. The interval is now typically a few days, and I need to reboot to get it back up. The Pi itself is still alive, as I can ssh to it, and all seems good. I’ve checked the emonhub logs but the log file doesn’t have enough history to get back to the time when it stopped logging (say 12 hr).
I did check the syslog though and found thiis when it stopped logging:
Oct 12 23:20:41 emonpi emonhub[509]: Exception in thread RFM2Pi:
Oct 12 23:20:41 emonpi emonhub[509]: Traceback (most recent call last):
Oct 12 23:20:41 emonpi emonhub[509]: File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner
Oct 12 23:20:41 emonpi emonhub[509]: self.run()
Oct 12 23:20:41 emonpi emonhub[509]: File "/home/pi/emonhub/src/interfacers/emonhub_interfacer.py", line 73, in run
Oct 12 23:20:41 emonpi emonhub[509]: rxc = self.read()
Oct 12 23:20:41 emonpi emonhub[509]: File "/home/pi/emonhub/src/interfacers/EmonHubJeeInterfacer.py", line 137, in read
Oct 12 23:20:41 emonpi emonhub[509]: c.nodeid = int(f[0]) + int(self._settings['nodeoffset'])
Oct 12 23:20:41 emonpi emonhub[509]: IndexError: list index out of range
Oct 12 23:20:43 emonpi Node-RED[498]: 12 Oct 23:20:43 - [info] [debug:View all 'emon/#' MQTT traffic]
Oct 12 23:20:43 emonpi Node-RED[498]: { 'emon/emontx3/vrms': '0', 'emon/12/3': 239.28 }
Oct 12 23:20:57 emonpi Node-RED[498]: 12 Oct 23:20:57 - [info] [mqtt-broker:23cea9f8.21ad86] Connection failed to broker: mqtt://10.0.0.113:1883
Oct 12 23:21:01 emonpi CRON[11861]: (pi) CMD (/home/pi/emonpi/service-runner >> /var/log/service-runner.log 2>&1)
Oct 12 23:21:15 emonpi Node-RED[498]: 12 Oct 23:21:15 - [info] [mqtt-broker:23cea9f8.21ad86] Connection failed to broker: mqtt://10.0.0.113:1883
Oct 12 23:21:33 emonpi Node-RED[498]: 12 Oct 23:21:33 - [info] [mqtt-broker:23cea9f8.21ad86] Connection failed to broker: mqtt://10.0.0.113:1883
Oct 12 23:21:51 emonpi Node-RED[498]: 12 Oct 23:21:51 - [info] [mqtt-broker:23cea9f8.21ad86] Connection failed to broker: mqtt://10.0.0.113:1883
prior to this MQTT traffic was good - after this connection failed to broker…
Can anyone help with finding a cause? The log entry suggests it arises from emonhub_interfacer.py
Thanks
Unfortunately there is not enough info there to establish anything concrete.
It actually points the finger at EmonHubJeeInterfacer.py with a IndexError: list index out of range error, if a list doesn’t have an index 0, it’s an empty list, so it should not be trying to process an empty list (frame) but we do not have any accompanying data to check what’s what, was there a incomplete frame? Was the frame somehow lost? is the error being reported faithfully? etc etc
What version of emonhub are you running? There were many issues with the emonpi variant of emonhub up to a year or so ago. The main issue then with debugging was that all the threads were being started within the RFM2Pi thread and therefore almost ALL issues appeared to be the EmonHubJeeInterfacer.py.
If you reboot, all logs are lost due to the fact they are held in RAM. As you have some log in syslog for that period I assum that is not the issue. The emonhub log files (unfortunately) get rotated by an aggressive logrotate setting and may only go back a couple of hours. You could check to see if there are any rotated emonhub.log files that go back further than the current file
ls -la /var/log/emonhub/
otherwise you will have to wait until it occurs again and hopefully get to the logs a little sooner.
This is the last commit:
7a3b2eb385472e6a60f319dbdf6cd32c6e375070
git status is:
On branch emon-pi
Your branch is behind 'origin/emon-pi' by 184 commits, and can be fast-forwarded.
(use "git pull" to update your local branch)
Changes not staged for commit:
(use "git add <file>..." to update what will be committed)
(use "git checkout -- <file>..." to discard changes in working directory)
modified: README.md
no changes added to commit (use "git add" and/or "git commit -a")
but can’t try the latest version:
pi@emonpi(rw):~/emonhub$ git pull origin emon-pi
From https://github.com/openenergymonitor/emonhub
* branch emon-pi -> FETCH_HEAD
Updating 7a3b2eb..899e526
error: Your local changes to the following files would be overwritten by merge:
README.md
Please, commit your changes or stash them before you can merge.
Aborting