Between @tim @beaylott @pb66 and I we have been working on fixing ongoing issues with the emon-pi variant of emonhub that has been resulting in crashes and ‘thread is dead’ errors, as reported multiple times on the forums e.g:
@beaylott has contributed a fix that implements thread restarting, restart counting, and logging of tracebacks within threads here, which I have briefly tested and appears to work well, see: Thread exception reporting, thread restarting, restart counting... by beaylott · Pull Request #36 · openenergymonitor/emonhub · GitHub
@tim has been working on a number of improvements including systemd support with watchdog reboot which is ongoing see: RFC - systemd integration with watchdog support and restart on thread failure - #17 by TrystanLea
In a PM between myself and @pb66 we have been discussing an ongoing issue that @pb66 bought to my attention where the intended separate threads of different interfacers where not being preserved. Our discussion has resulted in a move away from using pydispatcher to a native message queue implementation, here’s my explanation copied from my post on github:
Explanation of native message queue
In order to move data between the interfacers in emonhub e.g data received by the JeeLib interfacer and then published by the MQTT interfacer, emonhub has been using a python library called pydispatcher.
I had thought that it was a neat solution, moving data between threads with the ease of use similar to MQTT. However @pb66 highlighted that there was an issue with the implementation, that the intention of one thread per interfacer was not being preserved, the MQTT interfacer in particular was actually running in the RFM2Pi thread!
Paul then described a solution that would remove the need for the external pydispatcher library with a simple native message queue implementation. Here is my attempted summary (with references to the lines of code that implement it):
We start with the main thread (in emonhub.py)
The main thread starts a scan of each interfacer (L143)
The main thread finds a pubchannel on the RFM2Pi interfacer called ‘ToEmonCMS’ (L149)
The main thread reads and removes one item from the ‘ToEmonCMS’ queue on the RFM2Pi interfacer. (L155)
The main thread iterates through each interfacer looking for subchannel entries called ‘ToEmonCMS’ (L158-L162)
The main thread posts a copy of the item read from RFM2Pi into the ‘ToEmonCMS’ queue on each subscribing interfacer. (L168)
The subscribing interfacer then reads and acts on the queue in its own time and in its own thread.
The main thread then goes on to the next interfacer checking its pubchannel’s, repeating steps 4 to 6 above.
A single item is read at a time in order to avoid the case where a runaway interfacer monopolizes the main thread which could block the other interfacers, it also allows for the simple use of pop and append.
Paul @pb66 has been quite busy over recent weeks but I have tried my best to implement his suggested approach, which appears to work well. The result is now available in the emon-pi-develop branch of emonhub and is ready for testing!
Testing emonhub emon-pi-develop branch
I would very much welcome help with testing the emon-pi-develop branch. To do this the steps are:
cd /home/pi/emonhub git pull git checkout emon-pi-develop sudo service emonhub restart
I will likely move forward with merging the development branch into the main emon-pi branch in a week or so if no large issues are found as the improvements are a big step forward over the current implementation.