Hi folks,
I’ve got a python3 port of emonhub running on my emonpi. Everything seems to work but I’m aware that I’m testing a really small subset of what emonhub can do.
I’d really appreciate it if people who are in the position to do so could test this on the own systems. I’m more than happy to help to debug problems.
In the process of getting used to the code I tidied up a few things. I hope these changes are welcome.
Bruce
@bwduncan, excellent.
@TrystanLea @glyn.hudson @pb66 will need to be involved in this I suspect and @Greebo has Python expertise as well IIRC.
Can you confirm what your test environment is? Stretch/Buster? EmonPi or RPi (which version)?
Starting at the top and looking at the README - much of the info regarding the installation of Mosquitto is now no longer necessary I suspect.
It may also be the case that we state that this is only supported for Buster onwards.
Hi @borpin
This is a Raspberry Pi 2 Model B Rev 1.1 running Buster. It’s a standard emonPi from many years ago.
I suspect it would be much more reliable to build a debian package for this and let apt deal with the mosquitto/python/etc dependencies. That’s outside my expertise for now but I may be learning how to do this for work in the new year!
If we are to go down the path of deprecating support for older versions, there are some other bits which would welcome a cleaup (interfacer modules with the wrong name, for example). There’s almost 5000 lines of code, a lot of which is repeated, and newer python features could make this smaller and more readable. All that might be more effort than it’s worth, though, unless someone smarter than me can come up with a test suite!
Anyway, the python3 port has been running on my system for about 12 hours now.
Bruce
Which RFM card?
RFM69Pi, I think. How do I check? emonhub is configured to talk at 38400 over the serial line, if that helps.
2019-12-09 13:24:30,544 INFO MainThread Logging level set to DEBUG
2019-12-09 13:24:30,545 INFO MainThread Creating EmonHubJeeInterfacer 'RFM2Pi'
2019-12-09 13:24:30,548 DEBUG MainThread Opening serial port: /dev/ttyAMA0 @ 38400 bits/s
2019-12-09 13:24:32,564 INFO MainThread RFM2Pi device firmware version: [emonPi.29.00]
2019-12-09 13:24:32,565 INFO MainThread RFM2Pi device current settings: E i5 g210 @ 433 MHz q1 USA 0
2019-12-09 13:24:32,567 INFO MainThread Setting RFM2Pi calibration: 230V (1p)
2019-12-09 13:24:33,569 DEBUG MainThread Setting RFM2Pi pubchannels: ['ToEmonCMS']
2019-12-09 13:24:33,571 DEBUG MainThread Setting RFM2Pi subchannels: ['ToRFM12']
2019-12-09 13:24:33,574 INFO MainThread Creating EmonHubMqttInterfacer 'MQTT'
2019-12-09 13:24:33,579 DEBUG RFM2Pi device settings updated: E i5 g210 @ 433 MHz q1 USA 0
Good question, not sure (It isn’t an EmonPi then). Look at your invoice (that is what I do!). I think there may be something in ‘Learn’ that helps identify it.
Indeed there is - photos of the various modules and pointers to the differences.
Learn→Electricity Monitoring→Networking→Which radio module→Identifying different Radio Modules
Haha ok I’m not opening it up to figure out what radio module it has. It’s whatever came with the emonPi in January 2016. The invoice just says “emonPi”, so…
OK, The EmonPi is different to the EmonBase, hence my question. The EmonBase is just an RFM card that goes on any Pi. The EmonPi is (effectively) an EmonTX with an RFM module built in all in one case.
So I’m trying this on an older B and an RFP12Pi. It is not picking up the config option for speed (which needs to be 9600).
2019-12-09 19:22:37,886 DEBUG MainThread Opening serial port: /dev/ttyAMA0 @ 38400 bits/s
Fuller log
2019-12-09 19:23:27,588 WARNING MainThread RFM2Pi thread is dead.
2019-12-09 19:23:27,593 WARNING MainThread Attempting to restart thread RFM2Pi (thread has been restarted 4 times...
2019-12-09 19:23:27,598 INFO MainThread Creating EmonHubJeeInterfacer 'RFM2Pi'
2019-12-09 19:23:27,604 DEBUG MainThread Opening serial port: /dev/ttyAMA0 @ 38400 bits/s
2019-12-09 19:23:29,612 WARNING MainThread Device communication error - check settings
2019-12-09 19:23:29,621 INFO MainThread Setting RFM2Pi baseid: 12 (12i)
2019-12-09 19:23:30,627 INFO MainThread Setting RFM2Pi frequency: 433 (4b)
2019-12-09 19:23:31,632 INFO MainThread Setting RFM2Pi group: 210 (210g)
2019-12-09 19:23:32,637 INFO MainThread Setting RFM2Pi quiet: 1 (1q)
2019-12-09 19:23:33,643 INFO MainThread Setting RFM2Pi calibration: 230V (1p)
2019-12-09 19:23:34,648 DEBUG MainThread Setting RFM2Pi pubchannels: ['ToEmonCMS']
2019-12-09 19:23:34,656 DEBUG MainThread Setting RFM2Pi subchannels: ['ToRFM12']
2019-12-09 19:23:37,432 WARNING RFM2Pi Exception caught in RFM2Pi thread. Traceback (most recent call last):
File "/opt/openenergymonitor/emonhub/src/emonhub_interfacer.py", line 32, in wrapper
return f(*args)
File "/opt/openenergymonitor/emonhub/src/emonhub_interfacer.py", line 99, in run
rxc = self.read()
File "/opt/openenergymonitor/emonhub/src/interfacers/EmonHubJeeInterfacer.py", line 84, in read
self._rx_buf = self._rx_buf + self._ser.readline().decode()
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x80 in position 1: invalid start byte
Interestingly (this was the first time I’d used this Pi for this), I then installed the emon-pi variant (which worked), but the exception disappeared! It now just does not work and sits.
2019-12-09 19:40:22,282 INFO MainThread EmonHub emonHub emon-pi variant v2.1.2
2019-12-09 19:40:22,291 INFO MainThread Opening hub...
2019-12-09 19:40:22,295 INFO MainThread Logging level set to DEBUG
2019-12-09 19:40:22,299 INFO MainThread Creating EmonHubJeeInterfacer 'RFM2Pi'
2019-12-09 19:40:22,308 DEBUG MainThread Opening serial port: /dev/ttyAMA0 @ 38400 bits/s
2019-12-09 19:40:24,342 ERROR MainThread Unable to create 'RFM2Pi' interfacer: 'utf-8' codec can't decode byte 0x80 in position 1: invalid start byte
2019-12-09 19:40:24,345 INFO MainThread Creating EmonHubMqttInterfacer 'MQTT'
2019-12-09 19:40:24,356 DEBUG MainThread Setting MQTT pubchannels: ['ToRFM12']
2019-12-09 19:40:24,359 DEBUG MainThread Setting MQTT subchannels: ['ToEmonCMS']
2019-12-09 19:40:24,362 INFO MainThread Setting MQTT node_format_enable: 1
2019-12-09 19:40:24,365 INFO MainThread Setting MQTT nodevar_format_enable: 1
2019-12-09 19:40:24,367 INFO MainThread Setting MQTT nodevar_format_basetopic: emon/
2019-12-09 19:40:24,374 INFO MainThread Creating EmonHubEmoncmsHTTPInterfacer 'emoncmsorg'
2019-12-09 19:40:24,379 DEBUG MainThread Setting emoncmsorg pubchannels: ['ToRFM12']
2019-12-09 19:40:24,382 DEBUG MainThread Setting emoncmsorg subchannels: ['ToEmonCMS']
2019-12-09 19:40:24,385 WARNING MainThread Setting emoncmsorg apikey: obscured
2019-12-09 19:40:24,388 INFO MainThread Setting emoncmsorg url: https://emoncms.org
2019-12-09 19:40:24,390 INFO MainThread Setting emoncmsorg senddata: 1
2019-12-09 19:40:24,393 INFO MainThread Setting emoncmsorg sendstatus: 1
Log of successful emon-pi start (same config file).
2019-12-09 19:46:36,631 INFO MainThread EmonHub emonHub emon-pi variant v2.1.2
2019-12-09 19:46:36,634 INFO MainThread Opening hub...
2019-12-09 19:46:36,637 INFO MainThread Logging level set to DEBUG
2019-12-09 19:46:36,641 INFO MainThread Creating EmonHubJeeInterfacer 'RFM2Pi'
2019-12-09 19:46:36,651 DEBUG MainThread Opening serial port: /dev/ttyAMA0 @ 9600 bits/s
2019-12-09 19:46:38,659 INFO MainThread RFM2Pi device firmware version & configuration: not available
2019-12-09 19:46:38,664 INFO MainThread Setting RFM2Pi frequency: 433 (4b)
2019-12-09 19:46:39,670 INFO MainThread Setting RFM2Pi group: 210 (210g)
2019-12-09 19:46:40,676 INFO MainThread Setting RFM2Pi quiet: 0 (0q)
2019-12-09 19:46:41,682 INFO MainThread Setting RFM2Pi baseid: 5 (5i)
2019-12-09 19:46:42,687 INFO MainThread Setting RFM2Pi calibration: 230V (1p)
2019-12-09 19:46:43,693 DEBUG MainThread Setting RFM2Pi subchannels: ['ToRFM12']
2019-12-09 19:46:43,702 DEBUG MainThread Setting RFM2Pi pubchannels: ['ToEmonCMS']
2019-12-09 19:46:43,709 INFO MainThread Creating EmonHubMqttInterfacer 'MQTT'
2019-12-09 19:46:43,725 DEBUG MainThread Setting MQTT subchannels: ['ToEmonCMS']
2019-12-09 19:46:43,715 DEBUG RFM2Pi acknowledged command: > 4b
2019-12-09 19:46:43,733 DEBUG MainThread Setting MQTT pubchannels: ['ToRFM12']
2019-12-09 19:46:43,738 INFO MainThread Setting MQTT nodevar_format_enable: 1
2019-12-09 19:46:43,742 INFO MainThread Setting MQTT node_format_enable: 1
2019-12-09 19:46:43,746 INFO MainThread Setting MQTT nodevar_format_basetopic: emon/
2019-12-09 19:46:43,752 INFO MainThread Creating EmonHubEmoncmsHTTPInterfacer 'emoncmsorg'
2019-12-09 19:46:43,760 DEBUG MainThread Setting emoncmsorg subchannels: ['ToEmonCMS']
2019-12-09 19:46:43,764 DEBUG MainThread Setting emoncmsorg pubchannels: ['ToRFM12']
2019-12-09 19:46:43,768 INFO MainThread Setting emoncmsorg url: https://emoncms.org
2019-12-09 19:46:43,772 INFO MainThread Setting emoncmsorg senddata: 1
2019-12-09 19:46:43,777 WARNING MainThread Setting emoncmsorg apikey: obscured
2019-12-09 19:46:43,781 INFO MainThread Setting emoncmsorg sendstatus: 1
2019-12-09 19:46:43,938 DEBUG RFM2Pi acknowledged command: > 210g
2019-12-09 19:46:44,153 DEBUG RFM2Pi 1 NEW FRAME : 19 205 0 0 0 124 1 23 0
2019-12-09 19:46:44,163 DEBUG RFM2Pi 1 Timestamp : 1575920804.15
2019-12-09 19:46:44,168 DEBUG RFM2Pi 1 From Node : 19
2019-12-09 19:46:44,171 DEBUG RFM2Pi 1 Values : [20.5, 0, 38, 2.3000000000000003]
2019-12-09 19:46:44,176 DEBUG RFM2Pi 1 Sent to channel(start)' : ToEmonCMS
2019-12-09 19:46:44,180 DEBUG RFM2Pi 1 Sent to channel(end)' : ToEmonCMS
@bwduncan would be useful if you changed the version your branch reports itself as.
Hi @borpin, good work, thank you very much. I’ve fixed that bug and bumped the version to 3 (hope that doesn’t step on any toes, it matches the compatible python version and it’s not like we’ll run out of numbers…). Thanks, Bruce
Version would I think be better as 3.0.0-beta so it can go to 3.0.0 when released. As it is a significant change, under semantic versioning, it should go to V3.0.0 when released.
Starts off OK
2019-12-10 07:39:40,523 INFO MainThread Opening hub...
2019-12-10 07:39:40,527 INFO MainThread Logging level set to DEBUG
2019-12-10 07:39:40,530 INFO MainThread Creating EmonHubJeeInterfacer 'RFM2Pi'
2019-12-10 07:39:40,540 DEBUG MainThread Opening serial port: /dev/ttyAMA0 @ 9600 bits/s
2019-12-10 07:39:42,549 INFO MainThread RFM2Pi device firmware version & configuration: not available
2019-12-10 07:39:42,553 INFO MainThread Setting RFM2Pi baseid: 5 (5i)
2019-12-10 07:39:43,558 INFO MainThread Setting RFM2Pi frequency: 433 (4b)
2019-12-10 07:39:44,563 INFO MainThread Setting RFM2Pi group: 210 (210g)
2019-12-10 07:39:45,568 INFO MainThread Setting RFM2Pi quiet: 1 (1q)
2019-12-10 07:39:46,577 INFO MainThread Setting RFM2Pi calibration: 230V (1p)
2019-12-10 07:39:47,582 DEBUG MainThread Setting RFM2Pi pubchannels: ['ToEmonCMS']
2019-12-10 07:39:47,585 DEBUG MainThread Setting RFM2Pi subchannels: ['ToRFM12']
2019-12-10 07:39:47,591 INFO MainThread Creating EmonHubMqttInterfacer 'MQTT'
2019-12-10 07:39:47,599 DEBUG RFM2Pi acknowledged command: > 5i
2019-12-10 07:39:47,607 DEBUG MainThread Setting MQTT pubchannels: ['ToRFM12']
2019-12-10 07:39:47,610 DEBUG MainThread Setting MQTT subchannels: ['ToEmonCMS']
2019-12-10 07:39:47,613 INFO MainThread Setting MQTT node_format_enable: 1
2019-12-10 07:39:47,616 INFO MainThread Setting MQTT nodevar_format_enable: 1
2019-12-10 07:39:47,618 INFO MainThread Setting MQTT nodevar_format_basetopic: emon/
2019-12-10 07:39:47,623 INFO MainThread Creating EmonHubEmoncmsHTTPInterfacer 'emoncmsorg'
2019-12-10 07:39:47,629 DEBUG MainThread Setting emoncmsorg pubchannels: ['ToRFM12']
2019-12-10 07:39:47,632 DEBUG MainThread Setting emoncmsorg subchannels: ['ToEmonCMS']
2019-12-10 07:39:47,635 INFO MainThread Setting emoncmsorg apikey: set
2019-12-10 07:39:47,639 INFO MainThread Setting emoncmsorg url: http://192.168.7.47/emoncms
2019-12-10 07:39:47,641 INFO MainThread Setting emoncmsorg senddata: 1
2019-12-10 07:39:47,645 INFO MainThread Setting emoncmsorg sendstatus: 1
2019-12-10 07:39:47,650 INFO MainThread Creating EmonHubEmoncmsHTTPInterfacer 'emoncm77'
2019-12-10 07:39:47,655 DEBUG MainThread Setting emoncm77 pubchannels: ['ToRFM12']
2019-12-10 07:39:47,658 DEBUG MainThread Setting emoncm77 subchannels: ['ToEmonCMS']
2019-12-10 07:39:47,661 INFO MainThread Setting emoncm77 apikey: set
2019-12-10 07:39:47,664 INFO MainThread Setting emoncm77 url: http://192.168.7.77/
2019-12-10 07:39:47,666 INFO MainThread Setting emoncm77 senddata: 1
2019-12-10 07:39:47,669 INFO MainThread Setting emoncm77 sendstatus: 1
2019-12-10 07:39:47,817 DEBUG RFM2Pi acknowledged command: > 4b
2019-12-10 07:39:48,029 DEBUG RFM2Pi acknowledged command: > 210g
2019-12-10 07:39:48,236 DEBUG RFM2Pi acknowledged command: > 1q
But dies quite quickly.
2019-12-10 07:40:26,387 DEBUG RFM2Pi 8 NEW FRAME : 10 204 11 0 0 0 0 0 0 123 93 48 117 48 117 48 117 48 117 48 117 48 117 0 0 0 0
2019-12-10 07:40:26,397 DEBUG RFM2Pi 8 Timestamp : 1575963626.386137
2019-12-10 07:40:26,402 DEBUG RFM2Pi 8 From Node : 10
2019-12-10 07:40:26,406 DEBUG RFM2Pi 8 Values : [3020, 0, 0, 0, 239.31, 300, 300, 300, 300, 300, 300, 0]
2019-12-10 07:40:26,410 DEBUG RFM2Pi 8 Sent to channel(start)' : ToEmonCMS
2019-12-10 07:40:26,414 DEBUG RFM2Pi 8 Sent to channel(end)' : ToEmonCMS
2019-12-10 07:40:26,567 DEBUG MQTT Publishing: emon/emontx111/power1 3020
2019-12-10 07:40:26,574 DEBUG MQTT Publishing: emon/emontx111/power2 0
2019-12-10 07:40:26,581 DEBUG MQTT Publishing: emon/emontx111/power3 0
2019-12-10 07:40:26,589 DEBUG MQTT Publishing: emon/emontx111/power4 0
2019-12-10 07:40:26,596 DEBUG MQTT Publishing: emon/emontx111/vrms 239.31
2019-12-10 07:40:26,603 DEBUG MQTT Publishing: emon/emontx111/temp1 300
2019-12-10 07:40:26,610 DEBUG MQTT Publishing: emon/emontx111/temp2 300
2019-12-10 07:40:26,616 DEBUG MQTT Publishing: emon/emontx111/temp3 300
2019-12-10 07:40:26,624 DEBUG MQTT Publishing: emon/emontx111/temp4 300
2019-12-10 07:40:26,631 DEBUG MQTT Publishing: emon/emontx111/temp5 300
2019-12-10 07:40:26,639 DEBUG MQTT Publishing: emon/emontx111/temp6 300
2019-12-10 07:40:26,645 DEBUG MQTT Publishing: emon/emontx111/pulse 0
2019-12-10 07:40:26,653 INFO MQTT Publishing: emonhub/rx/10/values 3020,0,0,0,239.31,300,300,300,300,300,300,0
2019-12-10 07:40:34,162 DEBUG RFM2Pi 9 NEW FRAME : 19 209 0 0 0 141 1 23 0
2019-12-10 07:40:34,175 WARNING RFM2Pi Exception caught in RFM2Pi thread. Traceback (most recent call last):
File "/opt/openenergymonitor/emonhub/src/emonhub_interfacer.py", line 32, in wrapper
return f(*args)
File "/opt/openenergymonitor/emonhub/src/emonhub_interfacer.py", line 101, in run
rxc = self._process_rx(rxc)
File "/opt/openenergymonitor/emonhub/src/emonhub_interfacer.py", line 355, in _process_rx
for i in range(count):
TypeError: 'float' object cannot be interpreted as an integer
2019-12-10 07:40:34,219 WARNING MainThread RFM2Pi thread is dead.
2019-12-10 07:40:34,224 WARNING MainThread Attempting to restart thread RFM2Pi (thread has been restarted 0 times...
2019-12-10 07:40:34,231 INFO MainThread Creating EmonHubJeeInterfacer 'RFM2Pi'
2019-12-10 07:40:34,240 DEBUG MainThread Opening serial port: /dev/ttyAMA0 @ 9600 bits/s
2019-12-10 07:40:36,250 INFO MainThread RFM2Pi device firmware version & configuration: not available
2019-12-10 07:40:36,256 INFO MainThread Setting RFM2Pi baseid: 5 (5i)
2019-12-10 07:40:37,262 INFO MainThread Setting RFM2Pi frequency: 433 (4b)
2019-12-10 07:40:38,268 INFO MainThread Setting RFM2Pi group: 210 (210g)
2019-12-10 07:40:39,274 INFO MainThread Setting RFM2Pi quiet: 1 (1q)
2019-12-10 07:40:40,285 INFO MainThread Setting RFM2Pi calibration: 230V (1p)
2019-12-10 07:40:41,291 DEBUG MainThread Setting RFM2Pi pubchannels: ['ToEmonCMS']
2019-12-10 07:40:41,296 DEBUG MainThread Setting RFM2Pi subchannels: ['ToRFM12']
2019-12-10 07:40:41,310 DEBUG RFM2Pi acknowledged command: > 5i
2019-12-10 07:40:41,526 DEBUG RFM2Pi acknowledged command: > 4b
2019-12-10 07:40:41,756 DEBUG RFM2Pi acknowledged command: > 210g
2019-12-10 07:40:41,965 DEBUG RFM2Pi acknowledged command: > 1q
2019-12-10 07:40:42,072 DEBUG RFM2Pi acknowledged command: > 1p
2019-12-10 07:40:42,535 DEBUG RFM2Pi acknowledged command: <nn> i - set node ID (standard node ids are 1..26)
2019-12-10 07:40:42,661 DEBUG RFM2Pi acknowledged command: <n> b - set MHz band (4 = 433, 8 = 868, 9 = 915)
2019-12-10 07:40:42,841 DEBUG RFM2Pi acknowledged command: <nnn> g - set network group (RFM12 only allows 212, 0 = any)
2019-12-10 07:40:42,968 DEBUG RFM2Pi acknowledged command: <n> c - set collect mode (advanced, normally 0)
2019-12-10 07:40:43,093 DEBUG RFM2Pi acknowledged command: ...,<nn> a - send data packet to node <nn>, with ack
2019-12-10 07:40:43,218 DEBUG RFM2Pi acknowledged command: ...,<nn> s - send data packet to node <nn>, no ack
2019-12-10 07:40:43,342 DEBUG RFM2Pi acknowledged command: <n> l - turn activity LED on DIG8 on or off
2019-12-10 07:40:43,571 DEBUG RFM2Pi device settings updated: 69 i5 g210 @ 433 MHz Lock: 1
2019-12-10 07:40:46,456 DEBUG RFM2Pi 13 NEW FRAME : 10 203 11 0 0 0 0 0 0 118 93 48 117 48 117 48 117 48 117 48 117 48 117 0 0 0 0
Just so you know, I have 2 x HTTP interfacers and one MQTT running.
That’s a good bug! Fixed and pushed. I changed the version to v3-beta so that it’s clearer in the logs.
So seems to be working fine here.
Thanks @bwduncan for doing this!
I created a pull request to help with testing. (https://github.com/openenergymonitor/emonhub/pull/87) If I then do the following:
cd /opt/openenergymonitor/emonhub
git checkout -b bwduncan-python3 emon-pi
git pull https://github.com/bwduncan/emonhub.git python3
sudo service emonhub restart
I get the error:
emonhub.service: Failed with result 'exit-code'.
Running emonhub manually:
pi@emonpi:/opt/openenergymonitor/emonhub $ /usr/local/bin/emonhub/emonhub.py --config-file=/etc/emonhub/emonhub.conf
Traceback (most recent call last):
File "/usr/local/bin/emonhub/emonhub.py", line 23, in <module>
import emonhub_setup as ehs
File "/opt/openenergymonitor/emonhub/src/emonhub_setup.py", line 13, in <module>
from configobj import ConfigObj
ModuleNotFoundError: No module named 'configobj'
I must be missing a step or testing the wrong version?
Is that the right order (mine was a fresh install)?
do the pull
I think you need to add the remote?
Have you installed all the python3 packages? I ran the new installer (on top of the old one).
Hi @TrystanLea,
Thanks for testing!
I think the python3-configobj package isn’t installed. The install.sh script installs this package and I think I was relying on that script being run as part of the update to get all the modules. Not sure if that’s a valid assumption. Anyway, can you try running the script and see if it works?
Cheers,
Bruce
I think… (but I need to rebuild the emonpi from scratch to be sure)
git remote add bwduncan https://github.com/bwduncan/emonhub.git
git fetch bwduncan
git checkout bwduncan/python3
I think you need to install everything for python3
sudo apt-get update
sudo apt-get install -y python3-serial python3-configobj python3-pip
sudo pip3 install paho-mqtt requests