Community
OpenEnergyMonitor

Community

Python3 for emonhub

Tags: #<Tag:0x00007f56b7b83728>

Hi folks,

I’ve got a python3 port of emonhub running on my emonpi. Everything seems to work but I’m aware that I’m testing a really small subset of what emonhub can do.

I’d really appreciate it if people who are in the position to do so could test this on the own systems. I’m more than happy to help to debug problems.

In the process of getting used to the code I tidied up a few things. I hope these changes are welcome.

Bruce

1 Like

@bwduncan, excellent.

@TrystanLea @glyn.hudson @pb66 will need to be involved in this I suspect and @Greebo has Python expertise as well IIRC.

Can you confirm what your test environment is? Stretch/Buster? EmonPi or RPi (which version)?

Starting at the top and looking at the README - much of the info regarding the installation of Mosquitto is now no longer necessary I suspect.

It may also be the case that we state that this is only supported for Buster onwards.

1 Like

Hi @borpin

This is a Raspberry Pi 2 Model B Rev 1.1 running Buster. It’s a standard emonPi from many years ago.

I suspect it would be much more reliable to build a debian package for this and let apt deal with the mosquitto/python/etc dependencies. That’s outside my expertise for now but I may be learning how to do this for work in the new year!

If we are to go down the path of deprecating support for older versions, there are some other bits which would welcome a cleaup (interfacer modules with the wrong name, for example). There’s almost 5000 lines of code, a lot of which is repeated, and newer python features could make this smaller and more readable. All that might be more effort than it’s worth, though, unless someone smarter than me can come up with a test suite!

Anyway, the python3 port has been running on my system for about 12 hours now.

Bruce

Which RFM card?

RFM69Pi, I think. How do I check? emonhub is configured to talk at 38400 over the serial line, if that helps.

2019-12-09 13:24:30,544 INFO     MainThread Logging level set to DEBUG
2019-12-09 13:24:30,545 INFO     MainThread Creating EmonHubJeeInterfacer 'RFM2Pi'
2019-12-09 13:24:30,548 DEBUG    MainThread Opening serial port: /dev/ttyAMA0 @ 38400 bits/s                                                                             
2019-12-09 13:24:32,564 INFO     MainThread RFM2Pi device firmware version: [emonPi.29.00]                                                                               
2019-12-09 13:24:32,565 INFO     MainThread RFM2Pi device current settings:  E i5 g210 @ 433 MHz q1 USA 0                                                                
2019-12-09 13:24:32,567 INFO     MainThread Setting RFM2Pi calibration: 230V (1p)
2019-12-09 13:24:33,569 DEBUG    MainThread Setting RFM2Pi pubchannels: ['ToEmonCMS']                                                                                    
2019-12-09 13:24:33,571 DEBUG    MainThread Setting RFM2Pi subchannels: ['ToRFM12']
2019-12-09 13:24:33,574 INFO     MainThread Creating EmonHubMqttInterfacer 'MQTT'
2019-12-09 13:24:33,579 DEBUG    RFM2Pi     device settings updated: E i5 g210 @ 433 MHz q1 USA 0                                                                        

Good question, not sure (It isn’t an EmonPi then). Look at your invoice (that is what I do!). I think there may be something in ‘Learn’ that helps identify it.

Indeed there is - photos of the various modules and pointers to the differences.
Learn→Electricity Monitoring→Networking→Which radio module→Identifying different Radio Modules

1 Like

Haha ok I’m not opening it up to figure out what radio module it has. It’s whatever came with the emonPi in January 2016. The invoice just says “emonPi”, so…

OK, The EmonPi is different to the EmonBase, hence my question. The EmonBase is just an RFM card that goes on any Pi. The EmonPi is (effectively) an EmonTX with an RFM module built in all in one case.

1 Like

So I’m trying this on an older B and an RFP12Pi. It is not picking up the config option for speed (which needs to be 9600).

2019-12-09 19:22:37,886 DEBUG    MainThread Opening serial port: /dev/ttyAMA0 @ 38400 bits/s

Fuller log

2019-12-09 19:23:27,588 WARNING  MainThread RFM2Pi thread is dead.
2019-12-09 19:23:27,593 WARNING  MainThread Attempting to restart thread RFM2Pi (thread has been restarted 4 times...
2019-12-09 19:23:27,598 INFO     MainThread Creating EmonHubJeeInterfacer 'RFM2Pi'
2019-12-09 19:23:27,604 DEBUG    MainThread Opening serial port: /dev/ttyAMA0 @ 38400 bits/s
2019-12-09 19:23:29,612 WARNING  MainThread Device communication error - check settings
2019-12-09 19:23:29,621 INFO     MainThread Setting RFM2Pi baseid: 12 (12i)
2019-12-09 19:23:30,627 INFO     MainThread Setting RFM2Pi frequency: 433 (4b)
2019-12-09 19:23:31,632 INFO     MainThread Setting RFM2Pi group: 210 (210g)
2019-12-09 19:23:32,637 INFO     MainThread Setting RFM2Pi quiet: 1 (1q)
2019-12-09 19:23:33,643 INFO     MainThread Setting RFM2Pi calibration: 230V (1p)
2019-12-09 19:23:34,648 DEBUG    MainThread Setting RFM2Pi pubchannels: ['ToEmonCMS']
2019-12-09 19:23:34,656 DEBUG    MainThread Setting RFM2Pi subchannels: ['ToRFM12']
2019-12-09 19:23:37,432 WARNING  RFM2Pi     Exception caught in RFM2Pi thread. Traceback (most recent call last):
  File "/opt/openenergymonitor/emonhub/src/emonhub_interfacer.py", line 32, in wrapper
    return f(*args)
  File "/opt/openenergymonitor/emonhub/src/emonhub_interfacer.py", line 99, in run
    rxc = self.read()
  File "/opt/openenergymonitor/emonhub/src/interfacers/EmonHubJeeInterfacer.py", line 84, in read
    self._rx_buf = self._rx_buf + self._ser.readline().decode()
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x80 in position 1: invalid start byte

Interestingly (this was the first time I’d used this Pi for this), I then installed the emon-pi variant (which worked), but the exception disappeared! It now just does not work and sits.

2019-12-09 19:40:22,282 INFO     MainThread EmonHub emonHub emon-pi variant v2.1.2
2019-12-09 19:40:22,291 INFO     MainThread Opening hub...
2019-12-09 19:40:22,295 INFO     MainThread Logging level set to DEBUG
2019-12-09 19:40:22,299 INFO     MainThread Creating EmonHubJeeInterfacer 'RFM2Pi'
2019-12-09 19:40:22,308 DEBUG    MainThread Opening serial port: /dev/ttyAMA0 @ 38400 bits/s
2019-12-09 19:40:24,342 ERROR    MainThread Unable to create 'RFM2Pi' interfacer: 'utf-8' codec can't decode byte 0x80 in position 1: invalid start byte
2019-12-09 19:40:24,345 INFO     MainThread Creating EmonHubMqttInterfacer 'MQTT'
2019-12-09 19:40:24,356 DEBUG    MainThread Setting MQTT pubchannels: ['ToRFM12']
2019-12-09 19:40:24,359 DEBUG    MainThread Setting MQTT subchannels: ['ToEmonCMS']
2019-12-09 19:40:24,362 INFO     MainThread Setting MQTT node_format_enable: 1
2019-12-09 19:40:24,365 INFO     MainThread Setting MQTT nodevar_format_enable: 1
2019-12-09 19:40:24,367 INFO     MainThread Setting MQTT nodevar_format_basetopic: emon/
2019-12-09 19:40:24,374 INFO     MainThread Creating EmonHubEmoncmsHTTPInterfacer 'emoncmsorg'
2019-12-09 19:40:24,379 DEBUG    MainThread Setting emoncmsorg pubchannels: ['ToRFM12']
2019-12-09 19:40:24,382 DEBUG    MainThread Setting emoncmsorg subchannels: ['ToEmonCMS']
2019-12-09 19:40:24,385 WARNING  MainThread Setting emoncmsorg apikey: obscured
2019-12-09 19:40:24,388 INFO     MainThread Setting emoncmsorg url: https://emoncms.org
2019-12-09 19:40:24,390 INFO     MainThread Setting emoncmsorg senddata: 1
2019-12-09 19:40:24,393 INFO     MainThread Setting emoncmsorg sendstatus: 1

Log of successful emon-pi start (same config file).

2019-12-09 19:46:36,631 INFO     MainThread EmonHub emonHub emon-pi variant v2.1.2
2019-12-09 19:46:36,634 INFO     MainThread Opening hub...
2019-12-09 19:46:36,637 INFO     MainThread Logging level set to DEBUG
2019-12-09 19:46:36,641 INFO     MainThread Creating EmonHubJeeInterfacer 'RFM2Pi'
2019-12-09 19:46:36,651 DEBUG    MainThread Opening serial port: /dev/ttyAMA0 @ 9600 bits/s
2019-12-09 19:46:38,659 INFO     MainThread RFM2Pi device firmware version & configuration: not available
2019-12-09 19:46:38,664 INFO     MainThread Setting RFM2Pi frequency: 433 (4b)
2019-12-09 19:46:39,670 INFO     MainThread Setting RFM2Pi group: 210 (210g)
2019-12-09 19:46:40,676 INFO     MainThread Setting RFM2Pi quiet: 0 (0q)
2019-12-09 19:46:41,682 INFO     MainThread Setting RFM2Pi baseid: 5 (5i)
2019-12-09 19:46:42,687 INFO     MainThread Setting RFM2Pi calibration: 230V (1p)
2019-12-09 19:46:43,693 DEBUG    MainThread Setting RFM2Pi subchannels: ['ToRFM12']
2019-12-09 19:46:43,702 DEBUG    MainThread Setting RFM2Pi pubchannels: ['ToEmonCMS']
2019-12-09 19:46:43,709 INFO     MainThread Creating EmonHubMqttInterfacer 'MQTT'
2019-12-09 19:46:43,725 DEBUG    MainThread Setting MQTT subchannels: ['ToEmonCMS']
2019-12-09 19:46:43,715 DEBUG    RFM2Pi     acknowledged command: > 4b
2019-12-09 19:46:43,733 DEBUG    MainThread Setting MQTT pubchannels: ['ToRFM12']
2019-12-09 19:46:43,738 INFO     MainThread Setting MQTT nodevar_format_enable: 1
2019-12-09 19:46:43,742 INFO     MainThread Setting MQTT node_format_enable: 1
2019-12-09 19:46:43,746 INFO     MainThread Setting MQTT nodevar_format_basetopic: emon/
2019-12-09 19:46:43,752 INFO     MainThread Creating EmonHubEmoncmsHTTPInterfacer 'emoncmsorg'
2019-12-09 19:46:43,760 DEBUG    MainThread Setting emoncmsorg subchannels: ['ToEmonCMS']
2019-12-09 19:46:43,764 DEBUG    MainThread Setting emoncmsorg pubchannels: ['ToRFM12']
2019-12-09 19:46:43,768 INFO     MainThread Setting emoncmsorg url: https://emoncms.org
2019-12-09 19:46:43,772 INFO     MainThread Setting emoncmsorg senddata: 1
2019-12-09 19:46:43,777 WARNING  MainThread Setting emoncmsorg apikey: obscured
2019-12-09 19:46:43,781 INFO     MainThread Setting emoncmsorg sendstatus: 1
2019-12-09 19:46:43,938 DEBUG    RFM2Pi     acknowledged command: > 210g
2019-12-09 19:46:44,153 DEBUG    RFM2Pi     1 NEW FRAME : 19 205 0 0 0 124 1 23 0
2019-12-09 19:46:44,163 DEBUG    RFM2Pi     1 Timestamp : 1575920804.15
2019-12-09 19:46:44,168 DEBUG    RFM2Pi     1 From Node : 19
2019-12-09 19:46:44,171 DEBUG    RFM2Pi     1    Values : [20.5, 0, 38, 2.3000000000000003]
2019-12-09 19:46:44,176 DEBUG    RFM2Pi     1 Sent to channel(start)' : ToEmonCMS
2019-12-09 19:46:44,180 DEBUG    RFM2Pi     1 Sent to channel(end)' : ToEmonCMS

@bwduncan would be useful if you changed the version your branch reports itself as.

Hi @borpin, good work, thank you very much. I’ve fixed that bug and bumped the version to 3 (hope that doesn’t step on any toes, it matches the compatible python version and it’s not like we’ll run out of numbers…). Thanks, Bruce

Version would I think be better as 3.0.0-beta so it can go to 3.0.0 when released. As it is a significant change, under semantic versioning, it should go to V3.0.0 when released.

Starts off OK

2019-12-10 07:39:40,523 INFO     MainThread Opening hub...
2019-12-10 07:39:40,527 INFO     MainThread Logging level set to DEBUG
2019-12-10 07:39:40,530 INFO     MainThread Creating EmonHubJeeInterfacer 'RFM2Pi'
2019-12-10 07:39:40,540 DEBUG    MainThread Opening serial port: /dev/ttyAMA0 @ 9600 bits/s
2019-12-10 07:39:42,549 INFO     MainThread RFM2Pi device firmware version & configuration: not available
2019-12-10 07:39:42,553 INFO     MainThread Setting RFM2Pi baseid: 5 (5i)
2019-12-10 07:39:43,558 INFO     MainThread Setting RFM2Pi frequency: 433 (4b)
2019-12-10 07:39:44,563 INFO     MainThread Setting RFM2Pi group: 210 (210g)
2019-12-10 07:39:45,568 INFO     MainThread Setting RFM2Pi quiet: 1 (1q)
2019-12-10 07:39:46,577 INFO     MainThread Setting RFM2Pi calibration: 230V (1p)
2019-12-10 07:39:47,582 DEBUG    MainThread Setting RFM2Pi pubchannels: ['ToEmonCMS']
2019-12-10 07:39:47,585 DEBUG    MainThread Setting RFM2Pi subchannels: ['ToRFM12']
2019-12-10 07:39:47,591 INFO     MainThread Creating EmonHubMqttInterfacer 'MQTT'
2019-12-10 07:39:47,599 DEBUG    RFM2Pi     acknowledged command: > 5i
2019-12-10 07:39:47,607 DEBUG    MainThread Setting MQTT pubchannels: ['ToRFM12']
2019-12-10 07:39:47,610 DEBUG    MainThread Setting MQTT subchannels: ['ToEmonCMS']
2019-12-10 07:39:47,613 INFO     MainThread Setting MQTT node_format_enable: 1
2019-12-10 07:39:47,616 INFO     MainThread Setting MQTT nodevar_format_enable: 1
2019-12-10 07:39:47,618 INFO     MainThread Setting MQTT nodevar_format_basetopic: emon/
2019-12-10 07:39:47,623 INFO     MainThread Creating EmonHubEmoncmsHTTPInterfacer 'emoncmsorg'
2019-12-10 07:39:47,629 DEBUG    MainThread Setting emoncmsorg pubchannels: ['ToRFM12']
2019-12-10 07:39:47,632 DEBUG    MainThread Setting emoncmsorg subchannels: ['ToEmonCMS']
2019-12-10 07:39:47,635 INFO     MainThread Setting emoncmsorg apikey: set
2019-12-10 07:39:47,639 INFO     MainThread Setting emoncmsorg url: http://192.168.7.47/emoncms
2019-12-10 07:39:47,641 INFO     MainThread Setting emoncmsorg senddata: 1
2019-12-10 07:39:47,645 INFO     MainThread Setting emoncmsorg sendstatus: 1
2019-12-10 07:39:47,650 INFO     MainThread Creating EmonHubEmoncmsHTTPInterfacer 'emoncm77'
2019-12-10 07:39:47,655 DEBUG    MainThread Setting emoncm77 pubchannels: ['ToRFM12']
2019-12-10 07:39:47,658 DEBUG    MainThread Setting emoncm77 subchannels: ['ToEmonCMS']
2019-12-10 07:39:47,661 INFO     MainThread Setting emoncm77 apikey: set
2019-12-10 07:39:47,664 INFO     MainThread Setting emoncm77 url: http://192.168.7.77/
2019-12-10 07:39:47,666 INFO     MainThread Setting emoncm77 senddata: 1
2019-12-10 07:39:47,669 INFO     MainThread Setting emoncm77 sendstatus: 1
2019-12-10 07:39:47,817 DEBUG    RFM2Pi     acknowledged command: > 4b
2019-12-10 07:39:48,029 DEBUG    RFM2Pi     acknowledged command: > 210g
2019-12-10 07:39:48,236 DEBUG    RFM2Pi     acknowledged command: > 1q

But dies quite quickly.

2019-12-10 07:40:26,387 DEBUG    RFM2Pi     8 NEW FRAME : 10 204 11 0 0 0 0 0 0 123 93 48 117 48 117 48 117 48 117 48 117 48 117 0 0 0 0
2019-12-10 07:40:26,397 DEBUG    RFM2Pi     8 Timestamp : 1575963626.386137
2019-12-10 07:40:26,402 DEBUG    RFM2Pi     8 From Node : 10
2019-12-10 07:40:26,406 DEBUG    RFM2Pi     8    Values : [3020, 0, 0, 0, 239.31, 300, 300, 300, 300, 300, 300, 0]
2019-12-10 07:40:26,410 DEBUG    RFM2Pi     8 Sent to channel(start)' : ToEmonCMS
2019-12-10 07:40:26,414 DEBUG    RFM2Pi     8 Sent to channel(end)' : ToEmonCMS
2019-12-10 07:40:26,567 DEBUG    MQTT       Publishing: emon/emontx111/power1 3020
2019-12-10 07:40:26,574 DEBUG    MQTT       Publishing: emon/emontx111/power2 0
2019-12-10 07:40:26,581 DEBUG    MQTT       Publishing: emon/emontx111/power3 0
2019-12-10 07:40:26,589 DEBUG    MQTT       Publishing: emon/emontx111/power4 0
2019-12-10 07:40:26,596 DEBUG    MQTT       Publishing: emon/emontx111/vrms 239.31
2019-12-10 07:40:26,603 DEBUG    MQTT       Publishing: emon/emontx111/temp1 300
2019-12-10 07:40:26,610 DEBUG    MQTT       Publishing: emon/emontx111/temp2 300
2019-12-10 07:40:26,616 DEBUG    MQTT       Publishing: emon/emontx111/temp3 300
2019-12-10 07:40:26,624 DEBUG    MQTT       Publishing: emon/emontx111/temp4 300
2019-12-10 07:40:26,631 DEBUG    MQTT       Publishing: emon/emontx111/temp5 300
2019-12-10 07:40:26,639 DEBUG    MQTT       Publishing: emon/emontx111/temp6 300
2019-12-10 07:40:26,645 DEBUG    MQTT       Publishing: emon/emontx111/pulse 0
2019-12-10 07:40:26,653 INFO     MQTT       Publishing: emonhub/rx/10/values 3020,0,0,0,239.31,300,300,300,300,300,300,0
2019-12-10 07:40:34,162 DEBUG    RFM2Pi     9 NEW FRAME : 19 209 0 0 0 141 1 23 0
2019-12-10 07:40:34,175 WARNING  RFM2Pi     Exception caught in RFM2Pi thread. Traceback (most recent call last):
  File "/opt/openenergymonitor/emonhub/src/emonhub_interfacer.py", line 32, in wrapper
    return f(*args)
  File "/opt/openenergymonitor/emonhub/src/emonhub_interfacer.py", line 101, in run
    rxc = self._process_rx(rxc)
  File "/opt/openenergymonitor/emonhub/src/emonhub_interfacer.py", line 355, in _process_rx
    for i in range(count):
TypeError: 'float' object cannot be interpreted as an integer

2019-12-10 07:40:34,219 WARNING  MainThread RFM2Pi thread is dead.
2019-12-10 07:40:34,224 WARNING  MainThread Attempting to restart thread RFM2Pi (thread has been restarted 0 times...
2019-12-10 07:40:34,231 INFO     MainThread Creating EmonHubJeeInterfacer 'RFM2Pi'
2019-12-10 07:40:34,240 DEBUG    MainThread Opening serial port: /dev/ttyAMA0 @ 9600 bits/s
2019-12-10 07:40:36,250 INFO     MainThread RFM2Pi device firmware version & configuration: not available
2019-12-10 07:40:36,256 INFO     MainThread Setting RFM2Pi baseid: 5 (5i)
2019-12-10 07:40:37,262 INFO     MainThread Setting RFM2Pi frequency: 433 (4b)
2019-12-10 07:40:38,268 INFO     MainThread Setting RFM2Pi group: 210 (210g)
2019-12-10 07:40:39,274 INFO     MainThread Setting RFM2Pi quiet: 1 (1q)
2019-12-10 07:40:40,285 INFO     MainThread Setting RFM2Pi calibration: 230V (1p)
2019-12-10 07:40:41,291 DEBUG    MainThread Setting RFM2Pi pubchannels: ['ToEmonCMS']
2019-12-10 07:40:41,296 DEBUG    MainThread Setting RFM2Pi subchannels: ['ToRFM12']
2019-12-10 07:40:41,310 DEBUG    RFM2Pi     acknowledged command: > 5i
2019-12-10 07:40:41,526 DEBUG    RFM2Pi     acknowledged command: > 4b
2019-12-10 07:40:41,756 DEBUG    RFM2Pi     acknowledged command: > 210g
2019-12-10 07:40:41,965 DEBUG    RFM2Pi     acknowledged command: > 1q
2019-12-10 07:40:42,072 DEBUG    RFM2Pi     acknowledged command: > 1p
2019-12-10 07:40:42,535 DEBUG    RFM2Pi     acknowledged command: <nn> i     - set node ID (standard node ids are 1..26)
2019-12-10 07:40:42,661 DEBUG    RFM2Pi     acknowledged command: <n> b      - set MHz band (4 = 433, 8 = 868, 9 = 915)
2019-12-10 07:40:42,841 DEBUG    RFM2Pi     acknowledged command: <nnn> g    - set network group (RFM12 only allows 212, 0 = any)
2019-12-10 07:40:42,968 DEBUG    RFM2Pi     acknowledged command: <n> c      - set collect mode (advanced, normally 0)
2019-12-10 07:40:43,093 DEBUG    RFM2Pi     acknowledged command: ...,<nn> a - send data packet to node <nn>, with ack
2019-12-10 07:40:43,218 DEBUG    RFM2Pi     acknowledged command: ...,<nn> s - send data packet to node <nn>, no ack
2019-12-10 07:40:43,342 DEBUG    RFM2Pi     acknowledged command: <n> l      - turn activity LED on DIG8 on or off
2019-12-10 07:40:43,571 DEBUG    RFM2Pi     device settings updated: 69 i5 g210 @ 433 MHz  Lock: 1
2019-12-10 07:40:46,456 DEBUG    RFM2Pi     13 NEW FRAME : 10 203 11 0 0 0 0 0 0 118 93 48 117 48 117 48 117 48 117 48 117 48 117 0 0 0 0

Just so you know, I have 2 x HTTP interfacers and one MQTT running.

1 Like

That’s a good bug! Fixed and pushed. I changed the version to v3-beta so that it’s clearer in the logs.

1 Like

So seems to be working fine here.

1 Like

Thanks @bwduncan for doing this!

I created a pull request to help with testing. (https://github.com/openenergymonitor/emonhub/pull/87) If I then do the following:

cd /opt/openenergymonitor/emonhub
git checkout -b bwduncan-python3 emon-pi 
git pull https://github.com/bwduncan/emonhub.git python3
sudo service emonhub restart

I get the error:

emonhub.service: Failed with result 'exit-code'.

Running emonhub manually:

[email protected]:/opt/openenergymonitor/emonhub $ /usr/local/bin/emonhub/emonhub.py --config-file=/etc/emonhub/emonhub.conf
Traceback (most recent call last):
  File "/usr/local/bin/emonhub/emonhub.py", line 23, in <module>
    import emonhub_setup as ehs
  File "/opt/openenergymonitor/emonhub/src/emonhub_setup.py", line 13, in <module>
    from configobj import ConfigObj
ModuleNotFoundError: No module named 'configobj'

I must be missing a step or testing the wrong version?

Is that the right order (mine was a fresh install)?

do the pull

I think you need to add the remote?

Have you installed all the python3 packages? I ran the new installer (on top of the old one).

1 Like

Hi @TrystanLea,

Thanks for testing!

I think the python3-configobj package isn’t installed. The install.sh script installs this package and I think I was relying on that script being run as part of the update to get all the modules. Not sure if that’s a valid assumption. Anyway, can you try running the script and see if it works?

Cheers,
Bruce

1 Like

I think… (but I need to rebuild the emonpi from scratch to be sure)

git remote add bwduncan https://github.com/bwduncan/emonhub.git
git fetch bwduncan
git checkout bwduncan/python3

I think you need to install everything for python3

sudo apt-get update
sudo apt-get install -y python3-serial python3-configobj python3-pip
sudo pip3 install paho-mqtt requests