Python3 for emonhub

djh · 2019-12-17T19:27:14.485Z

bwduncan:

I do need to make another change to the pull request because python3 tries harder than python2 did to output accurate floating point numbers, so we end up getting numbers represented with more precision than they actually have. … The solution is to give the output exact rounding, but I need to take some time to figure out all the affected output paths.

I’d suggest putting in the pull request as is with a release note asking testers to report cases where rounding is necessary. That’ll be an easier and probably more reliable way to fix all the formatting and will expose the rest of the changes to testing earlier.

edit: It’s pretty annoying to have the system automatically make changes to my post without letting me review it first. I’d appreciate changing the system setting so it doesn’t do that, if possible.

borpin · 2019-12-17T19:31:13.519Z

djh:

edit: It’s pretty annoying to have the system automatically make changes to my post without letting me review it first. I’d appreciate changing the system setting so it doesn’t do that, if possible.

What did it change?

Oh, I see. Simple, don’t quote a whole post! It is a Discourse thing.

TrystanLea · 2019-12-17T20:21:09.616Z

We’ve been running the Python3 version in the office without any issues evident so far and I’ve just updated my home system with this now. Thanks again @bwduncan for your work on this.

I’ve created a python3 branch on the main emonhub respository and merged your changes in there for the time being as testing is ongoing https://github.com/openenergymonitor/emonhub/tree/python3

The next step is to add the required python3 installation steps into the EmonSripts emonhub update script here EmonScripts/emonhub.sh at master · openenergymonitor/EmonScripts · GitHub and test this process end to end to make sure the automatic update process is seamless for people.

borpin · 2019-12-31T07:58:27.083Z

Running the @bwduncan branch for a couple of weeks here.

Pi 3B - nothing else running.

image1376×186 18.6 KB

pi@raspberrypi:/usr/local/bin/emonhub $ systemctl status emonhub.service
● emonhub.service - emonHub service description
   Loaded: loaded (/opt/openenergymonitor/emonhub/service/emonhub.service; enabled; vendor preset: enabled)
   Active: active (running) since Tue 2019-12-10 15:21:29 GMT; 2 weeks 6 days ago
  Process: 2380 ExecStartPre=/bin/mkdir -p /var/log/emonhub/ (code=exited, status=0/SUCCESS)
  Process: 2381 ExecStartPre=/bin/chgrp -R emonhub /var/log/emonhub/ (code=exited, status=0/SUCCESS)
  Process: 2382 ExecStartPre=/bin/chmod 775 /var/log/emonhub/ (code=exited, status=0/SUCCESS)
 Main PID: 2383 (python3)
   Memory: 19.1M
   CGroup: /system.slice/emonhub.service
           └─2383 python3 /usr/local/bin/emonhub/emonhub.py --config-file=/etc/emonhub/emonhub.conf --logfile=/var/log/emonhub/emonhub.log

Dec 10 15:21:28 raspberrypi systemd[1]: Starting emonHub service description...
Dec 10 15:21:29 raspberrypi systemd[1]: Started emonHub service description.

In that time EmonCMS reports Quality of 99% overall. Long periods of 100%

All seems good to me.

One note is that the CPU load is reasonably high. I discovered on my lower power Orange Pi, one of the issues I was having was high CPU load causing missed readings.

@TrystanLea, I would like to see the ability to switch off the internal logrotate so a larger logging period can be made available.

bwduncan · 2020-01-01T01:03:55.510Z

That’s great, thanks for the feedback.

I have noticed that emonhub does some tight loops with short timeouts in some places. For example I think there is a 100ms sleep in a loop somewhere which is only there to check if the config file has changed. No user can change their config so quickly! Usually one can reasonably expect to have to ask for a config file to be reloaded, but changing that would be a big user experience change! I think it would be reasonable to reduce the polling to 1s or 10s, for example, without impacting anyone.

The first rule of optimisation is to measure, though, and I’d be happy to help with that, too.

Re logging, maybe that should be discussed in a new thread? I’d recommend turning on syslog logging to avoid losing logs. Send them wherever makes sense for your setup.

Happy new year!
Bruce

borpin · 2020-01-02T15:55:05.059Z

3 posts were split to a new topic: Emonhub logging options

alexandrecuer · 2020-01-24T16:06:02.191Z

running also fine from my side

if creating a fresh new card with the scripts, in order to have the emonPiLCD service running, you still need to install manually :

sudo pip install paho-mqtt requests

as install.sh from emonhub does not include it anymore using pip but pip3

@TrystanLea : maybe a temporary fix could be to add the command sudo pip install paho-mqtt requests in the emonPiLCD_detect.sh script from the emonpi repo…do yu want a PR in that way ?

best

Alex

bwduncan · 2020-02-06T18:21:37.834Z

Hi Alex,

Sorry for the slow reply. Discourse didn’t notify me for some reason despite

You will receive notifications because you created this topic.

Thanks for the feedback! It might be easier just to move emonPiLCD to python3 at the same time. I have ported this already so it should be pretty quick to test.

I’ll make another PR.

Bruce

alexandrecuer · 2020-03-19T11:54:28.942Z

@bwduncan: for emonpiLCD python3, I suppose I have to use your branch python3-fixes ?
[edit]
I’ve tested your branch and managed to make it work

The class IPAddress seems not to work any more with python3 :

class IPAddress:
    def __init__(self):
        self.sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)

    def get_ip_address(self, ifname):
        try:
            return socket.inet_ntoa(fcntl.ioctl(
                self.sock.fileno(),
                0x8915,  # SIOCGIFADDR
                struct.pack('256s', ifname[:15])
            )[20:24])
        except Exception:
            return 0

this is working :

def get_ip():
    s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
    try:
        # doesn't even have to be reachable
        s.connect(('10.255.255.255', 1))
        IP = s.getsockname()[0]
    except:
        IP = '127.0.0.1'
    finally:
        s.close()
    return IP

Moreover, each time the script calls r.get, I had to add a decode(‘utf8’)
cf page 3 for example :

lcd[0] = feed1_name + ':' + r.get("feed1").decode('utf8') + feed1_unit

or page 4 and 5 :

basedata = r.get("basedata").decode('utf8')

moreover there is some hardcoded paths, like in the LCD class…it should be something like that :

lcd_status = subprocess.check_output([path+"/emonPiLCD_detect.sh", "%s" % i2c_address])

with path defined at the top of the script by :

path=os.path.dirname(os.path.realpath(__file__))

anyway the subprocess.check_output command is not working…

best

Alex

bwduncan · 2020-03-19T23:25:11.884Z

Hi Alex,

Thanks, that’s very helpful! I don’t think we need to throw out IPAddress. It just needs ifname[:15] to be replaced with ifname[:15].encode('utf-8'). That except clause has always been too broad. We should probably just remove it.

It seems like redis is equipped to handle text data. We should probably add charset="utf-8" and decode_responses=True to the Redis object. Can you test that? This might be useful Python-redis keys() returns list of bytes objects instead of strings - Stack Overflow

I agree about the hard-coded paths, and the use of subprocess at all. There’s nothing that couldn’t be done using Python in there, but I think it’s outside the scope of porting to Python 3.

Thanks,
Bruce

alexandrecuer · 2020-03-20T08:15:32.555Z

checked out to your master branch and did only the following mods to make the emonPiLCD.py file works :

r = redis.Redis(host=redis_host, port=redis_port, db=0, charset="utf-8", decode_responses=True)

class IPAddress:

struct.pack('256s', ifname[:15].encode('utf-8'))

class LCD

lcd_status=subprocess.run([path+"/emonPiLCD_detect.sh", "%s" % i2c_address])
            if lcd_status:
            ...........

if not lcd_status:

to make things clean, I’ve created a PR to your master branch of emonpi

github.com/bwduncan/emonpi

python3 fixes next

bwduncan:master ← dromotherm:master

opened 03:40PM - 20 Mar 20 UTC

alexandrecuer

+11 -13

a simple PR with all changes tested to make emonPiLCD.service work with python3 … tested on SD created with the EmonScripts best Alex

EDIT : @TrystanLea
some things will have to be modified in the EmonScripts, in the install/emonpilcd.sh file

sudo apt-get install -y python3-smbus i2c-tools python3-rpi.gpio python3-gpiozero
sudo pip3 install xmltodict

or

sudo apt-get install python3-smbus i2c-tools python3-rpi.gpio python3-pip redis-server  python3-gpiozero -y
sudo pip3 install redis paho-mqtt xmltodict requests

best

Alex

borpin · 2020-03-24T14:52:57.720Z

@bwduncan, just trying the serialinterfacer - I see you have a note it may need more debugging!

github.com/openenergymonitor/emonhub

python3 - serial interfacer

opened 02:49PM - 24 Mar 20 UTC

closed 08:13PM - 09 Jun 20 UTC

borpin

@bwduncan Using either the `EmonHubSerialInterfacer` or the `EmonHubTx3eInter…facer` the following error is seen; ``` 2020-03-24 14:43:08,760 WARNING MainThread Attempting to restart thread SerialTx (thread has been restarted 68 times... 2020-03-24 14:43:08,763 INFO MainThread Creating EmonHubSerialInterfacer 'SerialTx' 2020-03-24 14:43:08,769 DEBUG MainThread Opening serial port: /dev/ttyAMA0 @ 115200 bits/s 2020-03-24 14:43:08,772 DEBUG MainThread Setting SerialTx pubchannels: ['ToEmonCMS'] 2020-03-24 14:43:18,742 WARNING SerialTx Exception caught in SerialTx thread. Traceback (most recent call last): File "/opt/openenergymonitor/emonhub/src/emonhub_interfacer.py", line 32, in wrapper return f(*args) File "/opt/openenergymonitor/emonhub/src/emonhub_interfacer.py", line 99, in run rxc = self.read() File "/opt/openenergymonitor/emonhub/src/interfacers/EmonHubSerialInterfacer.py", line 90, in read c.nodeid = int(f[0]) ValueError: invalid literal for int() with base 10: 'MSG:71923,Vrms:246.78,P1:157,E1:16346,pulse:15' 2020-03-24 14:43:18,758 WARNING MainThread SerialTx thread is dead. ``` This is a shop emonTX - no mods.

[edit]
Definitely a python 3 issue - went back to the old version and it works fine.

TrystanLea · 2020-05-26T12:36:59.393Z

Trying this again today, I seem to be back to the same error that I had above. This time I did run the install script for python3 dependencies but it didnt sort the issue as it did last time… a bit strange…

pi@emonpi:/opt/openenergymonitor/emonhub $ python3 src/emonhub.py --config-file=/etc/emonhub/emonhub.conf
Traceback (most recent call last):
  File "src/emonhub.py", line 23, in <module>
    import emonhub_setup as ehs
  File "/opt/openenergymonitor/emonhub/src/emonhub_setup.py", line 13, in <module>
    from configobj import ConfigObj

borpin · 2020-05-26T12:42:39.642Z

I modified the emonscript ini file so it ran the right installer as I ran the install. I did run the installer separately last time and it worked fine.

TBH, I think I just checked out the new branch and ran the installer and restarted emonhub service IIRC.

I suspect the update path of this is going to be the most problematic part of this.

borpin · 2020-05-26T12:43:18.014Z

Oh and the problem here was solved on GitHub.

TrystanLea · 2020-05-26T12:43:27.099Z

Oops, silly mistake, I didnt run sudo apt-get update before running install working now!

TrystanLea · 2020-05-26T13:04:28.170Z

I’ve added python3 support to the tesla powerwall interfacer - may need verify = false in the request…
fix python3 support for tesla powerwall interfacer · openenergymonitor/emonhub@da27a4a · GitHub

borpin · 2020-05-26T13:13:13.890Z

@TrystanLea, Is this a good opportunity to move to a ‘stable’ branch and do releases from there? The Python 2 emonpi branch could then be left alone.

TrystanLea · 2020-05-26T13:18:02.408Z

Good point, yes that would be good

TrystanLea · 2020-05-26T14:44:11.348Z

I think this should be all that is needed to both install the python3 version of emonhub on new installs and update to python3 on existing systems via emonPi/base update: Install and update python3 version of emonhub by TrystanLea · Pull Request #87 · openenergymonitor/EmonScripts · GitHub

Leaving that as an open pull request now for discussion and testing.

Would you also like me to merge Python3 fixes by TrystanLea · Pull Request #99 · openenergymonitor/emonhub · GitHub @bwduncan?