emonPi new SD image - emonHub failing

boydo · 2020-06-22T09:29:27.868Z

borpin:

python src/emonhub.py --config-file=/etc/emonhub/emonhub.conf

Yep that’s now running after that command. On restart it does not run.

Tockley · 2020-06-22T09:42:48.472Z

TrystanLea:

python3 src/emonhub.py --config-file=/etc/emonhub/emonhub.conf

Yep, back up and running until ending process

TrystanLea · 2020-06-22T09:48:30.489Z

Thanks all

@Tockley @boydo did you just do the 3 commands I mentioned or did you run pip3 install paho-mqtt as well?

Tockley · 2020-06-22T09:54:58.714Z

TrystanLea:

python3 src/emonhub.py --config-file=/etc/emonhub/emonhub.conf

Just the 3 commands and the process runs live in terminal, MySolar app starts updating, etc. But of course closing terminal or interrupting process stops emonhub.

TrystanLea · 2020-06-22T09:56:11.197Z

ok great but also strange:

sudo service emonhub restart

should bring it back up in a way that allows it to continue once you exit.

Given that it has stopped before there is likely still an issue.

TrystanLea · 2020-06-22T10:01:16.342Z

Ok I’ve been able to replicate this here on a new system. Emonhub appears to work fine from the command line but the service fails after the first second or two. Im looking into it.

Tockley · 2020-06-22T10:03:47.675Z

TrystanLea:

ok great but also strange:
sudo service emonhub restart
should bring it back up in a way that allows it to continue once you exit.

Given that it has stopped before there is likely still an issue.

Alas no, does not come back up.

TrystanLea · 2020-06-22T10:17:26.819Z

Ok the quick fix is, similar to @robinrottier thanks! It needs the sudo:

sudo pip3 install paho-mqtt requests
sudo service emonhub restart

We will fix up the update process to do this automatically next

Jaddache · 2020-06-22T10:28:51.299Z

Thank you Trystan. Those two lines appear to have sorted my emonbase out. The emonhub service is now running again and the inputs are updating.

TrystanLea · 2020-06-22T10:42:53.252Z

Running emonPI/base update from the admin page will also now automatically fix this issue.

robinrottier · 2020-06-22T11:31:58.844Z

Were there any other changes to do with the emonhub socket listener? I have emonhub config entry like this:

[[mysocketlistener]]
        Type = EmonHubSocketInterfacer
        [[[init_settings]]]
                port_nb = 8080
        [[[runtimesettings]]]
                pubchannels = ToEmonCMS,

And now on startup it logs this the first time it receives something from the socket:

2020-06-22 11:57:21,546 WARNING  mysocketlistener Exception caught in mysocketlistener thread. Traceback (most recent call last):
  File "/opt/openenergymonitor/emonhub/src/emonhub_interfacer.py", line 32, in wrapper
    return f(*args)
  File "/opt/openenergymonitor/emonhub/src/emonhub_interfacer.py", line 99, in run
    rxc = self.read()
  File "/opt/openenergymonitor/emonhub/src/interfacers/EmonHubSocketInterfacer.py", line 78, in read
    self._sock_rx_buf = self._sock_rx_buf + conn.recv(1024)
TypeError: can only concatenate str (not "bytes") to str

The socket thread is then dead… it tries to restart that listener but its now stuck ina loop saying the socket already exists.

Before the update this listener was working fine receiving content posted from a python script on my victron/venus controller

To fix it I edited EmonHubSocketInterfacer.py line 78 as per the error and changed to:

        # Read data
        self._sock_rx_buf = self._sock_rx_buf + conn.recv(1024).decode("utf-8")

adding the decode to turn bytes into string. Restarted and its collecting values from the listener again.

boydo · 2020-06-22T12:27:22.643Z

TrystanLea:

Running emonPI/base update from the admin page will also now automatically fix this issue.

Sorted, many thanks.

borpin · 2020-06-22T13:22:12.732Z

robinrottier:

To fix it I edited EmonHubSocketInterfacer.py line 78 as per the error and changed to:
        # Read data
        self._sock_rx_buf = self._sock_rx_buf + conn.recv(1024).decode("utf-8")
adding the decode to turn bytes into string. Restarted and its collecting values from the listener again.

Could you open a PR on GitHub for this? Just edit the file in GitHub itself is the easiest way.

github.com

openenergymonitor/emonhub/blob/57aeef9a10d96ed25e5ca819014c7809ca90bed5/src/interfacers/EmonHubSocketInterfacer.py#L78-L78


      
          self._sock_rx_buf = self._sock_rx_buf + conn.recv(1024)

TrystanLea · 2020-06-22T13:55:03.645Z

Thanks @borpin @robinrottier I’ve applied the decode fix.

borpin · 2020-06-22T14:05:01.392Z

All the similar decodes do not specify the character set - perhaps this should not either to be consistent.

robinrottier · 2020-06-22T16:32:49.444Z

you’re probably right – I only used that it of code because I saw it somewhere on google and it worked for me, without having to think too much and I wanted my system to be back working asap on this nice sunny day.

I just changed my edit to :

self._sock_rx_buf = self._sock_rx_buf + conn.recv(1024).decode()

and it still parses my output in the socket listener, so great!

I’d suggest for a generic implementation you would read that string parameter value from the config of that instance of each socket listener, so different senders could write in whatever encoding they wanted to the socket listener; and have a missing or blank config to pick up the default, whatever that is (Id guess utf8 is default but I dont know). Maybe there might be something somewhere sending strings in utf16 or some other encoding and that would let you handle it with little extra effort

my real wonder would be why did it stop working – what changed in your update that meant the deocde() was now necessary and are there going to be other places that did not have a decode() that also now dont work?

TrystanLea · 2020-06-22T17:55:10.946Z

robinrottier:

my real wonder would be why did it stop working – what changed in your update that meant the deocde() was now necessary and are there going to be other places that did not have a decode() that also now dont work?

it’s probably something python3 related, that’s been the main and large change? @bwduncan do you know what it might be?

borpin · 2020-06-22T19:44:53.302Z

It was Bruce who told me to include the decode() statement in the Serial TX3e interfacer when that didn’t play. I added it to another decoder as well at some point I think.

bwduncan · 2020-06-22T21:32:19.989Z

Yes it could be that some bytes slipped through where text was expected. There really isn’t a reliable automatic way to detect all the places where this happens. You just have to look through all the code and know which interfaces are dealing with bytes and which are dealing with strings. Or you let them fail in production, which sadly seems to be the way to find them in this case Sorry about that.

I’ll have another human look at some of the interfacers and see if I can spot any suspect code.

UTF-8 is the default encoding in Python since like 3.5 I think. Almost everything we’ll be dealing with is using UTF-8, or ASCII which is a subset. I believe windows uses UTF-16 internally still, but I’d be surprised if you can show me a device which is sending something other than UTF-8!

Tockley · 2020-06-22T22:17:30.012Z

TrystanLea:

Running emonPI/base update from the admin page will also now automatically fix this issue.

Confirmed back and working through update.