I do need to make another change to the pull request because python3 tries harder than python2 did to output accurate floating point numbers, so we end up getting numbers represented with more precision than they actually have. For example, my emonpi t1 probe is registering 8.200000000000001, when clearly the emonpi only recorded a value of 8.2. Since this can’t be represented accurately as a floating point number it gets munged on its way through emonhub. The solution is to give the output exact rounding, but I need to take some time to figure out all the affected output paths.
1 Like
I’d suggest putting in the pull request as is with a release note asking testers to report cases where rounding is necessary. That’ll be an easier and probably more reliable way to fix all the formatting and will expose the rest of the changes to testing earlier.
edit: It’s pretty annoying to have the system automatically make changes to my post without letting me review it first. I’d appreciate changing the system setting so it doesn’t do that, if possible.
What did it change?
Oh, I see. Simple, don’t quote a whole post! It is a Discourse thing.
We’ve been running the Python3 version in the office without any issues evident so far and I’ve just updated my home system with this now. Thanks again @bwduncan for your work on this.
I’ve created a python3 branch on the main emonhub respository and merged your changes in there for the time being as testing is ongoing https://github.com/openenergymonitor/emonhub/tree/python3
The next step is to add the required python3 installation steps into the EmonSripts emonhub update script here EmonScripts/emonhub.sh at master · openenergymonitor/EmonScripts · GitHub and test this process end to end to make sure the automatic update process is seamless for people.
1 Like
Running the @bwduncan branch for a couple of weeks here.
Pi 3B - nothing else running.
pi@raspberrypi:/usr/local/bin/emonhub $ systemctl status emonhub.service
● emonhub.service - emonHub service description
Loaded: loaded (/opt/openenergymonitor/emonhub/service/emonhub.service; enabled; vendor preset: enabled)
Active: active (running) since Tue 2019-12-10 15:21:29 GMT; 2 weeks 6 days ago
Process: 2380 ExecStartPre=/bin/mkdir -p /var/log/emonhub/ (code=exited, status=0/SUCCESS)
Process: 2381 ExecStartPre=/bin/chgrp -R emonhub /var/log/emonhub/ (code=exited, status=0/SUCCESS)
Process: 2382 ExecStartPre=/bin/chmod 775 /var/log/emonhub/ (code=exited, status=0/SUCCESS)
Main PID: 2383 (python3)
Memory: 19.1M
CGroup: /system.slice/emonhub.service
└─2383 python3 /usr/local/bin/emonhub/emonhub.py --config-file=/etc/emonhub/emonhub.conf --logfile=/var/log/emonhub/emonhub.log
Dec 10 15:21:28 raspberrypi systemd[1]: Starting emonHub service description...
Dec 10 15:21:29 raspberrypi systemd[1]: Started emonHub service description.
In that time EmonCMS reports Quality of 99% overall. Long periods of 100%
All seems good to me.
One note is that the CPU load is reasonably high. I discovered on my lower power Orange Pi, one of the issues I was having was high CPU load causing missed readings.
@TrystanLea, I would like to see the ability to switch off the internal logrotate so a larger logging period can be made available.
1 Like
That’s great, thanks for the feedback.
I have noticed that emonhub does some tight loops with short timeouts in some places. For example I think there is a 100ms sleep in a loop somewhere which is only there to check if the config file has changed. No user can change their config so quickly! Usually one can reasonably expect to have to ask for a config file to be reloaded, but changing that would be a big user experience change! I think it would be reasonable to reduce the polling to 1s or 10s, for example, without impacting anyone.
The first rule of optimisation is to measure, though, and I’d be happy to help with that, too.
Re logging, maybe that should be discussed in a new thread? I’d recommend turning on syslog logging to avoid losing logs. Send them wherever makes sense for your setup.
Happy new year!
Bruce
1 Like
3 posts were split to a new topic: Emonhub logging options
running also fine from my side
if creating a fresh new card with the scripts, in order to have the emonPiLCD service running, you still need to install manually :
sudo pip install paho-mqtt requests
as install.sh from emonhub does not include it anymore using pip but pip3
@TrystanLea : maybe a temporary fix could be to add the command sudo pip install paho-mqtt requests in the emonPiLCD_detect.sh script from the emonpi repo…do yu want a PR in that way ?
best
Alex
1 Like
Hi Alex,
Sorry for the slow reply. Discourse didn’t notify me for some reason despite
You will receive notifications because you created this topic.
Thanks for the feedback! It might be easier just to move emonPiLCD to python3 at the same time. I have ported this already so it should be pretty quick to test.
I’ll make another PR.
Bruce
1 Like
@bwduncan: for emonpiLCD python3, I suppose I have to use your branch python3-fixes ?
[edit]
I’ve tested your branch and managed to make it work
The class IPAddress seems not to work any more with python3 :
class IPAddress:
def __init__(self):
self.sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
def get_ip_address(self, ifname):
try:
return socket.inet_ntoa(fcntl.ioctl(
self.sock.fileno(),
0x8915, # SIOCGIFADDR
struct.pack('256s', ifname[:15])
)[20:24])
except Exception:
return 0
this is working :
def get_ip():
s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
try:
# doesn't even have to be reachable
s.connect(('10.255.255.255', 1))
IP = s.getsockname()[0]
except:
IP = '127.0.0.1'
finally:
s.close()
return IP
Moreover, each time the script calls r.get, I had to add a decode(‘utf8’)
cf page 3 for example :
lcd[0] = feed1_name + ':' + r.get("feed1").decode('utf8') + feed1_unit
or page 4 and 5 :
basedata = r.get("basedata").decode('utf8')
moreover there is some hardcoded paths, like in the LCD class…it should be something like that :
lcd_status = subprocess.check_output([path+"/emonPiLCD_detect.sh", "%s" % i2c_address])
with path defined at the top of the script by :
path=os.path.dirname(os.path.realpath(__file__))
anyway the subprocess.check_output command is not working…
best
Alex
Hi Alex,
Thanks, that’s very helpful! I don’t think we need to throw out IPAddress. It just needs ifname[:15] to be replaced with ifname[:15].encode('utf-8'). That except clause has always been too broad. We should probably just remove it.
It seems like redis is equipped to handle text data. We should probably add charset="utf-8" and decode_responses=True to the Redis object. Can you test that? This might be useful Python-redis keys() returns list of bytes objects instead of strings - Stack Overflow
I agree about the hard-coded paths, and the use of subprocess at all. There’s nothing that couldn’t be done using Python in there, but I think it’s outside the scope of porting to Python 3.
Thanks,
Bruce
checked out to your master branch and did only the following mods to make the emonPiLCD.py file works :
r = redis.Redis(host=redis_host, port=redis_port, db=0, charset="utf-8", decode_responses=True)
class IPAddress:
struct.pack('256s', ifname[:15].encode('utf-8'))
class LCD
lcd_status=subprocess.run([path+"/emonPiLCD_detect.sh", "%s" % i2c_address])
if lcd_status:
...........
if not lcd_status:
to make things clean, I’ve created a PR to your master branch of emonpi
EDIT : @TrystanLea
some things will have to be modified in the EmonScripts, in the install/emonpilcd.sh file
sudo apt-get install -y python3-smbus i2c-tools python3-rpi.gpio python3-gpiozero
sudo pip3 install xmltodict
or
sudo apt-get install python3-smbus i2c-tools python3-rpi.gpio python3-pip redis-server python3-gpiozero -y
sudo pip3 install redis paho-mqtt xmltodict requests
best
Alex
1 Like
@bwduncan, just trying the serialinterfacer - I see you have a note it may need more debugging!
[edit]
Definitely a python 3 issue - went back to the old version and it works fine.
Trying this again today, I seem to be back to the same error that I had above. This time I did run the install script for python3 dependencies but it didnt sort the issue as it did last time… a bit strange…
pi@emonpi:/opt/openenergymonitor/emonhub $ python3 src/emonhub.py --config-file=/etc/emonhub/emonhub.conf
Traceback (most recent call last):
File "src/emonhub.py", line 23, in <module>
import emonhub_setup as ehs
File "/opt/openenergymonitor/emonhub/src/emonhub_setup.py", line 13, in <module>
from configobj import ConfigObjI modified the emonscript ini file so it ran the right installer as I ran the install. I did run the installer separately last time and it worked fine.
TBH, I think I just checked out the new branch and ran the installer and restarted emonhub service IIRC.
I suspect the update path of this is going to be the most problematic part of this.
1 Like
Oh and the problem here was solved on GitHub.
Oops, silly mistake, I didnt run sudo apt-get update before running install working now!
1 Like
I’ve added python3 support to the tesla powerwall interfacer - may need verify = false in the request…
fix python3 support for tesla powerwall interfacer · openenergymonitor/emonhub@da27a4a · GitHub
@TrystanLea, Is this a good opportunity to move to a ‘stable’ branch and do releases from there? The Python 2 emonpi branch could then be left alone.
Good point, yes that would be good
1 Like