High CPU consumption caused by checking settings in emonhub

Tricx · 17 November 2016 20:38

I’m running the emonpi SD card image on a Raspberry Pi 1 and B+
Ok, it’s a little outdated but its purpose it to collect som edata and push it to my local emoncms server.

Unfortunately emonhub is producing quit high CPU load.
With default setinngs is consumes 30% of CPU time.
After disabling all interfacers, emonhub uses still 25-30% CPU time.

Is this a typical value?

So, I went through the run loop of class EmonHub and identified self._setup.run() as the cause.
Is there a need to check for updated settings with this frequency?
I don’t think it is worth spending so much MIPS on it, even on a RPi 2/3.

Thanks,
TriCX

pb66 · 18 November 2016 17:42

Whilst I have never looked or tested the CPU load consumed by emonHub, I am aware the settings are being checked a lot more frequently than was originally set.

Somewhere along the way the min once per second throttling has been removed, you could try editing the “>0” to “>1” in this line in emonhub_setup.py.

github.com

openenergymonitor/emonhub/blob/emon-pi/src/emonhub_setup.py#L124

    
      
          
          
def check_settings(self):
              """Check settings
              
              Update attribute settings and return True if modified.
              
              """
              
              # Check settings only once per second (could be extended if processing power is scarce)
              now = time.time()
              if now - self._settings_update_timestamp < 1:
                  return
              # Update timestamp
              self._settings_update_timestamp = now
              
              # Backup settings
              settings = dict(self.settings)
              
              # Get settings from file
              try:
                  if self._fileformat == "ConfigObj":

glyn.hudson · 21 November 2016 10:32

Do you think this should be a change we should make to emonhub? I see no reason why the settings need to be checked more frequently than every few seconds. However, emonhub has been proven running stable long term on all models of RasPi hardware with no issues.

sheffieldnick · 1 February 2017 22:31

I would definitely like to see this changed I’ve just done some tests on a Raspberry Pi 3 with nothing else running (just out-of-the-box emonpi SD), this is the emonhub python process CPU usage:

< 0: 11.9% - 13.9% (i.e., current sources)
< 1: 3.0 - 4.0%
< 5: 0.7 - 1.7%
<10: 0.3 - 1.3%

so it does make a huge difference even on a ‘fast’ Pi. I’m a novice with emonhub so I don’t know why it needs to keep checking to see if the config file has changed at all? Don’t most daemons read their config files at load time, and then only reload with a SIGHUP or SIGUSR? or “/etc/init.d/emonhub reload” type command?

Minimising the CPU and memory load just means that the box can do more tasks without being overloaded, so it is always worth a bit of optimisation IMHO! (I plan on monitoring several hires networked security cameras as well as running emonhub, for example.)

Thanks

glyn.hudson · 1 February 2017 23:42

Nice testing, thanks. Yes, this sounds sub-optimal. We are spoilt by the amount of processing power on a raspi3!

Could you try @pb66 suggestion?

github.com

openenergymonitor/emonhub/blob/emon-pi/src/emonhub_setup.py#L124


      
          
          def check_settings(self):
              """Check settings
              
              Update attribute settings and return True if modified.
              
              """
              
              # Check settings only once per second (could be extended if processing power is scarce)
              now = time.time()
              if now - self._settings_update_timestamp < 1:
                  return
              # Update timestamp
              self._settings_update_timestamp = now
              
              # Backup settings
              settings = dict(self.settings)
              
              # Get settings from file
              try:
                  if self._fileformat == "ConfigObj":

sheffieldnick · 1 February 2017 23:59

I assumed @pb66 had made a typo. Wouldn’t:

if now - self._settings_update_timestamp > 1:
    return

never update the config after the first second??

That is why I tried changing it to <1 or <5 or <10 in my tests.

pb66 · 2 February 2017 02:21

He did indeed! the suggestion was basically try swapping the 0 for a 1, I hadn’t noticed I got the arrows around the other way.

Thanks for supplying some data on the various times. The original value (from the OEMgateway) was 1s, this worked well and is the intended setting, I do not recall at what point the 1 became a 0, but the original emonhub was corrected back to a 1 many months ago.

Many of the settings in emonhub.conf are runtime settings, they can be changed on the fly when emonhub is running, that is an intentional feature.

The reason I would want to avoid the time being extended to 5 or 10 secs is the delay after saving the conf file before the settings are picked up can cause additional confusion, especially when the applied settings only effect data or features on a slow update 10s til the edited file is read plus 10s or more til the next data packet into emonhub, plus travel time to emoncms and screen update time, it can be a long time before a recent change is seen, in which time another edit of the settings could be applied etc so keeping the responsiveness of the runtimesettings sharp is beneficial.

This has proven to be even more necessary when it is possible to edit the settings from multiple places eg via emoncms. (especially when done correctly using configobj rather than editing the text file directly).

I think a 70% reduction on the current consumption with no loss of responsiveness is fine, each further second saved beyond that has only a minor impact.

The way emonhub settings are intended to work is to use predominantly default settings and only the alterations and personalised settings (eg url and apikey) NEED to be defined, this combined with a vast number of redundant node definitions means there is a lot of unnecessary conf file entries being checked for differences on the emonpi.

There is also a settings file for the init.d daemon located at /etc/default/emonhub, but this is a different thing altogether.

I’ve submitted PR for a return to 1s with a note that it could be extended if processing power is scarce, it hasn’t been problematic thus far so there seems little point making too drastic a change.

glyn.hudson · 2 February 2017 10:47

Tested and merged, thanks. Would it be possible to run-run your CPU test @sheffieldnick with the updated applied (Setup > Update emonPi in local Emoncms).

Tricx · 2 February 2017 10:47

Besides a simple timer to check for updates, I propose to make use of file system features instead or in addition.
What about monitoring the config-file’s “modified time”?
We could check it manually
https://docs.python.org/3/library/os.path.html#os.path.getmtime
or make use of a lib e.g.

Regard,
Tricx

sheffieldnick · 2 February 2017 11:02

Agreed! I was just about to suggest the same thing - test the file modified timestamp on the config file and only load it if it has changed. Far more efficient.

sheffieldnick · 2 February 2017 11:12

I’ve run Setup → Update emonPi but it won’t overwrite my modified emonhub_setup.py file. How do I force it forget about my edit and give me your latest release?

Thanks

glyn.hudson · 2 February 2017 11:14

running

$ git stash

in the ~/emonhub folder should do it

sheffieldnick · 2 February 2017 11:22

Ta, that did the trick (I had to do git config --global user.email "root@localhost" as well).

I can confirm that the update brings the CPU usage down to 3% - 4% on my Pi3. Not great, but a big improvement

Thanks

pb66 · 2 February 2017 13:54

Watching the files “modified time” and only running the check then is a good idea, I’m unsure how essential it is though.

I’m not against the idea, in fact I have just added it to my long emonhub “todo” list and linked this discussion.

I do not have the time look into it yet and I have concerns about making further alteration to a file that currently works well and has done so for a long time, just because I am currently fairly familiar with it and have several plans for improving and extending it’s feature set.

For example I would like to introduce another method of managing the settings via MQTT as part of a larger overhaul to make emonhub more geared up for use with smaller network packets to make gsm use more efficient and affordable. It would also mean that if emonCMS(.org) makes moves towards running a MQTT broker in place or as an alternative to http(s) for remote devices, as long as the remote device can publish data to emonCMS, it can also subscribe to emonCMS for configuration settings, as well as for control features without any complicated port forwarding or running a server on the remote.

Independently of the above I also made some progress some time ago with an emoncms “emonhub” module which would provide remote management of one or more emonHubs from one or more emonCMS servers, my implementation was different to the current implementation as it conversed directly with emonhub and emonhub then saved any configuration changes to the local conf file rather than needing access to the file directly…

Command line settings were also in the pipeline eg it is a little known fact that emonhub has provision to pause interfacers and/or delete and rebuild interfacers, this is currently managed by editing the config file but there is no reason an interfacer couldn’t be paused directly via the command line (or from within another script) eg when updating rfm2pi firmware automatically stop the rfm2pi interfacer only and continue to get data from other sources rather than stopping emonhub.

None of these things prevent the idea you suggest, I just do not see a huge amount of benefit in changing something that works for minimal gain and therefore cannot justify spending time on it right now, especially when there are other changes planned and there are some much higher priority things to tackle first.

But if you wish to continue I will happily review and at least provide feedback on any proposed changes. I do try to keep emonhub cross-platform, although a quick glance at the modules linked didn’t suggest an issue there, but prior to the “emonpi” variant emonhub ran perfectly on windows too, I have not tried this version though.

glyn.hudson · 2 February 2017 14:19

This would be great

@TrystanLea is working on this, currently testing with EmonESP

github.com

emoncms/development/blob/master/mymqttcloud/readme.md

# MQTT Cloud Example

End to end demo of an MQTT Cloud control of ESP8266 Wifi Relay, including:

- Multi-user authentication and access list control using jpmens mosquitto auth plugin with MySQL database.
- Basic PHP server app and javascript client for user creation, login, device list and control.
- Simplified Wifi Relay firmware based on emonESP that runs on Martin Harizanov's [WIFI relay board](https://shop.openenergymonitor.com/wifi-mqtt-relay-thermostat).

![screenshot.png](screenshot.png)

Notes on setting up mosquitto & mosquitto-auth-plug + basic demos: [learn_mqtt/readme.md](learn_mqtt/readme.md)

### EmonESP WIFI Relay Firmware

See folder: EmonESP_WIFIRelay

### Demo Database setup

Create database:

This file has been truncated. show original

ttps://github.com/emoncms/development/blob/master/mymqttcloud/

sheffieldnick · 2 February 2017 14:52

For a few lines of code I think it is worth making it run up to 10x faster

Here is some improved code for emonhub_setup.py that checks the file modified time:

import os

and

    # Initialize update timestamp
    self._settings_update_timestamp = 0
    self._retry_time_interval = 5
    self._settings_modified_timestamp = 0

and

    # Check settings only once per second (could be extended if processing power is scarce)
    now = time.time()
    if now - self._settings_update_timestamp < 1:
        return

    # Check if file has been modified since last parsed
    try:
        modified = os.path.getmtime(self._filename)
        if modified == self._settings_modified_timestamp:
            return
    except IOError:
        # Pass on any file errors to handlers below
        pass

    # Update timestamp
    self._settings_update_timestamp = now
    self._settings_modified_timestamp = modified

In my tests this reduces the CPU load from 3.0 - 4.0% (1 second reparse) to only 0.3 - 0.7% (1 second check file modified, reparse if changed).

pb66 · 2 February 2017 15:06

That doesn’t look so bad

i’ll take a closer look and give it a try when I can, do you know off-hand if “os.path.getmtime(self._filename)” is ok with windows too?

sheffieldnick · 2 February 2017 15:18

Looks like it should work fine on windows? This (old) thread says filemtime() is a good cross-platform function, but http://stackoverflow.com/questions/237079/how-to-get-file-creation-modification-date-times-in-python

pb66 · 2 February 2017 15:24

That looks promising, proof’s in the pudding! I’ll try it on windows too when I get a mo.

pb66 · 2 February 2017 15:41

Great, It would be good to know his thoughts and intention’s so we can pull in the same direction, is this a private project or a community collaboration?