Emonhub - Reduce CPU usage

Hi All,

I was digging through emonhub code for an unrelated issue and found this in emonhub_setup.py

  # Check settings only once per second (could be extended if processing power is scarce)
        now = time.time()
        if now - self._settings_update_timestamp < 1:
            return
        # Update timestamp
        self._settings_update_timestamp = now

     #more code which reparses the config follows...

Looking deeper at the code it appears to re-read the config file every second and check for changes.
The comment at the top of emonhub_setup.py seems to confirm this:


The check_settings() method is run regularly as well. It checks the settings
and returns True is settings were changed.

On my pi-Zero-W emonhub takes around 12% cpu.
I changed the 1 second timeout above to 15 and emonhub dropped to circa 2% CPU.

If this does indeed reparse the config every second, I think weā€™d agree this is overkill and a significant saving could be made by optimising it.

Can someone more knowledgable than me verify this ?

If Iā€™m correct, it is certainly worth changing (most people I believe would be happy to wait 30s or longer for a config change to take effect)

1 Like

Ideally it would be configurable, Iā€™d suggest. On most platforms it wouldnā€™t matter.

I think most people would think it wasnā€™t working if they saved the change just after the 30 s refresh had happened.

Iā€™m not sure everyone realises or knows when they need to restart and when they donā€™t. Personally I always restart just to be on the safe side.

Since ā€œautoā€ has been around for a while, it will probably cause problems if itā€™s removed. But yes, itā€™s unnecessary - and could be removed completely provided everyone is told that itā€™s gone. I think complete removal is better than a possible 30 s wait - if you change it to that, youā€™ll get a stream of ā€œIt sometimes works and sometimes doesnā€™tā€ type of complaints.

What does ā€œSaveā€ do - apart from the obvious? Can the act of saving set a flag that would be very cheap to poll? In which case, 0.1 s would be feasible.

Iā€™m an old Unix hand, and the way that signalling something had changed to a process (change debug level, reread config etc) was usually done byā€¦ sending it a signal via killā€¦ e.g. kill -SIGHUP .

For me, rereading and parsing config file regularly isnā€™t efficient.
Most users expect to have to restart after changing config, and most do so anyway. I would doubt most people know about this feature.

Also, the emonhub.conf default file is fairly large as it has lots of examples of things that may or may not be used. All needs parsed. Every second. My own emonhub.conf is 391 lines, and most are used.

Yes, its not an issue on more powerful platforms, but even on my prod pi3B, I went from 7% to 1.2% usage. Those cycles would be better used somewhere else.

My vote is to remove this ā€˜featureā€™ and expect users should restart after making changes (at a time of their convenience).

The cost of an rpi4 adds to the cost of an emonpi by at least Ā£40. If a pi-Zero-2 could do the same for Ā£15, thatā€™s a heck of a saving.
The Zero-2 has roughly the same compute and memory as a 2b, which is more than enough to run emoncms and the full LAMP stack. (My emonpi had a 2B in it for years).

Let the debate continue.

Iā€™d agree. I was simply warning what I expect will happen if, as is usual round here, the change is done and nobody is told that thereā€™s even been a change, let alone what it was.

Could the check just look at the last modified time of the config file?

Keep track of the last modified time of the file when the config was last readā€¦ if the modified time has changed, re-parse the file and update the ā€œlast parsed file modified timestampā€ variable.

2 Likes

That sounds like a good scheme. Would you mind suggesting some code to do so?

Just pushed some changes that add an option to increase the reload interval.

@Greebo your suggestion might be better!

I think so too - surely thereā€™s a OS API call to get the file details which include creation and modification times? Reading and parsing the file reads more like a sledgehammer & nut solution to me.

I didnā€™t say it was a good solution, just a solution. And this feature (reloading settings) has been here since itā€™s inception I think.

Interesting, but it still defaults to 1 second.
Could we put a less aggressive figure in as default ?

Iā€™m still for stopping the behaviour altogether as I think itā€™s bad practice, and if we look at other projects (eg apache, dnsmasqā€¦ in fact most) the behaviour is not to reload unless toldā€¦ and Iā€™d argue thatā€™s what users expect.

No, as this way it isnā€™t a breaking change.

Yep, the logic is fairly simple, the procedure calls are simple and I thought I could quickly chuck a suggested change togetherā€¦ but then I started trying to understand the construction of the existing code and it all got too much for the amount of time I had available to look at it.

I havenā€™t given up, but it might have to wait a few days until I can find sufficient time to focus on it.

1 Like

Probably - the ones who havenā€™t noticed that it reloads anyway. But for those that have and have come to rely on it, as Brian says itā€™s a breaking change, and weā€™re the ones who have to supply the answers on the forum.

Or maybe leave the default as-is but add a switch that allows you to change the behaviour to suite your needs?

Iā€™m pretty sure Iā€™ve used daemons that behave both ways (i.e. many need a SIGHUP or manual restart, while others auto-reload their configs upon change).

Iā€™m struggling to recall an example of the latter, but I do remember that it even logs it with a friendly ā€œconfig file changed, reloadingā€.

Linux offers an inotify facility for those that want to know when their config files have changed. Iā€™m no python programmer but it looks like itā€™s supported there too: https://pypi.org/project/inotify/

1 Like

Which is exactly what I have done. It could wait 3 million seconds if you wanted!

Iā€™ll say it again though, it has been like this forever, so is hardly something to afflict folk generally!

True Brian. However, it is using more CPU than it really needs to. And that uses Watts, and those Watts add up. The whole point of energy monitoring is to lower usage. How may Piā€™s are running EmonCMS?
How many kWhā€™s per year does that at up to? Even if the default was changed to 5 seconds, the CPU usage would drop significantly and most users would be unaware of the wait time.

Yes, canā€™t argue with that :slight_smile: