Emonhub service increasing CPU load permanently as it runs

Titof · 2025-01-07T11:14:20.650Z

Hi all
Emoncms is running local on a Pi 3B, version is emonSD-20Nov23
I noticed comming back from Christmas holidays that Mysolarbattery Dashboard was extremly slow.
Using top, I was surpised that the emonhub service uses 70 to 80% of the CPU.
Emonhub is used only to get the data from my SMA inverter via bluetooth
After a reboot, it drops to normal 3-5%, but we see it increasing with the on time.
I implemented a workaround by rebooting automatically at 2 in the night each day. Now after around 10h up, I see:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
561 pi 20 0 89432 26780 8788 S 18.5 2.8 66:55.78 python3 /usr/local/bin/emonhub/emonhub.py --config-file=/etc

The last modification I made to the setup in end of november was adding an other python service using asyncio, that execute each 2 minutes (with a auto restart after completion), get the feeds by http and generates a plot with matplotlib, converts the jpg into base64 and send it to an external mqtt broker so I can see the basic graph with Solar, SOC and Load on my phone when not at home.
I have also 2 other python services running, one each 5 minutes to get the values from the cloud for a second inverter and push it on the emonpi mqtt broker, and a second one that execute at midnight to get the Solcast forecasts and push it also on the emonpi mqtt broker. These 2 scripts are also using asyncio in order to avoid blocking.

Any idea how to understand root cause and improve the situation ?

Thanks for your support

Christophe

.

awjlogan · 2025-01-08T09:56:43.120Z

Hi @Titof - have you modified CMS at all? If you disable the other services you’ve added, does the problem persist?

Titof · 2025-01-09T09:31:15.078Z

Hi.
The only mod I made to emoncms is the slight modification of the kW digits I made and discussed here MySolarBattery kW digits settings
I made some tests yesterday disabling the added services:

it seems that the CPU usage of emonhub does not increase.
then I added my ExternalMqtt service which was the latest added and CPU usage does not increase either. But this script do not push data to the emonpi mqtt broker, only to an external broker.

Do you think the problem is related to the fact that the 2 other python script services are publishing data to the emonpi mqtt broker like emonhub does (even if I do not know) ? I use aiomqtt module to do it with asyncio.
What could be the workaround ? pushing data with http requests ?
Thanks for your support !

awjlogan · 2025-01-10T13:32:05.669Z

Titof:

The only mod I made to emoncms is the slight modification of the kW digits I made and discussed here MySolarBattery kW digits settings

Looking at it, this wouldn’t make any difference.

It sounded like there was some retry mechanism being blocked, and that removing the other publishers fixed it goes someway to confirming that. There might be something in the log. If you run journalctl -u emonhub that should give you the emonHub log.

Titof · 2025-01-10T14:40:11.034Z

Yes there is plenty of retries in the log:

2025-01-10 15:31:49,272 ERROR SMASolar [Errno 104] Connection reset by peer
2025-01-10 15:31:49,273 ERROR SMASolar Bluetooth error while connecting to 00:80:25:27:B1:99
2025-01-10 15:31:49,274 ERROR SMASolar ‘BluetoothError’ object is not subscriptable
2025-01-10 15:31:49,276 ERROR SMASolar [‘Traceback (most recent call last):\n’, ’ File “”, line 3, in connect\n’, “_bluetooth.error: (104, ‘Connection reset by peer’)\n”, ‘\nDuring handling of the above exception, another exception occurred:\n\n’, ‘Traceback (most recent call last):\n’, ’ File “/opt/openenergymonitor/emonhub/src/interfacers/EmonHubSMASolarInterfacer.py”, line 132, in _open_bluetooth\n btSocket.connect((inverteraddress, port))\n’, ’ File “”, line 5, in connect\n’, ‘bluetooth.btcommon.BluetoothError: [Errno 104] Connection reset by peer\n’, ‘\nDuring handling of the above exception, another exception occurred:\n\n’, ‘Traceback (most recent call last):\n’, ’ File “/opt/openenergymonitor/emonhub/src/interfacers/EmonHubSMASolarInterfacer.py”, line 223, in read\n self._login_inverter()\n’, ’ File “/opt/openenergymonitor/emonhub/src/interfacers/EmonHubSMASolarInterfacer.py”, line 64, in _login_inverter\n self._btSocket = self._open_bluetooth(self._inverteraddress, self._port)\n’, ’ File “/opt/openenergymonitor/emonhub/src/interfacers/EmonHubSMASolarInterfacer.py”, line 140, in _open_bluetooth\n errCode = eval(err[0])[0]\n’, “TypeError: ‘BluetoothError’ object is not subscriptable\n”]
2025-01-10 15:32:00,038 ERROR SMASolar Bluetooth Error
2025-01-10 15:32:00,039 ERROR SMASolar timed out

But it is the same without my other mqtt services running

Titof · 2025-01-10T15:02:12.473Z

So I have created a new service to log the cpu usage of emnhub service. Up to now in Mqtt, but from now via http.
We cleary see a ramp.

image1357×685 50.2 KB

My idea is to be able to see if it change something when I will stop the other mqtt publishing services.

Titof · 2025-01-10T17:38:30.525Z

image1136×656 66.5 KB

The other Mqtt service that execute each 5 minutes has been turned off at 17h15.
No difference in the ramp.

awjlogan · 2025-01-11T09:53:06.662Z

And with both extra services disabled, does the CPU load stay stable? (Sorry, think you mentioned this before but it’s nice to see on the feed!).

Also worth trying using a different MQTT client for the external services - potentially there’s some conflict between the broker and the clients.

After that, I don’t currently have a working setup, but you’d have to start more thoroughly tracing and logging events.

Titof · 2025-01-11T15:12:32.000Z

What is the mqtt client used in emonhub?
I use an async version of paho.
But this situation is perhaps better linked to the SMAsolar Bluetooth interfacer?

awjlogan · 2025-01-12T21:36:59.587Z

Titof:

But this situation is perhaps better linked to the SMAsolar Bluetooth interfacer?

I was loathe to suggest it as it doesn’t seem to fit with load increasing over time, but it is in the logs so let’s deal with it. Additionally, the error is in a retry section which does fit the available data

Easy check - assuming you’re not using the SMA Solar system, try temporarily moving ./src/interfacers/EmonHubSMASolarInterfacer.py somewhere else and restart EmonHub. It loads all the interfacers in the src/interfacers folder at start up, so if it’s not in there it won’t be loaded.

I haven’t looked in detail at the bluetooth Python module, but the error is being caused by the error being accessed as an array (“object is not subscriptable”).

Titof · 2025-01-13T06:25:45.891Z

OK. I got it!!
Not a matter of mqtt, not a matter of bluetooth…
Looking at emonhub I found in the doc that serial is enabled by default in emonhub.conf.
And what I forgot is that in december I tried to spy which registers my battery inverter access on the gridmeter with a modbus sniffer script. It works well with my laptop but not with the pi and now I understand why.
The USB converter was still attached to the pi and connected to the modbus so receiving datas.
So it is the EmonOEMinterfacer that increase CPU usage when receiving wrong datas. I disabled it.
Now the problem is solved, CPU usage flat at 3% during the night!

awjlogan · 2025-01-13T10:23:05.948Z

Titof:

OK. I got it!!

Glad you found it Christophe Interesting, though, that this is causing high CPU usage - the interfacer should be able to handle bad data (one should always treat input data from sensors as potentially bad), so definitely worth looking into that more in future.