Proxmox EmonCMS Heavy Resource Usage

So, after fixing the MQTT issue that was preventing my OpenEVSEs from talking with EmonCMS I’ve noticed a new interesting issue. It seems after my EmonCMS runs for a long period that when I go back to take a look at things that it’s response is dragged to snail pace slow. More like, the EmonCMS webpage will not load and when I look at Proxmox to see what’s going on I show the CPU maxed at 98% and memory usage also showing at 98%. I need to check the logs, but I recall @borpin mentioning swap usage with Ubuntu Server 22.04 was odd at times. I haven’t gotten a look at it, but it’s worth considering. That said, I wondered if anyone else has experienced this sort of thing with EmonCMS and might know if there is a memory leak of something else that occurs. At this point, I only have Inputs and no Feeds created yet. I have 2 IotaWatts connected, but only 1 is sending information as the other has an issue right now and is not connected at the moment. I also have 2 OpenEVSE connected via MQTT.

What does ps ax or htop say is causing the issue?

That is my next check. The Container install of Ubuntu Server 22.04 is a bit sparse so I might have to install htop, but I will check those when I get home.

Well, got home and it appears the EmonCMS is not running. ps ax shows

pi@EmonCMS-CT:/$ ps ax
      1 ?        Ss    13:48 /sbin/init
     42 ?        Ss    12:21 /lib/systemd/systemd-journald
     96 ?        Ss     7:57 /lib/systemd/systemd-networkd
    100 ?        Ss     4:29 /lib/systemd/systemd-resolved
    102 ?        Ss    10:09 /usr/sbin/cron -f -P
    103 ?        Ss     1:15 @dbus-daemon --system --address=systemd: --nofork --nopidfile --syste
    106 ?        Ss     0:24 /usr/bin/python3 /usr/bin/networkd-dispatcher --run-startup-triggers
    107 ?        Ssl    0:20 /usr/sbin/rsyslogd -n -iNONE
    108 ?        Ss     2:51 /lib/systemd/systemd-logind
    130 ?        Ssl   17:48 /usr/bin/redis-server
    132 pts/0    Ss+    0:00 /sbin/agetty -o -p -- \u --noclear --keep-baud console 115200,38400,9
    133 pts/1    Ss     0:00 /bin/login -p --
    134 pts/2    Ss+    0:00 /sbin/agetty -o -p -- \u --noclear --keep-baud tty2 115200,38400,9600
    148 ?        Ss    14:26 /usr/sbin/mosquitto -c /etc/mosquitto/mosquitto.conf
    177 ?        Ss     3:33 /usr/sbin/apache2 -k start
    201 ?        Ssl   70:13 /usr/sbin/mariadbd
    223 ?        Ss     0:00 /usr/bin/python3 /var/www/emoncms/scripts/services/service-runner/ser
    577 pts/1    S      0:00 -bash
   1595 ?        S      2:50 /usr/sbin/apache2 -k start
   1603 ?        S      2:31 /usr/sbin/apache2 -k start
   1605 ?        S      2:10 /usr/sbin/apache2 -k start
   1607 ?        S      2:00 /usr/sbin/apache2 -k start
   1630 ?        S      0:17 /usr/sbin/apache2 -k start
   1649 ?        S      0:42 /usr/sbin/apache2 -k start
   1651 ?        S      0:44 /usr/sbin/apache2 -k start
   1652 ?        S      0:42 /usr/sbin/apache2 -k start
   1654 ?        S      0:39 /usr/sbin/apache2 -k start
   1686 ?        S      0:00 /usr/sbin/apache2 -k start
   1775 ?        Ss     0:00 /usr/bin/php /var/www/emoncms/scripts/feedwriter.php
   1778 ?        Ss     0:04 /usr/bin/php /opt/emoncms/modules/demandshaper/demandshaper_run.php
   1783 ?        Ss     0:06 /usr/bin/php /var/www/emoncms/scripts/services/emoncms_mqtt/emoncms_m
   2558 pts/1    R+     0:00 ps ax

htop install appears to not work as the servers are down. Will try that tomorrow.

Got htop installed and filtered to see what memory usage is. Right now the top user is MQTT.

And a little time later.

It would appear that EmonCMS_MQTT might be having a memory leak. Wondering if it’s the version I am on by chance or something with EmonCMS.

So from what I can see EmonCMS_MQTT is using a bunch of RAM. Now, I’m not sure if this module is EmonCMS or Mosquitto, but that is what appears to be happening. Only started to happen once I got Mosquitto working, so I think I found another oddball bug. My question is, what changed? Is the EmonCMS installed via script the same as EmonSD or others?

Looks like Proxmox is reporting having to kill the CT and it calls out which process is using up all the memory.

Jul 21 11:27:32 pve kernel: oom-kill:constraint=CONSTRAINT_MEMCG,nodemask=(null),cpuset=ns,mems_allowed=0,oom_memcg=/lxc/110,task_memcg=/lxc/110/ns/system.slice/emoncms_mqtt.service,task=php,pid=2581952,uid=100000
Jul 21 11:27:32 pve kernel: Memory cgroup out of memory: Killed process 2581952 (php) total-vm:3669052kB, anon-rss:3419640kB, file-rss:0kB, shmem-rss:0kB, UID:100000 pgtables:7156kB oom_score_adj:0

I checked the mosquitto log and it doesn’t show anything that would point to what’s going on.

1 Like

So here is the last couple of entries from the log for EmonCMS

2023-07-21 19:27:29.420|INFO|emoncms_mqtt.php|emon/openevse_south/total_week 49.00181934
2023-07-21 19:27:29.431|INFO|emoncms_mqtt.php|emon/openevse_south/total_month 193.4396401
2023-07-21 19:27:29.442|INFO|emoncms_mqtt.php|emon/openevse_south/total_year 229.5857772
2023-07-21 19:27:29.453|INFO|emoncms_mqtt.php|emon/openevse_south/total_switches 47
2023-07-21 19:28:57.-350|INFO|emoncms_mqtt.php|484 Messages processed in last 5 minutes
2023-07-21 19:33:58.-301|INFO|emoncms_mqtt.php|0 Messages processed in last 5 minutes
2023-07-21 19:38:59.-253|INFO|emoncms_mqtt.php|0 Messages processed in last 5 minutes
2023-07-21 19:44:00.-210|INFO|emoncms_mqtt.php|0 Messages processed in last 5 minutes
2023-07-21 19:49:01.-160|INFO|emoncms_mqtt.php|0 Messages processed in last 5 minutes
2023-07-21 19:54:02.-118|INFO|emoncms_mqtt.php|0 Messages processed in last 5 minutes

The log in there is LOADED with MQTT reports. Not sure if that is normal, but it is loaded.


So, as I read up on this. I’m starting to wonder if MQTT is hanging onto the messages instead of removing them once delivered.

So I went ahead and decided to do a clean install. Wiped the Container and reinstalled Ubuntu 22.04 and EmonCMS. The only edit I’ve made to a configuration file was added the listener 1883 to the mosquitto.conf. So with this day 1 install I did the normal create a user on the main page and my mqtt connections came back up. As they are running now for about an hour now and I’m seeing the emoncms_mqtt memory usage climbing slowly. But it is climbing. Looks like the issue is withing the current package of emonscripts. Not sure if there is a new setting for mosquitto to have it clear out old messages from memory or maybe to give it a maximum memory limit before clearing them out.

Just a side thought. I have two OpenEVSEs using MQTT to send information to EmonCMS. Both use the same user name and password for the connection but with two different topics. Now, I’m wondering if it’s possible that the way those open the connection and close it if the two user name might be causing an odd bug to show up where the connection doesn’t close and it just keeps opening a new one. See how up above I had the log LOADED with those connections. Like for each and every one, which seemed excessive to me.

I don’t know anything about Linux / Proxmox but I do use MQTT extensively. I have numerous devices that use the same MQTT user name and password to access my MQTT broker(s). Each device has its own topic structure for identification. I have never had any memory issues running MQTT on a variety of devices such as Raspi’s, Home Assistant, ESP* devices etc. I think MQTT is generally a very robust system.

Agreed. All my MQTT devices use the same username/password, maybe a dozen.

The only issue I’ve had with MQTT is overloading it, that was dragging down the performance of the whole machine BUT I’m running EmonCMS on Pi2 and the device was publishing 60+ bits of info in a single MQTT call. Splitting up call into 4 calls and dropping unneeded info fixed that.

How often are you EVSE’s posting data to MQTT?

OpenEVSE publishes to MQTT every 30 seconds. I don’t believe that this being in Proxmox in either containers or VM is causing the issue. It would appear to be an MQTT or EmonCMS issue that is in the latest @TrystanLea or @glyn.hudson, any chance you folk might have something for me to check? I’m curious @Vster, how did you split yours up? I see some Mosquitto options for limiting data and I thought I saw in one of the EmonCMS config files some MQTT options as well that aren’t called out in default setting, but I can’t seem to locate those again to review them.

I was pushing data from a Pi1 that’s connected to my home battery system.
A Python program pulled the data from each of the four batteries and pushed it to EmonCMS over MQTT.
Each battery has BatteryNumber, Volts, Current, Temp, TempLow, TempHigh, VoltLow, VoltHigh, Status, VoltStatus, CurrStatus, TempStatus, SOC, Time, BVStatus, BTStatus. So 16 x 4 key/value pairs of data. It proved too much for MQTT on the EmonCMS (remember it’s only a Pi2 and also runs NodeRed).
Splitting the MQTT push into 4 parts, i,e 4 lots of 16 key/value pairs rather then 1 lot of 64, brought the load down a lot. There is a lot of other stuff flowing into EmonCMS too.

Actually I later changed it send the data with a HTTP call instead as that lowered the load even more.

If you’re running Ubuntu, can I recommend you install “MQTT Explorer”.
Set it up to subscribe to the EmonCMS and you’ll be able to see what’s going on, it may help sort the situation?

Actually, I do have MQTT Explorer. Can’t say I’ve used it much at all so I’m not totally familiar with it, but I will see if maybe it give me some incite into what is going on. I don’t suppose it would show what is retaining information would it?

So I took a look with MQTT Explorer to see if maybe it would offer some insight. I can’t say that it did. But I did pull up htop to see how emoncms_mqtt was up to 15% of memory usage. Decided to just stop the service on the emoncms web page. When I did that I saw the memory usage unload and when I restart it I could watch the memory usage grow again. This does appear to be an issue within emoncms_mqtt. I’m unsure if maybe there’s a configuration file that is maybe incorrect, but would check all of those if someone could point out which ones emoncms uses.

How exactly have you set this up, and on what hardware.

You variously mention containers, VMs and Proxmox.

Did you use the Ubuntu 22.04 base LXC as the starting point?

I have Proxmox installed on an old laptop, emoncms installed with the scripts (less various modules such as demandshaper, Mosquitto, emonhub etc) on an Ubuntu LXC (20.04 I think) and a separate LXC for a Mosquitto MQTT Broker.

@borpin, currently I have this installed on an Ubuntu 22.04 container. I have tested it with an Ubuntu 20.04 container and also VMs with Ubuntu Server 20.04 and 22.04. I was doing this just to see if it might be the installation type or version. Mostly to rule those being the problem out.

I installed EmonCMS using the Emonscript installation method. So, if you used the same method than we should be the same. Although, I don’t know how long ago you installed yours and what might have changed since then till now.