So, after fixing the MQTT issue that was preventing my OpenEVSEs from talking with EmonCMS I’ve noticed a new interesting issue. It seems after my EmonCMS runs for a long period that when I go back to take a look at things that it’s response is dragged to snail pace slow. More like, the EmonCMS webpage will not load and when I look at Proxmox to see what’s going on I show the CPU maxed at 98% and memory usage also showing at 98%. I need to check the logs, but I recall @borpin mentioning swap usage with Ubuntu Server 22.04 was odd at times. I haven’t gotten a look at it, but it’s worth considering. That said, I wondered if anyone else has experienced this sort of thing with EmonCMS and might know if there is a memory leak of something else that occurs. At this point, I only have Inputs and no Feeds created yet. I have 2 IotaWatts connected, but only 1 is sending information as the other has an issue right now and is not connected at the moment. I also have 2 OpenEVSE connected via MQTT.
What does ps ax or htop say is causing the issue?
That is my next check. The Container install of Ubuntu Server 22.04 is a bit sparse so I might have to install htop, but I will check those when I get home.
Well, got home and it appears the EmonCMS is not running. ps ax shows
pi@EmonCMS-CT:/$ ps ax
PID TTY STAT TIME COMMAND
1 ? Ss 13:48 /sbin/init
42 ? Ss 12:21 /lib/systemd/systemd-journald
96 ? Ss 7:57 /lib/systemd/systemd-networkd
100 ? Ss 4:29 /lib/systemd/systemd-resolved
102 ? Ss 10:09 /usr/sbin/cron -f -P
103 ? Ss 1:15 @dbus-daemon --system --address=systemd: --nofork --nopidfile --syste
106 ? Ss 0:24 /usr/bin/python3 /usr/bin/networkd-dispatcher --run-startup-triggers
107 ? Ssl 0:20 /usr/sbin/rsyslogd -n -iNONE
108 ? Ss 2:51 /lib/systemd/systemd-logind
130 ? Ssl 17:48 /usr/bin/redis-server 127.0.0.1:6379
132 pts/0 Ss+ 0:00 /sbin/agetty -o -p -- \u --noclear --keep-baud console 115200,38400,9
133 pts/1 Ss 0:00 /bin/login -p --
134 pts/2 Ss+ 0:00 /sbin/agetty -o -p -- \u --noclear --keep-baud tty2 115200,38400,9600
148 ? Ss 14:26 /usr/sbin/mosquitto -c /etc/mosquitto/mosquitto.conf
177 ? Ss 3:33 /usr/sbin/apache2 -k start
201 ? Ssl 70:13 /usr/sbin/mariadbd
223 ? Ss 0:00 /usr/bin/python3 /var/www/emoncms/scripts/services/service-runner/ser
577 pts/1 S 0:00 -bash
1595 ? S 2:50 /usr/sbin/apache2 -k start
1603 ? S 2:31 /usr/sbin/apache2 -k start
1605 ? S 2:10 /usr/sbin/apache2 -k start
1607 ? S 2:00 /usr/sbin/apache2 -k start
1630 ? S 0:17 /usr/sbin/apache2 -k start
1649 ? S 0:42 /usr/sbin/apache2 -k start
1651 ? S 0:44 /usr/sbin/apache2 -k start
1652 ? S 0:42 /usr/sbin/apache2 -k start
1654 ? S 0:39 /usr/sbin/apache2 -k start
1686 ? S 0:00 /usr/sbin/apache2 -k start
1775 ? Ss 0:00 /usr/bin/php /var/www/emoncms/scripts/feedwriter.php
1778 ? Ss 0:04 /usr/bin/php /opt/emoncms/modules/demandshaper/demandshaper_run.php
1783 ? Ss 0:06 /usr/bin/php /var/www/emoncms/scripts/services/emoncms_mqtt/emoncms_m
2558 pts/1 R+ 0:00 ps ax
htop install appears to not work as the servers are down. Will try that tomorrow.
Got htop installed and filtered to see what memory usage is. Right now the top user is MQTT.
And a little time later.
It would appear that EmonCMS_MQTT might be having a memory leak. Wondering if it’s the version I am on by chance or something with EmonCMS.
So from what I can see EmonCMS_MQTT is using a bunch of RAM. Now, I’m not sure if this module is EmonCMS or Mosquitto, but that is what appears to be happening. Only started to happen once I got Mosquitto working, so I think I found another oddball bug. My question is, what changed? Is the EmonCMS installed via script the same as EmonSD or others?
Looks like Proxmox is reporting having to kill the CT and it calls out which process is using up all the memory.
Jul 21 11:27:32 pve kernel: oom-kill:constraint=CONSTRAINT_MEMCG,nodemask=(null),cpuset=ns,mems_allowed=0,oom_memcg=/lxc/110,task_memcg=/lxc/110/ns/system.slice/emoncms_mqtt.service,task=php,pid=2581952,uid=100000
Jul 21 11:27:32 pve kernel: Memory cgroup out of memory: Killed process 2581952 (php) total-vm:3669052kB, anon-rss:3419640kB, file-rss:0kB, shmem-rss:0kB, UID:100000 pgtables:7156kB oom_score_adj:0
I checked the mosquitto log and it doesn’t show anything that would point to what’s going on.
So here is the last couple of entries from the log for EmonCMS
2023-07-21 19:27:29.420|INFO|emoncms_mqtt.php|emon/openevse_south/total_week 49.00181934
2023-07-21 19:27:29.431|INFO|emoncms_mqtt.php|emon/openevse_south/total_month 193.4396401
2023-07-21 19:27:29.442|INFO|emoncms_mqtt.php|emon/openevse_south/total_year 229.5857772
2023-07-21 19:27:29.453|INFO|emoncms_mqtt.php|emon/openevse_south/total_switches 47
2023-07-21 19:28:57.-350|INFO|emoncms_mqtt.php|484 Messages processed in last 5 minutes
2023-07-21 19:33:58.-301|INFO|emoncms_mqtt.php|0 Messages processed in last 5 minutes
2023-07-21 19:38:59.-253|INFO|emoncms_mqtt.php|0 Messages processed in last 5 minutes
2023-07-21 19:44:00.-210|INFO|emoncms_mqtt.php|0 Messages processed in last 5 minutes
2023-07-21 19:49:01.-160|INFO|emoncms_mqtt.php|0 Messages processed in last 5 minutes
2023-07-21 19:54:02.-118|INFO|emoncms_mqtt.php|0 Messages processed in last 5 minutes
The log in there is LOADED with MQTT reports. Not sure if that is normal, but it is loaded.
Anyone?
So, as I read up on this. I’m starting to wonder if MQTT is hanging onto the messages instead of removing them once delivered.
So I went ahead and decided to do a clean install. Wiped the Container and reinstalled Ubuntu 22.04 and EmonCMS. The only edit I’ve made to a configuration file was added the listener 1883 to the mosquitto.conf. So with this day 1 install I did the normal create a user on the main page and my mqtt connections came back up. As they are running now for about an hour now and I’m seeing the emoncms_mqtt memory usage climbing slowly. But it is climbing. Looks like the issue is withing the current package of emonscripts. Not sure if there is a new setting for mosquitto to have it clear out old messages from memory or maybe to give it a maximum memory limit before clearing them out.
Just a side thought. I have two OpenEVSEs using MQTT to send information to EmonCMS. Both use the same user name and password for the connection but with two different topics. Now, I’m wondering if it’s possible that the way those open the connection and close it if the two user name might be causing an odd bug to show up where the connection doesn’t close and it just keeps opening a new one. See how up above I had the log LOADED with those connections. Like for each and every one, which seemed excessive to me.
I don’t know anything about Linux / Proxmox but I do use MQTT extensively. I have numerous devices that use the same MQTT user name and password to access my MQTT broker(s). Each device has its own topic structure for identification. I have never had any memory issues running MQTT on a variety of devices such as Raspi’s, Home Assistant, ESP* devices etc. I think MQTT is generally a very robust system.
Agreed. All my MQTT devices use the same username/password, maybe a dozen.
The only issue I’ve had with MQTT is overloading it, that was dragging down the performance of the whole machine BUT I’m running EmonCMS on Pi2 and the device was publishing 60+ bits of info in a single MQTT call. Splitting up call into 4 calls and dropping unneeded info fixed that.
How often are you EVSE’s posting data to MQTT?
OpenEVSE publishes to MQTT every 30 seconds. I don’t believe that this being in Proxmox in either containers or VM is causing the issue. It would appear to be an MQTT or EmonCMS issue that is in the latest @TrystanLea or @glyn.hudson, any chance you folk might have something for me to check? I’m curious @Vster, how did you split yours up? I see some Mosquitto options for limiting data and I thought I saw in one of the EmonCMS config files some MQTT options as well that aren’t called out in default setting, but I can’t seem to locate those again to review them.
I was pushing data from a Pi1 that’s connected to my home battery system.
A Python program pulled the data from each of the four batteries and pushed it to EmonCMS over MQTT.
Each battery has BatteryNumber, Volts, Current, Temp, TempLow, TempHigh, VoltLow, VoltHigh, Status, VoltStatus, CurrStatus, TempStatus, SOC, Time, BVStatus, BTStatus. So 16 x 4 key/value pairs of data. It proved too much for MQTT on the EmonCMS (remember it’s only a Pi2 and also runs NodeRed).
Splitting the MQTT push into 4 parts, i,e 4 lots of 16 key/value pairs rather then 1 lot of 64, brought the load down a lot. There is a lot of other stuff flowing into EmonCMS too.
Actually I later changed it send the data with a HTTP call instead as that lowered the load even more.
If you’re running Ubuntu, can I recommend you install “MQTT Explorer”.
Set it up to subscribe to the EmonCMS and you’ll be able to see what’s going on, it may help sort the situation?
Actually, I do have MQTT Explorer. Can’t say I’ve used it much at all so I’m not totally familiar with it, but I will see if maybe it give me some incite into what is going on. I don’t suppose it would show what is retaining information would it?
So I took a look with MQTT Explorer to see if maybe it would offer some insight. I can’t say that it did. But I did pull up htop to see how emoncms_mqtt was up to 15% of memory usage. Decided to just stop the service on the emoncms web page. When I did that I saw the memory usage unload and when I restart it I could watch the memory usage grow again. This does appear to be an issue within emoncms_mqtt. I’m unsure if maybe there’s a configuration file that is maybe incorrect, but would check all of those if someone could point out which ones emoncms uses.
How exactly have you set this up, and on what hardware.
You variously mention containers, VMs and Proxmox.
Did you use the Ubuntu 22.04 base LXC as the starting point?
I have Proxmox installed on an old laptop, emoncms installed with the scripts (less various modules such as demandshaper, Mosquitto, emonhub etc) on an Ubuntu LXC (20.04 I think) and a separate LXC for a Mosquitto MQTT Broker.
@borpin, currently I have this installed on an Ubuntu 22.04 container. I have tested it with an Ubuntu 20.04 container and also VMs with Ubuntu Server 20.04 and 22.04. I was doing this just to see if it might be the installation type or version. Mostly to rule those being the problem out.
I installed EmonCMS using the Emonscript installation method. So, if you used the same method than we should be the same. Although, I don’t know how long ago you installed yours and what might have changed since then till now.