I’ve setup a script to post 200 inputs every second:
<?php
$mqtt_client = new Mosquitto\Client("pubtest",true);
$mqtt_client->setCredentials('emonpi', 'emonpimqtt2016');
$mqtt_client->connect("localhost", 1883, 5);
while (true) {
$mqtt_client->loop();
for ($i=0; $i<200; $i++) {
$mqtt_client->publish("emon/test/p$i", 100, 0, false);
}
sleep(1);
}
I’ve logged 6 feeds and 6 power to kwh feeds.
So far so good on memory I think:
pi@emonpi:~ $ ps aux | grep emoncms_mqtt
root 1477 18.4 1.2 93448 24088 ? Ss 10:43 0:45 /usr/bin/php /var/www/emoncms/scripts/services/emoncms_mqtt/emoncms_mqtt.php
If I push up to 1000 inputs a second, I notice that only inputs up to about 400 are updating on time. Memory appears stable at 1.2% CPU is increasing now at 20.9%
No problem. Here is a full screenshot of the two OpenEVSE’s inputs into EmonCMS.
Appologies for the long cat style of it. Wanted to capture all of it in one.
Originally yes, it’s the standard emonSD image available for download. I realise that might not be the same version as a newly build version today but wanted to check the stability first with high input count or high post rate
My test of 200 inputs updating 10 times a second is still stable, memory at 1.2%
The test script testing_code/MQTTlog.py exposes x inputs via MQTT to EmonCMS. The value of the input is the memory currently in use by emoncms_mqtt.php.
The Emonscripts install increases memory regularly when running. This happens already for 1 input every 10 seconds but also for higher number of inputs. Just scrolling through the logs emonscripts_install/emoncms.log gives an idea of the increase in memory usage.
The EmonSD install only increases memory once after boot and stays constant after that even at 10 inputs every 10 seconds. So the EmonSD 10Nov22 works perfectly.
For both tests, one input is logged to a feed every 10 seconds. The graphs and logs can be found in emonscripts_install/ and emonSD_10Nov22/.
The delay of 10ms has no influence on the increase of memory.
In conclusion, I agree with @borpin that the problem is likely in the installation with the EmonScripts, version of some package…
As I used the scripts both on Debian Bullseye and Buster which both had this MQTT issue for me, I feel like the problem might be in versioning/install of the EmonScripts.
Thanks all for the problem solving, I have my system running stable with the EmonSD_10Nov22 version.
(I originally wanted to control the partition sizes which was why I used the scripts, but now I have resized the EmonSD partitions)
Have you done an apt update/upgrade of the SD Image?
The SD Image was built using the scripts! I think it is more likely a newer version of something that the script will use by default is the cause. Probably either PHP or Mosquitto. Doing an update/upgrade doesn’t update PHP to a new version IIRC, even if available.
I just checked the update log. The full update only runs apt-get update, however I have now run the upgrade (117 packages upgraded) which upgraded apache and php among other things.
Running the test script with 10 inputs every 10 seconds still does not increase the memory usage of emoncms_mqtt.php on this now ‘upgraded’ EmonSD. So it also works fine after upgrade.
PHP 8.1.12 (Zend Version 4.1.12) -> Version: 8.1.21 (Zend Version 4.1.21)
Mosquitto 2.0.11 (not upgraded)
Apache/2.4.54 (Raspbian) -> Apache/2.4.56 (Raspbian)
and some other packages
I built a new system with EmonScripts today and was able to replicate this issue quite clearly. I ultimately tracked the issue down to a piece of code in Modules/input/input_model.php, function redis_get_inputs.
There appears to be a memory leak within the latest version of the phpredis library when using either hGetAll or hMGet with php 8.1.23. I will create an issue on the phpredis repo to ask about this, I cant see any reports of the same issue elsewhere…
I’ve changed the implementation back to the less CPU efficient implementation that does not use a pipeline and that at least keeps memory stable.
The fix , hopefully temporary is available in a emoncms branch called ‘fix_emoncms_mqtt_memory_leak’
cd /var/www/emoncms
git checkout fix_emoncms_mqtt_memory_leak
CPU does appear to be 4x as high unfortunately, in my test of 50x inputs every second it’s the difference between about 5% of CPU and 20% for the emoncms_mqtt process.
