Have you done an apt update/upgrade of the SD Image?
The SD Image was built using the scripts! I think it is more likely a newer version of something that the script will use by default is the cause. Probably either PHP or Mosquitto. Doing an update/upgrade doesn’t update PHP to a new version IIRC, even if available.
I ran the “Full update” button in admin/update on the EmonSD 2022 Nov 10 before testing.
I can’t remember if it actually does an apt update/upgrade (I know it says it does). Did you check the update log?.
I just checked the update log. The full update only runs apt-get update, however I have now run the upgrade (117 packages upgraded) which upgraded apache and php among other things.
Running the test script with 10 inputs every 10 seconds still does not increase the memory usage of emoncms_mqtt.php on this now ‘upgraded’ EmonSD. So it also works fine after upgrade.
PHP 8.1.12 (Zend Version 4.1.12) -> Version: 8.1.21 (Zend Version 4.1.21)
Mosquitto 2.0.11 (not upgraded)
Apache/2.4.54 (Raspbian) -> Apache/2.4.56 (Raspbian)
and some other packages
I have a feeling on a fresh install, you get 8.2. It only updates within the X.X version installed.
Thanks for the detailed testing on this. Looks like the next step for me is to do a new EmonScripts install!
Wanted to see if you had a chance to do the EmonScripts install test yet.
Apologies, still haven’t got back to this and Im away for a week next week. If you dont hear from me in just over a weeks time give me a nudge!
No problem. Will check in again in a week.
I built a new system with EmonScripts today and was able to replicate this issue quite clearly. I ultimately tracked the issue down to a piece of code in Modules/input/input_model.php, function redis_get_inputs.
$pipe = $this->redis->multi(Redis::PIPELINE);
foreach ($inputids as $id) {
$pipe->hGetAll("input:$id");
}
$result = $pipe->exec();
There appears to be a memory leak within the latest version of the phpredis library when using either hGetAll or hMGet with php 8.1.23. I will create an issue on the phpredis repo to ask about this, I cant see any reports of the same issue elsewhere…
I’ve changed the implementation back to the less CPU efficient implementation that does not use a pipeline and that at least keeps memory stable.
The fix , hopefully temporary is available in a emoncms branch called ‘fix_emoncms_mqtt_memory_leak’
cd /var/www/emoncms
git checkout fix_emoncms_mqtt_memory_leak
CPU does appear to be 4x as high unfortunately, in my test of 50x inputs every second it’s the difference between about 5% of CPU and 20% for the emoncms_mqtt process.
Could we revert to a previous version of either phpredis or PHP and hold that version until it is fixed upstream?
Seem to remember we did this for something else for a while.
Yes that might be the best option, my guess is that we’d need to roll back the php minor version but would need to test…
Thanks for taking a look. Will reinstall and give it a test run tonight if I get a chance. Was thinking it might be related to a package that was called out. Don’t think my Ryzen will care to much about a little more load, but hopefully this bug will get fixed and we can go back to your more efficient code set.
I ran into the same problem after upgrading my RPi 4 from buster to bullseye and upgrading to PHP8.2. After a few hours, my RPi4 was becoming unresponsive. I tracked the problem down to PHP that was taking huge amounts of memory…
Thanks for sharing your solution. It solved my problem. Looking forward for a fix in redis, though.
Have you had a chance to test?
[edit]
My EmonPi is on 8.1.15 and is OK.
pi@emonpi:~ $ sudo apt list --installed | grep php
WARNING: apt does not have a stable CLI interface. Use with caution in scripts.
libapache2-mod-php8.1/now 8.1.15-1+0~20230203.33+debian11~1.gbp878288 armhf [installed,upgradable to: 8.1.23-1+0~20230904.54+debian11~1.gbpe6c1a6]
php-common/now 2:93+0~20221211.45+debian11~1.gbpdb4dcc all [installed,upgradable to: 2:93+0~20230409.46+debian11~1.gbpdb4dcc]
php-pear/now 1:1.10.13+submodules+notgz+2022032202-2+0~20221209.38+debian11~1.gbpfd4c1d all [installed,upgradable to: 1:1.10.13+submodules+notgz+2022032202-2+0~20230612.39+debian11~1.gbpfd4c1d]
php-xml/now 2:8.2+93+0~20221211.45+debian11~1.gbpdb4dcc all [installed,upgradable to: 2:8.2+93+0~20230409.46+debian11~1.gbpdb4dcc]
php8.1-cli/now 8.1.15-1+0~20230203.33+debian11~1.gbp878288 armhf [installed,upgradable to: 8.1.23-1+0~20230904.54+debian11~1.gbpe6c1a6]
php8.1-common/now 8.1.15-1+0~20230203.33+debian11~1.gbp878288 armhf [installed,upgradable to: 8.1.23-1+0~20230904.54+debian11~1.gbpe6c1a6]
php8.1-curl/now 8.1.15-1+0~20230203.33+debian11~1.gbp878288 armhf [installed,upgradable to: 8.1.23-1+0~20230904.54+debian11~1.gbpe6c1a6]
php8.1-dev/now 8.1.15-1+0~20230203.33+debian11~1.gbp878288 armhf [installed,upgradable to: 8.1.23-1+0~20230904.54+debian11~1.gbpe6c1a6]
php8.1-gd/now 8.1.15-1+0~20230203.33+debian11~1.gbp878288 armhf [installed,upgradable to: 8.1.23-1+0~20230904.54+debian11~1.gbpe6c1a6]
php8.1-mbstring/now 8.1.15-1+0~20230203.33+debian11~1.gbp878288 armhf [installed,upgradable to: 8.1.23-1+0~20230904.54+debian11~1.gbpe6c1a6]
php8.1-mysql/now 8.1.15-1+0~20230203.33+debian11~1.gbp878288 armhf [installed,upgradable to: 8.1.23-1+0~20230904.54+debian11~1.gbpe6c1a6]
php8.1-opcache/now 8.1.15-1+0~20230203.33+debian11~1.gbp878288 armhf [installed,upgradable to: 8.1.23-1+0~20230904.54+debian11~1.gbpe6c1a6]
php8.1-readline/now 8.1.15-1+0~20230203.33+debian11~1.gbp878288 armhf [installed,upgradable to: 8.1.23-1+0~20230904.54+debian11~1.gbpe6c1a6]
php8.1-xml/now 8.1.15-1+0~20230203.33+debian11~1.gbp878288 armhf [installed,upgradable to: 8.1.23-1+0~20230904.54+debian11~1.gbpe6c1a6]
php8.1/now 8.1.15-1+0~20230203.33+debian11~1.gbp878288 all [installed,upgradable to: 8.1.23-1+0~20230904.54+debian11~1.gbpe6c1a6]
php8.2-common/now 8.2.2-1+0~20230203.13+debian11~1.gbp8ebd00 armhf [installed,upgradable to: 8.2.10-1+0~20230904.33+debian11~1.gbp2dc84c]
php8.2-xml/now 8.2.2-1+0~20230203.13+debian11~1.gbp8ebd00 armhf [installed,upgradable to: 8.2.10-1+0~20230904.33+debian11~1.gbp2dc84c]
pkg-php-tools/oldstable,now 1.40 all [installed,automatic]
Ubuntu 22.04 (Ithink) (OS: Linux 5.15.74-1-pve) - PHP Version: 8.1.2-1ubuntu2.13 (Zend Version 4.1.2)
On the emonpi, 8.1.23 and 8.2.10 is available so potentially an upgrade could break the system (if it is PHP)
pi@emonpi:~ $ sudo apt list --upgradable | grep php
WARNING: apt does not have a stable CLI interface. Use with caution in scripts.
libapache2-mod-php8.1/bullseye 8.1.23-1+0~20230904.54+debian11~1.gbpe6c1a6 armhf [upgradable from: 8.1.15-1+0~20230203.33+debian11~1.gbp878288]
php-common/bullseye 2:93+0~20230409.46+debian11~1.gbpdb4dcc all [upgradable from: 2:93+0~20221211.45+debian11~1.gbpdb4dcc]
php-pear/bullseye 1:1.10.13+submodules+notgz+2022032202-2+0~20230612.39+debian11~1.gbpfd4c1d all [upgradable from: 1:1.10.13+submodules+notgz+2022032202-2+0~20221209.38+debian11~1.gbpfd4c1d]
php-xml/bullseye 2:8.2+93+0~20230409.46+debian11~1.gbpdb4dcc all [upgradable from: 2:8.2+93+0~20221211.45+debian11~1.gbpdb4dcc]
php8.1-cli/bullseye 8.1.23-1+0~20230904.54+debian11~1.gbpe6c1a6 armhf [upgradable from: 8.1.15-1+0~20230203.33+debian11~1.gbp878288]
php8.1-common/bullseye 8.1.23-1+0~20230904.54+debian11~1.gbpe6c1a6 armhf [upgradable from: 8.1.15-1+0~20230203.33+debian11~1.gbp878288]
php8.1-curl/bullseye 8.1.23-1+0~20230904.54+debian11~1.gbpe6c1a6 armhf [upgradable from: 8.1.15-1+0~20230203.33+debian11~1.gbp878288]
php8.1-dev/bullseye 8.1.23-1+0~20230904.54+debian11~1.gbpe6c1a6 armhf [upgradable from: 8.1.15-1+0~20230203.33+debian11~1.gbp878288]
php8.1-gd/bullseye 8.1.23-1+0~20230904.54+debian11~1.gbpe6c1a6 armhf [upgradable from: 8.1.15-1+0~20230203.33+debian11~1.gbp878288]
php8.1-mbstring/bullseye 8.1.23-1+0~20230904.54+debian11~1.gbpe6c1a6 armhf [upgradable from: 8.1.15-1+0~20230203.33+debian11~1.gbp878288]
php8.1-mysql/bullseye 8.1.23-1+0~20230904.54+debian11~1.gbpe6c1a6 armhf [upgradable from: 8.1.15-1+0~20230203.33+debian11~1.gbp878288]
php8.1-opcache/bullseye 8.1.23-1+0~20230904.54+debian11~1.gbpe6c1a6 armhf [upgradable from: 8.1.15-1+0~20230203.33+debian11~1.gbp878288]
php8.1-readline/bullseye 8.1.23-1+0~20230904.54+debian11~1.gbpe6c1a6 armhf [upgradable from: 8.1.15-1+0~20230203.33+debian11~1.gbp878288]
php8.1-xml/bullseye 8.1.23-1+0~20230904.54+debian11~1.gbpe6c1a6 armhf [upgradable from: 8.1.15-1+0~20230203.33+debian11~1.gbp878288]
php8.1/bullseye 8.1.23-1+0~20230904.54+debian11~1.gbpe6c1a6 all [upgradable from: 8.1.15-1+0~20230203.33+debian11~1.gbp878288]
php8.2-common/bullseye 8.2.10-1+0~20230904.33+debian11~1.gbp2dc84c armhf [upgradable from: 8.2.2-1+0~20230203.13+debian11~1.gbp8ebd00]
php8.2-xml/bullseye 8.2.10-1+0~20230904.33+debian11~1.gbp2dc84c armhf [upgradable from: 8.2.2-1+0~20230203.13+debian11~1.gbp8ebd00]
@TrystanLea - can you try an emonSD base image and an upgrade and see what happens?
[edit2]
Looks like 8.3 is available in Bullseye. Perhaps skipping forward will solve it.
Another developer has replied on the issue I created on the phpredis repo and has tracked the issue down to a specific commit in the phpredis library and/or using redis 6.0 Redis pipeline memory leak? · Issue #2376 · phpredis/phpredis · GitHub, he has created a number of other related issues including Segmentation Fault while just calling `->exec()` · Issue #2390 · phpredis/phpredis · GitHub.
The interim solution might be to install redis 5.3.7
In trying to install php 8.3 before reading the above reply suggesting that the redis version was the issue. I seem to have broken my phpredis library. I cant install it, I get the error:
Configuring for:
PHP Api Version: 20230831
Zend Module Api No: 20230831
Zend Extension Api No: 420230831
cp: cannot stat 'shtool': No such file or directory
cp: cannot stat 'config.guess': No such file or directory
cp: cannot stat 'config.sub': No such file or directory
cp: cannot stat 'ltmain.sh': No such file or directory
chmod: cannot access '/home/pi/phpredis/build/shtool': No such file or directory
shtool at '/home/pi/phpredis/build/shtool' does not exist or is not executable.
Make sure that the file exists and is executable and then rerun this script.
and now Im struggling to recover my image from that position! argh 
If anyone else can try switching to redis 5.3.7 that would be much appreciated!
It must be a combination of the two as this combination is fine.
Redis Server 6.0.16
PHP Redis 6.0.0-dev
Version 8.1.15 (Zend Version 4.1.15)
The PHPRedis developers have updated the issue to say that the issue is now fixed (September 23th) Redis pipeline memory leak? · Issue #2376 · phpredis/phpredis · GitHub
I’ve been testing a new image build for about 3 week and it seems to have been stable. Testing the 200 input every 1s script above works fine without memory build up so I think we can safely close this issue now.