Emoncms_mqtt.php very high ram use

borpin · 2023-08-13T17:52:01.175Z

Rutger0000:

In conclusion, I agree with @borpin that the problem is likely in the installation with the EmonScripts, version of some package…

Have you done an apt update/upgrade of the SD Image?

Rutger0000:

As I used the scripts both on Debian Bullseye and Buster which both had this MQTT issue for me, I feel like the problem might be in versioning/install of the EmonScripts.

The SD Image was built using the scripts! I think it is more likely a newer version of something that the script will use by default is the cause. Probably either PHP or Mosquitto. Doing an update/upgrade doesn’t update PHP to a new version IIRC, even if available.

Rutger0000 · 2023-08-13T18:08:21.242Z

I ran the “Full update” button in admin/update on the EmonSD 2022 Nov 10 before testing.

borpin · 2023-08-13T19:33:16.660Z

I can’t remember if it actually does an apt update/upgrade (I know it says it does). Did you check the update log?.

Rutger0000 · 2023-08-13T20:03:05.611Z

I just checked the update log. The full update only runs apt-get update, however I have now run the upgrade (117 packages upgraded) which upgraded apache and php among other things.

Running the test script with 10 inputs every 10 seconds still does not increase the memory usage of emoncms_mqtt.php on this now ‘upgraded’ EmonSD. So it also works fine after upgrade.

PHP 8.1.12 (Zend Version 4.1.12) -> Version: 8.1.21 (Zend Version 4.1.21)
Mosquitto 2.0.11 (not upgraded)
Apache/2.4.54 (Raspbian) -> Apache/2.4.56 (Raspbian)
and some other packages

borpin · 2023-08-14T20:38:00.618Z

Rutger0000:

PHP 8.1.12

I have a feeling on a fresh install, you get 8.2. It only updates within the X.X version installed.

TrystanLea · 2023-08-16T07:18:18.528Z

Thanks for the detailed testing on this. Looks like the next step for me is to do a new EmonScripts install!

Ryan_S · 2023-08-25T16:41:55.222Z

Wanted to see if you had a chance to do the EmonScripts install test yet.

TrystanLea · 2023-08-25T20:29:38.820Z

Apologies, still haven’t got back to this and Im away for a week next week. If you dont hear from me in just over a weeks time give me a nudge!

Ryan_S · 2023-08-25T21:25:08.911Z

No problem. Will check in again in a week.

TrystanLea · 2023-09-04T15:56:15.804Z

I built a new system with EmonScripts today and was able to replicate this issue quite clearly. I ultimately tracked the issue down to a piece of code in Modules/input/input_model.php, function redis_get_inputs.

    $pipe = $this->redis->multi(Redis::PIPELINE);
    foreach ($inputids as $id) {
        $pipe->hGetAll("input:$id");
    }
    $result = $pipe->exec();

There appears to be a memory leak within the latest version of the phpredis library when using either hGetAll or hMGet with php 8.1.23. I will create an issue on the phpredis repo to ask about this, I cant see any reports of the same issue elsewhere…

I’ve changed the implementation back to the less CPU efficient implementation that does not use a pipeline and that at least keeps memory stable.

The fix , hopefully temporary is available in a emoncms branch called ‘fix_emoncms_mqtt_memory_leak’

cd /var/www/emoncms
git checkout fix_emoncms_mqtt_memory_leak

CPU does appear to be 4x as high unfortunately, in my test of 50x inputs every second it’s the difference between about 5% of CPU and 20% for the emoncms_mqtt process.

borpin · 2023-09-05T06:21:21.711Z

TrystanLea:

either hGetAll or hMGet with php 8.1.23

Could we revert to a previous version of either phpredis or PHP and hold that version until it is fixed upstream?

Seem to remember we did this for something else for a while.

TrystanLea · 2023-09-07T15:57:55.358Z

Yes that might be the best option, my guess is that we’d need to roll back the php minor version but would need to test…

Ryan_S · 2023-09-07T18:58:00.232Z

Thanks for taking a look. Will reinstall and give it a test run tonight if I get a chance. Was thinking it might be related to a package that was called out. Don’t think my Ryzen will care to much about a little more load, but hopefully this bug will get fixed and we can go back to your more efficient code set.

odaigle · 2023-09-19T19:55:28.376Z

I ran into the same problem after upgrading my RPi 4 from buster to bullseye and upgrading to PHP8.2. After a few hours, my RPi4 was becoming unresponsive. I tracked the problem down to PHP that was taking huge amounts of memory…

Thanks for sharing your solution. It solved my problem. Looking forward for a fix in redis, though.

borpin · 2023-09-19T20:10:30.491Z

TrystanLea:

my guess is that we’d need to roll back the php minor version

Have you had a chance to test?

[edit]
My EmonPi is on 8.1.15 and is OK.

pi@emonpi:~ $ sudo apt list --installed | grep php

WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

libapache2-mod-php8.1/now 8.1.15-1+0~20230203.33+debian11~1.gbp878288 armhf [installed,upgradable to: 8.1.23-1+0~20230904.54+debian11~1.gbpe6c1a6]
php-common/now 2:93+0~20221211.45+debian11~1.gbpdb4dcc all [installed,upgradable to: 2:93+0~20230409.46+debian11~1.gbpdb4dcc]
php-pear/now 1:1.10.13+submodules+notgz+2022032202-2+0~20221209.38+debian11~1.gbpfd4c1d all [installed,upgradable to: 1:1.10.13+submodules+notgz+2022032202-2+0~20230612.39+debian11~1.gbpfd4c1d]
php-xml/now 2:8.2+93+0~20221211.45+debian11~1.gbpdb4dcc all [installed,upgradable to: 2:8.2+93+0~20230409.46+debian11~1.gbpdb4dcc]
php8.1-cli/now 8.1.15-1+0~20230203.33+debian11~1.gbp878288 armhf [installed,upgradable to: 8.1.23-1+0~20230904.54+debian11~1.gbpe6c1a6]
php8.1-common/now 8.1.15-1+0~20230203.33+debian11~1.gbp878288 armhf [installed,upgradable to: 8.1.23-1+0~20230904.54+debian11~1.gbpe6c1a6]
php8.1-curl/now 8.1.15-1+0~20230203.33+debian11~1.gbp878288 armhf [installed,upgradable to: 8.1.23-1+0~20230904.54+debian11~1.gbpe6c1a6]
php8.1-dev/now 8.1.15-1+0~20230203.33+debian11~1.gbp878288 armhf [installed,upgradable to: 8.1.23-1+0~20230904.54+debian11~1.gbpe6c1a6]
php8.1-gd/now 8.1.15-1+0~20230203.33+debian11~1.gbp878288 armhf [installed,upgradable to: 8.1.23-1+0~20230904.54+debian11~1.gbpe6c1a6]
php8.1-mbstring/now 8.1.15-1+0~20230203.33+debian11~1.gbp878288 armhf [installed,upgradable to: 8.1.23-1+0~20230904.54+debian11~1.gbpe6c1a6]
php8.1-mysql/now 8.1.15-1+0~20230203.33+debian11~1.gbp878288 armhf [installed,upgradable to: 8.1.23-1+0~20230904.54+debian11~1.gbpe6c1a6]
php8.1-opcache/now 8.1.15-1+0~20230203.33+debian11~1.gbp878288 armhf [installed,upgradable to: 8.1.23-1+0~20230904.54+debian11~1.gbpe6c1a6]
php8.1-readline/now 8.1.15-1+0~20230203.33+debian11~1.gbp878288 armhf [installed,upgradable to: 8.1.23-1+0~20230904.54+debian11~1.gbpe6c1a6]
php8.1-xml/now 8.1.15-1+0~20230203.33+debian11~1.gbp878288 armhf [installed,upgradable to: 8.1.23-1+0~20230904.54+debian11~1.gbpe6c1a6]
php8.1/now 8.1.15-1+0~20230203.33+debian11~1.gbp878288 all [installed,upgradable to: 8.1.23-1+0~20230904.54+debian11~1.gbpe6c1a6]
php8.2-common/now 8.2.2-1+0~20230203.13+debian11~1.gbp8ebd00 armhf [installed,upgradable to: 8.2.10-1+0~20230904.33+debian11~1.gbp2dc84c]
php8.2-xml/now 8.2.2-1+0~20230203.13+debian11~1.gbp8ebd00 armhf [installed,upgradable to: 8.2.10-1+0~20230904.33+debian11~1.gbp2dc84c]
pkg-php-tools/oldstable,now 1.40 all [installed,automatic]

Ubuntu 22.04 (Ithink) (OS: Linux 5.15.74-1-pve) - PHP Version: 8.1.2-1ubuntu2.13 (Zend Version 4.1.2)

On the emonpi, 8.1.23 and 8.2.10 is available so potentially an upgrade could break the system (if it is PHP)

pi@emonpi:~ $ sudo apt list --upgradable | grep php

WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

libapache2-mod-php8.1/bullseye 8.1.23-1+0~20230904.54+debian11~1.gbpe6c1a6 armhf [upgradable from: 8.1.15-1+0~20230203.33+debian11~1.gbp878288]
php-common/bullseye 2:93+0~20230409.46+debian11~1.gbpdb4dcc all [upgradable from: 2:93+0~20221211.45+debian11~1.gbpdb4dcc]
php-pear/bullseye 1:1.10.13+submodules+notgz+2022032202-2+0~20230612.39+debian11~1.gbpfd4c1d all [upgradable from: 1:1.10.13+submodules+notgz+2022032202-2+0~20221209.38+debian11~1.gbpfd4c1d]
php-xml/bullseye 2:8.2+93+0~20230409.46+debian11~1.gbpdb4dcc all [upgradable from: 2:8.2+93+0~20221211.45+debian11~1.gbpdb4dcc]
php8.1-cli/bullseye 8.1.23-1+0~20230904.54+debian11~1.gbpe6c1a6 armhf [upgradable from: 8.1.15-1+0~20230203.33+debian11~1.gbp878288]
php8.1-common/bullseye 8.1.23-1+0~20230904.54+debian11~1.gbpe6c1a6 armhf [upgradable from: 8.1.15-1+0~20230203.33+debian11~1.gbp878288]
php8.1-curl/bullseye 8.1.23-1+0~20230904.54+debian11~1.gbpe6c1a6 armhf [upgradable from: 8.1.15-1+0~20230203.33+debian11~1.gbp878288]
php8.1-dev/bullseye 8.1.23-1+0~20230904.54+debian11~1.gbpe6c1a6 armhf [upgradable from: 8.1.15-1+0~20230203.33+debian11~1.gbp878288]
php8.1-gd/bullseye 8.1.23-1+0~20230904.54+debian11~1.gbpe6c1a6 armhf [upgradable from: 8.1.15-1+0~20230203.33+debian11~1.gbp878288]
php8.1-mbstring/bullseye 8.1.23-1+0~20230904.54+debian11~1.gbpe6c1a6 armhf [upgradable from: 8.1.15-1+0~20230203.33+debian11~1.gbp878288]
php8.1-mysql/bullseye 8.1.23-1+0~20230904.54+debian11~1.gbpe6c1a6 armhf [upgradable from: 8.1.15-1+0~20230203.33+debian11~1.gbp878288]
php8.1-opcache/bullseye 8.1.23-1+0~20230904.54+debian11~1.gbpe6c1a6 armhf [upgradable from: 8.1.15-1+0~20230203.33+debian11~1.gbp878288]
php8.1-readline/bullseye 8.1.23-1+0~20230904.54+debian11~1.gbpe6c1a6 armhf [upgradable from: 8.1.15-1+0~20230203.33+debian11~1.gbp878288]
php8.1-xml/bullseye 8.1.23-1+0~20230904.54+debian11~1.gbpe6c1a6 armhf [upgradable from: 8.1.15-1+0~20230203.33+debian11~1.gbp878288]
php8.1/bullseye 8.1.23-1+0~20230904.54+debian11~1.gbpe6c1a6 all [upgradable from: 8.1.15-1+0~20230203.33+debian11~1.gbp878288]
php8.2-common/bullseye 8.2.10-1+0~20230904.33+debian11~1.gbp2dc84c armhf [upgradable from: 8.2.2-1+0~20230203.13+debian11~1.gbp8ebd00]
php8.2-xml/bullseye 8.2.10-1+0~20230904.33+debian11~1.gbp2dc84c armhf [upgradable from: 8.2.2-1+0~20230203.13+debian11~1.gbp8ebd00]

@TrystanLea - can you try an emonSD base image and an upgrade and see what happens?

[edit2]
Looks like 8.3 is available in Bullseye. Perhaps skipping forward will solve it.

TrystanLea · 2023-09-20T11:25:35.564Z

Another developer has replied on the issue I created on the phpredis repo and has tracked the issue down to a specific commit in the phpredis library and/or using redis 6.0 Redis pipeline memory leak? · Issue #2376 · phpredis/phpredis · GitHub, he has created a number of other related issues including Segmentation Fault while just calling `->exec()` · Issue #2390 · phpredis/phpredis · GitHub.

The interim solution might be to install redis 5.3.7

TrystanLea · 2023-09-20T11:34:10.448Z

In trying to install php 8.3 before reading the above reply suggesting that the redis version was the issue. I seem to have broken my phpredis library. I cant install it, I get the error:

Configuring for:
PHP Api Version:         20230831
Zend Module Api No:      20230831
Zend Extension Api No:   420230831
cp: cannot stat 'shtool': No such file or directory
cp: cannot stat 'config.guess': No such file or directory
cp: cannot stat 'config.sub': No such file or directory
cp: cannot stat 'ltmain.sh': No such file or directory
chmod: cannot access '/home/pi/phpredis/build/shtool': No such file or directory
shtool at '/home/pi/phpredis/build/shtool' does not exist or is not executable.
Make sure that the file exists and is executable and then rerun this script.

and now Im struggling to recover my image from that position! argh

TrystanLea · 2023-09-20T11:36:15.137Z

If anyone else can try switching to redis 5.3.7 that would be much appreciated!

borpin · 2023-09-26T08:33:03.512Z

It must be a combination of the two as this combination is fine.

REDIS

Redis Server 6.0.16
PHP Redis 6.0.0-dev

PHP

Version 8.1.15 (Zend Version 4.1.15)

TrystanLea · 2023-11-16T17:03:06.014Z

The PHPRedis developers have updated the issue to say that the issue is now fixed (September 23th) Redis pipeline memory leak? · Issue #2376 · phpredis/phpredis · GitHub

I’ve been testing a new image build for about 3 week and it seems to have been stable. Testing the 200 input every 1s script above works fine without memory build up so I think we can safely close this issue now.