High server load

Robert.Wall · 2021-10-06T21:22:12.350Z

I know the Iotawatt does send stored data after it loses the connection, and so I wondered whether that would be the sort of reply you’d get. @TrystanLea is your best hope here, as the creator of emonCMS. Hopefully, he’ll get a message about this and respond.

TrystanLea · 2021-10-06T21:24:45.963Z

Hello @sisim do you have redis installed and enabled? And what feed engines are you using?

TrystanLea · 2021-10-06T21:26:31.094Z

Redis should be enabled in emoncms/settings.ini

sisim · 2021-10-06T21:26:59.993Z

Hello,

I have the feeling that something can be done on the MySQL (mariaDB in my case) management, don’t you think? I was also advised to ping @TrystanLea , maybe you have an idea?

Also, is it possible that I have a redis issue? In the admin page, it looks like it’s working:

But if I run redis-cli ping, I receive an error message:
Could not connect to Redis at 127.0.0.1:6379: Connection refused

Also, maybe redis is not able to manage different services on the same server? (I have Nextcloud installed on the same server, also using redis)

TrystanLea · 2021-10-06T21:29:21.563Z

Sounds like there may be a redis configuration issue, is redis enabled in emoncms/settings.ini

sisim · 2021-10-06T21:34:53.478Z

Hello, thanks for your quick reply!

I don’t have any settings.ini, I have only settings.php. And I see:

// Redis Database (used as a cache for improved performance)
"redis"=>array(
    'enabled' => false
),

I modified it to true, but then I can’t run emoncms anymore:

 Fatal error: Uncaught RedisException: Connection refused in /var/www/html/emoncms/index.php:34 Stack trace: #0 /var/www/html/emoncms/index.php(34): Redis->connect() #1 {main} thrown in /var/www/html/emoncms/index.php on line 34

TrystanLea · 2021-10-06T21:38:15.877Z

Ok, if you can fix the reason it won’t connect and enable redis that will make a large difference to your server performance. The issue is outside of emoncms and something to do with your redis configuration so I wouldn’t know where to start. But hopefully that gives you a clue to investigate …

sisim · 2021-10-06T21:45:39.257Z

Thank you very much. I just would like to check if it’s the right place to activate redis? (settings.php instead of settings.ini)

I’ll investigate further. But why is the administration page showing that redis is running… if it’s not running properly? I didn’t look into this direction, because I thought that redis was running correctly.

sisim · 2021-10-07T12:42:31.719Z

Hello,
I have solved the redis config and it’s running now. But the load is still high, mysqld being replaced by redis-server:

Would you have any idea to investigate further? Do you have such issues with the IoTaWatt connected to your own server?

NB: I’d be interested to know why the administration page on emoncms was showing that redis was running, while it was not running properly? I didn’t look into this direction, because I thought that redis was running correctly.

TrystanLea · 2021-10-07T14:03:15.726Z

Great to hear that you managed to get redis running, surprised that it hasn’t made a significant difference for you, the snap shot that you share there does suggest redis at ~37% cpu and 0.1% mem compared to 81.5% and 3.8% with mysql… is redis always at ~37%? if so there may be another factor that is contributing to the high server load.

What feed engines are you using to record the data?

is this a dedicated machine , does it have SSD storage?

You could also enable the redis buffer here emoncms/default-settings.php at master · emoncms/emoncms · GitHub, you need to have the feedwriter service installed: emoncms/install-service-feedwriter.md at master · emoncms/emoncms · GitHub

20 Iotawatts should be no problem at all with the right configuration. If they are uploading historic data after an outage things can slow down for a bit, but it’s usually only temporary.

sisim · 2021-10-07T15:31:56.835Z

Thank you so much for you advice!

The majority of the server load is due to emoncms, as you’ll see in the screenshots below. This first one is with IWs connected:

This second one is just after disconnection of IWs (stopped emoncms):

image694×162 7.82 KB

Redis-server is most of the time between 20 and 40%. Nothing else uses Redis on the server (when I stop emoncms, redis-server is at 0).

We’re using the standard feed engine to record the data (PHPFINA). And 7 VIRTUAL feeds.

It is not a dedicated machine (also Nextcloud is running on it). But as you see in the screenshot above, the load is mainly due to emoncms. It has no SSD, only HDD in RAID. Could it be the reason?

I’ll have a look into the redis buffer options and feedwriter service and keep you updated.

We have constant internet issues with IWs: they use mobile internet and we have a poor connection (we are in Africa and this is due to the mobile network). This makes IWs uploading historic data frequently. Could it create an issue related to the persistence setup of redis? (I have RDB persistence default settings in redis).

TrystanLea · 2021-10-07T16:30:07.799Z

sisim:

Could it create an issue related to the persistence setup of redis?

Yes that may be contributing to it, can you try disabling persistence in /etc/redis/redis.conf

#save 900 1
#save 300 1
#save 60 1

sisim:

It is not a dedicated machine

Is this your own machine/hardware or hosted with someone else? what type of hosting is it?

sisim · 2021-10-07T16:31:56.888Z

I tried to launch the feedwriter service, following the instructions in emoncms/install-service-feedwriter.md at master · emoncms/emoncms · GitHub

The first step of the instructions is Removal of /etc/init.d script, while I don’t have any /etc/init.d/feedwriter. Is it normal?

My blocking point is the following error when I try to launch the feedwriter service:

Started Emoncms feedwriter script.
/bin/chown: invalid user: ‘pi’
feedwriter.service: Control process exited, code=exited, status=1/FAILURE
feedwriter.service: Failed with result 'exit-code'.

I guess that the script was done for a Raspberry Pi and not adapted to a standard Debian install?

sisim · 2021-10-07T16:43:48.971Z

I adapted the script to work (changing the user and the path of the script), but I have now the following error:

Fatal error: Uncaught RedisException: Connection refused in /var/www/html/emoncms/scripts/feedwriter.php:34

TrystanLea · 2021-10-07T19:40:41.114Z

Hello @sisim it sounds like a similar error to the one you had earlier,how did you solve that?

Did turning off redis persistance help things?

sisim · 2021-10-07T20:03:30.363Z

I solved it by correcting the port in the redis.conf (for some reason with another software, I changed the port which wasn’t 6379 anymore: putting it back to 6379 solved the problem). I didn’t change anything, redis is working, I see it in the active processes.

Maybe the user isn’t the right one in the script? I changed it from pi to sisim (the main user on my server), and also tried root or www-data (line 52 of /var/www/html/emoncms/scripts/services/feedwriter/feedwriter.service). But no result.

sisim · 2021-10-07T20:06:58.870Z

I’ll try tomorrow to disable persistence: I don’t have access to the IWs now (I need to reconnect them and test).

The machine is our own machine: this is a server with an old AMD E350D CPU, running Debian 10.

MyForest · 2021-10-09T21:04:16.701Z

Just to give you some hope that it will work out in the end, I have EmonCMS running with MySQL and Redis with about 100 PHPFINA feeds and it’s using about 1% CPU. I have data coming in from the car, solar, heat pump and weather station and some of that is at 6-second granularity.

Oh, and it’s all running inside Docker so that’ll no doubt be draining a bit too.

Apparently I have 653 processes running on this machine!

I’m sure with the help of this community you will prevail.

sisim · 2021-10-11T03:43:15.077Z

Thank you for your encouragement @MyForest ! In our situation, there’s the specific case of our IoTaWatts’ management, which is causing the server load issue specifically when uploading historic data. I’ll continue to investigate…

sisim · 2021-12-08T12:52:13.601Z

Hello all, I didn’t find the perfect configuration on the server side, so I used the hardware solution: we upgraded the CPU and RAM and we’re fine now.

Thank you all for your support!