High server load

Hello,

We have emoncms (10.8.8) installed on our server (Debian Buster) and we love this software, thank you for your work! We use 20 IoTaWatts (“IW”), connected to emoncms. Unfortunately, the server load is super high when IW are connected, making the server unusable for the other services installed.
image

As you can see on the screenshot above, the database (MariaDB) is the problem. When I stop the emoncms server (just by renaming the emoncms folder into emoncms2, so that Apache cannot find it anymore), the load goes back to normal (around 0.5) while other services are still running (notably Nextcloud). This is why I’m sure that the load issue is related to emoncms.

It is not confirmed yet, but we have the feeling that the load is higher when several IWs are uploading data which was not uploaded yet (for example after an interruption of the internet connection of the IW).

Would you have an advice to solve this?

Thanks in advance!

Here is a few more information about our config:

  • Debian 10 (buster)
  • Server version: Apache/2.4.38 (Debian)
  • PHP 8.0.11 (cli) (built: Sep 23 2021 22:03:11) ( NTS )
  • mariadb Ver 15.1 Distrib 10.3.29-MariaDB
  • CPU: AMD E350D x86_64 (2 cores)

There are few here who know anything about the IoTaWatt, so as it sounds as if it is that which is generating the excessive load on your server, I think the IoTaWatt forum might be a better place to ask. That is managed by Bob Lemaire, its creator, and he and the other users there are more likely to have the specific knowledge to help you.

Thanks for your quick feedback! I also asked on IoTaWatt side, but I’m not sure who to ask now :wink: (High server load - Support - IoTaWatt User Community)

His answer is the following:

Sorry I can’t be of more help, but this is an issue I think better addressed by the Open Energy Monitor folks (EMonCMS) and/or a Linux/Apache expert. I had suggested to the EmonCMS folks to put in an upload throttling feedback header in Emoncms to reduce load from many IoTaWatt recovering from an outage, but that was not adopted.
There may be configuration parameters in Apache that would work to reduce Emoncms load.I seem to recall reading years back that early versions of Emoncms used MySQL for the data store but that it became too slow. Subsequently the phpfina and other methods were developed to store the feed data more efficiently. The metadata for those feeds is probably stored in the sql database, but I don’t see where that would be a bottleneck unless they update with each write.
As I understand it, the actual data store in the Emoncms.org server uses queues to handle the traffic across multiple threads. I don’t know if the OSS version can do that.

I know the Iotawatt does send stored data after it loses the connection, and so I wondered whether that would be the sort of reply you’d get. @TrystanLea is your best hope here, as the creator of emonCMS. Hopefully, he’ll get a message about this and respond.

Hello @sisim do you have redis installed and enabled? And what feed engines are you using?

Redis should be enabled in emoncms/settings.ini

Hello,

I have the feeling that something can be done on the MySQL (mariaDB in my case) management, don’t you think? I was also advised to ping @TrystanLea , maybe you have an idea?

Also, is it possible that I have a redis issue? In the admin page, it looks like it’s working:
image

But if I run redis-cli ping, I receive an error message:
Could not connect to Redis at 127.0.0.1:6379: Connection refused

Also, maybe redis is not able to manage different services on the same server? (I have Nextcloud installed on the same server, also using redis)

Sounds like there may be a redis configuration issue, is redis enabled in emoncms/settings.ini

Hello, thanks for your quick reply!

I don’t have any settings.ini, I have only settings.php. And I see:

// Redis Database (used as a cache for improved performance)
"redis"=>array(
    'enabled' => false
),

I modified it to true, but then I can’t run emoncms anymore:

 Fatal error: Uncaught RedisException: Connection refused in /var/www/html/emoncms/index.php:34 Stack trace: #0 /var/www/html/emoncms/index.php(34): Redis->connect() #1 {main} thrown in /var/www/html/emoncms/index.php on line 34

Ok, if you can fix the reason it won’t connect and enable redis that will make a large difference to your server performance. The issue is outside of emoncms and something to do with your redis configuration so I wouldn’t know where to start. But hopefully that gives you a clue to investigate …

1 Like

Thank you very much. I just would like to check if it’s the right place to activate redis? (settings.php instead of settings.ini)

I’ll investigate further. But why is the administration page showing that redis is running… if it’s not running properly? I didn’t look into this direction, because I thought that redis was running correctly.

Hello,
I have solved the redis config and it’s running now. But the load is still high, mysqld being replaced by redis-server:
image

Would you have any idea to investigate further? Do you have such issues with the IoTaWatt connected to your own server?

NB: I’d be interested to know why the administration page on emoncms was showing that redis was running, while it was not running properly? I didn’t look into this direction, because I thought that redis was running correctly.

Great to hear that you managed to get redis running, surprised that it hasn’t made a significant difference for you, the snap shot that you share there does suggest redis at ~37% cpu and 0.1% mem compared to 81.5% and 3.8% with mysql… is redis always at ~37%? if so there may be another factor that is contributing to the high server load.

What feed engines are you using to record the data?

is this a dedicated machine , does it have SSD storage?

You could also enable the redis buffer here emoncms/default-settings.php at master · emoncms/emoncms · GitHub, you need to have the feedwriter service installed: emoncms/install-service-feedwriter.md at master · emoncms/emoncms · GitHub

20 Iotawatts should be no problem at all with the right configuration. If they are uploading historic data after an outage things can slow down for a bit, but it’s usually only temporary.

Thank you so much for you advice!

The majority of the server load is due to emoncms, as you’ll see in the screenshots below. This first one is with IWs connected:
image

This second one is just after disconnection of IWs (stopped emoncms):

Redis-server is most of the time between 20 and 40%. Nothing else uses Redis on the server (when I stop emoncms, redis-server is at 0).

We’re using the standard feed engine to record the data (PHPFINA). And 7 VIRTUAL feeds.

It is not a dedicated machine (also Nextcloud is running on it). But as you see in the screenshot above, the load is mainly due to emoncms. It has no SSD, only HDD in RAID. Could it be the reason?

I’ll have a look into the redis buffer options and feedwriter service and keep you updated.

We have constant internet issues with IWs: they use mobile internet and we have a poor connection (we are in Africa and this is due to the mobile network). This makes IWs uploading historic data frequently. Could it create an issue related to the persistence setup of redis? (I have RDB persistence default settings in redis).

Yes that may be contributing to it, can you try disabling persistence in /etc/redis/redis.conf

#save 900 1
#save 300 1
#save 60 1 

Is this your own machine/hardware or hosted with someone else? what type of hosting is it?

I tried to launch the feedwriter service, following the instructions in emoncms/install-service-feedwriter.md at master · emoncms/emoncms · GitHub

The first step of the instructions is Removal of /etc/init.d script, while I don’t have any /etc/init.d/feedwriter. Is it normal?

My blocking point is the following error when I try to launch the feedwriter service:

Started Emoncms feedwriter script.
/bin/chown: invalid user: ‘pi’
feedwriter.service: Control process exited, code=exited, status=1/FAILURE
feedwriter.service: Failed with result 'exit-code'.

I guess that the script was done for a Raspberry Pi and not adapted to a standard Debian install?

I adapted the script to work (changing the user and the path of the script), but I have now the following error:

Fatal error: Uncaught RedisException: Connection refused in /var/www/html/emoncms/scripts/feedwriter.php:34

Hello @sisim it sounds like a similar error to the one you had earlier,how did you solve that?

Did turning off redis persistance help things?

I solved it by correcting the port in the redis.conf (for some reason with another software, I changed the port which wasn’t 6379 anymore: putting it back to 6379 solved the problem). I didn’t change anything, redis is working, I see it in the active processes.

Maybe the user isn’t the right one in the script? I changed it from pi to sisim (the main user on my server), and also tried root or www-data (line 52 of /var/www/html/emoncms/scripts/services/feedwriter/feedwriter.service). But no result.