RAM use increases continuously to failure

Emoncms
1

I’m running emoncms low-write 10.5.6 version on a Pi 3 Model B (Rev 1.2) with 1 GB memory. I’ve got about 140 Feeds running and about 12 dashboards hosting about 20 multicharts. This setup operates very nicely, except that RAM use continuously increases. After running about 10 days, RAM use reaches ca. 70% and Swap use starts increasing. After a couple of more days, Swap reaches 100% and soon the emon server fails (that is, I cannot login to emon and/or Feeds no longer accumulate). This chart shows two cycles, one with default Swappiness of 60, and a cycle that finished last night with Swappiness set to 1.

image
image720×365 95.9 KB

Mem_used (aka RAM) is scaled on the right side axis while Swap_used and var_log_used is scaled on the left axis. I’m recording the RAM and Swap as revealed by free and the var_log use is log2ram use from df –h.
The good news (I think), is that var/log is behaving as expected.

What could be causing the continuous increase in Mem use? Using htop, I see there are 40 mysql processes each listed as using 6.5% of MEM. But I am not experienced enough to troubleshoot memory use problems further. I’m happy to diagnose this memory problem more if someone can give me pointers on what to do.

2

Hi Steve,

Are you sure that the feedwriter is actually writing the data to disc?
If you go to admin → Server information
on the line for feedwriter what does it say?
Should be something like ::
Active Running - sleep 300s 508 feed points pending write
Whats yours got?

3

I believe the feedwriter is behaving. At the moment, it shows 51 feed points pending write:

image
image1356×807 61.4 KB

50 or so feeds points pending is typical, from what I remember.

Meanwhile, it’s been 34 hours since reboot and the RAM use (as reported on the Admin page) has risen from 19% (right after reboot) to 31%.

image
image1395×743 48.2 KB

4

Worrying your emonhub looks upset.
Is there anything in the section " Emoncms Log" ?

It obviously seems like there is a memory leak somewhere. I was hoping that it was a build up of data waiting to be written to disc - ie feedwriter not writing for whatever reason but that seems OK.

Emonhub is worrying me

1 Like
5

Nothing in the emon log since the reboot. These are the last lines, reflecting the problems on 06-21 when RAM and Swap were close to 100% the other day:

2021-06-21 02:36:17.321|ERROR|demandshaper_run.php|Can't connect to database, please verify credentials/configuration in settings.php
2021-06-21 02:37:34.118|ERROR|demandshaper_run.php|Can't connect to database, please verify credentials/configuration in settings.php
2021-06-21 02:38:49.599|ERROR|demandshaper_run.php|Can't connect to database, please verify credentials/configuration in settings.php
2021-06-21 02:39:59.273|ERROR|demandshaper_run.php|Can't connect to database, please verify credentials/configuration in settings.php
2021-06-21 02:41:37.405|WARN|emoncms_mqtt.php|Not connected, retrying connection
2021-06-21 02:41:37.454|WARN|emoncms_mqtt.php|Connecting to MQTT server: Connection Accepted.: code: 0
2021-06-21 02:47:38.290|WARN|emoncms_mqtt.php|Not connected, retrying connection
2021-06-21 02:47:38.314|WARN|emoncms_mqtt.php|Connecting to MQTT server: Connection Accepted.: code: 0
2021-06-21 02:50:24.785|WARN|feed_model.php|Feed model: Requested feed does not exist feedid=179
2021-06-21 03:02:11.817|WARN|feed_model.php|Feed model: Requested feed does not exist feedid=179

As for emonhub, I believe I am not using any of its services.

6

If you can log into the Pi - can you do

cat /var/log/emonhub/emonhub.log

Thanks!

1 Like
7

Well, the emonhub log doesn’t look good!

2021-06-22 10:29:38,321 CRITICAL MainThread Error parsing config file "/etc/emonhub/emonhub.conf": Parsing failed with several errors.
First error at line 243.
2021-06-22 10:29:45,543 CRITICAL MainThread Error parsing config file "/etc/emonhub/emonhub.conf": Parsing failed with several errors.
First error at line 243.

Here’s the block of emonhub.conf starting at line 243:

[[5]]
    nodename = emonPi
    firmware = emonPi_RFM69CW_RF12Demo_DiscreteSampling.ino
    hardware = emonpi
    [[[rx]]]
        names = power1,power2,power1_plus_power2,Vrms,T1,T2,T3,T4,T5,T6,pulseCount
        datacodes = h, h, h, h, h, h, h, h, h, h, L
        scales = 1,1,1,0.01,0.1,0.1,0.1,0.1,0.1,0.1,1
        units = W,W,W,V,C,C,C,C,C,C,p

FWIW, I am not using emonhub; all of my inputs are sent thru the API. Maybe I should kill emonhub and see what happens to RAM use?

8

Not that it’s going to solve your issue, but for reference, and to clear up confusion about swappiness…

The biggest misconception about the swappiness parameter is that it controls the amount of RAM used before swapping starts. This article explains what swappiness really does:

https://www.howtogeek.com/449691/what-is-swapiness-on-linux-and-how-to-change-it/

9

The fact it errors at [[5]] would suggest you have 2 node 5’s defined. Comment out one of those blocks of settings to remedy that error.

But if you don’t want to use emonhub then stop and disable the emonhub service using systemctl.

Emonhub, feedwriter, mosquitto and log2ram all use RAM to queue data so be sure to tie off loose ends when not using certain aspects.

Are you using mySQL for anything other than emoncms? 40 each at 6.5% is 260% so something smells fishy there. MySQL usually misbehaves in RAM when there is little/no space on disc, is there space on all partitions?

10

I just disabled emonhub and demandshaper services using

sudo systemctl stop emonhub.service

and

sudo systemctl stop demandshaper.service

I’ll watch and see if there is any change to RAM use over the next few days.

I too am puzzled by the Mem use of mysql processes reported by htop. I’m not using MySQL for anything else on this Pi. This Pi is only running emoncms and Dataplicity.

Here’s current Disk usage - I don’t think it’s running out of disk space:

image
image1159×481 31 KB

11

Just in case you mis-typed when you said “disabled”, that command just stops them, they’ll still try and autostart next reboot or if a dependant service starts them. To actually disable them, you need to use either:

sudo systemctl disable emonub

or

sudo systemctl mask emonhub

The difference between the two is explained succinctly here or the full detail is in the man page here (search for disable or mask)

12

My emonhub.config contained three identical sets of nodes defined. I deleted two of them, rebooted, and RAM use has remained stable at 20% (188 MB used) for the last 12 hours. So looks like cleaning up the emonhub.config solved the memory leak. Thanks for all the help!

1 Like
13

Good to hear that sorted the issue, the script that was adding in duplicate node entries has recently been disabled, could you check what the admin page states is your EmonScripts version? should be v1.2.2

14

On your new “Components” page, I see “EmonScripts version 1.2.4.”
I don’t know why it indicates it’s been locally modified.

image
image1486×107 7.48 KB

Let me know if you’d like to see anything else

15

If you can log into the machine cd into that directory and run
git status

that will display any local changes.

16

Sorry yes that is correct, looks like you have the latest version. those local changes are changes to file permissions and can be ignored. Let us know if this happens again.

17

Hi,

Can you give details how are you generating this grapsh about mem, cpu… with emoncms with raspberry?
thanks

18

I’m using Python to run the system commands df and free, parse the values I want with partition and slice, then send to my emon server using the API. Note that the emon Sys Admin calculates RAM use slightly differently from the Mem output of free. Here’s the code:

#!/usr/bin/python3
# Call this script by cron as desired.

import subprocess
import requests
from datetime import datetime, date

# Run 'df -h' command and extract size of /var/log
# Following examples from https://askubuntu.com/questions/1063412/check-available-memory-in-linux 
df_output = subprocess.run(['df', '-h'], capture_output=True)  # echos result to screen
# convert to string
df_output_txt = str(df_output)
# Use 'partition' to twice. First cut everything before, then cut every everything after desired value
(discard, separator, remainder) = df_output_txt.partition('50M  ')
(var_log_used_M, separator, discard) = remainder.partition('M')
print('var_log_used_M: ', var_log_used_M)

# Run 'free --kilo' and extract sizes of Mem and Swap
free_output = subprocess.run(["free", "--kilo"], capture_output=True)
print('free_output: ', free_output)
free_output = str(free_output)

# Extract Mem value using partition then slice
(discard, separator, remainder) = free_output.partition('Mem:        ')
print('remainder after Mem: ', remainder)
Mem_used_kilo = remainder[13:19]  #this should catch all the digits, but might need adjusting on different systems

#print('Mem_kilo: ', Mem_kilo)
Mem_used_M = int(Mem_used_kilo) / 1000
print('Mem_used_M: ', str(Mem_used_M))

# Extract Swap value
(discard, separator, remainder) = free_output.partition('Swap:        ')
Swap_used_k = remainder[17:22]  #might need adjusting on different systems
Swap_used_M = int(Swap_used_k) / 1000
print('Swap_used_M: ', str(Swap_used_M))

# Prepare to send data to emonPi
# Determine epochtime
epochtime = str(int((datetime.utcnow() - datetime(1970, 1, 1)).total_seconds()))

# Build data to send to emonPi
datapack = '{Mem_used_M:' + str(Mem_used_M) + ',Swap_used_M:' + str(Swap_used_M) + ',var_log_used_M:' + str(var_log_used_M) +'}'
print(datapack)

# Combine the url with the time and datapack into "fullurl":
fullurl = 'http://192.168.x.xx/input/post?time=' + epochtime + '&node=emonPi&json=' + datapack + "&apikey=useyourkeyhere"
print('emonPi fullurl: ' + fullurl)

# Now send the message to emomPi
r = requests.post(fullurl)
19

hi,

I have tried but appears an error, any suggestion:

./test.pl

Traceback (most recent call last):
File “./test.pl”, line 19, in
df_output = subprocess.run([‘df’, ‘-h’], capture_output=True) # echos result to screen
File “/usr/lib/python3.5/subprocess.py”, line 383, in run
with Popen(*popenargs, **kwargs) as process:
TypeError: init() got an unexpected keyword argument ‘capture_output’
root@emonpi:/tmp# dpkg -l |grep pyth
ii dh-python 2.20170125 all Debian helper tools for packaging Python libraries and applications
ii libpython-all-dev:armhf 2.7.13-2 armhf package depending on all supported Python development packages
ii libpython-dev:armhf 2.7.13-2 armhf header files and a static library for Python (default)
ii libpython-stdlib:armhf 2.7.13-2 armhf interactive high-level object-oriented language (default python version)
ii libpython2.7:armhf 2.7.13-2+deb9u3 armhf Shared Python runtime library (version 2.7)
ii libpython2.7-dev:armhf 2.7.13-2+deb9u3 armhf Header files and a static library for Python (v2.7)
ii libpython2.7-minimal:armhf 2.7.13-2+deb9u3 armhf Minimal subset of the Python language (version 2.7)
ii libpython2.7-stdlib:armhf 2.7.13-2+deb9u3 armhf Interactive high-level object-oriented language (standard library, version 2.7)
ii libpython3-dev:armhf 3.5.3-1 armhf header files and a static library for Python (default)
ii libpython3-stdlib:armhf 3.5.3-1 armhf interactive high-level object-oriented language (default python3 version)
ii libpython3.5:armhf 3.5.3-1+deb9u1 armhf Shared Python runtime library (version 3.5)
ii libpython3.5-dev:armhf 3.5.3-1+deb9u1 armhf Header files and a static library for Python (v3.5)
ii libpython3.5-minimal:armhf 3.5.3-1+deb9u1 armhf Minimal subset of the Python language (version 3.5)
ii libpython3.5-stdlib:armhf 3.5.3-1+deb9u1 armhf Interactive high-level object-oriented language (standard library, version 3.5)
ii python 2.7.13-2 armhf interactive high-level object-oriented language (default version)
ii python-all 2.7.13-2 armhf package depending on all supported Python runtime versions
ii python-all-dev 2.7.13-2 armhf package depending on all supported Python development packages
ii python-apt-common 1.1.0~beta5 all Python interface to libapt-pkg (locales)
ii python-can 1.5.2-3 all Controller Area Network (CAN) interface module - Python 2.x
ii python-cffi-backend 1.9.1-2 armhf Foreign Function Interface for Python calling C code - backend
ii python-chardet 2.3.0-2 all universal character encoding detector for Python2
ii python-colorzero 1.1 all Construct, convert, and manipulate colors in a Pythonic manner.
ii python-configobj 5.0.6-2 all simple but powerful config file reader and writer for Python 2
ii python-configparser 3.3.0r2-2 all backport of the enhanced config parser introduced in Python 3.2
ii python-crypto 2.6.1-7 armhf cryptographic algorithms and protocols for Python
ii python-cryptography 1.7.1-3+deb9u1 armhf Python library exposing cryptographic recipes and primitives (Python 2)
ii python-dbus 1.2.4-1 armhf simple interprocess messaging system (Python interface)
ii python-dev 2.7.13-2 armhf header files and a static library for Python (default)
ii python-enum34 1.1.6-1 all backport of Python 3.4’s enum package
ii python-gi 3.22.0-2 armhf Python 2.x bindings for gobject-introspection libraries
ii python-gpiozero 1.5.0 all Simple API for controlling devices attached to a Pi’s GPIO pins.
ii python-idna 2.2-1 all Python IDNA2008 (RFC 5891) handling (Python 2)
ii python-ipaddress 1.0.17-1 all Backport of Python 3 ipaddress module (Python 2)
ii python-keyring 10.1-1 all store and access your passwords safely
ii python-keyrings.alt 1.3-1 all alternate backend implementations for python-keyring
ii python-minimal 2.7.13-2 armhf minimal subset of the Python language (default version)
ii python-openssl 16.2.0-1 all Python 2 wrapper around the OpenSSL library
ii python-pika 0.10.0-1 all AMQP client library for Python 2
ii python-pip 9.0.1-2+rpt2 all Python package installer
ii python-pip-whl 9.0.1-2+rpt2 all Python package installer
ii python-pkg-resources 33.1.1-1 all Package Discovery and Resource Access using pkg_resources
ii python-pyasn1 0.1.9-2 all ASN.1 library for Python (Python 2 module)
ii python-requests 2.12.4-1 all elegant and simple HTTP library for Python2, built for human beings
ii python-rpi.gpio 0.6.5~stretch-1 armhf Python GPIO module for Raspberry Pi
ii python-secretstorage 2.3.1-2 all Python module for storing secrets - Python 2.x version
ii python-serial 3.2.1-1 all pyserial - module encapsulating access for the serial port
ii python-setuptools 33.1.1-1 all Python Distutils Enhancements
ii python-six 1.12.0-0+rpt1 all Python 2 and 3 compatibility utilities (Python 2)
ii python-smbus:armhf 3.1.2-3 armhf Python bindings for Linux SMBus access through i2c-dev
ii python-spidev 20170223~145721-1 all Bindings for Linux SPI access through spidev (Python 2)
ii python-urllib3 1.19.1-1 all HTTP library with thread-safe connection pooling for Python
ii python-wheel 0.29.0-2 all built-package format for Python
ii python-xdg 0.25-4 all Python 2 library to access freedesktop.org standards
ii python2.7 2.7.13-2+deb9u3 armhf Interactive high-level object-oriented language (version 2.7)
ii python2.7-dev 2.7.13-2+deb9u3 armhf Header files and a static library for Python (v2.7)
ii python2.7-minimal 2.7.13-2+deb9u3 armhf Minimal subset of the Python language (version 2.7)
ii python3 3.5.3-1 armhf interactive high-level object-oriented language (default python3 version)
ii python3-apt 1.1.0~beta5 armhf Python 3 interface to libapt-pkg
ii python3-cffi-backend 1.9.1-2 armhf Foreign Function Interface for Python 3 calling C code - runtime
ii python3-chardet 2.3.0-2 all universal character encoding detector for Python3
ii python3-crypto 2.6.1-7 armhf cryptographic algorithms and protocols for Python 3
ii python3-cryptography 1.7.1-3+deb9u1 armhf Python library exposing cryptographic recipes and primitives (Python 3)
ii python3-dbus 1.2.4-1 armhf simple interprocess messaging system (Python 3 interface)
ii python3-dev 3.5.3-1 armhf header files and a static library for Python (default)
ii python3-gi 3.22.0-2 armhf Python 3 bindings for gobject-introspection libraries
ii python3-gpiozero 1.4.1 all Simple API for controlling devices attached to a Pi’s GPIO pins.
ii python3-idna 2.2-1 all Python IDNA2008 (RFC 5891) handling (Python 3)
ii python3-keyring 10.1-1 all store and access your passwords safely - Python 3 version of the package
ii python3-keyrings.alt 1.3-1 all alternate backend implementations for python3-keyring
ii python3-minimal 3.5.3-1 armhf minimal subset of the Python language (default python3 version)
ii python3-pika 0.10.0-1 all AMQP client library for Python 3
ii python3-pip 9.0.1-2+rpt2 all Python package installer
ii python3-pkg-resources 33.1.1-1 all Package Discovery and Resource Access using pkg_resources
ii python3-pyasn1 0.1.9-2 all ASN.1 library for Python (Python 3 module)
ii python3-requests 2.12.4-1 all elegant and simple HTTP library for Python3, built for human beings
ii python3-rpi.gpio 0.6.5~stretch-1 armhf Python 3 GPIO module for Raspberry Pi
ii python3-secretstorage 2.3.1-2 all Python module for storing secrets - Python 3.x version
ii python3-serial 3.2.1-1 all pyserial - module encapsulating access for the serial port
ii python3-setuptools 33.1.1-1 all Python3 Distutils Enhancements
ii python3-six 1.12.0-0+rpt1 all Python 2 and 3 compatibility utilities (Python 3)
ii python3-spidev 20170223~145721-1 all Bindings for Linux SPI access through spidev (Python 3)
ii python3-urllib3 1.19.1-1 all HTTP library with thread-safe connection pooling for Python3
ii python3-wheel 0.29.0-2 all built-package format for Python
ii python3-xdg 0.25-4 all Python 3 library to access freedesktop.org standards
ii python3.5 3.5.3-1+deb9u1 armhf Interactive high-level object-oriented language (version 3.5)
ii python3.5-dev 3.5.3-1+deb9u1 armhf Header files and a static library for Python (v3.5)
ii python3.5-minimal 3.5.3-1+deb9u1 armhf Minimal subset of the Python language (version 3.5)

Only is needed to run this script as scheduled or is required any action on emoncms to configure this mem and cpu monitoring?

Thanks

Missatge de Steve Roof via OpenEnergyMonitor Community <[email protected]> del dia ds., 26 de juny 2021 a les 15:08:

20

Try replacing
df_output = subprocess.run(['df', '-h'], capture_output=True)
with
df_output = subprocess.run(['df', '-h'], stdout=subprocess.PIPE)

capture_output=True works only for Python 3.7+

Next time you need to troubleshoot your code, try Googling the line that gives an error. In this case, Googling “df_output = subprocess.run([‘df’, ‘-h’], capture_output=True)” led me to this page that provides good explanations and alternative approaches:

1 Like