RAM use increases over time

CelsoHenriques · 11 April 2019 09:14

Thanks.
I confirm that everything is working good except one thing:

Did you noticed a ram increase over time? My emonpi have almost 8 days working without interruption and i notice ram increase every single day. A average of +/- 50 MB each day.
Only restore the ram usage by reboot the emonpi.
Do that happen with you too?

borpin · 18 April 2019 10:13

What does

df -h

and

ls -la /var/log

tell us?

CelsoHenriques · 18 April 2019 23:53

Sorry for the late reply. i wasn’t at home.
Here it is:

df -h

pi@emonpi:~ $ df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/root       3.9G  1.9G  1.9G  50% /
devtmpfs        484M     0  484M   0% /dev
tmpfs           489M     0  489M   0% /dev/shm
tmpfs           489M   50M  440M  11% /run
tmpfs           5.0M  4.0K  5.0M   1% /run/lock
tmpfs           489M     0  489M   0% /sys/fs/cgroup
tmpfs           1.0M     0  1.0M   0% /var/tmp
tmpfs            50M   50M     0 100% /var/log
tmpfs            30M  4.0K   30M   1% /tmp
/dev/mmcblk0p1   43M   22M   21M  52% /boot
/dev/mmcblk0p3  3.3G  115M  3.0G   4% /home/pi/data
tmpfs            98M     0   98M   0% /run/user/1000

pi@emonpi:~ $ ls -la /var/log
total 51152
drwxr-xr-x 11 root      root           520 Apr 18 06:25 .
drwxr-xr-x 12 root      root          4096 Aug  2  2018 ..
drwxr-xr-x  2 root      adm            100 Apr 12 13:39 apache2
-rw-r-----  1 root      adm          32768 Apr 18 23:50 auth.log
-rw-r--r--  1 root      root          6796 Apr 12 13:39 boot.log
-rw-------  1 root      utmp             0 Apr 12 13:39 btmp
-rw-r-----  1 root      adm       26103808 Apr 18 23:52 daemon.log
-rw-r-----  1 root      adm           1177 Apr 12 13:39 debug
-rw-rw-rw-  1 root      root           328 Apr 12 13:40 emoncms.log
drwxr-xr-x  2 pi        root            60 Apr 12 13:39 emonpilcd
-rw-r-----  1 root      adm          32768 Apr 15 04:56 kern.log
drwxr-xr-x  2 pi        pi              80 Apr 18 06:25 logrotate
-rw-r-----  1 root      adm          28672 Apr 18 23:52 messages
drwxr-xr-x  2 mosquitto mosquitto       60 Apr 12 13:39 mosquitto
-rw-rw-rw-  1 root      root             0 Apr 12 13:39 mqtt_input.log
drwxr-xr-x  2 mysql     adm             60 Apr 12 13:39 mysql
-rw-rw-rw-  1 root      root             0 Apr 12 13:39 mysql.log
-rw-rw-rw-  1 root      root             0 Apr 12 13:39 ntp_update.log
drwxr-xr-x  2 root      root            40 Apr 12 13:39 openhab
drwxr-xr-x  2 root      root            40 Apr 12 13:39 openhab2
drwxr-xr-x  2 redis     redis           60 Apr 12 13:39 redis
-rw-rw-rw-  1 root      root             0 Apr 12 13:39 service-runner.log
drwxr-xr-x  2 root      root            60 Apr 12 13:39 supervisor
-rw-r-----  1 root      adm       26152960 Apr 18 23:52 syslog
-rw-r-----  1 root      adm            190 Apr 12 13:39 user.log
-rw-rw-r--  1 root      utmp          1920 Apr 18 23:50 wtmp

or this if it is more easy:

borpin · 19 April 2019 06:21

First problem is the /var/log is full. quick fix is to delete daemon.log (it will come back).

Second (and unrelated actually), is this a 4Gb SD Card?

[edit]
What I then meant to say is that it could be that as it cannot write to disk rsyslog may be keeping the logs in RAM. If you make some space then restart the service sudo systemctl restart rsyslog.service does RAM useage drop?

Keep an eye on /var/log and see if the RAM use returns when it fills up again.

I mention a change to the rsyslog.conf file to stop the daemon.log file filling up here emonSD next steps: log2ram - #8 by borpin

pb66 · 19 April 2019 08:20

No it must be at least an 8G as the root partition is 3.9G and the data partition is 3.3G

In addition to ls you should use something like

sudo du /var/log -d1 -h

to report the size of the logs in subfolders

CelsoHenriques · 19 April 2019 08:33

Yes, i am using 8GB sd card. But i didn’t did the emonsd expand comand. Since is a 8GB, didn’t think was needed.

pb66 · 19 April 2019 08:37

I think it happens automatically

[edit - having said that I can find no evidence of reference to that now, maybe it’s not the case]

CelsoHenriques · 19 April 2019 08:45

But shouldn’t the log be deleted automaticaly when that happens?
Some auto “flush”?
The problema i see is even if i make more space, the log file will continue filling up space.

pb66 · 19 April 2019 08:52

There is another discussion in progress about possibly adding a safety net to do just that, but it is not definitely going to happen and if it does it won’t be mediate.

The cause of the log partition filling up needs to be address as a bug, the size of log partition and the logrotation must be matched to the usual logfile creation rates (with some headroom). Brian thinks he’s found the issue with the logs filling up. Did you look at the link he gave you?

CelsoHenriques · 19 April 2019 09:34

I dont know exactly how it works.

See: https://github.com/openenergymonitor/emonpi/wiki/emonSD-pre-built-SD-card-Download-%26-Change-Log

"

Expand data partition to fill SD card:
By default the ‘data’ partition used to store Emoncms data is 1GB, this can be expanded to fill the remaining space on the SD card. This has already been done on shop bought emonPi / emonBase. To expand the data partition:
Connect via SSH and run $ sudo emonSDexpand and then follow prompts

“Did you look at the link he gave you?”

Yes. No need to further investigate this topic.

It really seems what is happening to me, is related to that.

Thanks for remind-me the link. Brian send it and i somehow, i missed that.

Thanks to all for all the help. I will wait and watch the other topic to see when will get fixed.