Swap File Growing - question on log rotate & demandshaper log

Pleased to report for the first time I am aware of the Swap has shrunk slightly.
Up time is now 4 days 1 hour and the swap is 3.25%

This is without any intervention by me.

pi@emonpi:/var/log $  df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/root       3.9G  1.8G  2.0G  48% /
devtmpfs        484M     0  484M   0% /dev
tmpfs           489M     0  489M   0% /dev/shm
tmpfs           489M   50M  440M  11% /run
tmpfs           5.0M  4.0K  5.0M   1% /run/lock
tmpfs           489M     0  489M   0% /sys/fs/cgroup
tmpfs           1.0M     0  1.0M   0% /var/tmp
tmpfs            50M  6.1M   44M  13% /var/log
tmpfs            30M   52K   30M   1% /tmp
/dev/mmcblk0p1   43M   22M   21M  52% /boot
/dev/mmcblk0p3   26G  2.0G   23G   9% /home/pi/data
tmpfs            98M     0   98M   0% /run/user/1000

After
pi@emonpi:/var/log $ sudo rm *.1

pi@emonpi:/var/log $  df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/root       3.9G  1.8G  2.0G  48% /
devtmpfs        484M     0  484M   0% /dev
tmpfs           489M     0  489M   0% /dev/shm
tmpfs           489M   50M  440M  11% /run
tmpfs           5.0M  4.0K  5.0M   1% /run/lock
tmpfs           489M     0  489M   0% /sys/fs/cgroup
tmpfs           1.0M     0  1.0M   0% /var/tmp
tmpfs            50M  4.1M   46M   9% /var/log
tmpfs            30M   52K   30M   1% /tmp
/dev/mmcblk0p1   43M   22M   21M  52% /boot
/dev/mmcblk0p3   26G  2.0G   23G   9% /home/pi/data
tmpfs            98M     0   98M   0% /run/user/1000

I have not had time to tell which logs are now increasing at the faster rate nor have I used truncate yet. I will check again tomorrow. But if the swap is shrinking I guess the log rotate system is now working correctly

@TrystanLea whats the best way to access demandshaper logs?

All looks quite healthy, plenty of room.

The demandshaper log is combined into /var/log/emoncms/emoncms.log. You wont see much in there as the logging is much reduced from what it used to be. It will now only log the logging level set in emoncms/settings.php.

Not so good today.
Up time 5 days, before any intervention swap is up to 9.75%

pi@emonpi:/var/log $ df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/root       3.9G  1.8G  2.0G  48% /
devtmpfs        484M     0  484M   0% /dev
tmpfs           489M     0  489M   0% /dev/shm
tmpfs           489M   50M  440M  11% /run
tmpfs           5.0M  4.0K  5.0M   1% /run/lock
tmpfs           489M     0  489M   0% /sys/fs/cgroup
tmpfs           1.0M     0  1.0M   0% /var/tmp
tmpfs            50M  4.3M   46M   9% /var/log
tmpfs            30M   64K   30M   1% /tmp
/dev/mmcblk0p1   43M   22M   21M  52% /boot
/dev/mmcblk0p3   26G  2.1G   23G   9% /home/pi/data
tmpfs            98M     0   98M   0% /run/user/1000
pi@emonpi:/var/log $

Running ls -lah /var/log does not show any large logs. Maybe there is another cause of swap growing.
What exactly is swap used for?

pi@emonpi:/var/log $ ls -lah /var/log
total 2.0M
drwxr-xr-x 13 root      root       540 Jun 25 14:45 .
drwxr-xr-x 12 root      root      4.0K Aug  2  2018 ..
drwxr-xr-x  2 root      adm        120 Jun 26 08:17 apache2
-rw-r-----  1 root      adm       374K Jun 26 08:40 auth.log
-rw-r--r--  1 root      root      6.6K Jun 21 10:24 boot.log
-rw-------  1 root      utmp      1.5K Jun 26 08:39 btmp
-rw-r-----  1 root      adm       441K Jun 26 08:39 daemon.log
-rw-r-----  1 root      adm       1.2K Jun 21 10:24 debug
drwxr-xr-x  2 pi        pi          80 Jun 26 06:27 emoncms
drwxrwxr-x  2 emonhub   emonhub     60 Jun 21 10:24 emonhub
drwxr-xr-x  2 pi        root        60 Jun 21 10:24 emonpilcd
-rw-r-----  1 root      adm       203K Jun 26 08:40 kern.log
drwxr-xr-x  2 pi        pi          80 Jun 26 08:17 logrotate
-rw-r-----  1 root      adm       121K Jun 26 08:40 messages
drwxr-xr-x  2 mosquitto mosquitto   40 Jun 21 10:24 mosquitto
-rw-rw-rw-  1 root      root         0 Jun 21 10:24 mqtt_input.log
drwxr-xr-x  2 mysql     adm         60 Jun 21 10:24 mysql
-rw-rw-rw-  1 root      root         0 Jun 21 10:24 mysql.log
-rw-rw-rw-  1 root      root         0 Jun 21 10:24 ntp_update.log
drwxr-xr-x  2 root      root        40 Jun 21 10:24 openhab
drwxr-xr-x  2 root      root        40 Jun 21 10:24 openhab2
drwxr-xr-x  2 redis     redis       60 Jun 21 10:24 redis
-rw-rw-rw-  1 root      root         0 Jun 21 10:24 service-runner.log
drwxr-xr-x  2 root      root        60 Jun 21 10:24 supervisor
-rw-r-----  1 root      adm       848K Jun 26 08:40 syslog
-rw-r-----  1 root      adm        396 Jun 21 10:24 user.log
-rw-rw-r--  1 root      utmp      7.9K Jun 26 08:39 wtmp
pi@emonpi:/var/log $

How are you monitoring the swap @ian?

I tried free -m and swapon --all, swapon --summary all showing 0 use on my system.

Your logs do look fine, only 9% full, suggests its something else.

This might be useful for finding out what is using the swap:

https://www.cyberciti.biz/faq/linux-which-process-is-using-swap/

These should help answer that question:

Many thanks. I am pleased to say the swap has shrunk slightly today to 6.25% so it looks as if I may not have an issue. Clearly the logs are not currently casing a problem.

I will now just monitor the swap and hopefully not see the freezing that occurred in the past.

I’ve mentioned this a couple of times elsewhere on the forum but the “swappiness” needs reigning in on the emonSD images.

The swappiness is a setting between 0 and 100 that adjusts how aggressively the os swaps page files out of ram on to disk (or sdcard in our case) where 100 is really aggressive and the os tries to free up as much ram as it possibly can and 0 is either off (kernels 3.5>) or “minimum swappiness without disabling it entirely” (kernels >3.5).

Whilst 60 is the Linux default, earlier Raspbian images had the swappiness explicitly set to “1”. More recent Raspbian images are missing the explicit setting and therefore default to 60.

New 2019-04-08 Raspbian lite

pi@raspberrypi:~ $ uname -a
Linux raspberrypi 4.14.98-v7+ #1200 SMP Tue Feb 12 20:27:48 GMT 2019 armv7l GNU/Linux
pi@raspberrypi:~ $ sysctl vm.swappiness
vm.swappiness = 60

verses a much older image

pi@raspberrypi~ $ uname -a
Linux raspberrypi 4.1.19+ #858 Tue Mar 15 15:52:03 GMT 2016 armv6l GNU/Linux
pi@raspberrypi~ $ sysctl vm.swappiness
vm.swappiness = 1

There are many discussions out there on this topic and many different opinions, mainly due to different applications and/or different hardware, some users running high traffic websites from a fast hdd might want high swappiness (for example), but for the emonSD image I think off or at least “1” is better for the life of the sdcard. For ages we used RO images so a swapfile wasn’t accessible regardless of the setting so we know we can run on zero swapfile, with the Oct2018 image the os is not RO so it is using the default “60” which is 60% of the way towards “swap everything you possibly can” and way too high!

Whether swap is used or not will depend on personal application and activity, but a setting of 60 is intended to try and swap out if possible without being overly aggressive about it, so the use of swap doesn’t indicate a fault by itself.

IMO, we are not big users of RAM on the average emonSD so if swap is being depended upon, there is an issue to be resolved (eg the recent emonhub.logs etc) and if there is an issue, it will quickly run out of swap space as we have seen. For the emonSD, the use of swap space does 2 things, prolongs the inevitable crash if there is an issue where memory is being filled and nibbles away at the sdcard life expectancy.

Whilst I agree we have seen the swap space used (filled) when the log partition fills up, but i struggle to understand how the memory can overflow by over 100MB (size of swap file) because the max 50M limit is reached by the logs. Surely if the logs max out at 50MB that leaves 950MB for the system (1050MB including the available swap) so there are possibly other issues at large here and the log partition filling up just reduces the space available to the OS plus adding to it’s load by not allowing any logs to be written, thus causing the jam.

We could just reduce the swappiness to ~1-10 to retain it as an emergency escape lane (which would probably remove Ians 6-10% usage), but IMO any systems reliant on the swap to keep running are simply burning away sdcard life and not addressing any issues that make the system dependent on swap.

If anyone wants to test a lower swappiness, the easy way to do so is using

sudo sysctl vm.swappiness=0

where “0” is the level of swappiness between 0 and 100 (current default 60), this will revert at reboot and/or can be run more than once to adjust further.

The more permanent way is to write a line to a new drop-in file to /etc/sysctl.d/ eg

printf "vm.swappiness = 0" | sudo tee /etc/sysctl.d/50-emonsd.conf

where “50-emonsd.conf” is the file name, must end in .conf and the “50” is a priority level (I chose 50 as a midpoint) .

to load the changes to that file without a reboot use

sudo sysctl --system

Assuming everything is tickety-boo and swap isn’t needed this could actually make the webpages more responsive because loading from ram cache is so much faster than loading from disk cache (ie swapped out pages), but if we are so short of ram we need swap it might make things slower as any caching will be affected, but do we want faster at the cost of sdcard reliability and life expectancy? It would be better to make sure OEM/emoncms can always work within the limits of the RAM rather than rely on the swapfile.

Just my tuppence worth!

2 Likes

Interesting I was not aware of that. Sounds like a good idea to reduce to 1 again.

Checking 6 systems I administer running the latest SD card image. 3 had zero swap use and 3 had ~63 out of 99mb. I’ve applied the swappiness setting and will monitor the result.

I had to reboot for the swap to be reset to zero it seems?

1 Like

I guess that would make sense as adjusting the swappiness is only altering when/how the page files are moved to disk, it doesn’t effect the files that are there already, perhaps over time those pages would be read back into ram and not returned due to the change in settings, but it seems the swap can be cleared with

sudo swapoff -a
sudo swapon -a

I’ve never (knowingly) applied this to a system that was actively utilising swap at the time.

It would be good to hear what results you get as I rarely, if ever see swap space being used so the impact on my setups is purely theoretical (afaik).

1 Like

Many thanks. I have set swappiness to 10 and rebooted.

I continue to monitor swap.