The whole logs in RAM and logrotate setup is under review, personally I think the current position needs some triage in the interim (and I have said as much) but I don’t think that’s on the cards.
Lets find out what the root issue is first, if it’s a runaway log, a bit of triage to the existing setup isn’t going to resolve it so tacking the source is probably the better plan at this point.
This morning the system has been up 22 hours.
Swap Used: 0.00%
That’s the first time I have seen empty swap after a days use!
For the record:-
pi@emonpi:~/demandshaper $ sudo ls -la /var/log/*/ /var/log
/var/log:
total 2624
drwxr-xr-x 13 root root 600 Jun 22 07:17 .
drwxr-xr-x 12 root root 4096 Aug 2 2018 ..
drwxr-xr-x 2 root adm 120 Jun 22 08:17 apache2
-rw-r----- 1 root adm 78945 Jun 22 08:46 auth.log
-rw-r--r-- 1 root root 6684 Jun 21 10:24 boot.log
-rw------- 1 root utmp 1152 Jun 22 08:42 btmp
-rw-r----- 1 root adm 76843 Jun 22 08:46 daemon.log
-rw-r----- 1 root adm 1095810 Jun 22 07:17 daemon.log.1
-rw-r----- 1 root adm 1177 Jun 21 10:24 debug
drwxr-xr-x 2 pi pi 80 Jun 21 16:01 emoncms
-rw-rw-rw- 1 root root 0 Jun 21 10:24 emoncms.log
drwxrwxr-x 2 emonhub emonhub 60 Jun 21 10:24 emonhub
drwxr-xr-x 2 pi root 60 Jun 21 10:24 emonpilcd
-rw-r----- 1 root adm 65157 Jun 22 08:45 kern.log
drwxr-xr-x 2 pi pi 80 Jun 22 08:17 logrotate
-rw-r----- 1 root adm 45753 Jun 22 08:45 messages
drwxr-xr-x 2 mosquitto mosquitto 40 Jun 21 10:24 mosquitto
-rw-rw-rw- 1 root root 0 Jun 21 10:24 mqtt_input.log
drwxr-xr-x 2 mysql adm 60 Jun 21 10:24 mysql
-rw-rw-rw- 1 root root 0 Jun 21 10:24 mysql.log
-rw-rw-rw- 1 root root 0 Jun 21 10:24 ntp_update.log
drwxr-xr-x 2 root root 40 Jun 21 10:24 openhab
drwxr-xr-x 2 root root 40 Jun 21 10:24 openhab2
drwxr-xr-x 2 redis redis 60 Jun 21 10:24 redis
-rw-rw-rw- 1 root root 0 Jun 21 10:24 service-runner.log
drwxr-xr-x 2 root root 60 Jun 21 10:24 supervisor
-rw-r----- 1 root adm 191337 Jun 22 08:46 syslog
-rw-r----- 1 root adm 1087784 Jun 22 05:17 syslog.1
-rw-r----- 1 root adm 396 Jun 21 10:24 user.log
-rw-rw-r-- 1 root utmp 4224 Jun 22 08:42 wtmp
/var/log/apache2/:
total 1692
drwxr-xr-x 2 root adm 120 Jun 22 08:17 .
drwxr-xr-x 13 root root 600 Jun 22 07:17 ..
-rw-r--r-- 1 root root 345540 Jun 22 08:46 access.log
-rw-r--r-- 1 root root 1377463 Jun 22 08:17 access.log.1
-rw-rw-rw- 1 root adm 1951 Jun 22 08:15 error.log
-rw-r--r-- 1 root root 0 Jun 21 10:24 other_vhosts_access.log
/var/log/emoncms/:
total 4
drwxr-xr-x 2 pi pi 80 Jun 21 16:01 .
drwxr-xr-x 13 root root 600 Jun 22 07:17 ..
-rw-rw-rw- 1 root root 1058 Jun 22 08:42 emoncms.log
lrwxrwxrwx 1 pi pi 30 Jun 21 16:01 emonpiupdate.log -> /home/pi/data/emonpiupdate.log
/var/log/emonhub/:
total 8
drwxrwxr-x 2 emonhub emonhub 60 Jun 21 10:24 .
drwxr-xr-x 13 root root 600 Jun 22 07:17 ..
-rwxrwxr-x 1 emonhub emonhub 6378 Jun 21 16:01 emonhub.log
/var/log/emonpilcd/:
total 4
drwxr-xr-x 2 pi root 60 Jun 21 10:24 .
drwxr-xr-x 13 root root 600 Jun 22 07:17 ..
-rw-r--r-- 1 pi root 306 Jun 21 16:01 emonpilcd.log
/var/log/logrotate/:
total 12
drwxr-xr-x 2 pi pi 80 Jun 22 08:17 .
drwxr-xr-x 13 root root 600 Jun 22 07:17 ..
-rw-r--r-- 1 root root 5506 Jun 22 08:17 logrotate.log
-rw-r--r-- 1 root root 1088 Jun 22 08:17 logrotate.status
/var/log/mosquitto/:
total 0
drwxr-xr-x 2 mosquitto mosquitto 40 Jun 21 10:24 .
drwxr-xr-x 13 root root 600 Jun 22 07:17 ..
/var/log/mysql/:
total 4
drwxr-xr-x 2 mysql adm 60 Jun 21 10:24 .
drwxr-xr-x 13 root root 600 Jun 22 07:17 ..
-rw-rw-rw- 1 mysql adm 1569 Jun 21 10:24 error.log
/var/log/openhab/:
total 0
drwxr-xr-x 2 root root 40 Jun 21 10:24 .
drwxr-xr-x 13 root root 600 Jun 22 07:17 ..
/var/log/openhab2/:
total 0
drwxr-xr-x 2 root root 40 Jun 21 10:24 .
drwxr-xr-x 13 root root 600 Jun 22 07:17 ..
/var/log/redis/:
total 4
drwxr-xr-x 2 redis redis 60 Jun 21 10:24 .
drwxr-xr-x 13 root root 600 Jun 22 07:17 ..
-rw-rw-rw- 1 redis redis 4017 Jun 21 10:24 redis-server.log
/var/log/supervisor/:
total 0
drwxr-xr-x 2 root root 60 Jun 21 10:24 .
drwxr-xr-x 13 root root 600 Jun 22 07:17 ..
-rw-rw-rw- 1 root root 0 Jun 21 10:24 supervisord.log
pi@emonpi:~/demandshaper $
There are quite a lot of messages going into the daemon.log and syslog. What we have discovered is that output to Std outputs end up here via rsyslog and are often duplicated.
What do these logs largely consist of (in terms of messages)? Is there a lot from the demandshaper or is it something else?
[edit]
As Paul said, these relate to the lwrfd service. If you are not using this service you should mask it so it will not run. I will guess the command to see if there is a service there (never used it);
systemctl status lwrfd
Thanks for responding
Nearly all lwrfd failing to connect to mqtt
pi@emonpi:~ $ systemctl status lwrfd
● lwrfd.service - LSB: LightwaveRF MQTT daemon startup
Loaded: loaded (/etc/init.d/lwrfd; generated; vendor preset: enabled)
Active: active (running) since Fri 2019-06-21 10:24:32 UTC; 1 day 6h ago
Docs: man:systemd-sysv-generator(8)
CGroup: /system.slice/lwrfd.service
└─1059 /usr/local/bin/lwrfmqtt
Jun 22 16:59:42 emonpi lwrfd[1059]: Reconnecting to MQTT ...
Jun 22 16:59:42 emonpi lwrfd[1059]: Failed to connect to MQTT, return code -1
Jun 22 16:59:52 emonpi lwrfd[1059]: Reconnecting to MQTT ...
Jun 22 16:59:52 emonpi lwrfd[1059]: Failed to connect to MQTT, return code -1
Jun 22 17:00:02 emonpi lwrfd[1059]: Reconnecting to MQTT ...
Jun 22 17:00:02 emonpi lwrfd[1059]: Failed to connect to MQTT, return code -1
Jun 22 17:00:12 emonpi lwrfd[1059]: Reconnecting to MQTT ...
Jun 22 17:00:13 emonpi lwrfd[1059]: Failed to connect to MQTT, return code -1
Jun 22 17:00:23 emonpi lwrfd[1059]: Reconnecting to MQTT ...
Jun 22 17:00:23 emonpi lwrfd[1059]: Failed to connect to MQTT, return code -1
pi@emonpi:~ $
What is the masking command format?
Try sudo systemctl mask --now lwrfd that should stop the service, disable it from starting at boot and mask it all in one hit, you can check it again with systemctl status lwrfd.
[Edit] or it might be sudo systemctl mask lwrfd --now I can’t be sure now.
I think it is
sudo systemctl --now mask lwrfd
From man systemctl
systemctl [OPTIONS...] COMMAND [NAME...]
Today up 2 days 7 minutes
Swap at 2% 2MB
pi@emonpi:~ $ sudo ls -la /var/log/*/ /var/log
/var/log:
total 3268
drwxr-xr-x 13 root root 600 Jun 23 04:17 .
drwxr-xr-x 12 root root 4096 Aug 2 2018 ..
drwxr-xr-x 2 root adm 120 Jun 23 09:17 apache2
-rw-r----- 1 root adm 156663 Jun 23 10:33 auth.log
-rw-r--r-- 1 root root 6684 Jun 21 10:24 boot.log
-rw------- 1 root utmp 1152 Jun 22 08:42 btmp
-rw-r----- 1 root adm 321377 Jun 23 10:33 daemon.log
-rw-r----- 1 root adm 1069647 Jun 23 04:17 daemon.log.1
-rw-r----- 1 root adm 1177 Jun 21 10:24 debug
drwxr-xr-x 2 pi pi 80 Jun 21 16:01 emoncms
-rw-rw-rw- 1 root root 0 Jun 21 10:24 emoncms.log
drwxrwxr-x 2 emonhub emonhub 60 Jun 21 10:24 emonhub
drwxr-xr-x 2 pi root 60 Jun 21 10:24 emonpilcd
-rw-r----- 1 root adm 99060 Jun 23 10:33 kern.log
drwxr-xr-x 2 pi pi 80 Jun 23 10:17 logrotate
-rw-r----- 1 root adm 63659 Jun 23 10:31 messages
drwxr-xr-x 2 mosquitto mosquitto 40 Jun 21 10:24 mosquitto
-rw-rw-rw- 1 root root 0 Jun 21 10:24 mqtt_input.log
drwxr-xr-x 2 mysql adm 60 Jun 21 10:24 mysql
-rw-rw-rw- 1 root root 0 Jun 21 10:24 mysql.log
-rw-rw-rw- 1 root root 0 Jun 21 10:24 ntp_update.log
drwxr-xr-x 2 root root 40 Jun 21 10:24 openhab
drwxr-xr-x 2 root root 40 Jun 21 10:24 openhab2
drwxr-xr-x 2 redis redis 60 Jun 21 10:24 redis
-rw-rw-rw- 1 root root 0 Jun 21 10:24 service-runner.log
drwxr-xr-x 2 root root 60 Jun 21 10:24 supervisor
-rw-r----- 1 root adm 503247 Jun 23 10:33 syslog
-rw-r----- 1 root adm 1082933 Jun 23 01:17 syslog.1
-rw-r----- 1 root adm 396 Jun 21 10:24 user.log
-rw-rw-r-- 1 root utmp 5760 Jun 23 10:33 wtmp
/var/log/apache2/:
total 2148
drwxr-xr-x 2 root adm 120 Jun 23 09:17 .
drwxr-xr-x 13 root root 600 Jun 23 04:17 ..
-rw-r--r-- 1 root root 836903 Jun 23 10:33 access.log
-rw-r--r-- 1 root root 1349436 Jun 23 09:17 access.log.1
-rw-rw-rw- 1 root adm 6153 Jun 23 09:30 error.log
-rw-r--r-- 1 root root 0 Jun 21 10:24 other_vhosts_access.log
/var/log/emoncms/:
total 4
drwxr-xr-x 2 pi pi 80 Jun 21 16:01 .
drwxr-xr-x 13 root root 600 Jun 23 04:17 ..
-rw-rw-rw- 1 root root 1544 Jun 23 10:31 emoncms.log
lrwxrwxrwx 1 pi pi 30 Jun 21 16:01 emonpiupdate.log -> /home/pi/data/emonpiupdate.log
/var/log/emonhub/:
total 8
drwxrwxr-x 2 emonhub emonhub 60 Jun 21 10:24 .
drwxr-xr-x 13 root root 600 Jun 23 04:17 ..
-rwxrwxr-x 1 emonhub emonhub 6378 Jun 21 16:01 emonhub.log
/var/log/emonpilcd/:
total 4
drwxr-xr-x 2 pi root 60 Jun 21 10:24 .
drwxr-xr-x 13 root root 600 Jun 23 04:17 ..
-rw-r--r-- 1 pi root 306 Jun 21 16:01 emonpilcd.log
/var/log/logrotate/:
total 12
drwxr-xr-x 2 pi pi 80 Jun 23 10:17 .
drwxr-xr-x 13 root root 600 Jun 23 04:17 ..
-rw-r--r-- 1 root root 4985 Jun 23 10:17 logrotate.log
-rw-r--r-- 1 root root 1088 Jun 23 10:17 logrotate.status
/var/log/mosquitto/:
total 0
drwxr-xr-x 2 mosquitto mosquitto 40 Jun 21 10:24 .
drwxr-xr-x 13 root root 600 Jun 23 04:17 ..
/var/log/mysql/:
total 4
drwxr-xr-x 2 mysql adm 60 Jun 21 10:24 .
drwxr-xr-x 13 root root 600 Jun 23 04:17 ..
-rw-rw-rw- 1 mysql adm 1569 Jun 21 10:24 error.log
/var/log/openhab/:
total 0
drwxr-xr-x 2 root root 40 Jun 21 10:24 .
drwxr-xr-x 13 root root 600 Jun 23 04:17 ..
/var/log/openhab2/:
total 0
drwxr-xr-x 2 root root 40 Jun 21 10:24 .
drwxr-xr-x 13 root root 600 Jun 23 04:17 ..
/var/log/redis/:
total 4
drwxr-xr-x 2 redis redis 60 Jun 21 10:24 .
drwxr-xr-x 13 root root 600 Jun 23 04:17 ..
-rw-rw-rw- 1 redis redis 4017 Jun 21 10:24 redis-server.log
/var/log/supervisor/:
total 0
drwxr-xr-x 2 root root 60 Jun 21 10:24 .
drwxr-xr-x 13 root root 600 Jun 23 04:17 ..
-rw-rw-rw- 1 root root 0 Jun 21 10:24 supervisord.log
pi@emonpi:~ $
lwrfd stopped so now see what another day does to swap.
pi@emonpi:~ $ systemctl status lwrfd
● lwrfd.service
Loaded: masked (/dev/null; bad)
Active: inactive (dead)
Jun 23 10:33:56 emonpi lwrfd[1059]: Reconnecting to MQTT ...
Jun 23 10:33:56 emonpi lwrfd[1059]: Failed to connect to MQTT, return code -1
Jun 23 10:34:06 emonpi lwrfd[1059]: Reconnecting to MQTT ...
Jun 23 10:34:06 emonpi lwrfd[1059]: Failed to connect to MQTT, return code -1
Jun 23 10:34:16 emonpi lwrfd[1059]: Reconnecting to MQTT ...
Jun 23 10:34:17 emonpi lwrfd[1059]: Failed to connect to MQTT, return code -1
Jun 23 10:34:27 emonpi lwrfd[1059]: Reconnecting to MQTT ...
Jun 23 10:34:27 emonpi lwrfd[1059]: Failed to connect to MQTT, return code -1
Jun 23 10:34:35 emonpi systemd[1]: Stopping lwrfd.service...
Jun 23 10:34:35 emonpi systemd[1]: Stopped lwrfd.service.
pi@emonpi:~ $
This morning up time 3 days.
Swap at 4.75MB
pi@emonpi:~ $ sudo ls -la /var/log/*/ /var/log
/var/log:
total 3544
drwxr-xr-x 13 root root 600 Jun 23 04:17 .
drwxr-xr-x 12 root root 4096 Aug 2 2018 ..
drwxr-xr-x 2 root adm 120 Jun 24 10:17 apache2
-rw-r----- 1 root adm 227308 Jun 24 10:22 auth.log
-rw-r--r-- 1 root root 6684 Jun 21 10:24 boot.log
-rw------- 1 root utmp 1152 Jun 22 08:42 btmp
-rw-r----- 1 root adm 362388 Jun 24 10:18 daemon.log
-rw-r----- 1 root adm 1069647 Jun 23 04:17 daemon.log.1
-rw-r----- 1 root adm 1177 Jun 21 10:24 debug
drwxr-xr-x 2 pi pi 80 Jun 21 16:01 emoncms
-rw-rw-rw- 1 root root 0 Jun 21 10:24 emoncms.log
drwxrwxr-x 2 emonhub emonhub 60 Jun 21 10:24 emonhub
drwxr-xr-x 2 pi root 60 Jun 21 10:24 emonpilcd
-rw-r----- 1 root adm 134130 Jun 24 10:18 kern.log
drwxr-xr-x 2 pi pi 80 Jun 24 10:17 logrotate
-rw-r----- 1 root adm 82685 Jun 24 10:18 messages
drwxr-xr-x 2 mosquitto mosquitto 40 Jun 21 10:24 mosquitto
-rw-rw-rw- 1 root root 0 Jun 21 10:24 mqtt_input.log
drwxr-xr-x 2 mysql adm 60 Jun 21 10:24 mysql
-rw-rw-rw- 1 root root 0 Jun 21 10:24 mysql.log
-rw-rw-rw- 1 root root 0 Jun 21 10:24 ntp_update.log
drwxr-xr-x 2 root root 40 Jun 21 10:24 openhab
drwxr-xr-x 2 root root 40 Jun 21 10:24 openhab2
drwxr-xr-x 2 redis redis 60 Jun 21 10:24 redis
-rw-rw-rw- 1 root root 0 Jun 21 10:24 service-runner.log
drwxr-xr-x 2 root root 60 Jun 21 10:24 supervisor
-rw-r----- 1 root adm 622520 Jun 24 10:20 syslog
-rw-r----- 1 root adm 1082933 Jun 23 01:17 syslog.1
-rw-r----- 1 root adm 396 Jun 21 10:24 user.log
-rw-rw-r-- 1 root utmp 6528 Jun 24 10:18 wtmp
/var/log/apache2/:
total 1544
drwxr-xr-x 2 root adm 120 Jun 24 10:17 .
drwxr-xr-x 13 root root 600 Jun 23 04:17 ..
-rw-r--r-- 1 root root 63989 Jun 24 10:22 access.log
-rw-r--r-- 1 root root 1414924 Jun 24 10:17 access.log.1
-rw-rw-rw- 1 root adm 94655 Jun 24 09:28 error.log
-rw-r--r-- 1 root root 0 Jun 21 10:24 other_vhosts_access.log
/var/log/emoncms/:
total 4
drwxr-xr-x 2 pi pi 80 Jun 21 16:01 .
drwxr-xr-x 13 root root 600 Jun 23 04:17 ..
-rw-rw-rw- 1 root root 1888 Jun 24 10:18 emoncms.log
lrwxrwxrwx 1 pi pi 30 Jun 21 16:01 emonpiupdate.log -> /home/pi/data/emonpiupdate.log
/var/log/emonhub/:
total 8
drwxrwxr-x 2 emonhub emonhub 60 Jun 21 10:24 .
drwxr-xr-x 13 root root 600 Jun 23 04:17 ..
-rwxrwxr-x 1 emonhub emonhub 6378 Jun 21 16:01 emonhub.log
/var/log/emonpilcd/:
total 4
drwxr-xr-x 2 pi root 60 Jun 21 10:24 .
drwxr-xr-x 13 root root 600 Jun 23 04:17 ..
-rw-r--r-- 1 pi root 306 Jun 21 16:01 emonpilcd.log
/var/log/logrotate/:
total 12
drwxr-xr-x 2 pi pi 80 Jun 24 10:17 .
drwxr-xr-x 13 root root 600 Jun 23 04:17 ..
-rw-r--r-- 1 root root 5506 Jun 24 10:17 logrotate.log
-rw-r--r-- 1 root root 1089 Jun 24 10:17 logrotate.status
/var/log/mosquitto/:
total 0
drwxr-xr-x 2 mosquitto mosquitto 40 Jun 21 10:24 .
drwxr-xr-x 13 root root 600 Jun 23 04:17 ..
/var/log/mysql/:
total 4
drwxr-xr-x 2 mysql adm 60 Jun 21 10:24 .
drwxr-xr-x 13 root root 600 Jun 23 04:17 ..
-rw-rw-rw- 1 mysql adm 1569 Jun 21 10:24 error.log
/var/log/openhab/:
total 0
drwxr-xr-x 2 root root 40 Jun 21 10:24 .
drwxr-xr-x 13 root root 600 Jun 23 04:17 ..
/var/log/openhab2/:
total 0
drwxr-xr-x 2 root root 40 Jun 21 10:24 .
drwxr-xr-x 13 root root 600 Jun 23 04:17 ..
/var/log/redis/:
total 4
drwxr-xr-x 2 redis redis 60 Jun 21 10:24 .
drwxr-xr-x 13 root root 600 Jun 23 04:17 ..
-rw-rw-rw- 1 redis redis 4017 Jun 21 10:24 redis-server.log
/var/log/supervisor/:
total 0
drwxr-xr-x 2 root root 60 Jun 21 10:24 .
drwxr-xr-x 13 root root 600 Jun 23 04:17 ..
-rw-rw-rw- 1 root root 0 Jun 21 10:24 supervisord.log
I don’t want to let the swap get much bigger as past experience suggests it will suddenly grow very rapidly and freeze the Pi.
What is the best way to clear the Swap? I have been rebooting.
What does df -h tell you?
[edit]
You can delete any log file named *.1
Use this truncate command to clear the biggest log files. Reboot Required Every 20 Days Or So To Restore Logging - #35 by borpin
Other than that, you need to investigate what is generating the messages.
Is this a recent EmonSD image as the base?
Interesting that swap is still increasing with the stopped lwrfd logging. seeing the output of df -h as @borpin suggests would be useful. Can you tell which logs are now increasing at the faster rate?
Just want to clear up that the demandshaper service does use the emoncms logger https://github.com/emoncms/demandshaper/blob/master/run.php#L34 but it looks like it missed catching a couple of errors above which then went to stdout.
Pleased to report for the first time I am aware of the Swap has shrunk slightly.
Up time is now 4 days 1 hour and the swap is 3.25%
This is without any intervention by me.
pi@emonpi:/var/log $ df -h
Filesystem Size Used Avail Use% Mounted on
/dev/root 3.9G 1.8G 2.0G 48% /
devtmpfs 484M 0 484M 0% /dev
tmpfs 489M 0 489M 0% /dev/shm
tmpfs 489M 50M 440M 11% /run
tmpfs 5.0M 4.0K 5.0M 1% /run/lock
tmpfs 489M 0 489M 0% /sys/fs/cgroup
tmpfs 1.0M 0 1.0M 0% /var/tmp
tmpfs 50M 6.1M 44M 13% /var/log
tmpfs 30M 52K 30M 1% /tmp
/dev/mmcblk0p1 43M 22M 21M 52% /boot
/dev/mmcblk0p3 26G 2.0G 23G 9% /home/pi/data
tmpfs 98M 0 98M 0% /run/user/1000
After
pi@emonpi:/var/log $ sudo rm *.1
pi@emonpi:/var/log $ df -h
Filesystem Size Used Avail Use% Mounted on
/dev/root 3.9G 1.8G 2.0G 48% /
devtmpfs 484M 0 484M 0% /dev
tmpfs 489M 0 489M 0% /dev/shm
tmpfs 489M 50M 440M 11% /run
tmpfs 5.0M 4.0K 5.0M 1% /run/lock
tmpfs 489M 0 489M 0% /sys/fs/cgroup
tmpfs 1.0M 0 1.0M 0% /var/tmp
tmpfs 50M 4.1M 46M 9% /var/log
tmpfs 30M 52K 30M 1% /tmp
/dev/mmcblk0p1 43M 22M 21M 52% /boot
/dev/mmcblk0p3 26G 2.0G 23G 9% /home/pi/data
tmpfs 98M 0 98M 0% /run/user/1000
I have not had time to tell which logs are now increasing at the faster rate nor have I used truncate yet. I will check again tomorrow. But if the swap is shrinking I guess the log rotate system is now working correctly
@TrystanLea whats the best way to access demandshaper logs?
All looks quite healthy, plenty of room.
The demandshaper log is combined into /var/log/emoncms/emoncms.log. You wont see much in there as the logging is much reduced from what it used to be. It will now only log the logging level set in emoncms/settings.php.
Not so good today.
Up time 5 days, before any intervention swap is up to 9.75%
pi@emonpi:/var/log $ df -h
Filesystem Size Used Avail Use% Mounted on
/dev/root 3.9G 1.8G 2.0G 48% /
devtmpfs 484M 0 484M 0% /dev
tmpfs 489M 0 489M 0% /dev/shm
tmpfs 489M 50M 440M 11% /run
tmpfs 5.0M 4.0K 5.0M 1% /run/lock
tmpfs 489M 0 489M 0% /sys/fs/cgroup
tmpfs 1.0M 0 1.0M 0% /var/tmp
tmpfs 50M 4.3M 46M 9% /var/log
tmpfs 30M 64K 30M 1% /tmp
/dev/mmcblk0p1 43M 22M 21M 52% /boot
/dev/mmcblk0p3 26G 2.1G 23G 9% /home/pi/data
tmpfs 98M 0 98M 0% /run/user/1000
pi@emonpi:/var/log $
Running ls -lah /var/log does not show any large logs. Maybe there is another cause of swap growing.
What exactly is swap used for?
pi@emonpi:/var/log $ ls -lah /var/log
total 2.0M
drwxr-xr-x 13 root root 540 Jun 25 14:45 .
drwxr-xr-x 12 root root 4.0K Aug 2 2018 ..
drwxr-xr-x 2 root adm 120 Jun 26 08:17 apache2
-rw-r----- 1 root adm 374K Jun 26 08:40 auth.log
-rw-r--r-- 1 root root 6.6K Jun 21 10:24 boot.log
-rw------- 1 root utmp 1.5K Jun 26 08:39 btmp
-rw-r----- 1 root adm 441K Jun 26 08:39 daemon.log
-rw-r----- 1 root adm 1.2K Jun 21 10:24 debug
drwxr-xr-x 2 pi pi 80 Jun 26 06:27 emoncms
drwxrwxr-x 2 emonhub emonhub 60 Jun 21 10:24 emonhub
drwxr-xr-x 2 pi root 60 Jun 21 10:24 emonpilcd
-rw-r----- 1 root adm 203K Jun 26 08:40 kern.log
drwxr-xr-x 2 pi pi 80 Jun 26 08:17 logrotate
-rw-r----- 1 root adm 121K Jun 26 08:40 messages
drwxr-xr-x 2 mosquitto mosquitto 40 Jun 21 10:24 mosquitto
-rw-rw-rw- 1 root root 0 Jun 21 10:24 mqtt_input.log
drwxr-xr-x 2 mysql adm 60 Jun 21 10:24 mysql
-rw-rw-rw- 1 root root 0 Jun 21 10:24 mysql.log
-rw-rw-rw- 1 root root 0 Jun 21 10:24 ntp_update.log
drwxr-xr-x 2 root root 40 Jun 21 10:24 openhab
drwxr-xr-x 2 root root 40 Jun 21 10:24 openhab2
drwxr-xr-x 2 redis redis 60 Jun 21 10:24 redis
-rw-rw-rw- 1 root root 0 Jun 21 10:24 service-runner.log
drwxr-xr-x 2 root root 60 Jun 21 10:24 supervisor
-rw-r----- 1 root adm 848K Jun 26 08:40 syslog
-rw-r----- 1 root adm 396 Jun 21 10:24 user.log
-rw-rw-r-- 1 root utmp 7.9K Jun 26 08:39 wtmp
pi@emonpi:/var/log $
How are you monitoring the swap @ian?
I tried free -m and swapon --all, swapon --summary all showing 0 use on my system.
Your logs do look fine, only 9% full, suggests its something else.
This might be useful for finding out what is using the swap:
https://www.cyberciti.biz/faq/linux-which-process-is-using-swap/
These should help answer that question:
Many thanks. I am pleased to say the swap has shrunk slightly today to 6.25% so it looks as if I may not have an issue. Clearly the logs are not currently casing a problem.
I will now just monitor the swap and hopefully not see the freezing that occurred in the past.
I’ve mentioned this a couple of times elsewhere on the forum but the “swappiness” needs reigning in on the emonSD images.
The swappiness is a setting between 0 and 100 that adjusts how aggressively the os swaps page files out of ram on to disk (or sdcard in our case) where 100 is really aggressive and the os tries to free up as much ram as it possibly can and 0 is either off (kernels 3.5>) or “minimum swappiness without disabling it entirely” (kernels >3.5).
Whilst 60 is the Linux default, earlier Raspbian images had the swappiness explicitly set to “1”. More recent Raspbian images are missing the explicit setting and therefore default to 60.
New 2019-04-08 Raspbian lite
pi@raspberrypi:~ $ uname -a
Linux raspberrypi 4.14.98-v7+ #1200 SMP Tue Feb 12 20:27:48 GMT 2019 armv7l GNU/Linux
pi@raspberrypi:~ $ sysctl vm.swappiness
vm.swappiness = 60
verses a much older image
pi@raspberrypi~ $ uname -a
Linux raspberrypi 4.1.19+ #858 Tue Mar 15 15:52:03 GMT 2016 armv6l GNU/Linux
pi@raspberrypi~ $ sysctl vm.swappiness
vm.swappiness = 1
There are many discussions out there on this topic and many different opinions, mainly due to different applications and/or different hardware, some users running high traffic websites from a fast hdd might want high swappiness (for example), but for the emonSD image I think off or at least “1” is better for the life of the sdcard. For ages we used RO images so a swapfile wasn’t accessible regardless of the setting so we know we can run on zero swapfile, with the Oct2018 image the os is not RO so it is using the default “60” which is 60% of the way towards “swap everything you possibly can” and way too high!
Whether swap is used or not will depend on personal application and activity, but a setting of 60 is intended to try and swap out if possible without being overly aggressive about it, so the use of swap doesn’t indicate a fault by itself.
IMO, we are not big users of RAM on the average emonSD so if swap is being depended upon, there is an issue to be resolved (eg the recent emonhub.logs etc) and if there is an issue, it will quickly run out of swap space as we have seen. For the emonSD, the use of swap space does 2 things, prolongs the inevitable crash if there is an issue where memory is being filled and nibbles away at the sdcard life expectancy.
Whilst I agree we have seen the swap space used (filled) when the log partition fills up, but i struggle to understand how the memory can overflow by over 100MB (size of swap file) because the max 50M limit is reached by the logs. Surely if the logs max out at 50MB that leaves 950MB for the system (1050MB including the available swap) so there are possibly other issues at large here and the log partition filling up just reduces the space available to the OS plus adding to it’s load by not allowing any logs to be written, thus causing the jam.
We could just reduce the swappiness to ~1-10 to retain it as an emergency escape lane (which would probably remove Ians 6-10% usage), but IMO any systems reliant on the swap to keep running are simply burning away sdcard life and not addressing any issues that make the system dependent on swap.
If anyone wants to test a lower swappiness, the easy way to do so is using
sudo sysctl vm.swappiness=0
where “0” is the level of swappiness between 0 and 100 (current default 60), this will revert at reboot and/or can be run more than once to adjust further.
The more permanent way is to write a line to a new drop-in file to /etc/sysctl.d/ eg
printf "vm.swappiness = 0" | sudo tee /etc/sysctl.d/50-emonsd.conf
where “50-emonsd.conf” is the file name, must end in .conf and the “50” is a priority level (I chose 50 as a midpoint) .
to load the changes to that file without a reboot use
sudo sysctl --system
Assuming everything is tickety-boo and swap isn’t needed this could actually make the webpages more responsive because loading from ram cache is so much faster than loading from disk cache (ie swapped out pages), but if we are so short of ram we need swap it might make things slower as any caching will be affected, but do we want faster at the cost of sdcard reliability and life expectancy? It would be better to make sure OEM/emoncms can always work within the limits of the RAM rather than rely on the swapfile.
Just my tuppence worth!
Interesting I was not aware of that. Sounds like a good idea to reduce to 1 again.
Checking 6 systems I administer running the latest SD card image. 3 had zero swap use and 3 had ~63 out of 99mb. I’ve applied the swappiness setting and will monitor the result.
I had to reboot for the swap to be reset to zero it seems?
I guess that would make sense as adjusting the swappiness is only altering when/how the page files are moved to disk, it doesn’t effect the files that are there already, perhaps over time those pages would be read back into ram and not returned due to the change in settings, but it seems the swap can be cleared with
sudo swapoff -a
sudo swapon -a
I’ve never (knowingly) applied this to a system that was actively utilising swap at the time.
It would be good to hear what results you get as I rarely, if ever see swap space being used so the impact on my setups is purely theoretical (afaik).
Many thanks. I have set swappiness to 10 and rebooted.
I continue to monitor swap.