/var/log filling up - system stops

I have just suffered the /var/log filling up issue. The symptom is everything seems to stop.

This is a emonSD-17Oct19 based system that I have just updated.

The issue is that logrotate has changed and this is creating an issue with the logrotate and the changes to emonhub user. This will happen if you update the OS (currently the update doesn’t do that).

To confirm use the command

[email protected]:/var/log $ df -h /var/log
Filesystem      Size  Used Avail Use% Mounted on
log2ram          50M   50M     0 100% /var/log

To find the largest file in the log folder

sudo find /var/log -type f -printf "%s\t%p\n" | sort -n | tail -1 | cut -f 2

If you try and manually trigger logrotate you get

[email protected]:/var/log $ sudo /usr/sbin/logrotate /etc/logrotate.conf
error: skipping "/var/log/emonhub/emonhub.log" because parent directory has insecure permissions (It's world writable or writable by group which is not "root") Set "su" directive in config file to tell logrotate which user/group should be used for rotation.

If everything has stopped, SSH into the Pi and

sudo rm /var/log/emonhub/emonhub.log.*
sudo chown pi:root emonhub
sudo systemctl restart emonhub
systemctl status emonhub.service
sudo /usr/sbin/logrotate /etc/logrotate.conf

Should clear it.

reboot to be sure.

1 Like

@TrystanLea one more reason to remove this release from the safe update list.

minimalmodbus also fails installing.

1 Like