@TrystanLea
I have 4 emonTx/RPi3’s running the Oct 2018 image updated to 9.9.8.
A fifth RPi3 (watchman) is also on the local network receiving data by HTTP from the 4 emonTx’s.
watchman has mail installed for another purpose which incidentally means I also get a daily email at 6:25 informing on the progress of the log rotate.
On 21 April, the email reported errors when writing – No space left on device.
Today’s email was …
/etc/cron.daily/logrotate:
reading config file /etc/logrotate.conf
Reading state from file: /var/log/logrotate/logrotate.status
Allocating hash table for state file, size 64 entries Creating new state Creating new state Creating new state Creating new state Creating new state Creating new state Creating new state Creating new state Creating new state Creating new state Creating new state Creating new state Creating new state Creating new state Creating new state Creating new state Creating new state Creating new state Creating new state Creating new state Creating new state Creating new state Creating new state
Handling 1 logs
rotating pattern: /var/log/*.log /var/log/*/*.log /var/log/syslog /var/log/messages /var/log/btmp 1048576 bytes (2 rotations) empty log files are not rotated, old logs are removed considering log /var/log/auth.log
Now: 2019-04-27 06:25
Last rotated at 2019-04-14 06:00
log does not need rotating (log size is below the 'size' threshold) considering log /var/log/boot.log
Now: 2019-04-27 06:25
Last rotated at 2019-04-14 06:00
log does not need rotating (log size is below the 'size' threshold) considering log /var/log/daemon.log
Now: 2019-04-27 06:25
Last rotated at 2019-04-23 06:25
log needs rotating
considering log /var/log/dataplicity.log
Now: 2019-04-27 06:25
Last rotated at 2019-04-14 06:00
log does not need rotating (log size is below the 'size' threshold) considering log /var/log/emoncms.log
Now: 2019-04-27 06:25
Last rotated at 2019-04-14 06:00
log does not need rotating (log size is below the 'size' threshold) considering log /var/log/kern.log
Now: 2019-04-27 06:25
Last rotated at 2019-04-14 06:00
log does not need rotating (log size is below the 'size' threshold) considering log /var/log/mail.log
Now: 2019-04-27 06:25
Last rotated at 2019-04-14 06:00
log does not need rotating (log size is below the 'size' threshold) considering log /var/log/mysql.log
Now: 2019-04-27 06:25
Last rotated at 2019-04-14 06:00
log does not need rotating (log size is below the 'size' threshold) considering log /var/log/ntp_update.log
Now: 2019-04-27 06:25
Last rotated at 2019-04-14 06:00
log does not need rotating (log size is below the 'size' threshold) considering log /var/log/service-runner.log
Now: 2019-04-27 06:25
Last rotated at 2019-04-14 06:00
log does not need rotating (log size is below the 'size' threshold) considering log /var/log/user.log
Now: 2019-04-27 06:25
Last rotated at 2019-04-14 06:00
log does not need rotating (log size is below the 'size' threshold) considering log /var/log/apache2/access.log
Now: 2019-04-27 06:25
Last rotated at 2019-04-23 06:25
log needs rotating
considering log /var/log/apache2/error.log
Now: 2019-04-27 06:25
Last rotated at 2019-04-14 06:00
log does not need rotating (log size is below the 'size' threshold) considering log /var/log/apache2/other_vhosts_access.log
Now: 2019-04-27 06:25
Last rotated at 2019-04-14 06:00
log does not need rotating (log size is below the 'size' threshold) considering log /var/log/emonpilcd/emonpilcd.log
Now: 2019-04-27 06:25
Last rotated at 2019-04-14 06:00
log does not need rotating (log size is below the 'size' threshold) considering log /var/log/logrotate/logrotate.log
Now: 2019-04-27 06:25
Last rotated at 2019-04-14 06:00
log does not need rotating (log size is below the 'size' threshold) considering log /var/log/mosquitto/mosquitto.log
Now: 2019-04-27 06:25
Last rotated at 2019-04-14 06:00
log does not need rotating (log size is below the 'size' threshold) considering log /var/log/mysql/error.log
Now: 2019-04-27 06:25
Last rotated at 2019-04-14 06:00
log does not need rotating (log size is below the 'size' threshold) considering log /var/log/redis/redis-server.log
Now: 2019-04-27 06:25
Last rotated at 2019-04-14 06:00
log does not need rotating (log size is below the 'size' threshold) considering log /var/log/supervisor/supervisord.log
Now: 2019-04-27 06:25
Last rotated at 2019-04-14 06:00
log does not need rotating (log size is below the 'size' threshold) considering log /var/log/syslog
Now: 2019-04-27 06:25
Last rotated at 2019-04-23 06:25
log needs rotating
considering log /var/log/messages
Now: 2019-04-27 06:25
Last rotated at 2019-04-14 06:00
log does not need rotating (log size is below the 'size' threshold) considering log /var/log/btmp
Now: 2019-04-27 06:25
Last rotated at 2019-04-14 06:00
log does not need rotating (log size is below the 'size' threshold) rotating log /var/log/daemon.log, log->rotateCount is 2 dateext suffix '-20190427'
glob pattern '-[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9]'
renaming /var/log/daemon.log.2 to /var/log/daemon.log.3 (rotatecount 2, logstart 1, i 2), old log /var/log/daemon.log.2 does not exist renaming /var/log/daemon.log.1 to /var/log/daemon.log.2 (rotatecount 2, logstart 1, i 1), old log /var/log/daemon.log.1 does not exist renaming /var/log/daemon.log.0 to /var/log/daemon.log.1 (rotatecount 2, logstart 1, i 0), old log /var/log/daemon.log.0 does not exist copying /var/log/daemon.log to /var/log/daemon.log.1
error: error writing to /var/log/daemon.log.1: No space left on device
error: error copying /var/log/daemon.log to /var/log/daemon.log.1: No space left on device rotating log /var/log/apache2/access.log, log->rotateCount is 2 dateext suffix '-20190427'
glob pattern '-[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9]'
renaming /var/log/apache2/access.log.2 to /var/log/apache2/access.log.3 (rotatecount 2, logstart 1, i 2), old log /var/log/apache2/access.log.2 does not exist renaming /var/log/apache2/access.log.1 to /var/log/apache2/access.log.2 (rotatecount 2, logstart 1, i 1), old log /var/log/apache2/access.log.1 does not exist renaming /var/log/apache2/access.log.0 to /var/log/apache2/access.log.1 (rotatecount 2, logstart 1, i 0), old log /var/log/apache2/access.log.0 does not exist copying /var/log/apache2/access.log to /var/log/apache2/access.log.1
error: error writing to /var/log/apache2/access.log.1: No space left on device
error: error copying /var/log/apache2/access.log to /var/log/apache2/access.log.1: No space left on device rotating log /var/log/syslog, log->rotateCount is 2 dateext suffix '-20190427'
glob pattern '-[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9]'
renaming /var/log/syslog.2 to /var/log/syslog.3 (rotatecount 2, logstart 1, i 2), old log /var/log/syslog.2 does not exist renaming /var/log/syslog.1 to /var/log/syslog.2 (rotatecount 2, logstart 1, i 1), old log /var/log/syslog.1 does not exist renaming /var/log/syslog.0 to /var/log/syslog.1 (rotatecount 2, logstart 1, i 0), old log /var/log/syslog.0 does not exist copying /var/log/syslog to /var/log/syslog.1
error: error writing to /var/log/syslog.1: No space left on device
error: error copying /var/log/syslog to /var/log/syslog.1: No space left on device
error: error creating temp state file /var/log/logrotate/logrotate.status.tmp: No space left on device
And the contents of /var/log are …
pi@watchman:/ $ du -ah /var/log
9.7M /var/log/syslog.3
5.9M /var/log/daemon.log.3
204K /var/log/mail.info
204K /var/log/mail.log
16K /var/log/dataplicity.log
0 /var/log/ntp_update.log
0 /var/log/service-runner.log
0 /var/log/mysql.log
136K /var/log/emoncms.log
4.0K /var/log/supervisor/supervisord.log
4.0K /var/log/supervisor
4.0K /var/log/mosquitto/mosquitto.log
4.0K /var/log/mosquitto
4.0K /var/log/logrotate/logrotate.status
8.0K /var/log/logrotate/logrotate.log
12K /var/log/logrotate
4.0K /var/log/mysql/error.log
4.0K /var/log/mysql
2.8M /var/log/apache2/access.log.3
1.9M /var/log/apache2/access.log
0 /var/log/apache2/other_vhosts_access.log
4.0K /var/log/apache2/error.log
4.7M /var/log/apache2
4.0K /var/log/redis/redis-server.log
4.0K /var/log/redis
704K /var/log/auth.log
4.0K /var/log/debug
28K /var/log/kern.log
32K /var/log/messages
4.0K /var/log/user.log
20M /var/log/daemon.log
9.6M /var/log/syslog
4.0K /var/log/emonpilcd/emonpilcd.log
4.0K /var/log/emonpilcd
0 /var/log/btmp
4.0K /var/log/wtmp
8.0K /var/log/boot.log
50M /var/log
The last entries posted to syslog & daemon.log were on 24 Apr at 2:53 when feedwriter was active.
I checked the other nodes (not set up to send emails) …
Three emonTx/RPi3 instances had /var/log full at 50M and had stopped writing to syslog on 14th, 23rd and 26th April. The last entry on one was when feedwriter was active. Two had the last log entry at 6:25 during the log rotate …
Apr 23 06:25:13 emonpi-node-11 liblogging-stdlog: action 'action 12' suspended, next retry is Tue Apr 23 06:25:43 2019 [v8.24.0 try http://www.rsyslog.com/e/2007 ]
I don’t want to try anything radical at present as I’m out of UK & communicating with the system via Dataplicity.
Is it best just to let things run and await the release of the new image that you & others are working on?
Thx
