EmonPi stopped updating inputs - /var/log/ full

johnbanks · 27 April 2019 20:37

I have 4 emonTx/RPi3’s running the Oct 2018 image updated to 9.9.8.

A fifth RPi3 (watchman) is also on the local network receiving data by HTTP from the 4 emonTx’s.

watchman has mail installed for another purpose which incidentally means I also get a daily email at 6:25 informing on the progress of the log rotate.

On 21 April, the email reported errors when writing – No space left on device.
Today’s email was …

/etc/cron.daily/logrotate:
reading config file /etc/logrotate.conf
Reading state from file: /var/log/logrotate/logrotate.status
Allocating hash table for state file, size 64 entries Creating new state Creating new state Creating new state Creating new state Creating new state Creating new state Creating new state Creating new state Creating new state Creating new state Creating new state Creating new state Creating new state Creating new state Creating new state Creating new state Creating new state Creating new state Creating new state Creating new state Creating new state Creating new state Creating new state

Handling 1 logs

rotating pattern: /var/log/*.log /var/log/*/*.log /var/log/syslog /var/log/messages /var/log/btmp  1048576 bytes (2 rotations) empty log files are not rotated, old logs are removed considering log /var/log/auth.log
  Now: 2019-04-27 06:25
  Last rotated at 2019-04-14 06:00
  log does not need rotating (log size is below the 'size' threshold) considering log /var/log/boot.log
  Now: 2019-04-27 06:25
  Last rotated at 2019-04-14 06:00
  log does not need rotating (log size is below the 'size' threshold) considering log /var/log/daemon.log
  Now: 2019-04-27 06:25
  Last rotated at 2019-04-23 06:25
  log needs rotating
considering log /var/log/dataplicity.log
  Now: 2019-04-27 06:25
  Last rotated at 2019-04-14 06:00
  log does not need rotating (log size is below the 'size' threshold) considering log /var/log/emoncms.log
  Now: 2019-04-27 06:25
  Last rotated at 2019-04-14 06:00
  log does not need rotating (log size is below the 'size' threshold) considering log /var/log/kern.log
  Now: 2019-04-27 06:25
  Last rotated at 2019-04-14 06:00
  log does not need rotating (log size is below the 'size' threshold) considering log /var/log/mail.log
  Now: 2019-04-27 06:25
  Last rotated at 2019-04-14 06:00
  log does not need rotating (log size is below the 'size' threshold) considering log /var/log/mysql.log
  Now: 2019-04-27 06:25
  Last rotated at 2019-04-14 06:00
  log does not need rotating (log size is below the 'size' threshold) considering log /var/log/ntp_update.log
  Now: 2019-04-27 06:25
  Last rotated at 2019-04-14 06:00
  log does not need rotating (log size is below the 'size' threshold) considering log /var/log/service-runner.log
  Now: 2019-04-27 06:25
  Last rotated at 2019-04-14 06:00
  log does not need rotating (log size is below the 'size' threshold) considering log /var/log/user.log
  Now: 2019-04-27 06:25
  Last rotated at 2019-04-14 06:00
  log does not need rotating (log size is below the 'size' threshold) considering log /var/log/apache2/access.log
  Now: 2019-04-27 06:25
  Last rotated at 2019-04-23 06:25
  log needs rotating
considering log /var/log/apache2/error.log
  Now: 2019-04-27 06:25
  Last rotated at 2019-04-14 06:00
  log does not need rotating (log size is below the 'size' threshold) considering log /var/log/apache2/other_vhosts_access.log
  Now: 2019-04-27 06:25
  Last rotated at 2019-04-14 06:00
  log does not need rotating (log size is below the 'size' threshold) considering log /var/log/emonpilcd/emonpilcd.log
  Now: 2019-04-27 06:25
  Last rotated at 2019-04-14 06:00
  log does not need rotating (log size is below the 'size' threshold) considering log /var/log/logrotate/logrotate.log
  Now: 2019-04-27 06:25
  Last rotated at 2019-04-14 06:00
  log does not need rotating (log size is below the 'size' threshold) considering log /var/log/mosquitto/mosquitto.log
  Now: 2019-04-27 06:25
  Last rotated at 2019-04-14 06:00
  log does not need rotating (log size is below the 'size' threshold) considering log /var/log/mysql/error.log
  Now: 2019-04-27 06:25
  Last rotated at 2019-04-14 06:00
  log does not need rotating (log size is below the 'size' threshold) considering log /var/log/redis/redis-server.log
  Now: 2019-04-27 06:25
  Last rotated at 2019-04-14 06:00
  log does not need rotating (log size is below the 'size' threshold) considering log /var/log/supervisor/supervisord.log
  Now: 2019-04-27 06:25
  Last rotated at 2019-04-14 06:00
  log does not need rotating (log size is below the 'size' threshold) considering log /var/log/syslog
  Now: 2019-04-27 06:25
  Last rotated at 2019-04-23 06:25
  log needs rotating
considering log /var/log/messages
  Now: 2019-04-27 06:25
  Last rotated at 2019-04-14 06:00
  log does not need rotating (log size is below the 'size' threshold) considering log /var/log/btmp
  Now: 2019-04-27 06:25
  Last rotated at 2019-04-14 06:00
  log does not need rotating (log size is below the 'size' threshold) rotating log /var/log/daemon.log, log->rotateCount is 2 dateext suffix '-20190427'
glob pattern '-[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9]'
renaming /var/log/daemon.log.2 to /var/log/daemon.log.3 (rotatecount 2, logstart 1, i 2), old log /var/log/daemon.log.2 does not exist renaming /var/log/daemon.log.1 to /var/log/daemon.log.2 (rotatecount 2, logstart 1, i 1), old log /var/log/daemon.log.1 does not exist renaming /var/log/daemon.log.0 to /var/log/daemon.log.1 (rotatecount 2, logstart 1, i 0), old log /var/log/daemon.log.0 does not exist copying /var/log/daemon.log to /var/log/daemon.log.1
error: error writing to /var/log/daemon.log.1: No space left on device
error: error copying /var/log/daemon.log to /var/log/daemon.log.1: No space left on device rotating log /var/log/apache2/access.log, log->rotateCount is 2 dateext suffix '-20190427'
glob pattern '-[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9]'
renaming /var/log/apache2/access.log.2 to /var/log/apache2/access.log.3 (rotatecount 2, logstart 1, i 2), old log /var/log/apache2/access.log.2 does not exist renaming /var/log/apache2/access.log.1 to /var/log/apache2/access.log.2 (rotatecount 2, logstart 1, i 1), old log /var/log/apache2/access.log.1 does not exist renaming /var/log/apache2/access.log.0 to /var/log/apache2/access.log.1 (rotatecount 2, logstart 1, i 0), old log /var/log/apache2/access.log.0 does not exist copying /var/log/apache2/access.log to /var/log/apache2/access.log.1
error: error writing to /var/log/apache2/access.log.1: No space left on device
error: error copying /var/log/apache2/access.log to /var/log/apache2/access.log.1: No space left on device rotating log /var/log/syslog, log->rotateCount is 2 dateext suffix '-20190427'
glob pattern '-[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9]'
renaming /var/log/syslog.2 to /var/log/syslog.3 (rotatecount 2, logstart 1, i 2), old log /var/log/syslog.2 does not exist renaming /var/log/syslog.1 to /var/log/syslog.2 (rotatecount 2, logstart 1, i 1), old log /var/log/syslog.1 does not exist renaming /var/log/syslog.0 to /var/log/syslog.1 (rotatecount 2, logstart 1, i 0), old log /var/log/syslog.0 does not exist copying /var/log/syslog to /var/log/syslog.1
error: error writing to /var/log/syslog.1: No space left on device
error: error copying /var/log/syslog to /var/log/syslog.1: No space left on device
error: error creating temp state file /var/log/logrotate/logrotate.status.tmp: No space left on device

And the contents of /var/log are …

pi@watchman:/ $ du -ah /var/log
9.7M    /var/log/syslog.3
5.9M    /var/log/daemon.log.3
204K    /var/log/mail.info
204K    /var/log/mail.log
16K     /var/log/dataplicity.log
0       /var/log/ntp_update.log
0       /var/log/service-runner.log
0       /var/log/mysql.log
136K    /var/log/emoncms.log
4.0K    /var/log/supervisor/supervisord.log
4.0K    /var/log/supervisor
4.0K    /var/log/mosquitto/mosquitto.log
4.0K    /var/log/mosquitto
4.0K    /var/log/logrotate/logrotate.status
8.0K    /var/log/logrotate/logrotate.log
12K     /var/log/logrotate
4.0K    /var/log/mysql/error.log
4.0K    /var/log/mysql
2.8M    /var/log/apache2/access.log.3
1.9M    /var/log/apache2/access.log
0       /var/log/apache2/other_vhosts_access.log
4.0K    /var/log/apache2/error.log
4.7M    /var/log/apache2
4.0K    /var/log/redis/redis-server.log
4.0K    /var/log/redis
704K    /var/log/auth.log
4.0K    /var/log/debug
28K     /var/log/kern.log
32K     /var/log/messages
4.0K    /var/log/user.log
20M     /var/log/daemon.log
9.6M    /var/log/syslog
4.0K    /var/log/emonpilcd/emonpilcd.log
4.0K    /var/log/emonpilcd
0       /var/log/btmp
4.0K    /var/log/wtmp
8.0K    /var/log/boot.log
50M     /var/log

The last entries posted to syslog & daemon.log were on 24 Apr at 2:53 when feedwriter was active.

I checked the other nodes (not set up to send emails) …

Three emonTx/RPi3 instances had /var/log full at 50M and had stopped writing to syslog on 14th, 23rd and 26th April. The last entry on one was when feedwriter was active. Two had the last log entry at 6:25 during the log rotate …

Apr 23 06:25:13 emonpi-node-11 liblogging-stdlog: action 'action 12' suspended, next retry is Tue Apr 23 06:25:43 2019 [v8.24.0 try http://www.rsyslog.com/e/2007 ]

I don’t want to try anything radical at present as I’m out of UK & communicating with the system via Dataplicity.

Is it best just to let things run and await the release of the new image that you & others are working on?

Thx

OlavKristiansen · 22 June 2020 12:24

I see something similar, and I am a bit concerned as we are out of the house for the next five weeks:
/var/log: Used: 96.05%
Total : 50 MB
Used : 48.02 MB
Free : 1.98 MB
Write Load : n/a

It has been on this value for the past couple of hours. Will it run full?

df - h:

Filesystem      Size  Used Avail Use% Mounted on
/dev/root       4.0G  2.2G  1.7G  58% /
devtmpfs        484M     0  484M   0% /dev
tmpfs           488M     0  488M   0% /dev/shm
tmpfs           488M  6.6M  482M   2% /run
tmpfs           5.0M  4.0K  5.0M   1% /run/lock
tmpfs           488M     0  488M   0% /sys/fs/cgroup
tmpfs           1.0M  8.0K 1016K   1% /var/lib/php/sessions
tmpfs            30M     0   30M   0% /tmp
tmpfs           1.0M     0  1.0M   0% /var/tmp
/dev/mmcblk0p3   10G  453M  9.1G   5% /var/opt/emoncms
/dev/mmcblk0p1  253M   52M  201M  21% /boot
log2ram          50M   49M  2.0M  97% /var/log
tmpfs            98M     0   98M   0% /run/user/1000

borpin · 22 June 2020 13:22

Have you updated?

OlavKristiansen · 22 June 2020 14:04

Yes, emoncms 10.2.3. I hit the Update all button last week and again today. I have not really observed the status of the /var/log until today, though, so I do not have any growth rate data.

OlavKristiansen · 22 June 2020 14:04

Server Information

Services

emonhub :- Active Running
emoncms_mqtt :- Active Running
feedwriter :- Active Running - sleep 300s 1466 feed points pending write
service-runner :- Active Running
emonPiLCD :- Active Running
redis-server :- Active Running
mosquitto :- Active Running
demandshaper :- Active Running

Emoncms

Version :- low-write 10.2.3
Modules :- Administration | App v2.1.6 | Backup v2.2.3 | EmonHub Config v2.0.5 | Dashboard v2.0.7 | DemandShaper v1.2.5 | Device v2.0.5 | EventProcesses | Feed | Graph v2.0.9 | Input | Postprocess v2.1.4 | CoreProcess | Schedule | Network Setup v1.0.0 | sync | Time | User | Visualisation | WiFi v2.0.2
Git :-
- URL :- GitHub - emoncms/emoncms: Web-app for processing, logging and visualising energy, temperature and other environmental data
- Branch :- * stable
- Describe :- 10.2.3

Server

OS :- Linux 4.19.75-v7+
Host :- emonpi | emonpi | (192.168.1.132)
Date :- 2020-06-22 15:02:34 BST
Uptime :- 15:02:34 up 2:39, 0 users, load average: 0.14, 0.13, 0.09

Memory

RAM :- Used: 18.97%
- Total :- 975.62 MB
- Used :- 185.07 MB
- Free :- 790.55 MB
Swap :- Used: 0.00%
- Total :- 100 MB
- Used :- 0 B
- Free :- 100 MB
  Write Load Period

Disk

/ :- Used: 54.23%
- Total :- 3.92 GB
- Used :- 2.13 GB
- Free :- 1.6 GB
- Write Load :- 353.66 B/s (2 hours 39 mins)
/var/opt/emoncms :- Used: 4.43%
- Total :- 9.98 GB
- Used :- 452.28 MB
- Free :- 9.03 GB
- Write Load :- 382.06 B/s (2 hours 39 mins)
/boot :- Used: 20.55%
- Total :- 252.05 MB
- Used :- 51.79 MB
- Free :- 200.25 MB
- Write Load :- 0.05 B/s (2 hours 39 mins)
/var/log :- Used: 96.12%
- Total :- 50 MB
- Used :- 48.06 MB
- Free :- 1.94 MB
- Write Load :- n/a

HTTP

Server :- Apache/2.4.38 (Raspbian) HTTP/1.1 CGI/1.1 80

MySQL

Version :- 5.5.5-10.3.17-MariaDB-0+deb10u1
Host :- localhost:6379 (127.0.0.1)
Date :- 2020-06-22 15:02:34 (UTC 01:00‌)
Stats :- Uptime: 9576 Threads: 14 Questions: 15670 Slow queries: 0 Opens: 48 Flush tables: 1 Open tables: 42 Queries per second avg: 1.636

Redis

Version :-
- Redis Server :- 5.0.3
- PHP Redis :- 5.0.2
Host :- localhost:6379
Size :- 532 keys (957.66K)
Uptime :- 0 days

MQTT Server

Version :- Mosquitto 1.5.7
Host :- localhost:1883 (127.0.0.1)

PHP

Version :- 7.3.9-1~deb10u1 (Zend Version 3.3.9)
Modules :- apache2handlercalendar Core ctype curl date dom v20031129exif fileinfo filter ftp gd gettext hash iconv json v1.7.0libxml mbstring mosquitto v0.4.0mysqli mysqlnd vmysqlnd 5.0.12-dev - 20150407 - $Id: 7cc7cc96e675f6d72e5cf0f267f48e167c2abb23 $openssl pcre PDO pdo_mysql Phar posix readline redis v5.0.2Reflection session shmop SimpleXML sockets sodium SPL standard sysvmsg sysvsem sysvshm tokenizer wddx xml xmlreader xmlwriter xsl Zend OPcache zlib

Pi

Model :- Raspberry Pi 3 Model B+ Rev 1.3 - 1GB (Sony UK)
Serial num. :- 56D9027C
CPU Temperature :- 55.31°C
GPU Temperature :- 55.8°C
emonpiRelease :- emonSD-17Oct19
File-system :- read-write

Client Information

HTTP

Browser :- Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.106 Safari/537.36
Language :- nb-NO,nb;q=0.9,en-NO;q=0.8,en;q=0.7,no;q=0.6,nn;q=0.5,en-US;q=0.4

Window

Size :- 1483 x 786

Screen

Resolution :- 1920 x 1200

borpin · 22 June 2020 14:07

This should fix it though I would have expected an update to clear out the folder. @TrystanLea

Do the truncate for any large logs.

OlavKristiansen · 22 June 2020 14:29

Hi, thanks, I followed to that thread and it seems to have fixed the log size for now. It was not the emoncms.log file but the /var/log/logrotate/logrotate.log file, so I trunctated that one and then ran the chown command.

48340   /var/log/logrotate
48336   /var/log/logrotate/logrotate.log
288     /var/log/lastlog
132     /var/log/daemon.log
124     /var/log/syslog
124     /var/log/auth.log
60      /var/log/wtmp
44      /var/log/emonpilcd/emonpilcd.log

This was the situation before:

Filesystem      Size  Used Avail Use% Mounted on
/dev/root       4.0G  2.2G  1.7G  58% /
devtmpfs        484M     0  484M   0% /dev
tmpfs           488M     0  488M   0% /dev/shm
tmpfs           488M  6.6M  482M   2% /run
tmpfs           5.0M  4.0K  5.0M   1% /run/lock
tmpfs           488M     0  488M   0% /sys/fs/cgroup
tmpfs           1.0M  4.0K 1020K   1% /var/lib/php/sessions
tmpfs            30M     0   30M   0% /tmp
tmpfs           1.0M     0  1.0M   0% /var/tmp
/dev/mmcblk0p3   10G  453M  9.1G   5% /var/opt/emoncms
/dev/mmcblk0p1  253M   52M  201M  21% /boot
log2ram          50M   49M  2.0M  97% /var/log
tmpfs            98M     0   98M   0% /run/user/1000

And after:

Filesystem      Size  Used Avail Use% Mounted on
/dev/root       4.0G  2.2G  1.7G  58% /
devtmpfs        484M     0  484M   0% /dev
tmpfs           488M     0  488M   0% /dev/shm
tmpfs           488M  6.6M  482M   2% /run
tmpfs           5.0M  4.0K  5.0M   1% /run/lock
tmpfs           488M     0  488M   0% /sys/fs/cgroup
tmpfs           1.0M  8.0K 1016K   1% /var/lib/php/sessions
tmpfs            30M     0   30M   0% /tmp
tmpfs           1.0M     0  1.0M   0% /var/tmp
/dev/mmcblk0p3   10G  453M  9.1G   5% /var/opt/emoncms
/dev/mmcblk0p1  253M   52M  201M  21% /boot
log2ram          50M  904K   50M   2% /var/log
tmpfs            98M     0   98M   0% /run/user/1000

OlavKristiansen · 22 June 2020 14:30

Is this going to be a permanent fix?

borpin · 22 June 2020 14:43

As long as the logrotate command does not give any errors then yes.

SteffanCook · 14 September 2020 11:59

My emonPi running the 17thOct’19 version of the firmware was reporting a full (50mb/50mb) /var/log.

borpin · 14 September 2020 12:02

I suggest you update emonSD-24Jul20 release