Apache2 Dies Frequently. No Errors in Log

borpin · 2020-05-20T15:50:42.287Z

brandock:

Any help would be much appreciated.

Try running a journalctl -f command in an SSH session and see what you get. Quite a good way of doing it when the logs are rotated so frequently. Putty keeps a fair bit of buffer.

It might not be Apache directly causing the problem.

brandock · 2020-05-20T16:48:52.487Z

I see lots of this:

May 20 11:47:58 emonpi rsyslogd[383]: file '7' write error: No space left on device [v8.1901.0 try https://www.rsyslog.com/e/2027 ]

May 20 11:47:58 emonpi rsyslogd[383]: action 'action-2-builtin:omfile' (module 'builtin:omfile') message lost, could not be processed. Check for additional error messages before this one. [v8.1901.0 try https://www.rsyslog.com/e/2027 ]

May 20 11:47:58 emonpi rsyslogd[383]: file '7' write error: No space left on device [v8.1901.0 try https://www.rsyslog.com/e/2027 ]

brandock · 2020-05-20T18:56:58.358Z

Following journalctl -f I just saw my certbot renewal job try to run and fail due to no space on disk. This makes sense because certbot stops apache then restarts it. So if this is the problem, the question is, why is my disk full? The problem must be /var/log?

pi@emonpi:/var/log/apache2 $ df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/root       4.0G  2.8G 1017M  74% /
devtmpfs        459M     0  459M   0% /dev
tmpfs           464M     0  464M   0% /dev/shm
tmpfs           464M   47M  417M  11% /run
tmpfs           5.0M  4.0K  5.0M   1% /run/lock
tmpfs           464M     0  464M   0% /sys/fs/cgroup
tmpfs           1.0M     0  1.0M   0% /var/tmp
tmpfs            30M     0   30M   0% /tmp
tmpfs           1.0M     0  1.0M   0% /var/lib/php/sessions
/dev/mmcblk0p3   10G  1.1G  8.4G  12% /var/opt/emoncms
/dev/mmcblk0p1  253M   53M  200M  21% /boot
log2ram          50M   50M     0 100% /var/log
tmpfs            93M     0   93M   0% /run/user/1000

Thelmike · 2020-05-20T19:35:21.228Z

Correct. Check with

du -sh /var/log/*

and you will see what directory or files consumes the most.

brandock · 2020-05-20T20:24:56.710Z

Looks like logrotate is the big winner (below). I see that there are various posts about problems with logrotate, but I am having trouble following them. Can anyone help point me to the next step here? Based on one of the posts I checked “sudo crontab -e” vs. “crontab -e”; neither of which have a servicerunner line, in case that makes a difference.

pi@emonpi : /var/log/apache2 $ sudo du -shl /var/log/*

0 /var/log/alternatives.log
0 /var/log/apache2
28K /var/log/apt
172K /var/log/auth.log
0 /var/log/bootstrap.log
0 /var/log/btmp
260K /var/log/daemon.log
4.0K /var/log/debug
0 /var/log/dpkg.log
2.8M /var/log/emoncms
8.0K /var/log/emonhub
4.0K /var/log/emonpilcd
24K /var/log/faillog
428K /var/log/kern.log
288K /var/log/lastlog
252K /var/log/letsencrypt
0 /var/log/log2ram.log
46M /var/log/logrotate
480K /var/log/messages
4.0K /var/log/mosquitto
0 /var/log/mysql
0 /var/log/private
0 /var/log/redis
0 /var/log/rotated_logs
12K /var/log/syslog
180K /var/log/ufw.log
4.0K /var/log/user.log
4.0K /var/log/wifiAP.log
88K /var/log/wtmp

borpin · 2020-05-21T08:06:58.458Z

/var/log filling up again Emoncms

If you are then sudo truncate -s 0 /var/log/logrotate/logrotate.log is the quick fix.

There is an open issue @TrystanLea.

to fix, include this in /opt/openenergymonitor/EmonScripts/defaults/etc/logrotate.d/00_defaults although this will block further update to the emonscripts folder by git.

maxsize 500k

/var/log/logrotate/*.log {
        rotate 7
        daily
        compress
        copytruncate
        size 100k
        nocreate
        missingok
        notifempty
        delaycompress
}

[edit] I have now done a PR to include this fix Fix var log filling up with logrotate log files by borpin · Pull Request #85 · openenergymonitor/EmonScripts · GitHub

brandock · 2020-05-22T03:23:05.701Z

So, I did the sudo truncate -s 0 /var/log/logrotate/logrotate.log for a quick fix. That knocked back /var/log/rotate from 45M to 4.0K.

Then I made the change to /opt/openenergymonitor/EmonScripts/defaults/etc/logrotate.d/00_defaults and (for good measure) restarted the machine.

Within about 12 hours var/log/rotate was back to 45M and apache2 went down again. Is there another step that is needed?

Thanks for your help with this!

borpin · 2020-05-22T06:26:16.651Z

brandock:

Within about 12 hours var/log/rotate was back to 45M and apache2 went down again. Is there another step that is needed?

Firstly Apache2 should not be generating that much log - did you check and see what the log messages are? Perhaps there is a problem that needs fixed.

Secondly, did you check the rotate logs to see if logrotate is failing to rotate the logs correctly?

Apache has it’s own rotate directives in /etc/logrotate.d/apache2. If you are getting excessive log messages you may need to modify this.

This is a problem with your setup and not an emoncms / emonSD issue.

brandock · 2020-05-22T21:42:52.385Z

Hi. I’m sorry. I think my previous post wasn’t clear. Let me clarify.
The directory that seems to be full when apache2 goes down is /var/log/logrotate. That’s the one that went from 4k to 46M in a day.

The apache2 log is actually empty, and has been this whole time. I think apache2 crashing is a side-effect of the log directory being full, and I shouldn’t brought it up again. My thought is that when the systemd job for certbot runs, which is twice a day, it stops apache2. If the certbot job dies because it can’t write to the log file (when the log directory gets full), then it doesn’t finish bringing apache2 back up. But neither certbot nor apache2 are measurably contributing to the size of /var/log.

I’ll keep watching and see what I can learn.

borpin · 2020-05-22T21:47:42.867Z

brandock:

The directory that seems to be full when apache2 goes down is /var/log/logrotate.

Which suggests there is an error being generated by logrotate. Have you checked the logs in that folder?

You can test logrotate with

sudo /usr/sbin/logrotate -v /etc/logrotate.conf

My bet is there is an incorrect folder permission somewhere.

brandock · 2020-05-25T14:52:28.069Z

borpin:

sudo /usr/sbin/logrotate -v /etc/logrotate.conf

Here is the output:

pi@emonpi:~$ sudo /usr/sbin/logrotate -v /etc/logrotate.conf
reading config file /etc/logrotate.conf
including /etc/logrotate.d
reading config file 00_defaults
reading config file 00_olddir
olddir is now /var/log.old
reading config file alternatives
reading config file apache2
reading config file apt
reading config file btmp
reading config file certbot
reading config file dpkg
reading config file emoncms
olddir is now /var/log.old/emoncms
reading config file emonhub
olddir is now /var/log.old/emonhub
reading config file log2ram
reading config file mosquitto
reading config file mysql-server
reading config file redis-server
reading config file rsyslog
reading config file ufw
reading config file wtmp
Reading state from file: /var/lib/logrotate/status
Allocating hash table for state file, size 64 entries
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state

Handling 18 logs

rotating pattern: /var/log/logrotate/*.log  102400 bytes (7 rotations)
empty log files are not rotated, log files >= 512000 are rotated earlier, old logs are removed
considering log /var/log/logrotate/logrotate.log
  Now: 2020-05-25 09:49
  Last rotated at 2020-05-24 00:00
  log does not need rotating (log size is below the 'size' threshold)

rotating pattern: /var/log/alternatives.log  monthly (12 rotations)
olddir is /var/log.old, empty log files are not rotated, log files >= 512000 are rotated earlier, old logs are removed
considering log /var/log/alternatives.log
  Now: 2020-05-25 09:49
  Last rotated at 2019-12-01 00:00
  log does not need rotating (log is empty)

rotating pattern: /var/log/apache2/*.log  after 1 days (14 rotations)
olddir is /var/log.old, empty log files are not rotated, log files >= 512000 are rotated earlier, old logs are removed
considering log /var/log/apache2/access.log
  Now: 2020-05-25 09:49
  Last rotated at 2019-11-16 22:00
  log does not need rotating (log is empty)
considering log /var/log/apache2/error.log
  Now: 2020-05-25 09:49
  Last rotated at 2019-11-17 00:00
  log does not need rotating (log is empty)
considering log /var/log/apache2/other_vhosts_access.log
  Now: 2020-05-25 09:49
  Last rotated at 2019-11-16 22:00
  log does not need rotating (log is empty)
not running prerotate script, since no logs will be rotated
not running postrotate script, since no logs were rotated

rotating pattern: /var/log/apt/term.log  monthly (12 rotations)
olddir is /var/log.old, empty log files are not rotated, log files >= 512000 are rotated earlier, old logs are removed
considering log /var/log/apt/term.log
  Now: 2020-05-25 09:49
  Last rotated at 2019-12-01 00:00
  log does not need rotating (log is empty)

rotating pattern: /var/log/apt/history.log  monthly (12 rotations)
olddir is /var/log.old, empty log files are not rotated, log files >= 512000 are rotated earlier, old logs are removed
considering log /var/log/apt/history.log
  Now: 2020-05-25 09:49
  Last rotated at 2019-12-01 00:00
  log does not need rotating (log is empty)

rotating pattern: /var/log/btmp  monthly (1 rotations)
olddir is /var/log.old, empty log files are rotated, log files >= 512000 are rotated earlier, old logs are removed
considering log /var/log/btmp
  Now: 2020-05-25 09:49
  Last rotated at 2020-05-01 00:00
  log does not need rotating (log has been rotated at 2020-5-1 0:0, that is not month ago yet)

rotating pattern: /var/log/letsencrypt/*.log  weekly (12 rotations)
olddir is /var/log.old, empty log files are rotated, log files >= 512000 are rotated earlier, old logs are removed
considering log /var/log/letsencrypt/letsencrypt.log
  Now: 2020-05-25 09:49
  Last rotated at 2020-05-24 00:00
  log does not need rotating (log has been rotated at 2020-5-24 0:0, that is not week ago yet)

rotating pattern: /var/log/dpkg.log  monthly (12 rotations)
olddir is /var/log.old, empty log files are not rotated, log files >= 512000 are rotated earlier, old logs are removed
considering log /var/log/dpkg.log
  Now: 2020-05-25 09:49
  Last rotated at 2019-12-01 00:00
  log does not need rotating (log is empty)

rotating pattern: /var/log/emoncms/*.log  weekly (4 rotations)
olddir is /var/log.old/emoncms, empty log files are rotated, log files >= 3145728 are rotated earlier, old logs are removed
considering log /var/log/emoncms/apache2-error.log
  Now: 2020-05-25 09:49
  Last rotated at 2020-05-24 00:00
  log does not need rotating (log has been rotated at 2020-5-24 0:0, that is not week ago yet)
considering log /var/log/emoncms/emoncms.log
  Now: 2020-05-25 09:49
  Last rotated at 2020-05-24 00:00
  log does not need rotating (log has been rotated at 2020-5-24 0:0, that is not week ago yet)
considering log /var/log/emoncms/emonpiupdate.log
  Now: 2020-05-25 09:49
  Last rotated at 2020-05-24 00:00
  log does not need rotating (log has been rotated at 2020-5-24 0:0, that is not week ago yet)
considering log /var/log/emoncms/emonupdate.log
  Now: 2020-05-25 09:49
  Last rotated at 2020-05-24 00:00
  log does not need rotating (log has been rotated at 2020-5-24 0:0, that is not week ago yet)
considering log /var/log/emoncms/importbackup.log
  Now: 2020-05-25 09:49
  Last rotated at 2020-05-24 00:00
  log does not need rotating (log has been rotated at 2020-5-24 0:0, that is not week ago yet)
considering log /var/log/emoncms/wificheck.log
  Now: 2020-05-25 09:49
  Last rotated at 2020-05-24 00:00
  log does not need rotating (log has been rotated at 2020-5-24 0:0, that is not week ago yet)

rotating pattern: /var/log/emonhub/emonhub.log  weekly (4 rotations)
olddir is /var/log.old/emonhub, empty log files are rotated, log files >= 3145728 are rotated earlier, old logs are removed
considering log /var/log/emonhub/emonhub.log
  Now: 2020-05-25 09:49
  Last rotated at 2020-05-24 00:00
  log does not need rotating (log has been rotated at 2020-5-24 0:0, that is not week ago yet)

rotating pattern: /var/log/log2ram.log
 after 1 days (7 rotations)
olddir is /var/log.old, empty log files are not rotated, log files >= 512000 are rotated earlier, old logs are removed
considering log /var/log/log2ram.log
  Now: 2020-05-25 09:49
  Last rotated at 2020-05-25 00:00
  log does not need rotating (log has been rotated at 2020-5-25 0:0, that is not day ago yet)

rotating pattern: /var/log/mosquitto/mosquitto.log  102400 bytes (7 rotations)
olddir is /var/log.old, empty log files are rotated, log files >= 512000 are rotated earlier, old logs are removed
considering log /var/log/mosquitto/mosquitto.log
  Now: 2020-05-25 09:49
  Last rotated at 2019-11-16 22:00
  log does not need rotating (log size is below the 'size' threshold)

rotating pattern: /var/log/mysql/mysql.log /var/log/mysql/mysql-slow.log /var/log/mysql/mariadb-slow.log /var/log/mysql/error.log  after 1 days (7 rotations)
olddir is /var/log.old, empty log files are rotated, log files >= 512000 are rotated earlier, old logs are removed
considering log /var/log/mysql/mysql.log
  log /var/log/mysql/mysql.log does not exist -- skipping
considering log /var/log/mysql/mysql-slow.log
  log /var/log/mysql/mysql-slow.log does not exist -- skipping
considering log /var/log/mysql/mariadb-slow.log
  log /var/log/mysql/mariadb-slow.log does not exist -- skipping
considering log /var/log/mysql/error.log
  Now: 2020-05-25 09:49
  Last rotated at 2020-05-25 00:00
  log does not need rotating (log has been rotated at 2020-5-25 0:0, that is not day ago yet)
not running postrotate script, since no logs were rotated

rotating pattern: /var/log/redis/redis-server*.log  weekly (12 rotations)
olddir is /var/log.old, empty log files are not rotated, log files >= 512000 are rotated earlier, old logs are removed
considering log /var/log/redis/redis-server.log
  Now: 2020-05-25 09:49
  Last rotated at 2020-05-19 00:00
  log does not need rotating (log has been rotated at 2020-5-19 0:0, that is not week ago yet)

rotating pattern: /var/log/syslog
 after 1 days (7 rotations)
olddir is /var/log.old, empty log files are not rotated, log files >= 512000 are rotated earlier, old logs are removed
considering log /var/log/syslog
  Now: 2020-05-25 09:49
  Last rotated at 2020-05-25 00:00
  log does not need rotating (log has been rotated at 2020-5-25 0:0, that is not day ago yet)

rotating pattern: /var/log/mail.info
/var/log/mail.warn
/var/log/mail.err
/var/log/mail.log
/var/log/daemon.log
/var/log/kern.log
/var/log/auth.log
/var/log/user.log
/var/log/lpr.log
/var/log/cron.log
/var/log/debug
/var/log/messages
 weekly (4 rotations)
olddir is /var/log.old, empty log files are not rotated, log files >= 512000 are rotated earlier, old logs are removed
considering log /var/log/mail.info
  log /var/log/mail.info does not exist -- skipping
considering log /var/log/mail.warn
  log /var/log/mail.warn does not exist -- skipping
considering log /var/log/mail.err
  log /var/log/mail.err does not exist -- skipping
considering log /var/log/mail.log
  log /var/log/mail.log does not exist -- skipping
considering log /var/log/daemon.log
  Now: 2020-05-25 09:49
  Last rotated at 2020-05-24 00:00
  log does not need rotating (log has been rotated at 2020-5-24 0:0, that is not week ago yet)
considering log /var/log/kern.log
  Now: 2020-05-25 09:49
  Last rotated at 2020-05-24 00:00
  log does not need rotating (log has been rotated at 2020-5-24 0:0, that is not week ago yet)
considering log /var/log/auth.log
  Now: 2020-05-25 09:49
  Last rotated at 2020-05-24 00:00
  log does not need rotating (log has been rotated at 2020-5-24 0:0, that is not week ago yet)
considering log /var/log/user.log
  Now: 2020-05-25 09:49
  Last rotated at 2020-05-24 00:00
  log does not need rotating (log has been rotated at 2020-5-24 0:0, that is not week ago yet)
considering log /var/log/lpr.log
  log /var/log/lpr.log does not exist -- skipping
considering log /var/log/cron.log
  log /var/log/cron.log does not exist -- skipping
considering log /var/log/debug
  Now: 2020-05-25 09:49
  Last rotated at 2020-05-24 00:00
  log does not need rotating (log has been rotated at 2020-5-24 0:0, that is not week ago yet)
considering log /var/log/messages
  Now: 2020-05-25 09:49
  Last rotated at 2020-05-24 00:00
  log does not need rotating (log has been rotated at 2020-5-24 0:0, that is not week ago yet)
not running postrotate script, since no logs were rotated

rotating pattern: /var/log/ufw.log
 weekly (4 rotations)
olddir is /var/log.old, empty log files are not rotated, log files >= 512000 are rotated earlier, old logs are removed
considering log /var/log/ufw.log
  Now: 2020-05-25 09:49
  Last rotated at 2020-05-24 00:00
  log does not need rotating (log has been rotated at 2020-5-24 0:0, that is not week ago yet)
not running postrotate script, since no logs were rotated

rotating pattern: /var/log/wtmp  monthly (1 rotations)
olddir is /var/log.old, empty log files are rotated, only log files >= 1048576 bytes are rotated, log files >= 512000 are rotated earlier, old logs are removed
considering log /var/log/wtmp
  Now: 2020-05-25 09:49
  Last rotated at 2019-10-17 09:00
  log does not need rotating ('minsize' directive is used and the log size is smaller than the minsize value)

borpin · 2020-05-25T14:58:50.026Z

That looks fine. It is odd that the file is increasing so quickly. It should only run once an hour.

brandock · 2020-05-25T15:00:04.283Z

I discovered that LetsEncrypt’s Certbot has a randomized systemd timer that it runs on, so they don’t get everyone’s renewal requests at once. That accounts for the random timing of apache2 going down. Every time certbot.timer runs, apache2 goes down a few minutes later and stays dead until I catch it and restart it.

So I ran sudo systemctl stop certbot.timer and now apache2 has stayed up for over 24 hours.

In the meantime, my /var/log is full again, and I am really having a hard time figuring out why.

pi@emonpi:~$ df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/root       4.0G  2.8G  1.1G  73% /
devtmpfs        459M     0  459M   0% /dev
tmpfs           464M     0  464M   0% /dev/shm
tmpfs           464M   47M  417M  11% /run
tmpfs           5.0M  4.0K  5.0M   1% /run/lock
tmpfs           464M     0  464M   0% /sys/fs/cgroup
tmpfs           1.0M     0  1.0M   0% /var/lib/php/sessions
tmpfs            30M  4.0K   30M   1% /tmp
tmpfs           1.0M     0  1.0M   0% /var/tmp
/dev/mmcblk0p3   10G  1.1G  8.4G  12% /var/opt/emoncms
/dev/mmcblk0p1  253M   53M  200M  21% /boot
log2ram          50M   50M     0 100% /var/log
tmpfs            93M     0   93M   0% /run/user/1000

pi@emonpi:~$ ls -l /var/log
total 1764
-rw-r--r-- 1 root      root         0 Dec  1 00:00 alternatives.log
drwxr-x--- 2 root      adm        100 Nov 17  2019 apache2
drwxr-xr-x 2 root      root       100 Dec  1 00:00 apt
-rw-r----- 1 root      adm      32768 May 25 09:56 auth.log
-rw-r--r-- 1 root      root         0 Sep 25  2019 bootstrap.log
-rw-rw---- 1 root      utmp         0 May 23 21:30 btmp
-rw-r----- 1 root      adm      28672 May 25 09:46 daemon.log
-rw-r----- 1 root      adm          0 May 24 00:00 debug
-rw-r--r-- 1 root      root         0 Dec  1 00:00 dpkg.log
drwxr-xr-x 2 pi        root       160 May 24 00:17 emoncms
drwxrwxr-x 2 root      emonhub     60 Oct 17  2019 emonhub
drwxr-xr-x 2 pi        root        60 Oct 17  2019 emonpilcd
-rw-r--r-- 1 root      root     24000 Oct 17  2019 faillog
-rw-r----- 1 root      adm     442368 May 25 09:57 kern.log
-rw-rw-r-- 1 root      utmp    292292 May 25 09:46 lastlog
drwx------ 2 root      root        60 May 24 00:17 letsencrypt
-rw-r--r-- 1 root      root         0 May 25 00:00 log2ram.log
drwxr-xr-x 2 root      adm        200 May 25 09:17 logrotate
-rw-r----- 1 root      adm     438272 May 25 09:57 messages
drwxr-xr-x 2 mosquitto root        60 Oct 17  2019 mosquitto
drwxr-s--- 2 mysql     adm         60 May 25 00:17 mysql
drwx------ 2 root      root        40 Sep 25  2019 private
drwxr-s--- 2 redis     adm         60 May 21 20:04 redis
lrwxrwxrwx 1 root      root        12 Oct 17  2019 rotated_logs -> /var/log.old
-rw-r----- 1 root      adm       4096 May 25 09:57 syslog
-rw-r----- 1 root      adm     442368 May 25 09:57 ufw.log
-rw-r----- 1 root      adm          0 May 24 00:00 user.log
-rw-r--r-- 1 root      root       882 May 21 20:04 wifiAP.log
-rw-rw-r-- 1 root      utmp     94080 May 25 09:46 wtmp

pi@emonpi:~$ sudo du -sh /var/log/*
0	/var/log/alternatives.log
0	/var/log/apache2
28K	/var/log/apt
32K	/var/log/auth.log
0	/var/log/bootstrap.log
0	/var/log/btmp
28K	/var/log/daemon.log
0	/var/log/debug
0	/var/log/dpkg.log
2.0M	/var/log/emoncms
0	/var/log/emonhub
4.0K	/var/log/emonpilcd
24K	/var/log/faillog
432K	/var/log/kern.log
288K	/var/log/lastlog
0	/var/log/letsencrypt
0	/var/log/log2ram.log
176K	/var/log/logrotate
428K	/var/log/messages
4.0K	/var/log/mosquitto
0	/var/log/mysql
0	/var/log/private
0	/var/log/redis
0	/var/log/rotated_logs
4.0K	/var/log/syslog
432K	/var/log/ufw.log
0	/var/log/user.log
4.0K	/var/log/wifiAP.log
92K	/var/log/wtmp

borpin · 2020-05-25T15:09:55.041Z

Try

sudo du -ah /var/log | sort -n -r | head -n 20

to find the largest files and sort them.

[edit] To just get the size of files use

sudo find /var/log -type f -exec du -ah {} + | sort -n -r | head -n 20

brandock · 2020-05-25T18:19:14.170Z

borpin:

To just get the size of files use
sudo find /var/log -type f -exec du -ah {} + | sort -n -r | head -n 20

pi@emonpi:~$ sudo find /var/log -type f -exec du -ah {} + | sort -n -r | head -n 20
432K	/var/log/ufw.log
432K	/var/log/kern.log
428K	/var/log/messages
288K	/var/log/lastlog
116K	/var/log/logrotate/logrotate.log.1
92K	/var/log/wtmp
32K	/var/log/auth.log
28K	/var/log/daemon.log
28K	/var/log/apt/eipp.log.xz
24K	/var/log/faillog
16K	/var/log/logrotate/logrotate.log
8.0K	/var/log/logrotate/logrotate.log.6.gz
8.0K	/var/log/logrotate/logrotate.log.5.gz
8.0K	/var/log/logrotate/logrotate.log.4.gz
8.0K	/var/log/logrotate/logrotate.log.3.gz
8.0K	/var/log/logrotate/logrotate.log.2.gz
4.0K	/var/log/wifiAP.log
4.0K	/var/log/syslog
4.0K	/var/log/mosquitto/mosquitto.log
4.0K	/var/log/logrotate/logrotate.status

Thanks.

Ufw was logging the following when it ran out of space. The output of tail /var/log/kern.log is identical to the output of tail /var/log/ufw.log. The content of /var/log/messages is also full of the UFW BLOCK messages, but they cut off earlier.

The blocked requests you see are from a Raspberry Pi that runs Dakboard and displays our solar production and use. I don’t know what they are, yet, because everything seems to be working on that machine.

May 25 00:07:04 emonpi kernel: [273756.672682] [UFW BLOCK] IN=eth0 OUT= MAC=b8:27:eb:3d:21:50:b8:27:eb:e2:7d:67:08:00:45:00:00:28:00:00:40:00:40:06:b7:07 SRC=192.168.1.80 DST=192.168.1.40 LEN=40 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=TCP SPT=1880 DPT=56340 WINDOW=0 RES=0x00 RST URGP=0 
May 25 00:07:19 emonpi kernel: [273772.031176] [UFW BLOCK] IN=eth0 OUT= MAC=b8:27:eb:3d:21:50:b8:27:eb:e2:7d:67:08:00:45:00:00:28:00:00:40:00:40:06:b7:07 SRC=192.168.1.80 DST=192.168.1.40 LEN=40 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=TCP SPT=1880 DPT=54144 WINDOW=0 RES=0x00 RST URGP=0 
May 25 00:07:36 emonpi kernel: [273789.304087] [UFW BLOCK] IN=eth0 OUT= MAC=b8:27:eb:3d:21:50:b8:27:eb:e2:7d:67:08:00:45:00:00:28:00:00:40:00:40:06:b7:07 SRC=192.168.1.80 DST=192.168.1.40 LEN=40 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=TCP SPT=1880 DPT=57176 WINDOW=0 RES=0x00 RST URGP=0 
May 25 00:08:09 emonpi kernel: [273821.698000] [UFW BLOCK] IN=eth0 OUT= MAC=b8:27:eb:3d:21:50:b8:27:eb:e2:7d:67:08:00:45:00:00:28:00:00:40:00:40:06:b7:07 SRC=192.168.1.80 DST=192.168.1.40 LEN=40 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=TCP SPT=1880 DPT=49278 WINDOW=0 RES=0x00 RST URGP=0 
May 25 00:08:22 emonpi kernel: [273835.388849] [UFW BLOCK] IN=eth0 OUT= MAC=b8:27:eb:3d:21:50:b8:27:eb:e2:7d:67:08:00:45:00:00:28:00:00:40:00:40:06:b7:07 SRC=192.168.1.80 DST=192.168.1.40 LEN=40 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=TCP SPT=1880 DPT=49278 WINDOW=0 RES=0x00 RST URGP=0 
May 25 00:08:38 emonpi kernel: [273851.276950] [UFW BLOCK] IN=eth0 OUT= MAC=b8:27:eb:3d:21:50:b8:27:eb:e2:7d:67:08:00:45:00:00:28:00:00:40:00:40:06:b7:07 SRC=192.168.1.80 DST=192.168.1.40 LEN=40 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=TCP SPT=1880 DPT=56748 WINDOW=0 RES=0x00 RST URGP=0 
May 25 00:09:04 emonpi kernel: [273876.988979] [UFW BLOCK] IN=eth0 OUT= MAC=b8:27:eb:3d:21:50:b8:27:eb:e2:7d:67:08:00:45:00:00:28:00:00:40:00:40:06:b7:07 SRC=192.168.1.80 DST=192.168.1.40 LEN=40 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=TCP SPT=1880 DPT=49278 WINDOW=0 RES=0x00 RST URGP=0 
May 25 00:09:38 emonpi kernel: [273911.502642] [UFW BLOCK] IN=eth0 OUT= MAC=b8:27:eb:3d:21:50:b8:27:eb:e2:7d:67:08:00:45:00:00:28:00:00:40:00:40:06:b7:07 SRC=192.168.1.80 DST=192.168.1.40 LEN=40 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=TCP SPT=1880 DPT=57676 WINDOW=0 RES=0x00 RST URGP=0 
May 25 00:09:39 emonpi kernel: [273911.731468] [UFW BLOCK] IN=eth0 OUT= MAC=b8:27:eb:3d:21:50:b8:27:eb:e2:7d:67:08:00:45:00:00:28:00:00:40:00:40:06:b7:07 SRC=192.168.1.80 DST=192.168.1.40 LEN=40 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=TCP SPT=1880 DPT=57676 WINDOW=0 RES=0x00 RST URGP=0 
May 25 00:10:09 emonpi kernel: [273942.147109] [UFW BLOCK] IN=eth0 OUT= MAC=b8:27:eb:3d:21:50:80:ea:96:e4:f5:53:08:00:45:00:00:2

borpin · 2020-05-25T18:28:59.911Z

Well your first move is to edit the file /etc/rsyslog.d/20-ufw.conf and remove the comment character before the &stop (TBH I’m not sure what the & does - just follow instructions).

This will mean the message is only logged once.

Next what are the ufw rules? I think

sudo ufw status verbose

will tell you.

Oh and clear out any rotated logs to increase space.

brandock · 2020-05-25T18:44:48.932Z

Made the change to 20-ufw.conf. Cleared out rotated logs (I think – more on that later).

pi@emonpi:~$ sudo ufw status verbose
Status: active
Logging: on (low)
Default: deny (incoming), allow (outgoing), disabled (routed)
New profiles: skip

To                         Action      From
--                         ------      ----
80/tcp                     ALLOW IN    Anywhere                  
443/tcp                    ALLOW IN    Anywhere                  
22/tcp                     ALLOW IN    Anywhere                  
1883/tcp                   ALLOW IN    Anywhere                  
80/tcp (v6)                ALLOW IN    Anywhere (v6)             
443/tcp (v6)               ALLOW IN    Anywhere (v6)             
22/tcp (v6)                ALLOW IN    Anywhere (v6)             
1883/tcp (v6)              ALLOW IN    Anywhere (v6)

borpin · 2020-05-25T19:26:52.672Z

brandock:

The blocked requests you see are from a Raspberry Pi that runs Dakboard and displays our solar production and use.

Has it just started to be an issue? What updates have been done?

Probably the next step is to install ‘tshark’ on the requesting Pi and see what traffic it is sending to that Pi.

However, do you really need UFW active?

[edit]
Does the remote machine have node-red on it?

TrystanLea · 2020-05-26T13:19:41.086Z

borpin:

There is an open issue @TrystanLea.

I’ve now merged this in and merged to stable, so a standard emonPi/base update will bring this in.

borpin · 2020-05-26T13:29:53.933Z

TrystanLea:

so a standard emonPi/base update will bring this in

Would it be possible to do a release / module info for this repo as well? When helping users it would be good to know what version that is sitting at as well as everything else.