Try running a journalctl -f
command in an SSH session and see what you get. Quite a good way of doing it when the logs are rotated so frequently. Putty keeps a fair bit of buffer.
It might not be Apache directly causing the problem.
Try running a journalctl -f
command in an SSH session and see what you get. Quite a good way of doing it when the logs are rotated so frequently. Putty keeps a fair bit of buffer.
It might not be Apache directly causing the problem.
I see lots of this:
May 20 11:47:58 emonpi rsyslogd[383]: file '7' write error: No space left on device [v8.1901.0 try https://www.rsyslog.com/e/2027 ]
May 20 11:47:58 emonpi rsyslogd[383]: action 'action-2-builtin:omfile' (module 'builtin:omfile') message lost, could not be processed. Check for additional error messages before this one. [v8.1901.0 try https://www.rsyslog.com/e/2027 ]
May 20 11:47:58 emonpi rsyslogd[383]: file '7' write error: No space left on device [v8.1901.0 try https://www.rsyslog.com/e/2027 ]
Following journalctl -f I just saw my certbot renewal job try to run and fail due to no space on disk. This makes sense because certbot stops apache then restarts it. So if this is the problem, the question is, why is my disk full? The problem must be /var/log?
pi@emonpi:/var/log/apache2 $ df -h
Filesystem Size Used Avail Use% Mounted on
/dev/root 4.0G 2.8G 1017M 74% /
devtmpfs 459M 0 459M 0% /dev
tmpfs 464M 0 464M 0% /dev/shm
tmpfs 464M 47M 417M 11% /run
tmpfs 5.0M 4.0K 5.0M 1% /run/lock
tmpfs 464M 0 464M 0% /sys/fs/cgroup
tmpfs 1.0M 0 1.0M 0% /var/tmp
tmpfs 30M 0 30M 0% /tmp
tmpfs 1.0M 0 1.0M 0% /var/lib/php/sessions
/dev/mmcblk0p3 10G 1.1G 8.4G 12% /var/opt/emoncms
/dev/mmcblk0p1 253M 53M 200M 21% /boot
log2ram 50M 50M 0 100% /var/log
tmpfs 93M 0 93M 0% /run/user/1000
Correct. Check with
du -sh /var/log/*
and you will see what directory or files consumes the most.
Looks like logrotate is the big winner (below). I see that there are various posts about problems with logrotate, but I am having trouble following them. Can anyone help point me to the next step here? Based on one of the posts I checked “sudo crontab -e” vs. “crontab -e”; neither of which have a servicerunner line, in case that makes a difference.
pi@emonpi : /var/log/apache2 $ sudo du -shl /var/log/*
0 /var/log/alternatives.log
0 /var/log/apache2
28K /var/log/apt
172K /var/log/auth.log
0 /var/log/bootstrap.log
0 /var/log/btmp
260K /var/log/daemon.log
4.0K /var/log/debug
0 /var/log/dpkg.log
2.8M /var/log/emoncms
8.0K /var/log/emonhub
4.0K /var/log/emonpilcd
24K /var/log/faillog
428K /var/log/kern.log
288K /var/log/lastlog
252K /var/log/letsencrypt
0 /var/log/log2ram.log
46M /var/log/logrotate
480K /var/log/messages
4.0K /var/log/mosquitto
0 /var/log/mysql
0 /var/log/private
0 /var/log/redis
0 /var/log/rotated_logs
12K /var/log/syslog
180K /var/log/ufw.log
4.0K /var/log/user.log
4.0K /var/log/wifiAP.log
88K /var/log/wtmp
There is an open issue @TrystanLea.
to fix, include this in /opt/openenergymonitor/EmonScripts/defaults/etc/logrotate.d/00_defaults
although this will block further update to the emonscripts folder by git.
maxsize 500k
/var/log/logrotate/*.log {
rotate 7
daily
compress
copytruncate
size 100k
nocreate
missingok
notifempty
delaycompress
}
[edit] I have now done a PR to include this fix Fix var log filling up with logrotate log files by borpin · Pull Request #85 · openenergymonitor/EmonScripts · GitHub
So, I did the sudo truncate -s 0 /var/log/logrotate/logrotate.log for a quick fix. That knocked back /var/log/rotate from 45M to 4.0K.
Then I made the change to /opt/openenergymonitor/EmonScripts/defaults/etc/logrotate.d/00_defaults and (for good measure) restarted the machine.
Within about 12 hours var/log/rotate was back to 45M and apache2 went down again. Is there another step that is needed?
Thanks for your help with this!
Firstly Apache2 should not be generating that much log - did you check and see what the log messages are? Perhaps there is a problem that needs fixed.
Secondly, did you check the rotate logs to see if logrotate is failing to rotate the logs correctly?
Apache has it’s own rotate directives in /etc/logrotate.d/apache2
. If you are getting excessive log messages you may need to modify this.
This is a problem with your setup and not an emoncms / emonSD issue.
Hi. I’m sorry. I think my previous post wasn’t clear. Let me clarify.
The directory that seems to be full when apache2 goes down is /var/log/logrotate. That’s the one that went from 4k to 46M in a day.
The apache2 log is actually empty, and has been this whole time. I think apache2 crashing is a side-effect of the log directory being full, and I shouldn’t brought it up again. My thought is that when the systemd job for certbot runs, which is twice a day, it stops apache2. If the certbot job dies because it can’t write to the log file (when the log directory gets full), then it doesn’t finish bringing apache2 back up. But neither certbot nor apache2 are measurably contributing to the size of /var/log.
I’ll keep watching and see what I can learn.
Which suggests there is an error being generated by logrotate. Have you checked the logs in that folder?
You can test logrotate with
sudo /usr/sbin/logrotate -v /etc/logrotate.conf
My bet is there is an incorrect folder permission somewhere.
Here is the output:
pi@emonpi:~$ sudo /usr/sbin/logrotate -v /etc/logrotate.conf
reading config file /etc/logrotate.conf
including /etc/logrotate.d
reading config file 00_defaults
reading config file 00_olddir
olddir is now /var/log.old
reading config file alternatives
reading config file apache2
reading config file apt
reading config file btmp
reading config file certbot
reading config file dpkg
reading config file emoncms
olddir is now /var/log.old/emoncms
reading config file emonhub
olddir is now /var/log.old/emonhub
reading config file log2ram
reading config file mosquitto
reading config file mysql-server
reading config file redis-server
reading config file rsyslog
reading config file ufw
reading config file wtmp
Reading state from file: /var/lib/logrotate/status
Allocating hash table for state file, size 64 entries
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Handling 18 logs
rotating pattern: /var/log/logrotate/*.log 102400 bytes (7 rotations)
empty log files are not rotated, log files >= 512000 are rotated earlier, old logs are removed
considering log /var/log/logrotate/logrotate.log
Now: 2020-05-25 09:49
Last rotated at 2020-05-24 00:00
log does not need rotating (log size is below the 'size' threshold)
rotating pattern: /var/log/alternatives.log monthly (12 rotations)
olddir is /var/log.old, empty log files are not rotated, log files >= 512000 are rotated earlier, old logs are removed
considering log /var/log/alternatives.log
Now: 2020-05-25 09:49
Last rotated at 2019-12-01 00:00
log does not need rotating (log is empty)
rotating pattern: /var/log/apache2/*.log after 1 days (14 rotations)
olddir is /var/log.old, empty log files are not rotated, log files >= 512000 are rotated earlier, old logs are removed
considering log /var/log/apache2/access.log
Now: 2020-05-25 09:49
Last rotated at 2019-11-16 22:00
log does not need rotating (log is empty)
considering log /var/log/apache2/error.log
Now: 2020-05-25 09:49
Last rotated at 2019-11-17 00:00
log does not need rotating (log is empty)
considering log /var/log/apache2/other_vhosts_access.log
Now: 2020-05-25 09:49
Last rotated at 2019-11-16 22:00
log does not need rotating (log is empty)
not running prerotate script, since no logs will be rotated
not running postrotate script, since no logs were rotated
rotating pattern: /var/log/apt/term.log monthly (12 rotations)
olddir is /var/log.old, empty log files are not rotated, log files >= 512000 are rotated earlier, old logs are removed
considering log /var/log/apt/term.log
Now: 2020-05-25 09:49
Last rotated at 2019-12-01 00:00
log does not need rotating (log is empty)
rotating pattern: /var/log/apt/history.log monthly (12 rotations)
olddir is /var/log.old, empty log files are not rotated, log files >= 512000 are rotated earlier, old logs are removed
considering log /var/log/apt/history.log
Now: 2020-05-25 09:49
Last rotated at 2019-12-01 00:00
log does not need rotating (log is empty)
rotating pattern: /var/log/btmp monthly (1 rotations)
olddir is /var/log.old, empty log files are rotated, log files >= 512000 are rotated earlier, old logs are removed
considering log /var/log/btmp
Now: 2020-05-25 09:49
Last rotated at 2020-05-01 00:00
log does not need rotating (log has been rotated at 2020-5-1 0:0, that is not month ago yet)
rotating pattern: /var/log/letsencrypt/*.log weekly (12 rotations)
olddir is /var/log.old, empty log files are rotated, log files >= 512000 are rotated earlier, old logs are removed
considering log /var/log/letsencrypt/letsencrypt.log
Now: 2020-05-25 09:49
Last rotated at 2020-05-24 00:00
log does not need rotating (log has been rotated at 2020-5-24 0:0, that is not week ago yet)
rotating pattern: /var/log/dpkg.log monthly (12 rotations)
olddir is /var/log.old, empty log files are not rotated, log files >= 512000 are rotated earlier, old logs are removed
considering log /var/log/dpkg.log
Now: 2020-05-25 09:49
Last rotated at 2019-12-01 00:00
log does not need rotating (log is empty)
rotating pattern: /var/log/emoncms/*.log weekly (4 rotations)
olddir is /var/log.old/emoncms, empty log files are rotated, log files >= 3145728 are rotated earlier, old logs are removed
considering log /var/log/emoncms/apache2-error.log
Now: 2020-05-25 09:49
Last rotated at 2020-05-24 00:00
log does not need rotating (log has been rotated at 2020-5-24 0:0, that is not week ago yet)
considering log /var/log/emoncms/emoncms.log
Now: 2020-05-25 09:49
Last rotated at 2020-05-24 00:00
log does not need rotating (log has been rotated at 2020-5-24 0:0, that is not week ago yet)
considering log /var/log/emoncms/emonpiupdate.log
Now: 2020-05-25 09:49
Last rotated at 2020-05-24 00:00
log does not need rotating (log has been rotated at 2020-5-24 0:0, that is not week ago yet)
considering log /var/log/emoncms/emonupdate.log
Now: 2020-05-25 09:49
Last rotated at 2020-05-24 00:00
log does not need rotating (log has been rotated at 2020-5-24 0:0, that is not week ago yet)
considering log /var/log/emoncms/importbackup.log
Now: 2020-05-25 09:49
Last rotated at 2020-05-24 00:00
log does not need rotating (log has been rotated at 2020-5-24 0:0, that is not week ago yet)
considering log /var/log/emoncms/wificheck.log
Now: 2020-05-25 09:49
Last rotated at 2020-05-24 00:00
log does not need rotating (log has been rotated at 2020-5-24 0:0, that is not week ago yet)
rotating pattern: /var/log/emonhub/emonhub.log weekly (4 rotations)
olddir is /var/log.old/emonhub, empty log files are rotated, log files >= 3145728 are rotated earlier, old logs are removed
considering log /var/log/emonhub/emonhub.log
Now: 2020-05-25 09:49
Last rotated at 2020-05-24 00:00
log does not need rotating (log has been rotated at 2020-5-24 0:0, that is not week ago yet)
rotating pattern: /var/log/log2ram.log
after 1 days (7 rotations)
olddir is /var/log.old, empty log files are not rotated, log files >= 512000 are rotated earlier, old logs are removed
considering log /var/log/log2ram.log
Now: 2020-05-25 09:49
Last rotated at 2020-05-25 00:00
log does not need rotating (log has been rotated at 2020-5-25 0:0, that is not day ago yet)
rotating pattern: /var/log/mosquitto/mosquitto.log 102400 bytes (7 rotations)
olddir is /var/log.old, empty log files are rotated, log files >= 512000 are rotated earlier, old logs are removed
considering log /var/log/mosquitto/mosquitto.log
Now: 2020-05-25 09:49
Last rotated at 2019-11-16 22:00
log does not need rotating (log size is below the 'size' threshold)
rotating pattern: /var/log/mysql/mysql.log /var/log/mysql/mysql-slow.log /var/log/mysql/mariadb-slow.log /var/log/mysql/error.log after 1 days (7 rotations)
olddir is /var/log.old, empty log files are rotated, log files >= 512000 are rotated earlier, old logs are removed
considering log /var/log/mysql/mysql.log
log /var/log/mysql/mysql.log does not exist -- skipping
considering log /var/log/mysql/mysql-slow.log
log /var/log/mysql/mysql-slow.log does not exist -- skipping
considering log /var/log/mysql/mariadb-slow.log
log /var/log/mysql/mariadb-slow.log does not exist -- skipping
considering log /var/log/mysql/error.log
Now: 2020-05-25 09:49
Last rotated at 2020-05-25 00:00
log does not need rotating (log has been rotated at 2020-5-25 0:0, that is not day ago yet)
not running postrotate script, since no logs were rotated
rotating pattern: /var/log/redis/redis-server*.log weekly (12 rotations)
olddir is /var/log.old, empty log files are not rotated, log files >= 512000 are rotated earlier, old logs are removed
considering log /var/log/redis/redis-server.log
Now: 2020-05-25 09:49
Last rotated at 2020-05-19 00:00
log does not need rotating (log has been rotated at 2020-5-19 0:0, that is not week ago yet)
rotating pattern: /var/log/syslog
after 1 days (7 rotations)
olddir is /var/log.old, empty log files are not rotated, log files >= 512000 are rotated earlier, old logs are removed
considering log /var/log/syslog
Now: 2020-05-25 09:49
Last rotated at 2020-05-25 00:00
log does not need rotating (log has been rotated at 2020-5-25 0:0, that is not day ago yet)
rotating pattern: /var/log/mail.info
/var/log/mail.warn
/var/log/mail.err
/var/log/mail.log
/var/log/daemon.log
/var/log/kern.log
/var/log/auth.log
/var/log/user.log
/var/log/lpr.log
/var/log/cron.log
/var/log/debug
/var/log/messages
weekly (4 rotations)
olddir is /var/log.old, empty log files are not rotated, log files >= 512000 are rotated earlier, old logs are removed
considering log /var/log/mail.info
log /var/log/mail.info does not exist -- skipping
considering log /var/log/mail.warn
log /var/log/mail.warn does not exist -- skipping
considering log /var/log/mail.err
log /var/log/mail.err does not exist -- skipping
considering log /var/log/mail.log
log /var/log/mail.log does not exist -- skipping
considering log /var/log/daemon.log
Now: 2020-05-25 09:49
Last rotated at 2020-05-24 00:00
log does not need rotating (log has been rotated at 2020-5-24 0:0, that is not week ago yet)
considering log /var/log/kern.log
Now: 2020-05-25 09:49
Last rotated at 2020-05-24 00:00
log does not need rotating (log has been rotated at 2020-5-24 0:0, that is not week ago yet)
considering log /var/log/auth.log
Now: 2020-05-25 09:49
Last rotated at 2020-05-24 00:00
log does not need rotating (log has been rotated at 2020-5-24 0:0, that is not week ago yet)
considering log /var/log/user.log
Now: 2020-05-25 09:49
Last rotated at 2020-05-24 00:00
log does not need rotating (log has been rotated at 2020-5-24 0:0, that is not week ago yet)
considering log /var/log/lpr.log
log /var/log/lpr.log does not exist -- skipping
considering log /var/log/cron.log
log /var/log/cron.log does not exist -- skipping
considering log /var/log/debug
Now: 2020-05-25 09:49
Last rotated at 2020-05-24 00:00
log does not need rotating (log has been rotated at 2020-5-24 0:0, that is not week ago yet)
considering log /var/log/messages
Now: 2020-05-25 09:49
Last rotated at 2020-05-24 00:00
log does not need rotating (log has been rotated at 2020-5-24 0:0, that is not week ago yet)
not running postrotate script, since no logs were rotated
rotating pattern: /var/log/ufw.log
weekly (4 rotations)
olddir is /var/log.old, empty log files are not rotated, log files >= 512000 are rotated earlier, old logs are removed
considering log /var/log/ufw.log
Now: 2020-05-25 09:49
Last rotated at 2020-05-24 00:00
log does not need rotating (log has been rotated at 2020-5-24 0:0, that is not week ago yet)
not running postrotate script, since no logs were rotated
rotating pattern: /var/log/wtmp monthly (1 rotations)
olddir is /var/log.old, empty log files are rotated, only log files >= 1048576 bytes are rotated, log files >= 512000 are rotated earlier, old logs are removed
considering log /var/log/wtmp
Now: 2020-05-25 09:49
Last rotated at 2019-10-17 09:00
log does not need rotating ('minsize' directive is used and the log size is smaller than the minsize value)
That looks fine. It is odd that the file is increasing so quickly. It should only run once an hour.
I discovered that LetsEncrypt’s Certbot has a randomized systemd timer that it runs on, so they don’t get everyone’s renewal requests at once. That accounts for the random timing of apache2 going down. Every time certbot.timer runs, apache2 goes down a few minutes later and stays dead until I catch it and restart it.
So I ran sudo systemctl stop certbot.timer and now apache2 has stayed up for over 24 hours.
In the meantime, my /var/log is full again, and I am really having a hard time figuring out why.
pi@emonpi:~$ df -h
Filesystem Size Used Avail Use% Mounted on
/dev/root 4.0G 2.8G 1.1G 73% /
devtmpfs 459M 0 459M 0% /dev
tmpfs 464M 0 464M 0% /dev/shm
tmpfs 464M 47M 417M 11% /run
tmpfs 5.0M 4.0K 5.0M 1% /run/lock
tmpfs 464M 0 464M 0% /sys/fs/cgroup
tmpfs 1.0M 0 1.0M 0% /var/lib/php/sessions
tmpfs 30M 4.0K 30M 1% /tmp
tmpfs 1.0M 0 1.0M 0% /var/tmp
/dev/mmcblk0p3 10G 1.1G 8.4G 12% /var/opt/emoncms
/dev/mmcblk0p1 253M 53M 200M 21% /boot
log2ram 50M 50M 0 100% /var/log
tmpfs 93M 0 93M 0% /run/user/1000
pi@emonpi:~$ ls -l /var/log
total 1764
-rw-r--r-- 1 root root 0 Dec 1 00:00 alternatives.log
drwxr-x--- 2 root adm 100 Nov 17 2019 apache2
drwxr-xr-x 2 root root 100 Dec 1 00:00 apt
-rw-r----- 1 root adm 32768 May 25 09:56 auth.log
-rw-r--r-- 1 root root 0 Sep 25 2019 bootstrap.log
-rw-rw---- 1 root utmp 0 May 23 21:30 btmp
-rw-r----- 1 root adm 28672 May 25 09:46 daemon.log
-rw-r----- 1 root adm 0 May 24 00:00 debug
-rw-r--r-- 1 root root 0 Dec 1 00:00 dpkg.log
drwxr-xr-x 2 pi root 160 May 24 00:17 emoncms
drwxrwxr-x 2 root emonhub 60 Oct 17 2019 emonhub
drwxr-xr-x 2 pi root 60 Oct 17 2019 emonpilcd
-rw-r--r-- 1 root root 24000 Oct 17 2019 faillog
-rw-r----- 1 root adm 442368 May 25 09:57 kern.log
-rw-rw-r-- 1 root utmp 292292 May 25 09:46 lastlog
drwx------ 2 root root 60 May 24 00:17 letsencrypt
-rw-r--r-- 1 root root 0 May 25 00:00 log2ram.log
drwxr-xr-x 2 root adm 200 May 25 09:17 logrotate
-rw-r----- 1 root adm 438272 May 25 09:57 messages
drwxr-xr-x 2 mosquitto root 60 Oct 17 2019 mosquitto
drwxr-s--- 2 mysql adm 60 May 25 00:17 mysql
drwx------ 2 root root 40 Sep 25 2019 private
drwxr-s--- 2 redis adm 60 May 21 20:04 redis
lrwxrwxrwx 1 root root 12 Oct 17 2019 rotated_logs -> /var/log.old
-rw-r----- 1 root adm 4096 May 25 09:57 syslog
-rw-r----- 1 root adm 442368 May 25 09:57 ufw.log
-rw-r----- 1 root adm 0 May 24 00:00 user.log
-rw-r--r-- 1 root root 882 May 21 20:04 wifiAP.log
-rw-rw-r-- 1 root utmp 94080 May 25 09:46 wtmp
pi@emonpi:~$ sudo du -sh /var/log/*
0 /var/log/alternatives.log
0 /var/log/apache2
28K /var/log/apt
32K /var/log/auth.log
0 /var/log/bootstrap.log
0 /var/log/btmp
28K /var/log/daemon.log
0 /var/log/debug
0 /var/log/dpkg.log
2.0M /var/log/emoncms
0 /var/log/emonhub
4.0K /var/log/emonpilcd
24K /var/log/faillog
432K /var/log/kern.log
288K /var/log/lastlog
0 /var/log/letsencrypt
0 /var/log/log2ram.log
176K /var/log/logrotate
428K /var/log/messages
4.0K /var/log/mosquitto
0 /var/log/mysql
0 /var/log/private
0 /var/log/redis
0 /var/log/rotated_logs
4.0K /var/log/syslog
432K /var/log/ufw.log
0 /var/log/user.log
4.0K /var/log/wifiAP.log
92K /var/log/wtmp
Try
sudo du -ah /var/log | sort -n -r | head -n 20
to find the largest files and sort them.
[edit] To just get the size of files use
sudo find /var/log -type f -exec du -ah {} + | sort -n -r | head -n 20
pi@emonpi:~$ sudo find /var/log -type f -exec du -ah {} + | sort -n -r | head -n 20
432K /var/log/ufw.log
432K /var/log/kern.log
428K /var/log/messages
288K /var/log/lastlog
116K /var/log/logrotate/logrotate.log.1
92K /var/log/wtmp
32K /var/log/auth.log
28K /var/log/daemon.log
28K /var/log/apt/eipp.log.xz
24K /var/log/faillog
16K /var/log/logrotate/logrotate.log
8.0K /var/log/logrotate/logrotate.log.6.gz
8.0K /var/log/logrotate/logrotate.log.5.gz
8.0K /var/log/logrotate/logrotate.log.4.gz
8.0K /var/log/logrotate/logrotate.log.3.gz
8.0K /var/log/logrotate/logrotate.log.2.gz
4.0K /var/log/wifiAP.log
4.0K /var/log/syslog
4.0K /var/log/mosquitto/mosquitto.log
4.0K /var/log/logrotate/logrotate.status
Thanks.
Ufw was logging the following when it ran out of space. The output of tail /var/log/kern.log is identical to the output of tail /var/log/ufw.log. The content of /var/log/messages is also full of the UFW BLOCK messages, but they cut off earlier.
The blocked requests you see are from a Raspberry Pi that runs Dakboard and displays our solar production and use. I don’t know what they are, yet, because everything seems to be working on that machine.
May 25 00:07:04 emonpi kernel: [273756.672682] [UFW BLOCK] IN=eth0 OUT= MAC=b8:27:eb:3d:21:50:b8:27:eb:e2:7d:67:08:00:45:00:00:28:00:00:40:00:40:06:b7:07 SRC=192.168.1.80 DST=192.168.1.40 LEN=40 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=TCP SPT=1880 DPT=56340 WINDOW=0 RES=0x00 RST URGP=0
May 25 00:07:19 emonpi kernel: [273772.031176] [UFW BLOCK] IN=eth0 OUT= MAC=b8:27:eb:3d:21:50:b8:27:eb:e2:7d:67:08:00:45:00:00:28:00:00:40:00:40:06:b7:07 SRC=192.168.1.80 DST=192.168.1.40 LEN=40 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=TCP SPT=1880 DPT=54144 WINDOW=0 RES=0x00 RST URGP=0
May 25 00:07:36 emonpi kernel: [273789.304087] [UFW BLOCK] IN=eth0 OUT= MAC=b8:27:eb:3d:21:50:b8:27:eb:e2:7d:67:08:00:45:00:00:28:00:00:40:00:40:06:b7:07 SRC=192.168.1.80 DST=192.168.1.40 LEN=40 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=TCP SPT=1880 DPT=57176 WINDOW=0 RES=0x00 RST URGP=0
May 25 00:08:09 emonpi kernel: [273821.698000] [UFW BLOCK] IN=eth0 OUT= MAC=b8:27:eb:3d:21:50:b8:27:eb:e2:7d:67:08:00:45:00:00:28:00:00:40:00:40:06:b7:07 SRC=192.168.1.80 DST=192.168.1.40 LEN=40 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=TCP SPT=1880 DPT=49278 WINDOW=0 RES=0x00 RST URGP=0
May 25 00:08:22 emonpi kernel: [273835.388849] [UFW BLOCK] IN=eth0 OUT= MAC=b8:27:eb:3d:21:50:b8:27:eb:e2:7d:67:08:00:45:00:00:28:00:00:40:00:40:06:b7:07 SRC=192.168.1.80 DST=192.168.1.40 LEN=40 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=TCP SPT=1880 DPT=49278 WINDOW=0 RES=0x00 RST URGP=0
May 25 00:08:38 emonpi kernel: [273851.276950] [UFW BLOCK] IN=eth0 OUT= MAC=b8:27:eb:3d:21:50:b8:27:eb:e2:7d:67:08:00:45:00:00:28:00:00:40:00:40:06:b7:07 SRC=192.168.1.80 DST=192.168.1.40 LEN=40 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=TCP SPT=1880 DPT=56748 WINDOW=0 RES=0x00 RST URGP=0
May 25 00:09:04 emonpi kernel: [273876.988979] [UFW BLOCK] IN=eth0 OUT= MAC=b8:27:eb:3d:21:50:b8:27:eb:e2:7d:67:08:00:45:00:00:28:00:00:40:00:40:06:b7:07 SRC=192.168.1.80 DST=192.168.1.40 LEN=40 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=TCP SPT=1880 DPT=49278 WINDOW=0 RES=0x00 RST URGP=0
May 25 00:09:38 emonpi kernel: [273911.502642] [UFW BLOCK] IN=eth0 OUT= MAC=b8:27:eb:3d:21:50:b8:27:eb:e2:7d:67:08:00:45:00:00:28:00:00:40:00:40:06:b7:07 SRC=192.168.1.80 DST=192.168.1.40 LEN=40 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=TCP SPT=1880 DPT=57676 WINDOW=0 RES=0x00 RST URGP=0
May 25 00:09:39 emonpi kernel: [273911.731468] [UFW BLOCK] IN=eth0 OUT= MAC=b8:27:eb:3d:21:50:b8:27:eb:e2:7d:67:08:00:45:00:00:28:00:00:40:00:40:06:b7:07 SRC=192.168.1.80 DST=192.168.1.40 LEN=40 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=TCP SPT=1880 DPT=57676 WINDOW=0 RES=0x00 RST URGP=0
May 25 00:10:09 emonpi kernel: [273942.147109] [UFW BLOCK] IN=eth0 OUT= MAC=b8:27:eb:3d:21:50:80:ea:96:e4:f5:53:08:00:45:00:00:2
Well your first move is to edit the file /etc/rsyslog.d/20-ufw.conf
and remove the comment character before the &stop
(TBH I’m not sure what the &
does - just follow instructions).
This will mean the message is only logged once.
Next what are the ufw rules? I think
sudo ufw status verbose
will tell you.
Oh and clear out any rotated logs to increase space.
Made the change to 20-ufw.conf. Cleared out rotated logs (I think – more on that later).
pi@emonpi:~$ sudo ufw status verbose
Status: active
Logging: on (low)
Default: deny (incoming), allow (outgoing), disabled (routed)
New profiles: skip
To Action From
-- ------ ----
80/tcp ALLOW IN Anywhere
443/tcp ALLOW IN Anywhere
22/tcp ALLOW IN Anywhere
1883/tcp ALLOW IN Anywhere
80/tcp (v6) ALLOW IN Anywhere (v6)
443/tcp (v6) ALLOW IN Anywhere (v6)
22/tcp (v6) ALLOW IN Anywhere (v6)
1883/tcp (v6) ALLOW IN Anywhere (v6)
Has it just started to be an issue? What updates have been done?
Probably the next step is to install ‘tshark’ on the requesting Pi and see what traffic it is sending to that Pi.
However, do you really need UFW active?
[edit]
Does the remote machine have node-red on it?
I’ve now merged this in and merged to stable, so a standard emonPi/base update will bring this in.
Would it be possible to do a release / module info for this repo as well? When helping users it would be good to know what version that is sitting at as well as everything else.