I’ve read all the posts related to /var/log filling up but it seems (with my limited understanding) this issue has not been resolved. Several ways to ‘fix’ this have been mentioned, but I’ve found none that work yet, unless I missed something, which is quite likely.
My Inputs are updating, but I reboot/update so often that RAM/Swap is not filling up.
I’m running 10.0.2 (master) and I hasn’t been resolved in that release. I think this issue’s importance should be raised as the problem of Inputs not updating greatly reduces the robustness of the system and therefore the reliability and trust.
My knowledge is somewhat limited, but I have plenty of time and can offer to help test or document a fix.
Hi Neil, there are several layers to the “log partition filling up” issue, yesterday Trystan pushed some changes to the repo’s to change the way the emoonhub.log’s are handled, that should crack the reason this issue has escalated dramatically most recently. AFAIA if you update today you should get those changes, so if your up for testing, you could try a new update today and check to see if there is an emonhub.log created
ls -la /var/log/emonhub
You could also just confirm for us the status of the logrotate by just checking the content of these 2 files
AFAIA the logrotate was maybe fixed already but that’s unconfirmed, the latest image (which are you running, assuming you are running an emonSD image?) was originally missing the logrotate mods and the images before that somehow got broken along the way, Both Trystan and Glyn have made changes recently but I’m unsure how effective they are as the emonhub log issue hid the logrotate issue a bit.
It would be good to confirm where we are with this at the moment if you are willing.
As for the longer term, there is other stuff in the pipeline to improve the whole logging side of things, the fixes above should just take us back to where the logging has been for the past few years, which has weaknesses but has in almost all cases worked fine most of the time. The future changes will hopefully give us a far more robust and capable setup, but we’re not there yet. Further testing will undoubtedly be useful thank you.
[edit] could you please also post a copy of your update log too?
I checked before updating and after updating. There was no difference in the data returned.
Here’s what I got:
pi@emonpi:~ $ ls -la /var/log/emonhub
ls: cannot access '/var/log/emonhub': No such file or directory
pi@emonpi:~ $ cat /etc/cron.hourly/logrotate
#!/bin/sh
test -x /usr/sbin/logrotate || exit 0
/usr/sbin/logrotate -v -s /var/log/logrotate/logrotate.status /etc/logrotate.conf 2>&1 | tee /var/log/logrotate/logrotate.log
pi@emonpi:~ $ cat /etc/logrotate.conf
# emoncms logrotate config
# default is daily run, move cron script to make this happen hourly:
# sudo mv /etc/cron.daily/logrotate /etc/cron.hourly/logrotate
# keep 2 rotations worth of backlogs (size x 2)
rotate 1
# Max size of log file (rotate log file when this size is reached)
size 1M
# Truncate the original log file in place after creating a copy
copytruncate
# Do not mail logfile
nomail
# log owners
su root root
# Don't rotate if empty or notify if missing
missingok
notifempty
# Don't nclude package specefic logrotate config
#include /etc/logrotate.d
# Include package specefic logrotate config
#include /etc/logrotate.d
# Include all logs including 1 level deep subdirectories
/var/log/*.log /var/log/*/*.log /var/log/syslog /var/log/messages /var/log/btmp /var/log/* {}
Please let me know if you need more info or testing.
If you look at the update log near the top, when it tries to update the first repo “emonpi” it fails due to some local changes
git pull /home/pi/emonpi
enable-ssh
enable-ssh-button
gpiozero-enable-ssh
greeebs-patch-1
* master
python-systemd-servicerunner
On branch master
Your branch is behind 'origin/master' by 27 commits, and can be fast-forwarded.
(use "git pull" to update your local branch)
. . .
error: Your local changes to the following files would be overwritten by merge:
lcd/emonPiLCD.cfg
Please commit your changes or stash them before you merge.
Aborting
although it does continue through the rest of the updates, it isn’t using the latest updater as that is part of the emonpi repo, ideally, when the emonpi repo fails to update the update routine should abort to avoid landing in some grey area, eg if there was some new code in emoncms or emonhub that also needed a symlink added or a config file edited, that could break a working setup.
In fact you are missing 27 commits, everything committed to the emonpi repo since April 15th which I think is most of the changes to the updater. actually most of them are for the installer that’s not being used yet.
I’m guessing you’ve fallen into one of those “grey area” I mentioned above?
The first thing to do is to get the updater to run properly by getting the emonpi repo to update, have you checked the file as suggested above? Do you recall what changes you made? Are they needed?
I changed ~/emonpi/lcd/emonPiLCD.cfg backlight_timeout = 0 to backlight_timeout = 30 several weeks ago. I’ve changed it back to 0 and performed an update again. My inputs are updating and the emoncms-mqtt failure is now Active: Running.
EmonHub log looks normal now.
I did this again:
pi@emonpi : ~/emonpi/lcd $ ls -la /var/log/emonhub
total 356
drwxr-xr-x 2 emonhub emonhub 60 May 22 09:02 .
drwxr-xr-x 12 root root 540 May 22 09:02 …
-rw-r–r-- 1 emonhub emonhub 362355 May 22 09:15 emonhub.log
**pi@emonpi** : **~/emonpi/lcd $** cat /etc/logrotate.conf
# emoncms logrotate config
# default is daily run, move cron script to make this happen hourly:
# sudo mv /etc/cron.daily/logrotate /etc/cron.hourly/logrotate
# keep 2 rotations worth of backlogs (size x 2)
rotate 1
# Max size of log file (rotate log file when this size is reached)
size 1M
# Truncate the original log file in place after creating a copy
copytruncate
# Do not mail logfile
nomail
# log owners
su root root
# Don't rotate if empty or notify if missing
missingok
notifempty
# Don't nclude package specefic logrotate config
#include /etc/logrotate.d
# Include package specefic logrotate config
#include /etc/logrotate.d
# Include all logs including 1 level deep subdirectories
/var/log/*.log /var/log/*/*.log /var/log/syslog /var/log/messages /var/log/btmp /var/log/* {}
Thanks Neil, I shall take a closer look at the logs later, but it seems the logrotate and emonhub.log is as expected so you should be ok now (fingers crossed )
It’s amazing how ‘worried’ I get when when a system that: (1) I continually tinker with, (2) is not critical in my application and, (3) and is only a hobby, doesn’t work perfectly!
But I’m learning a lot thanks to everyone who contributes here.
Please let me know if you find anything out of whack in the logs.
------------------------------------------
Updating backup module
------------------------------------------
- git branch: stable
- git tags: 1.1.5-10-g3dcb153
- git status:
On branch stable
Your branch is up-to-date with 'origin/stable'.
Changes not staged for commit:
(use "git add <file>..." to update what will be committed)
(use "git checkout -- <file>..." to discard changes in working directory)
modified: config.cfg
modified: emoncms-export.sh
this tells us you have some local changes to 2 files that could block the backup module from being updated. It looks like there were not updates to pull in at the time, so it wasn’t a problem, but it’s something to be aware of as it probably will cause a problem at some point.
@TrystanLea - If you check Neils update logs they both contain
Installing emoncms logrotate...
Now setting up Logrotate...
Backing up old logrotate configuration...
Linked to new logrotate configuration...
Backing up old logrotate cron job...
mv: cannot stat '/etc/cron.daily/logrotate': No such file or directory
Linked to new logrotate cron job...
Completed
Completed
Running logrotate...
set log rotate config owner to root
the 2 "Backing up old . . . " lines suggest the backed up files are being over written on every update, there no point backing up the original files if they are going to be overwritten again with the content of the symlinked files, ideally this needs to be multi-run safe by making a backup only if a real file is found (not a symlink) as I proposed
I also think the whole update should be aborted if the emonpi repo git pull fails and failures need to be passed to the user somehow. Perhaps a flag “there were errors check your update log” could be bannered across the emoncms gui and or on the emonpi lcd?
I changed the filename date format in emoncms-export.sh and the backup location in config.cfg. I have not been able to backup to a remote server (yet) but I am auto-backing-up to a USB drive for now. I wanted to backup to the server so that, in turn, that would be backed up to Time Machine and I would have backups going back for years then, without intervention from me. Perhaps I’m being too paranoid though
I am surprised that changing these files would cause an update failure. But there may be reasons beyond my knowledge.
Log rotation seems to be working properly now, /var/log is not filling up.
I am available for some testing or documentation for any issues that arise so please feel free to ask.
Basically it’s due to the the way the updates are done via git pulls. If an update (git pull) is initiated on a repository folder it will “pull in the changes” between your existing version and the latest version. If you have made any edits to the files it want s to change, it doesn’t know the priority, do you want to retain your edits (if that’s even possible) or do you want the latest “stock” code. At which point it doesn’t do either, it just tells you there are reasons it didn’t do it so you can “stash” or “commit” your changes or tell git it just overwrite them.
Looking at the logs it does look ok’ish, logrotate is working. But there are a couple of anomalies such several log files not being rotated, could you show us the next level down (ie in the sub-folders)
sudo ls -la /var/log/*/
and also give us a copy of ‘/var/log/logrotate/logrotate.log’ to look at?
Is your logrotate setup still “as standard” for an emonSD? Have you applied any of those “fixes” you mentioned?
I’m going to make a couple of comments here just to document some thoughts, these may or may not affect you and/or may or may not get looked into further or addressed by other future development. Some of it has been brought up before and some of it will not be relevant if/when log2ram (and the use of olddir) is introduced, but if that’s not rolled out to old images, some of these things could easily be addressed. I just want to make notes while it’s fresh in my mind, to refer back to.
mysql.log doesn’t appear to be used or even needed, it’s created by rc.local and doesn’t appear in any of my (non-emonsd) installs.
several logs are just hanging around using up ram. the logrotate.log is full of entries saying “log does not need rotating (log size is below the ‘size’ threshold)” which means at some point there could be several files sat at 990K for days or weeks on end as there is no daily/weekly/monthly alternative to the size limit of 1M.
several logs created by rc.local are not needed on the oct18 image as openhab(2) etc aren’t installed and the likes of mqtt_input has been replaced by emoncms_mqtt.
the logrotate log folder is the wrong ownership. It’s set in rc.local as pi:pi but don’t know why, both logs in that folder are root:root.
“notifempty” setting seems pointless with the “size 1M” setting, effectively a “not if less than 1M” which I believe covers “empty” too.
If a weekly/monthly period was left set then logs with no recent movement but have a rotated file sat there, would get rotated eventually (even though they are small/empty) so the rotated log is removed.
some log files like boot.log and debug.log are less likely to reach 1M so they are just hogging RAM, these need to be caught by either a lower size (eg 100K) or eventually by a weekly/monthly rotation.
emonhub.logs are too short at >1M, the size setting needs to be at least 2-3M for enonhub. Looking at the logs above we can see emonhub.log has generated over 305K in just 11mins, that’s over 1.5M per hour as seen by the emonhub.log.1 file size, essentially the 1M setting limits us to just 2 hours emonhub.log.
The command-line in the cron entry diverts stdout and stderr to logrotate.log using tee, this overwrites previous rotations so is of limited use, tee -a would append the output to the log.
copytruncate will block rotation if log partition is nearly full, looking at the rotation of emonhub in the log, it shows the steps where emonhub.log is duplicated to create the rotated file and then the original file is truncated, this can only happen if there is room, in this case it would have needed a t least 1.6M of free space to rotate.
logrotate.status doesn’t need to be in the log partition on the RW oct18 image and it could be symlinked to /tmp on the previous RO images rather than straying from the default location.
A duplicate entry is reported in the log “error: /etc/logrotate.conf:32 duplicate log entry for /var/log/auth.log”. Line 32 of logrotate.conf is the string of wildcards (emoncms/scripts/logger/logrotate.conf at 10.0.2 · emoncms/emoncms · GitHub) so why does it just error for auth.log, why not boot.log (for example).
@Neil - Sorry to ask again, but could you show us the output of
sudo ls -la /etc/logrotate*
sudo ls -la /etc/cron.*/logrotate
@pb66, I didn’t apply any of the fixes that have been discussed, except for deleting the daemon.log to make space, and I did install log2ram but I removed it before really testing it. I hope that didn’t break anything.
Sorry for the delay, here’s the info you requested.
pi@emonpi:~ $ sudo ls -la /etc/logrotate*
lrwxrwxrwx 1 root root 51 May 22 09:02 /etc/logrotate.conf -> /var/www/html/emoncms/scripts/logger/logrotate.conf
/etc/logrotate.d:
total 48
drwxr-xr-x 2 root root 4096 May 18 07:05 .
drwxr-xr-x 99 root root 4096 May 22 09:02 ..
-rw-r--r-- 1 root root 433 Jun 2 2018 apache2
-rw-r--r-- 1 root root 173 Sep 13 2017 apt
-rw-r--r-- 1 root root 79 Apr 18 2017 aptitude
-rw-r--r-- 1 root root 232 Jun 9 2015 dpkg
-rw-r--r-- 1 root root 419 Jan 14 2017 lighttpd
-rw-r--r-- 1 root root 154 Nov 3 2016 mosquitto
-rw-r--r-- 1 root root 802 Jun 7 2017 mysql-server
-rw-r--r-- 1 root root 124 Jun 14 2018 redis-server
-rw-r--r-- 1 root root 515 Jan 18 2017 rsyslog
-rw-r--r-- 1 root root 178 Aug 7 2014 ufw
And.
**pi@emonpi** : **~ $** sudo ls -la /etc/cron.*/logrotate
lrwxrwxrwx 1 root root 46 May 22 09:02 /etc/cron.hourly/logrotate -> /var/www/html/emoncms/scripts/logger/logrotate
)