No, definitely not!
Firstly, whilst I was keen to use the logrotate.CUSTOM file name for a custom logrotate.conf, that is only IF we have a custom logrotate.conf, at the time of that discussion, I felt it was most likely going to be necessary. But I would prefer to use the standard conf if possible, I’m just not entirely convinced we can. But I’m less hung up on whether we do or don’t use the standard conf than I am about getting it to work correctly. First we need to find the right configuration, and once we know what rules we need we can decide about how that is achieved.
Secondly, I disagree with many of the rules in the emonSD logrotate.conf, so even if we do arrive at a custom conf of my liking, it wouldn’t look like the current conf.
notifempty is pointless in conjunction with no timed rotation and size only (empty is less than 1M!)
We must have a timed rotation, not just rely on size, a 999k log file with zero activity could sit there in ram for ever, better it is rotated out after a week/month.
The rotate 1 is definitely no good.
I don’t think we need nomail or su root root.
we shouldn’t really be using compress without delaycompress if wildcards are being used, if a service is still writing to the rotated file when it is compressed it might crash that service, delaycompress only compresses some.log.2 onwards and is safer unless you know it’s ok to compress the new rotation.
The renamecopy directive is something I brought up to be used in conjunction with the olddir directive so that files can be moved out of RAM. By default olddir can only be used on the same device (ie the rotations could go to another folder, but still in RAM) Either renamecopy, copy or copytruncate can be used to facilitate moving files across devices using olddir.
The copytruncate on the emonSD was necessary because the include drop-ins were disabled so apache2, mosquitto and rsyslog were unable to trigger the post-rotate scripts that made the services let go of the old log handle and start logging to the new file, this isn’t good practice.
The loose wildcards are also a bit of a no no as you’ve discovered with the duplicate logs warnings. This could be less of an issue once we are rotating out of RAM as the rotated files will no longer be present in /var/log.
It is indeed better to avoid using copytruncate if possible, but it is far better to potentially loose a log message or 2, than loose actual data. copytruncate is a low quality rotate method to be used when it can’t be avoided.
Is that a typo? AFAIK the rule should be size 1M (not MB) or better still size 100k, 1M is way too big for some logs, small size is both less space used in ram and less for log2ram (rsync) to calculate each hour.
Regarding emonhub and copytruncate, I wasn’t aware that emonhub clung onto the old file handle, it makes sense now I think about it, but this has been masked for years by the fact that emonSD used copytruncate across the board.
Apache uses it’s “reload” (aka graceful restart) to release old log file handles and mosquitto seems to respond to a HUP signal by releasing the file handles without a restart, as it stands, emonhub doesn’t yet have that specific facility, I’m not great with “signals” so I’m not 100% sure, but I believe any signal able to make emonhub release the logfile handles will cause it to restart and we definitely do not want that to happen, data in transit could be lost and buffered data too. In time maybe a “reload” could be added to emonhub, not previously needed as it loads any new settings whilst running. So in the short term copytruncate maybe necessary for emonhub.
However! I’ve previously also mentioned emonhub’s “5M self-rotation” kicking in if the logs run away and that if there is a emonhub.log.1 present, this is something that needs handling in the logrotation definition, either in the pre or post rotation. So another possible solution in the immediate short term might be to (reduce the in-built rotation to 3M and) only rotate the emonhub.log.1 file, yes it means there will be 2 emonhub.log files in RAM, but it also means that there is no complication with a) the file handle as emonhub will rotate it’s own log (in ram) so we can avoid copytruncate and b) the potential of finding and having to handle 2 emonhub.log files at logrotate time is removed too.
so maybe something along the lines of
/var/log/emonhub/emonhub.log {
# Dummy definition to ensure emonhub.log is NOT rotated
# only rotate if over 10M (not possible with inbuilt rotation)
size 10M
}
/var/log/emonhub/emonhub.log.1 {
daily
rotate 28
size 3M
compress
delaycompress
create 775 emonhub emonhub
missingok
notifempty
# rotate emonhub log files to an emonhub folder in log.old
olddir /var/log.old/emonhub
createolddir 775 root emonhub
renamecopy
sharedscripts
postrotate
# possibly some concatenation into daily files
endscript
}
The explicit definition for emonhub.log would prevent a wildcard picking it up, I haven’t tested this though.
The specific mention of olddir in the emonhub (and emoncms?) drop-in is so that at least our main logs are segregated, unless we edit/replace each drop-in, all the rotated files will land in the same folder.
Some of the finer points such as length of retention and whether logs should be concatenated can be played with, the main point is to get it working well, whether we have a week or a years worth of data at this point is irrelevant unless the sdcard fills up.
As for the globals, we already need the olddir directive defined in logrotate.d/00-olddir, you could just try adding size 100k to that as it is loaded first. But any wildcards will need to go in a file prefixed with “ZZZ-something” so it is loaded last, but even then it may cause conflicts with the wtmp and btmp definitions in the stock config and we do need some form of wildcard I imagine as there are several logs that slip through the net.
The above might allow us to have a stock logrotate.conf but the main things that currently work against that idea is that all the non-emon rotated log files will be dumped in a single folder, so /var/log/mysql/error.log and /var/log/apache2/error.log (for example) will clash as both will want to be just /var/log/error.log.1 after rotation, the only way to avoid this AFAIA is to add the olddir path in the drop-ins as I have with the emonhub one above. Plus the wildcard cannot be placed after the 2 definitions included in the stock conf and TBH the “000” and “ZZZ” prefixes look a tad amateurish and are not very infallible, a user adds “0-something” it will slip in first and likewise with “ZZZZ-something” slipping in after the wildcards.
Plus, something esle we need to factor in is that not all installs will be using L2R/tmpfs logs, so the logrotate on a non-sdcard install should be different, the olddir stuff won’t be needed, the size values won’t be needed, the wildcards probably won’t be needed.
Are you sure it has stopped logging? It should still be logging to the original file, even if it has moved or changed name (ie is it logging to emonhub.log.1?) as the file descriptor will still be intact, unless the file is compressed or deleted (incl moving across devices).