On my emonSD-30Oct19 system without log persistence. /proc/diskstats its been writing at ~0.5kb/s for the last day with the 60 feeds. I havent put the numbers into the equation in the paper yet…
Yes, /home/pi/data/emonhub-log/ - from the link to my logging setup, you can see the path in the logrotate postrotate directive.
On time, I think that anything over 5 years is likely to be for an unsupported base OS so a new image/SD card could be required. 10 years I suggest is more than enough. There is an 80-20 rule here…
What’s the relative probability of an sdhc ‘wearing out’ vs an untoward damaging event? - power failure due to the Grid or home RCD tipping, etc.
With that in mind - will the extensive work that you guys are doing eventually be applicable to data backups?
2 Likes
Yes that’s a good question, im not sure. Im a bit concerned that our recent switch to read-write OS partition increases the likelihood of power failure issues. I think ideally we would have a short term UPS that would allow for a graceful shutdown.
@pb66, I must be missing it, but I cannot see that this is true. olddir does not appear in the repo and your symbolic link is only active if there is a rotated_logs folder.
I think we should differentiate between
- A desire to have a system of persistent logs for all services which includes a mechanism to restore the /var/log structure so is behaves on startup for any additional software installed
- A desire to hold emon* logs for longer.
I have to say I am not convinced of L2R, it seems convoluted to me and I really do not like a Cron job calling a service by reload! I was reminded of this work by @jsroques Mosquitto won't start on boot after raspbian and emonsd update - #41 by jsroques which is much the same.
I also think we need to start from scratch (on the new Install process - consider retro fitting later).
- Get rid of all the rc.local stuff and the rsyslog & logrotate modifications.
- Setup the management of a tmpfs /var/log/ for persistence and management of other systems that are installed (e.g. mosquitto that just assumes the log file is there form the install process).
- Develop a mechanism to generate emon* system logs
- Rotate and preserve emon* logs.
1 Like
@pb66, I must be missing it, but I cannot see that this is true. olddir does not appear in the repo and your symbolic link is only active if there is a rotated_logs folder.
I think we should differentiate between
- A desire to have a system of persistent logs for all services which includes a mechanism to restore the /var/log structure so is behaves on startup for any additional software installed
- A desire to hold emon* logs for longer.
I have to say I am not convinced of L2R, it seems convoluted to me and I really do not like a Cron job calling a service by reload! I was reminded of this work by @jsroques Mosquitto won't start on boot after raspbian and emonsd update - #41 by jsroques which is much the same.
I also think we need to start from scratch (on the new Install process - consider retro fitting later).
- Get rid of all the rc.local stuff and the rsyslog & logrotate modifications.
- Setup the management of a tmpfs /var/log/ for persistence and management of other systems that are installed (e.g. mosquitto that just assumes the log file is there form the install process).
- Develop a mechanism to generate emon* system logs
- Rotate and preserve emon* logs.
I already explained how the symlink and rotated files directory work with L2R in the “emonSD next steps: log2ram - #5 by pb66” thread.
If that folder is not there, then the rotation hasn’t run (correctly) yet. This is logrotation by logrotate and has nothing to do with L2R other than the additional config for logrotate is installed by the L2R installer as part of the L2R logging solution. I would like to see L2R and logrotate work closer, ie triggered as a single operation, check and rotate first, then sync just what’s left in ram and report stats (to log and/or emonhub too).
[edit - assuming you are using a emonSD/Pi/Base I think there are still some issues with the logrotation on that image. also assuming the image I have here in front of me is as intended, there seems to be a number of anomalies.
-
The logrotation used to be hourly on the emonSD, it is currently daily
pi@emonpi:~ $ sudo ls -la /etc/cron*/logrotate lrwxrwxrwx 1 root root 46 Apr 23 10:45 /etc/cron.daily/logrotate -> /var/www/html/emoncms/scripts/logger/logrotateIt seems the symlink isn’t moved as per the instruction in the custom conf file (emoncms/scripts/logger/logrotate.conf at ddb08542b5ccb9ae29b2822cea32f3beb20733ed · emoncms/emoncms · GitHub)
-
The logrotaion cron seems to try and use a custom logrotate.ststus file that “should” apparently be in
/var/log./usr/sbin/logrotate -v -s /var/log/logrotate/logrotate.status /etc/logrotate.conf 2>&1 | tee /var/log/logrotate/logrotate.logSince
/var/logis in tmpfs and any file there will be lost on reboot, this logrotate command fails due to not being able to find the nominated ststus file, thus logrotate doesn’t run. -
The custom logrotate.conf file disables all other logrotate configurations provided by other packages, (including L2R). In fact it disables the inclusion twice! (emoncms/scripts/logger/logrotate.conf at ddb08542b5ccb9ae29b2822cea32f3beb20733ed · emoncms/emoncms · GitHub)
# Don't nclude package specefic logrotate config #include /etc/logrotate.d # Include package specefic logrotate config #include /etc/logrotate.d -
Even though the config says
# keep 2 rotations worth of backlogs (size x 2)A recent change means we now only keep one rotation! So when running as an hourly rotation, only 1M per log is retained following each each hourly rotation.
But nowhere does it change the olddir setting (AFAICS). I think you have picked up that config change from somewhere else.
The config installed by log2ram for logrotate is to rotate the log2ram log file not to rotate all files into a new directory.
I know there are issues and that is why it needs removing - simply, it doesn’t work.
Just in general, I think the saving of logs (assuming they are written to /var/log/ on tmpfs) is irrelevant to the question of the type of FS for the data logging. Writing a few files once an hour, is not what kills SD Cards.
No, I know exactly what functionality I have added to L2R by my own hand!
Here is the PR or diff that I previously provided showing the changes made by me so that logrotate uses the olddir directive to rotate the files out of ram using the renamecopy option.
Here is a discussion about it on the L2R repo
There are 2 additional etc/logrotate.d files installed by L2R (as I’ve previously said these could be merged) one is for the L2R.log rotations and the other is for configuring olddir globally and therefore must have a name prefixed with 00 to ensure it is the first read conf file, namely 00_olddir. The content of said file is
# These settings will save the rotated logfiles to a seperate location.
# This file needs to be located in /etc/logrotate.d.
# The "00" prefix in the file name is required to ensure this file is
# loaded before any program specific logrotate settings are loaded as
# contents of /etc/logrotate.d are read in alphabetical order.
# The path & name for rotated logfiles folder.
# "/var/log.old" will `ls` next to "/var/log" and "/var/log.bak"
# This must mirror the OLD_LOG variable in the log2ram script.
olddir /var/log.old
# Create the path if it doesn't exist
createolddir 755 root root
# To allow the files to be "moved" (ie copied and original deleted)
# to another device/partition (eg out of RAM)
renamecopy
I’d missed the second file - apologies. Strangely a search of the repo for ‘olddir’ had not thrown it up!
- The use of copytruncate over just the default create may block rotations when /var/log is near or at capacity since copytruncate tries to copy the existing file and then delete the content of the original, if there is not enough space for the file being rotated to be duplicated this cannot happen. The default create function just renames the original and creates a new empty file, so whilst it does need some room (to create a blank file) it doesn’t need to double the file size to work.
That is a quirk of GitHub, you can not search forks, only the upstream repo and this doesn’t exist on the upstream repo, only on my fork.
1 Like
Yes, that’s great, as you’re not logging much. Some of us are logging 150+ feeds. And I’m planning on another 50+ if I can just find the time to get it coded…
So that’s my SD card down from 294 months to 21 months? Maybe?
OK, I may be a big of an outlier, but I’m willing to bet there are many on here who log dozens of feeds.
1 Like
I found this yesterday suggesting copytruncate was very bad (and that from the author of rsyslog apparently). rsyslog with logrotate: reload rsyslog vs copytruncate - Server Fault
Not had a chance to play with the reload alternative.
Promising results with the log2ram rsync modifications you suggested @pb66
rsync -aXv --inplace --no-whole-file --delete --links $RAM_LOG/ $HDD_LOG/ 2>&1 | $LOG_OUTPUT
1 Like
In addition to your changes here Comparing azlux:master...pb66:pb66 · azlux/log2ram · GitHub @pb66 have you changed /etc/logrotate.conf? do you keep the /etc/logrotate.d config set for all the modules?
With the above rsync results will you still use cp yourself or will you use the above modifications?
I have now on my test system:
- restored /etc/logrotate.conf and /etc/cron-hourly/logrotate to default
- installed your fork of log2ram @pb66
- applied the rsync modifcation
- set rsync=true in log2ram.conf
- added the logfile to the service file:
ExecStart=/opt/emon/emonhub/src/emonhub.py --config-file=/opt/emon/data/emonhub.conf --logfile=/var/log/emonhub/emonhub.log
Created a /etc/logrotate.d/emonhub configuration containing:
/var/log/emonhub/emonhub.log
{
rotate 2
size 1M
nomail
copytruncate
missingok
notifempty
compress
}
1M so that I can see this rotating sooner than later.
Is that similar to the configuration that you are using @pb66?
O dear something seems to have just deleted all the logs in /var/log
$ cat log2ram.log
sending incremental file list
deleting samba/
deleting redis/redis-server.log
deleting redis/
deleting mysql/error.log
deleting mysql/
deleting mosquitto/mosquitto.log
deleting mosquitto/
deleting logrotate/logrotate.status
deleting logrotate/logrotate.log
deleting logrotate/
deleting emonpilcd/emonpilcd.log
deleting emonpilcd/
deleting apt/term.log
deleting apt/history.log
deleting apt/eipp.log.xz
deleting apt/
deleting apache2/other_vhosts_access.log
deleting apache2/error.log
deleting apache2/access.log
deleting apache2/
deleting wtmp
deleting user.log
deleting syslog
deleting rotated_logs
deleting messages
deleting lastlog
deleting kern.log
deleting faillog
deleting emoncms.log
deleting dpkg.log
deleting debug
deleting daemon.log
deleting btmp
deleting bootstrap.log
deleting boot.log
deleting boot
deleting auth.log.1.gz
deleting auth.log
deleting alternatives.log
./
sent 72 bytes received 631 bytes 1,406.00 bytes/sec
total size is 30 speedup is 0.04
and
pi@emonpi:/var/log $ ls -lh
total 4.0K
-rw-r--r-- 1 root root 972 Apr 23 21:52 log2ram.log
lrwxrwxrwx 1 root root 12 Apr 23 21:52 rotated_logs -> /var/log.old