Yes, /home/pi/data/emonhub-log/ - from the link to my logging setup, you can see the path in the logrotate postrotate directive.
On time, I think that anything over 5 years is likely to be for an unsupported base OS so a new image/SD card could be required. 10 years I suggest is more than enough. There is an 80-20 rule here…
Yes that’s a good question, im not sure. Im a bit concerned that our recent switch to read-write OS partition increases the likelihood of power failure issues. I think ideally we would have a short term UPS that would allow for a graceful shutdown.
@pb66, I must be missing it, but I cannot see that this is true. olddir does not appear in the repo and your symbolic link is only active if there is a rotated_logs folder.
I think we should differentiate between
A desire to have a system of persistent logs for all services which includes a mechanism to restore the /var/log structure so is behaves on startup for any additional software installed
I also think we need to start from scratch (on the new Install process - consider retro fitting later).
Get rid of all the rc.local stuff and the rsyslog & logrotate modifications.
Setup the management of a tmpfs /var/log/ for persistence and management of other systems that are installed (e.g. mosquitto that just assumes the log file is there form the install process).
@pb66, I must be missing it, but I cannot see that this is true. olddir does not appear in the repo and your symbolic link is only active if there is a rotated_logs folder.
I think we should differentiate between
A desire to have a system of persistent logs for all services which includes a mechanism to restore the /var/log structure so is behaves on startup for any additional software installed
I also think we need to start from scratch (on the new Install process - consider retro fitting later).
Get rid of all the rc.local stuff and the rsyslog & logrotate modifications.
Setup the management of a tmpfs /var/log/ for persistence and management of other systems that are installed (e.g. mosquitto that just assumes the log file is there form the install process).
If that folder is not there, then the rotation hasn’t run (correctly) yet. This is logrotation by logrotate and has nothing to do with L2R other than the additional config for logrotate is installed by the L2R installer as part of the L2R logging solution. I would like to see L2R and logrotate work closer, ie triggered as a single operation, check and rotate first, then sync just what’s left in ram and report stats (to log and/or emonhub too).
[edit - assuming you are using a emonSD/Pi/Base I think there are still some issues with the logrotation on that image. also assuming the image I have here in front of me is as intended, there seems to be a number of anomalies.
The logrotation used to be hourly on the emonSD, it is currently daily
The logrotaion cron seems to try and use a custom logrotate.ststus file that “should” apparently be in /var/log.
/usr/sbin/logrotate -v -s /var/log/logrotate/logrotate.status /etc/logrotate.conf 2>&1 | tee /var/log/logrotate/logrotate.log
Since /var/log is in tmpfs and any file there will be lost on reboot, this logrotate command fails due to not being able to find the nominated ststus file, thus logrotate doesn’t run.
A recent change means we now only keep one rotation! So when running as an hourly rotation, only 1M per log is retained following each each hourly rotation.
Just in general, I think the saving of logs (assuming they are written to /var/log/ on tmpfs) is irrelevant to the question of the type of FS for the data logging. Writing a few files once an hour, is not what kills SD Cards.
No, I know exactly what functionality I have added to L2R by my own hand!
Here is the PR or diff that I previously provided showing the changes made by me so that logrotate uses the olddir directive to rotate the files out of ram using the renamecopy option.
Here is a discussion about it on the L2R repo
There are 2 additional etc/logrotate.d files installed by L2R (as I’ve previously said these could be merged) one is for the L2R.log rotations and the other is for configuring olddir globally and therefore must have a name prefixed with 00 to ensure it is the first read conf file, namely 00_olddir. The content of said file is
# These settings will save the rotated logfiles to a seperate location.
# This file needs to be located in /etc/logrotate.d.
# The "00" prefix in the file name is required to ensure this file is
# loaded before any program specific logrotate settings are loaded as
# contents of /etc/logrotate.d are read in alphabetical order.
# The path & name for rotated logfiles folder.
# "/var/log.old" will `ls` next to "/var/log" and "/var/log.bak"
# This must mirror the OLD_LOG variable in the log2ram script.
olddir /var/log.old
# Create the path if it doesn't exist
createolddir 755 root root
# To allow the files to be "moved" (ie copied and original deleted)
# to another device/partition (eg out of RAM)
renamecopy
The use of copytruncate over just the default create may block rotations when /var/log is near or at capacity since copytruncate tries to copy the existing file and then delete the content of the original, if there is not enough space for the file being rotated to be duplicated this cannot happen. The default create function just renames the original and creates a new empty file, so whilst it does need some room (to create a blank file) it doesn’t need to double the file size to work.
Yes, that’s great, as you’re not logging much. Some of us are logging 150+ feeds. And I’m planning on another 50+ if I can just find the time to get it coded…
So that’s my SD card down from 294 months to 21 months? Maybe?
OK, I may be a big of an outlier, but I’m willing to bet there are many on here who log dozens of feeds.
brings it down to ~11 sectors/minute 0.09 kb/s, in the previous 6 hours before the modifications, the write rate had increased to 0.86 kb/s so a 10x reduction.
added the logfile to the service file:
ExecStart=/opt/emon/emonhub/src/emonhub.py --config-file=/opt/emon/data/emonhub.conf --logfile=/var/log/emonhub/emonhub.log
Created a /etc/logrotate.d/emonhub configuration containing:
Those calculations just confirm that you’re far more likely to fill up your 4GB partition and need to buy a new card before you would wear the card out due to exceeding the available writes.
What size card do you have, and how much space is currently being used by your emonCMS data?