I am interested also! Once per hour I rsync the log files to a USB drive so they don’t get lost.
Hello,
I just finished a first step after struggling a bit with systemd…
Everything is here : GitHub - jsiobj/emonpi at var-log-files
Restoring /var/log file structure at boot
I created a small script and associated service which restores the /var/log/ file structure at boot.
The service is configured to run quite early in the boot process, before the end of basic.target which is before most “apps” services (using WantedBy=basic.target) but after all fs are mounted (using After=local-fs.target)
I found a nice cp trick : --attributes-only which only copies directories and files with proper attributes but with no data, that is only create empty files. That is a quick and efficient way of backing up and restoring the structure.
Here is boot process logs summary
Dec 8 22:01:29 emonpi rsyslogd: [origin software="rsyslogd" swVersion="8.4.2" x-pid="522" x-info="http://www.rsyslog.com"] start
Dec 8 22:01:29 emonpi systemd-modules-load[112]: Inserted module 'i2c_bcm2708'
Dec 8 22:01:29 emonpi systemd-modules-load[112]: Inserted module 'i2c_dev'
Dec 8 22:01:29 emonpi systemd[1]: Mounted Configuration File System.
...
Dec 8 22:01:29 emonpi systemd[1]: Reached target Local File Systems.
...
Dec 8 22:01:29 emonpi varlog-init.sh[290]: ‘/home/pi/emonpi/logtools/log.template/apache2’ -> ‘/var/log/apache2’
Dec 8 22:01:29 emonpi varlog-init.sh[290]: ‘/home/pi/emonpi/logtools/log.template/apache2/error.log’ -> ‘/var/log/apache2/error.log’
Dec 8 22:01:29 emonpi varlog-init.sh[290]: ‘/home/pi/emonpi/logtools/log.template/emoncms.log’ -> ‘/var/log/emoncms.log’
Dec 8 22:01:29 emonpi varlog-init.sh[290]: ‘/home/pi/emonpi/logtools/log.template/logrotate’ -> ‘/var/log/logrotate’
Dec 8 22:01:29 emonpi varlog-init.sh[290]: ‘/home/pi/emonpi/logtools/log.template/mosquitto’ -> ‘/var/log/mosquitto’
Dec 8 22:01:29 emonpi varlog-init.sh[290]: ‘/home/pi/emonpi/logtools/log.template/mqtt_input.log’ -> ‘/var/log/mqtt_input.log’
Dec 8 22:01:29 emonpi varlog-init.sh[290]: ‘/home/pi/emonpi/logtools/log.template/mysql’ -> ‘/var/log/mysql’
Dec 8 22:01:29 emonpi varlog-init.sh[290]: ‘/home/pi/emonpi/logtools/log.template/mysql/error.log’ -> ‘/var/log/mysql/error.log’
Dec 8 22:01:29 emonpi varlog-init.sh[290]: ‘/home/pi/emonpi/logtools/log.template/mysql.log’ -> ‘/var/log/mysql.log’
Dec 8 22:01:29 emonpi varlog-init.sh[290]: ‘/home/pi/emonpi/logtools/log.template/ntp_update.log’ -> ‘/var/log/ntp_update.log’
Dec 8 22:01:29 emonpi varlog-init.sh[290]: ‘/home/pi/emonpi/logtools/log.template/oem-varlog.flag’ -> ‘/var/log/oem-varlog.flag’
Dec 8 22:01:29 emonpi varlog-init.sh[290]: ‘/home/pi/emonpi/logtools/log.template/openhab’ -> ‘/var/log/openhab’
Dec 8 22:01:29 emonpi varlog-init.sh[290]: ‘/home/pi/emonpi/logtools/log.template/redis’ -> ‘/var/log/redis’
Dec 8 22:01:29 emonpi varlog-init.sh[290]: ‘/home/pi/emonpi/logtools/log.template/redis/redis-server.log’ -> ‘/var/log/redis/redis-server.log’
Dec 8 22:01:29 emonpi varlog-init.sh[290]: ‘/home/pi/emonpi/logtools/log.template/service-runner.log’ -> ‘/var/log/service-runner.log’
Dec 8 22:01:29 emonpi varlog-init.sh[290]: ‘/home/pi/emonpi/logtools/log.template/supervisor’ -> ‘/var/log/supervisor’
Dec 8 22:01:29 emonpi varlog-init.sh[290]: ‘/home/pi/emonpi/logtools/log.template/supervisor/supervisord.log’ -> ‘/var/log/supervisor/supervisord.log’
Dec 8 22:01:29 emonpi systemd[1]: Started Creates required directories and file in /var/log.
...
Dec 8 22:01:29 emonpi systemd[1]: Reached target Basic System.
Dec 8 22:01:29 emonpi systemd[1]: Starting Mosquitto MQTT Broker...
Dec 8 22:01:29 emonpi systemd[1]: Started Mosquitto MQTT Broker.
Dec 8 22:01:29 emonpi systemd[1]: Starting Node-RED graphical event wiring tool....
Dec 8 22:01:29 emonpi systemd[1]: Started Node-RED graphical event wiring tool..
So it seems to work as expected : log files are ready before apps services startup.
I have created a ~/emonpi/logtools directory where there is
- log.template which contains the “default” set of log files
- oem-varlog.install : a (very) simple install script for the service
- oem-varlog.service : the systemd unit file
- varlog-init.sh : the service script which is doing all the work (I finally created a script because of the limitations of systemd interpreter)
On startup, 2 directories (there structures and empty files) are copied :
-
~/emonpi/logtools/log.templatewhich is the default set of logs -
~/data/log.templatewhich is (will be) the backed up file structure from actual/var/log
That way, default log files are in the RO part so the user can’t mess it up and actual log files are stored in the RW fs. Using--attributes-onlycombine with-n(no overwrite) makes it a very light process.
But… as of now, backup of structure does not occurs on system shutdown or reboot 
and I have hard time to track down the issue… as I have no persistent log !
Anyway… I’ll keep trying. If ou have any clue of the reason why it won’t work, please let me know.
Full log files backup
Digging a little bit more into logrotate, although it cannot use another fs to rotate logs, it has a postrotate feature which could be used to run some rsync for instance. It is to be noted that rotated files should not / can not be moved from their origin directory as it will probably prevent logrotate to work properly.
But thinking about it anyway, it is probably not possible to backup all log files on shutdown, especially syslog, as you may not be able to back it up late enough in the shutdown process. But still, it shoud work well for apps logs.
That’s all for today. Any comment is welcome.
JS
1 Like
Hello,
Here is the status of my tests
I finally found out what was going wrong with the service
In the unit file, to “restore” the /var/log file structure early enough (and especially before user services) I set defaultDependencies=no. But I did not realize that this would also remove the dependency that makes systemd to run ExecStop on shutdown. So I added Conflicts=shutdown.target which is solving this.
Also, I moved the unit file to /etc/systemd/system instead of /lib/systemd/system. As I understand it /lib is for “packaged” apps, not user defined services.
I’ll will now move on to test a full backup of logfile using postrotate feature of logrotate and rsync.
The updated files are there : GitHub - jsiobj/emonpi at var-log-files
And, it is not yet too late so best wishes to OpenEnergyMonitor community 
JS
1 Like
Hello,
I just did some new tests and now log files are copied over ~/data on shutdown.
I tried to run this backup using postrotate option in logrotate but I could not have it to work. There are serveral logrotate.conf file and I am not sure which one to use. Actually, I could not find how logrotate was run… Is it from emoncms ?
JS
Logrotate is triggered from a cron job, usually a daily task, but I think it’s a hourly task on the emonSD. Try
ls -la /etc/cron.*/logrotate
to find the cron entry. This is from a stock Raspbian Stretch
pi@raspberrypi:~ $ ls -la /etc/cron.*/logrotate
-rwxr-xr-x 1 root root 89 May 5 2015 /etc/cron.daily/logrotate
I think the main conf file is /etc/logrotate.conf while the best practice for other applications is to add a file to /etc/logrotate.d/ and logrotate compiles it’s configuration by parsing all the conf files.
pi@raspberrypi:~ $ ls -la /etc/logrotate.conf
-rw-r--r-- 1 root root 599 May 5 2015 /etc/logrotate.conf
pi@raspberrypi:~ $ ls -la /etc/logrotate.d
total 24
drwxr-xr-x 2 root root 4096 Nov 29 01:17 .
drwxr-xr-x 112 root root 4096 Feb 20 11:34 ..
-rw-r--r-- 1 root root 173 Sep 13 16:47 apt
-rw-r--r-- 1 root root 79 Apr 18 2017 aptitude
-rw-r--r-- 1 root root 232 Jun 10 2015 dpkg
-rw-r--r-- 1 root root 515 Jan 18 2017 rsyslog
you can get more info on logrotate via the command-line with
logrotate -?
or
man logrotate
Sorry for going a bit quiet on this, I have been following your posts with the intention of “giving it a spin” but I never quite get to do it. But from what I read it looks impressive what you’ve done so far.
Hello @pb66,
Ok… I did not know there was such directories ! I just used crontab -l to check…
So indeed, there is an hourly task :
pi@emonpi(ro):~$ ls -l /etc/cron.hourly/
total 8
-rwxr-xr-x 1 root root 207 Jan 22 2016 fake-hwclock
lrwxrwxrwx 1 root root 41 Jan 26 2016 logrotate -> /var/www/emoncms/scripts/logger/logrotate
-rwxr-xr-x 1 root root 300 Jan 22 2016 ntp-backup
Looking at the script, I can see which conf file is used
pi@emonpi(ro):~$ cat /var/www/emoncms/scripts/logger/logrotate
#!/bin/sh
test -x /usr/sbin/logrotate || exit 0
/usr/sbin/logrotate -v -s /var/log/logrotate/logrotate.status /etc/logrotate.conf 2>&1 | tee /var/log/logrotate/logrotate.log
pi@emonpi(ro):~$ ls -l /etc/logrotate.conf
lrwxrwxrwx 1 root root 46 Apr 28 2016 /etc/logrotate.conf -> /var/www/emoncms/scripts/logger/logrotate.conf
It is using a specifc conf file. I find it strange though for the logrotate.conf to be located in /var/www/emoncms as it is more OS related stuff and not really website related.
Anyway, I’ll try to make this work and give the results.
Thank for the reply. And no worries for the delay… I had been busy myself and just restart to work on this a few days ago.
JS
Me too!
I had a quick look and found some related files in the emoncms repo, there’s a readme too.(emoncms/scripts/logger at master · emoncms/emoncms · GitHub) but in the emonPi repo, which is essentially the emonSD repo, there are also some logrotate files but there is a note in one saying
# NOT USED - SEE https://github.com/emoncms/emoncms/blob/master/scripts/logger/
why that is I cannot say. But there doesn’t seem to be any mention of logrotate in the build guide or update script so I’m not at all sure how and when it’s implemented. Or why the files are even in the emoncms repo, the “rc.local” file that is in that same directory is not used, the latest rc.local is in the emonPi repo. So I can only guess the rc.local is added by the emoncms logger install script and then overwritten by the emonpi update script.
Absolutely. I agree. I would like to eventually separate out all the “read-only” stuff to a separate install script/guide so that users can implement read-only or not as they choose and everything for “read-only” is in one place.
[edit]
Found a ref to the logger install script in the “low-write” install guide
Hello,
I finally did not use postrotate feature of logrotate as is only runs if it effectively rotate at least one file. So you’re not sure there will be a backup every hour. So I just added an hourly cron which runs every hour.
As of now, here is what is done :
- At startup : Content of ~/emonpi/logtools/log.template and ~/data/logtools/log.template are copied over to /var/log/
- On shutdown : Structure (directories and files with no content) is copied to ~/data/logtools/log.template so it can be restored on next boot
- Every hour (crontab.hourly) : /var/log is archived to ~/data/logtools/log.archive
I put all scripts, service unit, templates… in a dedicated directory ~/emonpi/logtools
Everything is there as usual : emonpi/logtools at var-log-files · jsiobj/emonpi · GitHub
There are still a few things to do :
- Thinking about it, on startup, not only the structure but the whole content should be restored. Or I need to “rotate” the log archive to that it is not erased with a new archive of empty files
- I wrote a litlle install script but I don’t know where this should be run from.
emonpiupdatescript ? - Thinking (again) about all this… shouldn’t a persistent fs be used for log files ?
This would make it simple. On my dev pi, I added a usb key which I mounted on /var/log
I think this is to reduce the number of write cycles to the SD card.
Can the archiving be done as part of the rotate cron script so only one cron event is called? Is there a risk of the archiving and rotating scripts getting horribly out of sync.
Can you cascade logrotate? so rotate the hourly into a daily into a weekly?
I agree, but this is a difficult question as any new logs created during the prior boot up will be overwritten by either the blank file structure or the log files back up, I cannot see a way around that unless the logfiles are parsed, eg the pre-existing auto created logs content is appended to the end of the restored back up logs.
Again a tough one. I too like to see the logs persisted and what we are trying to achieve with backing up the ram logs every hour is a “low-write” persisted log, rather than several processes writing and rewriting log files in real time we are buffering the writes and “backing up” the logfiles periodically.
There is a risk here that something gets lost if the Pi is not shutdown gracefully in the 59th minuet of the hour, but the alternative is too increase the backup frequency to every 1min or 10 mins etc, which may add to the workload but the emoncms buffered write is already every 1min (AFAICR).
I would like to consider another feature for this logfile manager, and that is too have the option to add a remote location to stash logfiles, using rsync or similar, a duplicate set of logfiles could be maintained remotely, the frequenct that these are synced could be more frequent than the integral backups, thus less data lost in a crash instance.
This is a good option, I suggested this for the 3rd writable “data” partition on an emonPi, the advantage is that if the increased writes are fatal to the USB stick, it doesn’t effect the root filesystem, and It can be easily pulled out to backup or read in another PC.
But for now I think you have a good system evolving, just the initial log content to debate.
That issue could be worked around by using ramlog. Here’s an excerpt from the ramlog web page:
Ramlog act as a system daemon. On startup it creates ramdisk, it copies files from /var/log into ramdisk and mounts ramdisk as /var/log. All logs after that will be updated on ramdisk. Logs on harddrive are kept in folder /var/log.hdd which is updated when ramlog is restarted or stopped. On shutdown it saves log files back to harddisk so logs are consistent. Ramlog 2.x is using tmpfs by default, ramfs and kernel ramdisk are suppored as well. Program rsync is used for log synchronization.
1 Like
Good call Bill, I wonder why you haven’t mentioned it before 
Some previous mentions
https://openenergymonitor.org/forum-archive/node/10773.html
https://openenergymonitor.org/forum-archive/node/11803.html
and it is used in DietPi too
https://openenergymonitor.org/forum-archive/node/12384.html
it hasn’t had a warm reception when Bill has previously suggested it and I’m not entirely sure why, perhaps it was considered too complex for simply writing the files to RAM. But now we are considering adding all this functionality, perhaps we should take another look.
I recall previously I wasn’t overly happy about the fact I can find no changes to the code (or the website) since 2010 and the info is fairly limited. that doesn’t mean it’s a non-runner for me, but I like to see activity and peruse the repos/issues. Plus I prefer installing via a repo such as debian or PECL or PIP etc rather than private zip’s, again not a show stopper.
I have since seen this project
which is still active (last updated 4 days ago). I haven’t looked at either in-depth so I cannot say if they would work for us or which is more suitable (or are there any others?)
Perhaps we can learn something from those projects even if we don’t fully embrace any one of them.
[Edit] - Just having a nose around the log2ram repo and see there are a couple of familiar faces using it @Tim (Current rsync usage may cause excessive SD card writes · Issue #45 · azlux/log2ram · GitHub) and Peter Scargill (Mosquitto and Apache · Issue #18 · azlux/log2ram · GitHub)
I have attached a copy of ramlog in a zipped folder for easier viewing. There is a file in there called “copying” and it states it’s ok to copy/distribute ramlog but not to modify it, I wonder if that includes repackaging it for systemd? I can see how/why log2ram might have started.
ramlog-2.0.0.zip (25.7 KB)
1 Like
I think this is what DietPi uses by default.
Indeed it is. PB made mention of that a couple of posts above this one.
Yip, I spotted that too late … 
Thanks Paul. 'preciate it.
YVW!
Hello,
I thought adding this to the script but is is located in emoncms dir, not emonpi dir. So it would require also an update in the emoncms repo. I finally found it simpler to add a new cron entry.
At worst, if it runs just before the rotation, an hour worth of log may be lost in case of a crash. But, it is just logs… The backup frequency could be increased though as Paul suggest :
The other risk I see is rsync running at the very same time. I’ll check if this could be avoid (using some flag.)
Not sure to understand what you mean.
I agree… to both statement 
I find ramlog quite complex (more than a thousand line of shell script… way too much shell for me to read ) Also, it is a bit old and seems not to be suited for systemd (may be it could be adapted ?)
I just had a look at log2ram which is way simpler ! It makes use of mount --bindand mount --make-private. I’ll have a look of what this does precisely.
And finally, digging around, I found this : Setting up overlayFS on Raspberry Pi. May be an overlay fs would would work, using SD card FS as the “lower” FS and ramfs as “upper” FS. Yet, a “commit” needs to be run from time to time to write upper FS updates to the lower FS.Do you think overlayFS is something worth looking at ?
I think this has changed on later versions of logrotate. The most recent man page says
The directory must be on the same physical device as the log file being rotated, unless copy, copytruncate or renamecopy option is used. (my emphasis)
From the ChangeLog I think it changed in version 3.8.8. however, checking an older Jessie based Pi it looks like the logrotate version is only 3.8.7 (and this appears to be the latest version in this repo).
Hello Brian,
We could use this to archive log files on a persistent fs. But there is still the log file backup on shutdown and restore at boot time to take care of. I could not find any option to run a rotation on shutdown but instead of running a backup script, we could run logrotate with a specific configuration so that it archive existing logs.
What would be the “policy” about upgrading logrotate on emonpi ?
I had a deeper look at log2ram. I find it quite similar to what I’ve done so far (apart from the fact I am only restoring file structure and not the whole logs for now…)
One main difference with what I am doing is the way it uses file systems. The scripts manages the tmpfs. It is not mounted using fstab. First it “binds” /var/log (in root fs) to /var/hdd.log and make the bind private so that it can mount a tmpfs to /var/log and still see the content of the “hidden” /var/log (because of the tmpfs mount) through /var/hdd.log.
hmmmm… not sure I am clear…