Since /var/log is mounted as tmpfs the contents including all subdirs are deleted on reboot. This is currently handled by having code in /etc/rc.local that recreates the directories and files that are required, and then restarting services.
There are several things I don’t like about this:
services being restarted unnecessarily
The list is liable to get out of date. for instance it creates /var/log/openhab, whereas OpenHAB2 requires /var/log/openhab2
some of the users do not exist until packages are installed. eg openhab. meaning you get errors when rc.local is run
it doesn’t handle other packages that the user might want to install
I think a better way would be to use tmpfiles.d and to automatically detect new files that are created in /var/log
It’s still a work in progress, but I’ve come up with a script that maintains a config in /usr/lib/tmpfiles.d for any new directories that are created in /var/log
This could be combined with a static config for the directories/files that are currently created by rc.local which would mean all the code that creates stuff in /var/log and restarts services could be removed.
#!/bin/sh
# Script that searches in /var/log for subdirectories and updates
# tmpfiles.d config file to ensure that all directories are recreated
# on boot
# This script could be called:
# * Manually after user has installed their own packages
# * During shutdown via systemd. see:
# https://superuser.com/questions/1016827/how-do-i-run-a-script-before-everything-else-on-shutdown-with-systemd
# * Automatically after installing packages via apt-get. see:
# https://www.cyberciti.biz/faq/debian-ubuntu-linux-hook-a-script-command-to-apt-get-upgrade-command/
# Requires /var/log and /usr/lib/tmpfiles.d
if [ ! -d /var/log ]; then
echo "/var/log does not exist"
exit 1;
fi
if [ ! -d /usr/lib/tmpfiles.d ]; then
echo "/usr/lib/tmpfiles.d does not exist"
exit 1;
fi
STATIC_CONF=/usr/lib/tmpfiles.d/emonpi.conf
DYNAMIC_CONF=/usr/lib/tmpfiles.d/emonpi-dynamic.conf
DYNAMIC_HEADER="# This file is part of emonpi
#
# Since /var/log is mounted as tmpfs, the contents do not persist across reboots.
#
# This file contains entries for directories in /var/log that have been discovered
# dynamically at runtime. For instance directories created by extra packages
# that have been installed by the user
#
# The update-tmpfiles-config.sh script will search /var/log and append entries
# to this file for any directories that exist that are not already managed by tmpfiles
# See tmpfiles.d(5) for details
# Type Path Mode UID GID Age Argument
"
# Make sure config files exist
# TODO: copy static file from /home/pi/emonpi/???
[ -f ${STATIC_CONF} ] || /usr/bin/touch ${STATIC_CONF}
[ -f ${DYNAMIC_CONF} ] || echo "${DYNAMIC_HEADER}" > ${DYNAMIC_CONF}
# Find any unknown directories that have been created in /var/log, and add
# dynamic entries for them
for d in `find /var/log -mindepth 1 -type d`; do
/bin/grep -q $d ${STATIC_CONF} ${DYNAMIC_CONF}
if [ $? -ne 0 ]; then
echo "Found new log dir in tmpfs mounted /var/log: $d"
/usr/bin/stat -c "d %n %#a %U %G" $d >>${DYNAMIC_CONF}
fi
done
After installing openhab2 and running the script, /usr/lib/tmpfiles.d/emonpi-dynamic.conf contains:
# This file is part of emonpi
#
# Since /var/log is mounted as tmpfs, the contents do not persist across reboots.
#
# This file contains entries for directories in /var/log that have been discovered
# dynamically at runtime. For instance directories created by extra packages
# that have been installed by the user
#
# The update-tmpfiles-config.sh script will search /var/log and append entries
# to this file for any directories that exist that are not already managed by tmpfiles
# See tmpfiles.d(5) for details
# Type Path Mode UID GID Age Argument
d /var/log/openhab2 0755 openhab openhab
d /var/log/apt 0755 root root
d /var/log/emonhub 0755 emonhub root
d /var/log/supervisor 0755 root root
d /var/log/mosquitto 0755 mosquitto mosquitto
d /var/log/logrotate 0755 pi pi
d /var/log/mysql 0755 mysql adm
d /var/log/apache2 0755 root adm
d /var/log/redis 0755 redis redis
d /var/log/emonpilcd 0755 pi root
I’ve just tested, this is a really tidy method to create /var/log tmpfs files.
I’ve created a dev branch and opend a PR on the emonPi repo to develop and test this:
So far, I’ve got your script in the emonPi repo emonpi/update-tmpfiles-config.sh and have tested running it manually to create /usr/lib/tmpfiles.d/emonpi-dynamic.conf. I then removed all the old file creating entires from rc.local and after a restart all the files were re-created and restarting servivces was not requires so startup was faster
As you say, the next step will be for this script to run automatically. Did you investigate either of these methods?
Ah I notice /var/log/emoncms.log was not picked up by the script. I’m guessing this is because it’s not in a folder. Is it possible for tmpfiles to create a file and set permissions?
I haven’t had a chance to look at automatic running of the script yet…
The script only looks for directories. My rationale for that is that most “well behaved” programs will create necessary files with the correct permissions as long as the directory exists. By including files, you end up with lots of unnecessary entries.
The script also checks for a emonpi.conf and will not add dynamic entries for anything in there. I think anything related to packages that are included as part of the standard emonsd should have entries in there, such that emonpi-dynamic.conf is empty unless the user manually adds their own packages (eg in my case openhab2)
If emoncms.log is required at start-up, you can add an entry to emonpi.conf
emonpi.conf should probably be included in /home/pi/emonpi and copied/linked to /usr/lib/tmpfiles.d That way it can be easily updated if required, as part of the standard update process
If a user installed package requires files to be created, I think it’s reasonable to consider that advanced usage, and require the user to create their own tmpfiles.d entries. It’s at least better than the current behaviour where the user has to create all directories as well (eg /var/log/openhab2)
I don’t think this should be limited to emonpi systems. Whilst that is a large proportion of installs, as it is non-destructive, there is no reason not to include it for all to use if they so wish.
AFAICS, you do not actually need the static conf file as the script will add any existing directories when it is run? The key would be to run it after any new install has taken place or prior to shutdown.
There was some discussion a while back on a script to save and restore the log files on a restart. Anyone remember that?
I also disliked the unnecessary restarting of services at boot although my solution wasn’t as nice as tmpfiles.d. Good work!
The file should be delivered into /etc/tmpfiles.d or should be delivered by deb.
While I was hacking, I noticed that if /etc/rc.local wasn’t restarting emonhub, it would fail to send data to emoncms.org because the kernel random number generator didn’t have enough entropy and so the HTTPS connection failed. The solution is obviously for emonhub to retry if it fails, but the error is deep enough in the system libraries that a workaround isn’t obvious and anyway it will surely just be fixed by the next OS upgrade. In the meantime I don’t have a neat solution, so please be aware of this issue.
If we ignore users installing their own packages for now, and just consider the standard emonpi image, I would consider it its own distribution (based on Debian)
From the tmpfiles.d man page:
Files in /etc/tmpfiles.d override files with the same name in /usr/lib/tmpfiles.d and /run/tmpfiles.d. Files in /run/tmpfiles.d override files with the same name in /usr/lib/tmpfiles.d. Packages should
install their configuration files in /usr/lib/tmpfiles.d. Files in /etc/tmpfiles.d are reserved for the local administrator, who may use this logic to override the configuration files installed by vendor
packages.
So, I think /usr/lib is the correct place for the the conf file(s).
Also, in that case, there is no need for the script, we can just ship /usr/lib/tmpfiles.d/emonpi.conf with all the required directories/files to handle /var/log being mounted as tmpfs.
Now, if we want to support (advanced) users installing their own packages, there are a few options:
provide script, and make it run automatically to detect new log files after packages are installed and/or during clean shutdown. In this case, I would consider the script to still be part of the emonpi “distribution” so I’d put the output in /usr/lib/tmpfiles.d
provide script to be run manually. In this case it could just output the config, and user could use it to update their own files in /etc/tmpfiles.d
Another longer term solution would be to provide upstream patches to packages to make them create their own /usr/lib/tmpfiles.d/<package>.conf when /var/log is mounted as tmpfs
I’ve tried log2ram and found it works really well, I have since modified a fork to improve things further by relocating the rotated files destination to disc rather than holding them in RAM.
Simply maintaining an on disk copy of the tmp log files/folders resolves many of the issues we face without overly complicating things.
All the files and folders are present at start up with the right structure,ownership and permission levels without maintaining any lists.
All logfiles are persisted across reboots as they are persisted at shutdown as well as hourly.
Works transparently with installing new software as any newly installed log files/folders will get copied within the hour or it can be triggered manually (although unnecessary if rebooting)
Plus
All the rotated files held on disc rather than in RAM reduces the RAM usage, minimizing any chance of the FS choking and a much longer history can be retained too, no longer would we need to debug within a couple of hours (without rebooting) in hope of finding some usable log entries before they are rotated out or lost in a reboot. The /var/log folder is much tidier too without the rotated logs cluttering the place up.
If there is any interest in reviewing or discussing this option too, We can start another thread.
My emonPi crashed sometime last night, it stopped logging and disconnected from wifi. Since the logs are all temporary, I don’t have any information to help debug this. I also only have about two hours’ history in the logs, which are absolutely full of very verbose debug logging from RFM2Pi and MQTT, and /var/log/syslog and /var/log/daemon are basically identical.
I would like to suggest the following:
Automatically rsync the logs to permanent storage every 5/15 minutes, e.g. from cron.
Restore from permanent storage at startup.
Compress rotated logs.
Disable the debug logging.
Route log messages to either /var/log/syslog or /var/log/daemon, not both.
I could submit a pull request if someone can point me to the correct source code on github, assuming that it’s not a deb package (I dislike debs intensely).
Hi @qris, have a search around on the forum and you will find why it has happened (change emonhub to be run by systemd) and a couple of fixes that can be applied (to rsyslog & logrotate, or to the service file).
The favoured route to sort this out permanently is to use Log2Ram, modify the service file and rework the Logrotate scheme.
I can definitely fix this on my own box now that I have SSH access, but that’s not my point. I think it creates a very bad user experience if most of the emonPis out there are crashing randomly and it’s just written off as a known problem without a solution. So I wanted to fix it for everyone, not just for me.
Log2ram seems controversial so I would not propose a solution that incorporates it right now. I’d favour simple changes to our default configurations to improve matters for everyone. It may make no difference to those who have already installed log2ram, but I’d guess they are the minority.
Agree with the first part but not the second. I think it is now accepted as a problem by @glyn.hudson & @TrystanLea and there is a solution (well a couple actually) it is just a solution has not yet been rolled out.
@borpin please could you point me to any known solution other than log2ram? I’m reluctant to install that one right now because it’s controversial, invasive and I’m an emonPi newbie (with ~20 years as a Unix developer) and do not want to wreck my new emonPi and lose my data.
I’m surprised that the developers are not placing more urgency on implementing some kind of fix or workaround in the standard image. In my opinion the priority of fixing this issue should be much higher than any new feature development.
Too much log data going into /var/log due to emonhub being switched to systemd and the logs not being cleared due to logrotation failing.
The simple and quick fix to the emonhub.log is to revert emonhub to use init.d rather than systemd. That way “normal” operation is resumed. But that breaks the emonhub log viewer in emoncms.
The “proper” way to fix the emonhub log is to correct the ExecStart line in the service unit to use the --logfile parameter in the command as intended. But this not only breaks the emoncms log viwer as above, but will also require further amendments (ie add emonhub to the rc.local folder creation and service restarts) as the service unit doesn’t create the log folder which the init.d does.
As for the logrotation part, there were some fixes rushed through that didn’t work, again this could be easily fixed in the short term by taking a couple of steps back.
The delay is now possibly because an all encompassing final solution is being sought rather than fixing the immediate problem. The delays to finding that total solution is predominantly due to people throwing in other partial solutions and raising concerns about the L2R solution.
We cannot debate L2R for an eternity AND not implement a hotfix in the interim, but that does seem to be the position.
Sorry Brian but that alone will not work as there is no folder created by the service unit. This is why I would not support the change to systemd until we had resolved the rc.local issue.
Maybe adding emonhub to the rc.local might work as a stop gap. But I do not know for sure what the ramifications might be for forcing a restart. It might well be ok, it might not.
Only if the logrotation remains broken.
I would prefer the rotation to be fixed (I know it wasn’t great when it did work, but it is essential to be working otherwise the logs will just grow and grow, rather than stopping within hours, it will stop after a few weeks maybe.
Yes, sorry, I’d forgotten I had added an rsyslog modification to the standard image which must create the right folders for you based on the service user - cannot see how else they are created for the emonhub user.
