Community
OpenEnergyMonitor

Community

Better handling of files in tmpfs /var/log


(Rohan Lloyd) #1

Since /var/log is mounted as tmpfs the contents including all subdirs are deleted on reboot. This is currently handled by having code in /etc/rc.local that recreates the directories and files that are required, and then restarting services.

There are several things I don’t like about this:

  • services being restarted unnecessarily
  • The list is liable to get out of date. for instance it creates /var/log/openhab, whereas OpenHAB2 requires /var/log/openhab2
  • some of the users do not exist until packages are installed. eg openhab. meaning you get errors when rc.local is run
  • it doesn’t handle other packages that the user might want to install

I think a better way would be to use tmpfiles.d and to automatically detect new files that are created in /var/log

It’s still a work in progress, but I’ve come up with a script that maintains a config in /usr/lib/tmpfiles.d for any new directories that are created in /var/log

This could be combined with a static config for the directories/files that are currently created by rc.local which would mean all the code that creates stuff in /var/log and restarts services could be removed.

#!/bin/sh

# Script that searches in /var/log for subdirectories and updates
# tmpfiles.d config file to ensure that all directories are recreated
# on boot

# This script could be called:
#   * Manually after user has installed their own packages
#   * During shutdown via systemd. see:
#     https://superuser.com/questions/1016827/how-do-i-run-a-script-before-everything-else-on-shutdown-with-systemd
#   * Automatically after installing packages via apt-get. see:
#     https://www.cyberciti.biz/faq/debian-ubuntu-linux-hook-a-script-command-to-apt-get-upgrade-command/

# Requires /var/log and /usr/lib/tmpfiles.d
if [ ! -d /var/log ]; then
  echo "/var/log does not exist"
  exit 1;
fi
if [ ! -d /usr/lib/tmpfiles.d ]; then
  echo "/usr/lib/tmpfiles.d does not exist"
  exit 1;
fi

STATIC_CONF=/usr/lib/tmpfiles.d/emonpi.conf
DYNAMIC_CONF=/usr/lib/tmpfiles.d/emonpi-dynamic.conf

DYNAMIC_HEADER="# This file is part of emonpi
#
# Since /var/log is mounted as tmpfs, the contents do not persist across reboots.
#
# This file contains entries for directories in /var/log that have been discovered
# dynamically at runtime. For instance directories created by extra packages
# that have been installed by the user
#
# The update-tmpfiles-config.sh script will search /var/log and append entries
# to this file for any directories that exist that are not already managed by tmpfiles

# See tmpfiles.d(5) for details

# Type Path    Mode UID  GID  Age Argument
"

# Make sure config files exist
# TODO: copy static file from /home/pi/emonpi/???
[ -f ${STATIC_CONF}  ] || /usr/bin/touch ${STATIC_CONF}

[ -f ${DYNAMIC_CONF} ] || echo "${DYNAMIC_HEADER}" > ${DYNAMIC_CONF}

# Find any unknown directories that have been created in /var/log, and add
# dynamic entries for them
for d in `find /var/log -mindepth 1 -type d`; do
  /bin/grep -q $d ${STATIC_CONF} ${DYNAMIC_CONF}
  if [ $? -ne 0 ]; then
    echo "Found new log dir in tmpfs mounted /var/log: $d"
    /usr/bin/stat -c "d %n %#a %U %G" $d >>${DYNAMIC_CONF}
  fi
done

(Rohan Lloyd) #2

After installing openhab2 and running the script, /usr/lib/tmpfiles.d/emonpi-dynamic.conf contains:

# This file is part of emonpi
#
# Since /var/log is mounted as tmpfs, the contents do not persist across reboots.
#
# This file contains entries for directories in /var/log that have been discovered
# dynamically at runtime. For instance directories created by extra packages
# that have been installed by the user
#
# The update-tmpfiles-config.sh script will search /var/log and append entries
# to this file for any directories that exist that are not already managed by tmpfiles

# See tmpfiles.d(5) for details

# Type Path    Mode UID  GID  Age Argument

d /var/log/openhab2 0755 openhab openhab
d /var/log/apt 0755 root root
d /var/log/emonhub 0755 emonhub root
d /var/log/supervisor 0755 root root
d /var/log/mosquitto 0755 mosquitto mosquitto
d /var/log/logrotate 0755 pi pi
d /var/log/mysql 0755 mysql adm
d /var/log/apache2 0755 root adm
d /var/log/redis 0755 redis redis
d /var/log/emonpilcd 0755 pi root

(Glyn Hudson) #3

That looks like a really nice idea, creating log file folders in rc.local is a pain. I’ll give your script a test next week.


(Glyn Hudson) #4

I’ve just tested, this is a really tidy method to create /var/log tmpfs files.

I’ve created a dev branch and opend a PR on the emonPi repo to develop and test this:

So far, I’ve got your script in the emonPi repo emonpi/update-tmpfiles-config.sh and have tested running it manually to create /usr/lib/tmpfiles.d/emonpi-dynamic.conf. I then removed all the old file creating entires from rc.local and after a restart all the files were re-created and restarting servivces was not requires so startup was faster :smiley:

As you say, the next step will be for this script to run automatically. Did you investigate either of these methods?


(Glyn Hudson) #6

Ah I notice /var/log/emoncms.log was not picked up by the script. I’m guessing this is because it’s not in a folder. Is it possible for tmpfiles to create a file and set permissions?


(Brian Orpin) #7

Or is it telling you it should really be in a folder :smile:


(Glyn Hudson) #8

Thanks


(Rohan Lloyd) #9

I haven’t had a chance to look at automatic running of the script yet…

The script only looks for directories. My rationale for that is that most “well behaved” programs will create necessary files with the correct permissions as long as the directory exists. By including files, you end up with lots of unnecessary entries.

The script also checks for a emonpi.conf and will not add dynamic entries for anything in there. I think anything related to packages that are included as part of the standard emonsd should have entries in there, such that emonpi-dynamic.conf is empty unless the user manually adds their own packages (eg in my case openhab2)

If emoncms.log is required at start-up, you can add an entry to emonpi.conf

emonpi.conf should probably be included in /home/pi/emonpi and copied/linked to /usr/lib/tmpfiles.d That way it can be easily updated if required, as part of the standard update process

If a user installed package requires files to be created, I think it’s reasonable to consider that advanced usage, and require the user to create their own tmpfiles.d entries. It’s at least better than the current behaviour where the user has to create all directories as well (eg /var/log/openhab2)


(Brian Orpin) #10

:laughing: :laughing::laughing::laughing:

I don’t think this should be limited to emonpi systems. Whilst that is a large proportion of installs, as it is non-destructive, there is no reason not to include it for all to use if they so wish.

AFAICS, you do not actually need the static conf file as the script will add any existing directories when it is run? The key would be to run it after any new install has taken place or prior to shutdown.

There was some discussion a while back on a script to save and restore the log files on a restart. Anyone remember that?


(Bruce) #11

I also disliked the unnecessary restarting of services at boot although my solution wasn’t as nice as tmpfiles.d. Good work!

The file should be delivered into /etc/tmpfiles.d or should be delivered by deb.

While I was hacking, I noticed that if /etc/rc.local wasn’t restarting emonhub, it would fail to send data to emoncms.org because the kernel random number generator didn’t have enough entropy and so the HTTPS connection failed. The solution is obviously for emonhub to retry if it fails, but the error is deep enough in the system libraries that a workaround isn’t obvious and anyway it will surely just be fixed by the next OS upgrade. In the meantime I don’t have a neat solution, so please be aware of this issue.

Bruce


(Rohan Lloyd) #12

If we ignore users installing their own packages for now, and just consider the standard emonpi image, I would consider it its own distribution (based on Debian)

From the tmpfiles.d man page:

Files in /etc/tmpfiles.d override files with the same name in /usr/lib/tmpfiles.d and /run/tmpfiles.d. Files in /run/tmpfiles.d override files with the same name in /usr/lib/tmpfiles.d. Packages should
install their configuration files in /usr/lib/tmpfiles.d. Files in /etc/tmpfiles.d are reserved for the local administrator, who may use this logic to override the configuration files installed by vendor
packages.

So, I think /usr/lib is the correct place for the the conf file(s).

Also, in that case, there is no need for the script, we can just ship /usr/lib/tmpfiles.d/emonpi.conf with all the required directories/files to handle /var/log being mounted as tmpfs.

Now, if we want to support (advanced) users installing their own packages, there are a few options:

  1. provide script, and make it run automatically to detect new log files after packages are installed and/or during clean shutdown. In this case, I would consider the script to still be part of the emonpi “distribution” so I’d put the output in /usr/lib/tmpfiles.d
  2. provide script to be run manually. In this case it could just output the config, and user could use it to update their own files in /etc/tmpfiles.d

Another longer term solution would be to provide upstream patches to packages to make them create their own /usr/lib/tmpfiles.d/<package>.conf when /var/log is mounted as tmpfs


(Paul) #13

The one that springs to mind is the Mosquitto won't start on boot after raspbian and emonsd update thread, but this topic has come up many times so there will be no doubt be several more.

I’ve tried log2ram and found it works really well, I have since modified a fork to improve things further by relocating the rotated files destination to disc rather than holding them in RAM.

Simply maintaining an on disk copy of the tmp log files/folders resolves many of the issues we face without overly complicating things.

  • All the files and folders are present at start up with the right structure,ownership and permission levels without maintaining any lists.
  • All logfiles are persisted across reboots as they are persisted at shutdown as well as hourly.
  • Works transparently with installing new software as any newly installed log files/folders will get copied within the hour or it can be triggered manually (although unnecessary if rebooting)

Plus

  • All the rotated files held on disc rather than in RAM reduces the RAM usage, minimizing any chance of the FS choking and a much longer history can be retained too, no longer would we need to debug within a couple of hours (without rebooting) in hope of finding some usable log entries before they are rotated out or lost in a reboot. The /var/log folder is much tidier too without the rotated logs cluttering the place up.

If there is any interest in reviewing or discussing this option too, We can start another thread.