Persistent journald logs

Tags: #<Tag:0x00007f8deee0dcc8>

I know there has been a lot of discussion about logging here, so much that although I’ve done a cursory search I admit I haven’t read all the threads so apologies if what I’m posting is a duplication or is no longer relevant.

Anyway I thought I’d just post details of a simple change I made to my system to record it for myself and so that others can use it if they wish.

On my system, which is still 9.9.8 low-write, /var/log is a tmpfs so logs don’t cause excessive writes to the SD card. With the log level settings I have and the standard logrotate config I don’t have problems with the logs becoming too big that some have experienced.

So I’m happy with the system with one exception. systemd-journald also logs into this tmpfs. Now systemd will happily log into a tmpfs under /run/log/journal and is designed to log into a persistent filesystem under /var/log/journal if a permanent record is desired. I would like a persistent journal in case there are problems when booting or shutting down, or other unexpected crashes.

Since it’s not possible to reconfigure journald’s log location, we need to make that location persistent. So here’s what I did:

  1. Made a directory /home/pi/data/journald.logs
  2. Set ownership to root:systemd-journal and modes to drwxr-sr-x
  3. Made a directory /var/log/journal
  4. Added a line to /etc/fstab:
    /home/pi/data/journald.logs /var/log/journal none defaults,bind 0 0
  5. sudo mount /var/log/journal
  6. sudo systemctl restart journald.service

I believe that in more recent systems /home/pi/data is no longer where data is stored, so it would be more appropriate to put it in /var/lib. I think there’s an error in the current setup in that there are separate directories under /var/lib for phpfina and phptimeseries. I think there should instead be a directory /var/lib/emoncms and the individual data directories and the journald logs should be placed under that. Applications keep their state information together in subdirectories.

Possibly journald logs would be better somewhere else - I’ll try to find out. Note that journald already has features to minimise writes and limit file sizes and suchlike, so there should be no worries about writing persistent logs to the SD card.

I am struggling in making the journald persistent. I tried your method, but when I try to mount /var/log/journal, it returns that it is not a directory, weird.
The following reboot was unsuccesful and I had to write the image onto the sd card again. I tried it twice with same result.

I also tried to simply change the /etc/systemd/journald.conf file, by setting the field “Storage” to “persistent”. It seems to be a widely used method out there. The configuration gets applied correctly as per the “systemctl status systemd-journald” command results. But after reboot, the journalctl only shows data since current boot.
Any ideas?

Sorry I don’t know how much you know about linux so I don’t know how to pitch my response. Could you post exactly what you did, and what the responses were? Did you follow my method exactly (indeed did I post an exact method, I don’t know for sure)!

It’s perhaps worth saying that mount has a -v (verbose) option that makes it explain any errors in more detail

Changing the /etc/systemd/journald.conf file won’t affect anything if the underlying filesystem is a tmpfs. That’s why you have to make sure it is somewhere permanent.

Hi Dave, I managed to get it working, I think I was missing a “d” in one of he “journald”, it is silly. Anyway thanks. Now I have persistent logs, it is a first step.

Well done! Now we just have to hope that it shows something useful in the logs. :grin:

Quick thought, without any form of pruning, the logs may well just grow and eventually fill the SD card. Worth keeping an eye on. It partly depends on the default settings used when journalctl was compiled.

This discussion will show how to check the defaults emonSD next steps: log2ram, and this how to change the max file size Best way it to create a drop-in so future system updates will not overwrite your settings.

The problem with this is the /val/log tree is mounted to tempfs (as you found out). @djh method uses the power of the system in that leaving the setting for storage at auto, journalctl checks for the folder /var/log/journal at boot and if it finds it, uses persistent storage.

However, it is better to create a drop-in for changes to configs and check on the max file size & number of files combo.

By default, SystemMaxUse= is 10% of partition. SystemMaxFileSize= defaults to 1/8th of SystemMaxUse=. However, I’m not convinced that these files will automatically be rotated as the max number of files defaults to 100!

Thanks for the research, Brian. However I don’t think there’s much to worry about so I certainly won’t be overriding the defaults on my system.

journald.conf is set to all defaults on my system, and I expect so on all raspbian systems. man journald.conf tells us:

       SystemMaxUse= and RuntimeMaxUse= control how much disk space the journal may use up at most.  SystemKeepFree= and RuntimeKeepFree= control how much disk space
       systemd-journald shall leave free for other uses.  systemd-journald will respect both limits and use the smaller of the two values.

       The first pair defaults to 10% and the second to 15% of the size of the respective file system, but each value is capped to 4G. If the file system is nearly
       full and either SystemKeepFree= or RuntimeKeepFree= are violated when systemd-journald is started, the limit will be raised to the percentage that is actually
       free. This means that if there was enough free space before and journal files were created, and subsequently something else causes the file system to fill up,
       journald will stop using more space, but it will not be removing existing files to reduce the footprint again, either.

Now /home/pi/data on my system is 11G and emoncms and journald together presently use 538M, so there’s a fair way to go before journald hits its 4G limit, and another 5G or so for emoncms to store data before I start having to worry.

journald is keeping the logs in files of 8M and there’s the limit of 100 files, so there’s another limit. And then there’s a limit of 8 files kept by default. I confess I don’t know which rule will eventually cause journald to discard some archived logs but I’m confident it will do that long before there’s any storage problem.

Good, I just wanted to point out that doing this is not without risk especially if the SD Card is smaller and an error state causes the amount of log messages to increase dramatically which has happened in the past.

As I understand it, the problems in the past have been with other logs, not journald?

That arch linux link you posted helpfully states:

If the journal is persistent (non-volatile), its size limit is set to a default value of 10% of the size of the underlying file system but capped at 4 GiB. For example, with /var/log/journal/ located on a 20 GiB partition, journal data may take up to 2 GiB. On a 50 GiB partition, it would max at 4 GiB.

Which makes it clear that there’s no risk of journald filling the system. At least some parts of systemd seem to be well thought out, and journald appears to be one of them.

As long as the storage isn’t filled with other stuff as the limit is a % of size not a % of free space. So there is not no risk, but I’ll agree, it is not a high risk strategy.