Persistent journald logs

I know there has been a lot of discussion about logging here, so much that although I’ve done a cursory search I admit I haven’t read all the threads so apologies if what I’m posting is a duplication or is no longer relevant.

Anyway I thought I’d just post details of a simple change I made to my system to record it for myself and so that others can use it if they wish.

On my system, which is still 9.9.8 low-write, /var/log is a tmpfs so logs don’t cause excessive writes to the SD card. With the log level settings I have and the standard logrotate config I don’t have problems with the logs becoming too big that some have experienced.

So I’m happy with the system with one exception. systemd-journald also logs into this tmpfs. Now systemd will happily log into a tmpfs under /run/log/journal and is designed to log into a persistent filesystem under /var/log/journal if a permanent record is desired. I would like a persistent journal in case there are problems when booting or shutting down, or other unexpected crashes.

Since it’s not possible to reconfigure journald’s log location, we need to make that location persistent. So here’s what I did:

  1. Made a directory /home/pi/data/journald.logs
  2. Set ownership to root:systemd-journal and modes to drwxr-sr-x
  3. Made a directory /var/log/journal
  4. Added a line to /etc/fstab:
    /home/pi/data/journald.logs /var/log/journal none defaults,bind 0 0
  5. sudo mount /var/log/journal
  6. sudo systemctl restart journald.service

I believe that in more recent systems /home/pi/data is no longer where data is stored, so it would be more appropriate to put it in /var/lib. I think there’s an error in the current setup in that there are separate directories under /var/lib for phpfina and phptimeseries. I think there should instead be a directory /var/lib/emoncms and the individual data directories and the journald logs should be placed under that. Applications keep their state information together in subdirectories.

Possibly journald logs would be better somewhere else - I’ll try to find out. Note that journald already has features to minimise writes and limit file sizes and suchlike, so there should be no worries about writing persistent logs to the SD card.

Hi,
I am struggling in making the journald persistent. I tried your method, but when I try to mount /var/log/journal, it returns that it is not a directory, weird.
The following reboot was unsuccesful and I had to write the image onto the sd card again. I tried it twice with same result.

I also tried to simply change the /etc/systemd/journald.conf file, by setting the field “Storage” to “persistent”. It seems to be a widely used method out there. The configuration gets applied correctly as per the “systemctl status systemd-journald” command results. But after reboot, the journalctl only shows data since current boot.
Any ideas?

Sorry I don’t know how much you know about linux so I don’t know how to pitch my response. Could you post exactly what you did, and what the responses were? Did you follow my method exactly (indeed did I post an exact method, I don’t know for sure)!

It’s perhaps worth saying that mount has a -v (verbose) option that makes it explain any errors in more detail

Changing the /etc/systemd/journald.conf file won’t affect anything if the underlying filesystem is a tmpfs. That’s why you have to make sure it is somewhere permanent.

Hi Dave, I managed to get it working, I think I was missing a “d” in one of he “journald”, it is silly. Anyway thanks. Now I have persistent logs, it is a first step.
Cheers

Well done! Now we just have to hope that it shows something useful in the logs. :grin:

Quick thought, without any form of pruning, the logs may well just grow and eventually fill the SD card. Worth keeping an eye on. It partly depends on the default settings used when journalctl was compiled.

This discussion will show how to check the defaults emonSD next steps: log2ram - #29 by borpin, and this how to change the max file size systemd/Journal - ArchWiki. Best way it to create a drop-in so future system updates will not overwrite your settings.

The problem with this is the /val/log tree is mounted to tempfs (as you found out). @djh method uses the power of the system in that leaving the setting for storage at auto, journalctl checks for the folder /var/log/journal at boot and if it finds it, uses persistent storage.

However, it is better to create a drop-in for changes to configs and check on the max file size & number of files combo.

[edit]
By default, SystemMaxUse= is 10% of partition. SystemMaxFileSize= defaults to 1/8th of SystemMaxUse=. However, I’m not convinced that these files will automatically be rotated as the max number of files defaults to 100!

Thanks for the research, Brian. However I don’t think there’s much to worry about so I certainly won’t be overriding the defaults on my system.

journald.conf is set to all defaults on my system, and I expect so on all raspbian systems. man journald.conf tells us:

       SystemMaxUse= and RuntimeMaxUse= control how much disk space the journal may use up at most.  SystemKeepFree= and RuntimeKeepFree= control how much disk space
       systemd-journald shall leave free for other uses.  systemd-journald will respect both limits and use the smaller of the two values.

       The first pair defaults to 10% and the second to 15% of the size of the respective file system, but each value is capped to 4G. If the file system is nearly
       full and either SystemKeepFree= or RuntimeKeepFree= are violated when systemd-journald is started, the limit will be raised to the percentage that is actually
       free. This means that if there was enough free space before and journal files were created, and subsequently something else causes the file system to fill up,
       journald will stop using more space, but it will not be removing existing files to reduce the footprint again, either.

Now /home/pi/data on my system is 11G and emoncms and journald together presently use 538M, so there’s a fair way to go before journald hits its 4G limit, and another 5G or so for emoncms to store data before I start having to worry.

journald is keeping the logs in files of 8M and there’s the limit of 100 files, so there’s another limit. And then there’s a limit of 8 files kept by default. I confess I don’t know which rule will eventually cause journald to discard some archived logs but I’m confident it will do that long before there’s any storage problem.

Good, I just wanted to point out that doing this is not without risk especially if the SD Card is smaller and an error state causes the amount of log messages to increase dramatically which has happened in the past.

As I understand it, the problems in the past have been with other logs, not journald?

That arch linux link you posted helpfully states:

If the journal is persistent (non-volatile), its size limit is set to a default value of 10% of the size of the underlying file system but capped at 4 GiB. For example, with /var/log/journal/ located on a 20 GiB partition, journal data may take up to 2 GiB. On a 50 GiB partition, it would max at 4 GiB.

Which makes it clear that there’s no risk of journald filling the system. At least some parts of systemd seem to be well thought out, and journald appears to be one of them.

As long as the storage isn’t filled with other stuff as the limit is a % of size not a % of free space. So there is not no risk, but I’ll agree, it is not a high risk strategy.

hi Dave, I see an issue with the plan. In the hypothetical case that can not boot from my SSD drive (because the pi crashed) I can not directly read the journal, of course. So I would need to boot from the SD card,mount the SSD and look for the files in it. I have tried to do that, but I see an empty /home/pi/data folder, where my journald.logs folder should be. All other folders in /home/pi/ have content.
When I boot back from the SSD, the jouranld.logs folder is right there.
Is this a permissions issue, any idea why this is?

I’m sorry but you haven’t given enough information about your system. You just mentioned that you have an SSD as well as the SD card, but you haven’t said how you have it configured, nor why hypothetically of course you wouldn’t be able to boot from it.

In particular, where do you expect the journals to be stored (i.e. directory name) and what filesystem is that directory a part of, and how is it mounted and where etc etc.

One thought is that perhaps you haven’t configured your SD card to mount the journald directory at the correct place?

oh yes, sure, let me put this a bit more into context… Actually I posted the issue I have in the following post, there you will see my file system. EmonPi crashes running OS on SSD - #8 by luisrodriguez7
I boot from the SD and then I have the operating system on the SSD. I have applied your proceudre in order to have persistent logs, so journals are stored in /home/pi/data/journald.logs
Here is my fstab:

pi@emonpi:~$ cat /etc/fstab
tmpfs           /tmp             tmpfs   nodev,nosuid,size=30M,mode=1777 0  0
tmpfs           /var/tmp         tmpfs   nodev,nosuid,size=1M,mode=1777  0  0
tmpfs           /var/log         tmpfs   nodev,nosuid,size=50M,mode=1777 0  0
tmpfs           /var/lib/dhcp    tmpfs   nodev,nosuid,size=1M,mode=1777  0  0
tmpfs           /var/lib/dhcpcd5 tmpfs   nodev,nosuid,size=1M,mode=1777  0  0
tmpfs           /var/lib/openhab tmpfs   nodev,nosuid,size=40M,mode=1777 0  0
#tmpfs           /var/lib/php/sessions    tmpfs nodev,nosuid,nofail,size=5M,mode=1777 0 0
proc            /proc            proc    defaults                        0  0
/dev/mmcblk0p1  /boot            vfat    defaults,noatime,nodiratime     0  2
/dev/disk/by-uuid/efbcb440-6661-4186-89cf-b7100fdddbbd /                ext4    defaults,ro,noatime,nodiratime,errors=remount-ro 0 1
/dev/disk/by-uuid/049fe126-0443-4393-98c5-40a043739bb8 /mnt             ext4    defaults,ro,noatime,nodiratime,errors=remount-ro 0 1
/dev/mmcblk0p3  /home/pi/data    ext2    defaults,rw,noatime,nodiratime,errors=remount-ro 0 2
/home/pi/data/journald.logs /var/log/journal none defaults,bind 0 0

Ah OK, I now see the links between the posts.

Can you post the output from

lsblk

It looks like home/pi/data is mounted on /dev/mmcblk0p3 which would be the normal setup for an EmonSD image. This means that log file entry is also still on the SD (so potentially increasing wear rates), but why it is not there after a reboot is a mystery. Do you check for it’s existence while the system is running?

Actually, as you are running an SSD, you can delete virtually all the tmpfs entries and just leave everything on the SSD. If you create the folder /var/log/journal, all the logs will be saved on the SSD. automatically. You also want to move /home/pi/data as well to the SSD.

I’m running Home Assistant on an SSD with a Raspbian image and the fstab looks like this.

pi@raspberrypi:~ $ cat /etc/fstab
proc            /proc           proc    defaults          0       0
PARTUUID=56cd6262-01  /boot           vfat    defaults          0       2
PARTUUID=56cd6262-02  /               ext4    defaults,noatime  0       1

The EmonSD image is not really designed for SSD/HDD operations. @TrystanLea perhaps that is something to be considered to be added to the EmonScripts as an option.

oh yes, I see! of course the /home/pi/data is still on the SD, I should read first before asking questions! and I confirm it is there when the system is running.

NAME        MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda           8:0    0 931.5G  0 disk
├─sda2        8:2    0 911.5G  0 part /mnt
└─sda1        8:1    0    20G  0 part /
mmcblk0     179:0    0  14.5G  0 disk
├─mmcblk0p2 179:2    0   3.4G  0 part
├─mmcblk0p3 179:3    0   3.9G  0 part /home/pi/data
└─mmcblk0p1 179:1    0    60M  0 part /boot