Sounds good. We could detect the required condition for the older rc.local and make that available where applicable. The update process could install the correct rc.local based on conditions.
It should be possible to avoid multiple images this way? correct me if im wrong.
I’ve just been advised about a journald feature I wasn’t aware of, which has changed how I suggest we proceed slightly.
SyncIntervalSec= The timeout before synchronizing journal files to disk. After syncing, journal files are placed in the OFFLINE state. Note that syncing is unconditionally done immediately after a log message of priority CRIT, ALERT or EMERG has been logged. This setting hence applies only to messages of the levels ERR, WARNING, NOTICE, INFO, DEBUG. The default timeout is 5 minutes.
If I’m reading that correctly, it means that journald will work exactly as we would like, straight out of the box. That is, if
/var/log/jounal exists, or we set its storage mode to persistent. Then journald will save everything to the disk/sdcard and it will only write to the device every five minutes, except in an emergency.
So all we then need to do is override the native logs of the various applications to somewhere like
I’ll do some experiments.
Ok, so what you are displaying there is the log messages created by systemd at start up, they are always in systemctl and the very latest have always been displayed with systemctl status.
What you do not see in your output is the logging created by the softwares themselves, a very different thing, that is going to /var/log for the services you displayed. This is not duplicate logging it is different logging in different places.
This discussion came up years ago about mosquitto and the mqtt_input services and it was decided to keep the 2 seperate as they are different things. Take emonhub for example, that produces a vast amount of helpful messages, therefore if this is going to journalctl rather than /var/log, the systemctl log messages are buried, you will need to use some filtering and regexes perhaps top find the systemd logs within the pages and pages of emonhub log messages. That’s if it’s still there, systemd will by default only keep upto 26mb of total logs, so vital boot up info will be quickly lost if every service is logging verbosely to that 26mb limit, yes it can be changed, but there is no need when the logs are kept separate as you demonstrate with your output, just the start and stops of the service(s) are displayed and retained as the faster moving log info is going to /var/log.
The start and stops are not displayed in my output above purely because the system has been up so long the logs have been rotated out (I only enabled persisting journalctl recently on this test server)
[email protected]:~$ systemctl status redis â redis-server.service - Advanced key-value store Loaded: loaded (/lib/systemd/system/redis-server.service; enabled; vendor pre set: enabled) Active: active (running) since Tue 2019-02-26 09:25:09 GMT; 2 week s 4 days ago Docs: http://redis.io/documentation, man:redis-server(1) Main PID: 971 (redis-server) Tasks: 3 (limit: 4915) CGroup: /system.slice/redis-server.service ââ971 /usr/bin/redis-server 127.0.0.1:6379 Warning: Journal has been rotated since unit was started. Log output is incomple te or unavailable. [email protected]:~$ sudo journalctl -u redis-server -- No entries -- [email protected]:~$ sudo ls -la /var/log/redis total 6180 drwxr-x--- 2 redis redis 4096 Mar 10 06:25 . drwxr-xr-x 11 root root 4096 Mar 16 06:25 .. -rw-rw---- 1 redis redis 2899641 Mar 16 15:06 redis-server.log -rw-rw---- 1 redis redis 351060 Jan 14 06:24 redis-server.log.10.gz -rw-rw---- 1 redis redis 263921 Jan 6 06:24 redis-server.log.11.gz -rw-rw---- 1 redis redis 351542 Dec 31 06:24 redis-server.log.12.gz -rw-rw---- 1 redis redis 262797 Mar 10 06:24 redis-server.log.1.gz -rw-rw---- 1 redis redis 350019 Mar 4 06:24 redis-server.log.2.gz -rw-rw---- 1 redis redis 265519 Feb 24 06:24 redis-server.log.3.gz -rw-rw---- 1 redis redis 354475 Feb 18 06:24 redis-server.log.4.gz -rw-rw---- 1 redis redis 64552 Feb 10 06:24 redis-server.log.5.gz -rw-rw---- 1 redis redis 200074 Feb 8 19:29 redis-server.log.6.gz -rw-rw---- 1 redis redis 351475 Feb 4 06:24 redis-server.log.7.gz -rw-rw---- 1 redis redis 307853 Jan 27 06:24 redis-server.log.8.gz -rw-rw---- 1 redis redis 264428 Jan 20 06:24 redis-server.log.9.gz [email protected]:~$
see the last line of the status
Warning: Journal has been rotated since unit was started. Log output is incomple te or unavailable.
I think you should just drop Jessie support. It is ancient history.
I kind of sympathise with that view, in fact it was my first thought too. But then I noticed (I was actually looking up which was newer - jessie or stretch - I’m not a Debian expert, remember) that jessie is an LTS release that has support until 2020 so there is an argument for maintaining software that runs on it for that long too. But I don’t think there’s any need to enhance it. If somebody wants new features they should upgrade to stretch.
I suppose that Trystan et al have a better understanding of how many people are dependent on jessie for some reason.
I think that might be a moot point in this scope, if we are supporting Stretch as it currently stands, you automatically get support for Jessie, unless you opt for some very latest feature that isn’t in the underlying version of systemd used in Jessie. The question is whether we continue support for Jessie and/or Stretch with restarting in the rc.local as it’s already out there on the Stretch base emonSD .
You’re quite right; my apologies. I guess we’re back to plan A for the moment then. In which case …
The appropriate way to create tmpfs and directories therein is systemd-tmpfiles https://jlk.fjfi.cvut.cz/arch/manpages/man/systemd-tmpfiles.8 and that way all the directories/files are created before the services are started and there’s no need for serialized service restarts to add to the boot time.
I’ll keep looking for better alternatives.
But the whole premise of this thread at present is to find ways to eliminate rc.local, so that’s not really a question. Sure there’d need to be support for existing systems in some form.
Only if the logs and folders are lost at reboot. Using log2ram would mean that all the folders and logfiles are there with the right permissions and ownership prior to the services starting, so it works transparently. Aside from the log2ram (systemd) service needing to do it’s bit right up front, it works exactly the same as when the logs are on disc and never lost.
even when you turn on persisting the journalctl logs it would allow them to be rotated out of ram onto disc so you can have ram maintained logs to preserve/extend SDcard life AND persisted logs for a prolonged period without keeping them in RAM, with zero tweaking of any services or softwares.
I really cannot see any downside or reason to look else where. I’m not suggesting it wioll be perfect from the outset, but we can get involved in it’s development or maintain our own version. By adopting something that is already out there we are hitting the ground running and not trying to reinvent the wheel as we have been for the last 2-3 years.
Indeed it is, eliminating rc.local is long overdue and that is the larger part of the root issue (the current method of logging to ram). But there are already Stretch systems out there with the rc.local restarting in use. We cannot say we will no longer support Jessie, only Stretch in order to only support no rc.local setups. To support all existing Stretch images we need to rollout an upgrade or maintain multiple images. By supporting existing Stretch images, one way or another, we also support Jessie, theres nothing different (on this front) between Jessie or Stretch thus far. Only from here on will the stretch images be different so you cannot split it strictly Jessie vs Stretch.
Well, losing the logs is the status quo, so we haven’t lost anything. I agree log2ram looks interesting (specifically https://github.com/azlux/log2ram/blob/master/log2ram ) but in the main cases of a system crash or a power failure (we had one this morning, they happen regularly) it won’t help. And in any case, as I said, saving service logs over reboot is not as useful as saving system logs.
The point is that journald does low-write ram-buffering out of the box. So you set it to persist to disk and it manages the writes every five minutes by default. I wouldn’t want to trust my system logs to log2ram until I have a lot more experience of it. It’s just not standard.
I’m not sure why “it won’t help” it’s miles better than what we have, only the logs since the last “persist to disk” would be lost, currently everything is lost.
The system logs will be saved, if you create /var/log/journal then all the system logs will be “persisted” (as systemd see’s it) to RAM and then log2ram will treat those systemd logs as any other, persisting them to disk at our chosen frequency and rotating them out of RAM to disk-only at a regular interval as per usual logrotates.
That’s great but it doesn’t fix the non-systemd logging eg emoncms, so we end up with 2 partial solutions to one problem.
That would need to be somewhere other than /var/log as we still need to keep /var/log in ram for the non-systemd logs.
That can be said for what ever solution we may concoct here, but log2ram has been around far longer than anything we haven’t yet come up with!
So run it on a dev system, or in parallel to something you do trust, If we never try it, it will never reach a “trusted” state. It is not possible for it to be worse than we have! It is a starting block, we will never find an established open-source commercial solution for what we need, ramlog has been suggested a few times but it is too complex, log2ram’s appeal is it’s simplicity. Don’t think of it as an application, but a gathering of a few commands in a service to automate logfile management in ram (specifically file and folder “creation”), that way it isn’t a million miles from what you are seeking, you would put trust in a handful of bash and/or systemd commands you/we cobbled together, we would then iorn out any issues that crop up. Using log2ram is just jumping in with a head start based on some ready made code!
You’ve missed the point. journald already does the low write buffering of system logs, but most importantly it writes-through to disk immediately any emergency messages as a system crashes, overriding the low-write policy. That’s exactly what’s needed* and what log2ram doesn’t do.
- well modulo those rare cases where you have to have a serially-connected console to capture the last kernel message.
log2ram does not persist logs in the case of a crash or power fail. journald does in the case of a crash and [I can think how to implement] could on a power fail. But system crash is the most important case. Power fails don’t usually need investigating, and service fails don’t usually need persistent logs.
I was doing a bit more investigation into how to switch applications to use the journal for logging to see how easy to use that would be. I decided that redis has a suitably small log to try so I investigated how it is set up.
But it clearly isn’t set up according to DigitalOcean’s instructions https://www.digitalocean.com/community/tutorials/how-to-install-and-secure-redis-on-debian-9 since that says it should be set up to use systemd to manage it, whilst on the emonSD it uses an init.d script. It’s also set to daemonize, which isn’t the default and isn’t mentioned on that web page. So where has the configuration come from? Has it been copied from jessie or somesuch? There’s also a warning in the log about 32-bit ness, although that doesn’t have any effect AFAICT.
You can rest assured, I understand fully what you are saying.
What you are now proposing doesn’t eliminate rc.local. To eliminate rc.local we need the log file and folder structure to be in place before the services are started, log2ram does that. Transparently for all softwares, systemd or not, included on the emonSD by default or not, Jessie or Stretch, with no tweaking, maintenance or hardcoded lists. log2 ram fixes the issues brought about by having logs in ram. It fixes what is currently broken. By broken I mean what has been lost (from a vanilla OS) by moving the logs into ram to protect the sdcard. We get all our services to start first time without assistance and the normal logs get persisted across boots. We are back were the OS started before OEM moved the log files. Remember, the default position of a Debian based distro is that systemd loses all the journalctl logs on reboot.
So log2ram fixes the root issue that results in us losing all our standard logs and forces us to use rc.local to ensure services start.
With me so far?
What I was explaining above was that, to that fix you could then “just add /var/log/journal” so that we can also persist our system logs across a reboot, this is a bonus, it is a “standard bonus” that is brought about by just adding a folder, just once and it adds so much advantage because system logs are not lost on reboot. This simple advance has zero cost and will be managed by log2ram with no additional settings or coding etc because log2ram works transparently for all logs in /var/log.
Ok? Still with me?
Now, if you want to go one step further and use the inbuilt “low write buffering of system logs” that’s great, I think that would be valuable. It (alone) doesn’t replace rc.local or fix the root issue, log2ram does. But these do not need to be competing idea’s.
Great, all we need to do is allow this to bypass log2ram. either we can change the place it persists to because log2ram is managing /var/log, after all we need to change the config to enable this feature set OR we can maybe just set up a symlink so systemd thinks it is persisting to
/var/log/journal when in fact it is actually persisting to
/var/log.journal (to fit in with the syslog naming Ive been using) directly on the disc. log2ram would ensure that symlink is always present in the ram held
/var/log no problem.
This adds to what is “standard” in Debian based distros and sounds great, it would work well with log2ram. log2ram fixes the root issues and on top of that we can indeed add the advanced system logging that’s not usually present in standard distro’s, so wasn’t really missed on the emonSD.
The addition of log2ram and even persisting system logs beyond a reboot requires no changes to systemd at all, except for adding the ‘/var/log/journal’ folder. everything up to this point remains stock/vanilla. But to add the advanced logging you now suggest would mean deviating from the stock settings. This isn’t a show stopper, it is just putting things into perspective.
Adding log2ram and persisting journal logs fixes the issues without deviating from stock configs. Then to add the additional feature of potentially catching crashes, does involve some deviation, but by the sounds of it, it could be worth doing. but it doesn’t negate the need for log2ram.
Does that make more sense?
systemd is totally backwards compatible. Many softwares are still init.d and systemd creates it’s own service units from the init.d script.
sudo ls -la /etc/init.d/ to see all the major players all still use init.d scripts. mysql, redis, mosquitto, apache2 and many others are all still init.d and systemd takes it from there. That’s why I don’t understand this need to convert all our (init.d) services even when they already work perfectly well in systemd (especially when systemd is handicapped by rc.local). We have many bigger fish to fry.
This is all totally new and foreign to me. Could you give me a reference that explains how this behaviour works, please? I ask because https://unix.stackexchange.com/questions/233468/how-does-systemd-use-etc-init-d-scripts/233581 suggests something different and because on my emonBase
/lib/systemd/system/ is full of service units which according to https://wiki.debian.org/systemd are supplied by Debian packages.
Indeed, but I’m trying to discover why there’s a difference rather than just edit the file and hope for the best.
: actually, I wouldn’t edit the file; I’d create a drop-in to modify it in /etc/systemd/system/redis-server.service.d
[edit2]: I was confused. It’s the redis-server.conf I need to change, but that doesn’t work so there’s something more complicated.