Startup & systemd - a personal view

(Paul) #30

Ok, so what you are displaying there is the log messages created by systemd at start up, they are always in systemctl and the very latest have always been displayed with systemctl status.

What you do not see in your output is the logging created by the softwares themselves, a very different thing, that is going to /var/log for the services you displayed. This is not duplicate logging it is different logging in different places.

This discussion came up years ago about mosquitto and the mqtt_input services and it was decided to keep the 2 seperate as they are different things. Take emonhub for example, that produces a vast amount of helpful messages, therefore if this is going to journalctl rather than /var/log, the systemctl log messages are buried, you will need to use some filtering and regexes perhaps top find the systemd logs within the pages and pages of emonhub log messages. That’s if it’s still there, systemd will by default only keep upto 26mb of total logs, so vital boot up info will be quickly lost if every service is logging verbosely to that 26mb limit, yes it can be changed, but there is no need when the logs are kept separate as you demonstrate with your output, just the start and stops of the service(s) are displayed and retained as the faster moving log info is going to /var/log.

The start and stops are not displayed in my output above purely because the system has been up so long the logs have been rotated out (I only enabled persisting journalctl recently on this test server)

[email protected]:~$ systemctl status redis
â redis-server.service - Advanced key-value store
   Loaded: loaded (/lib/systemd/system/redis-server.service; enabled; vendor pre
set: enabled)
   Active: active (running) since Tue 2019-02-26 09:25:09 GMT; 2 week
s 4 days ago
 Main PID: 971 (redis-server)
    Tasks: 3 (limit: 4915)
   CGroup: /system.slice/redis-server.service
           ââ971 /usr/bin/redis-server

Warning: Journal has been rotated since unit was started. Log output is incomple
te or unavailable.
[email protected]:~$ sudo journalctl -u redis-server
-- No entries --
[email protected]:~$ sudo ls -la /var/log/redis
total 6180
drwxr-x---  2 redis redis    4096 Mar 10 06:25 .
drwxr-xr-x 11 root  root     4096 Mar 16 06:25 ..
-rw-rw----  1 redis redis 2899641 Mar 16 15:06 redis-server.log
-rw-rw----  1 redis redis  351060 Jan 14 06:24 redis-server.log.10.gz
-rw-rw----  1 redis redis  263921 Jan  6 06:24 redis-server.log.11.gz
-rw-rw----  1 redis redis  351542 Dec 31 06:24 redis-server.log.12.gz
-rw-rw----  1 redis redis  262797 Mar 10 06:24 redis-server.log.1.gz
-rw-rw----  1 redis redis  350019 Mar  4 06:24 redis-server.log.2.gz
-rw-rw----  1 redis redis  265519 Feb 24 06:24 redis-server.log.3.gz
-rw-rw----  1 redis redis  354475 Feb 18 06:24 redis-server.log.4.gz
-rw-rw----  1 redis redis   64552 Feb 10 06:24 redis-server.log.5.gz
-rw-rw----  1 redis redis  200074 Feb  8 19:29 redis-server.log.6.gz
-rw-rw----  1 redis redis  351475 Feb  4 06:24 redis-server.log.7.gz
-rw-rw----  1 redis redis  307853 Jan 27 06:24 redis-server.log.8.gz
-rw-rw----  1 redis redis  264428 Jan 20 06:24 redis-server.log.9.gz
[email protected]:~$

see the last line of the status

Warning: Journal has been rotated since unit was started. Log output is incomple
te or unavailable.

(Brian Orpin) #31

I think you should just drop Jessie support. It is ancient history.

(Dave Howorth) #32

I kind of sympathise with that view, in fact it was my first thought too. But then I noticed (I was actually looking up which was newer - jessie or stretch - I’m not a Debian expert, remember) that jessie is an LTS release that has support until 2020 so there is an argument for maintaining software that runs on it for that long too. But I don’t think there’s any need to enhance it. If somebody wants new features they should upgrade to stretch.

I suppose that Trystan et al have a better understanding of how many people are dependent on jessie for some reason.

(Paul) #33

I think that might be a moot point in this scope, if we are supporting Stretch as it currently stands, you automatically get support for Jessie, unless you opt for some very latest feature that isn’t in the underlying version of systemd used in Jessie. The question is whether we continue support for Jessie and/or Stretch with restarting in the rc.local as it’s already out there on the Stretch base emonSD .

(Dave Howorth) #34

You’re quite right; my apologies. I guess we’re back to plan A for the moment then. In which case …

The appropriate way to create tmpfs and directories therein is systemd-tmpfiles and that way all the directories/files are created before the services are started and there’s no need for serialized service restarts to add to the boot time.

I’ll keep looking for better alternatives.

(Dave Howorth) #35

But the whole premise of this thread at present is to find ways to eliminate rc.local, so that’s not really a question. Sure there’d need to be support for existing systems in some form.

(Paul) #36

Only if the logs and folders are lost at reboot. Using log2ram would mean that all the folders and logfiles are there with the right permissions and ownership prior to the services starting, so it works transparently. Aside from the log2ram (systemd) service needing to do it’s bit right up front, it works exactly the same as when the logs are on disc and never lost.

even when you turn on persisting the journalctl logs it would allow them to be rotated out of ram onto disc so you can have ram maintained logs to preserve/extend SDcard life AND persisted logs for a prolonged period without keeping them in RAM, with zero tweaking of any services or softwares.

I really cannot see any downside or reason to look else where. I’m not suggesting it wioll be perfect from the outset, but we can get involved in it’s development or maintain our own version. By adopting something that is already out there we are hitting the ground running and not trying to reinvent the wheel as we have been for the last 2-3 years.

(Paul) #37

Indeed it is, eliminating rc.local is long overdue and that is the larger part of the root issue (the current method of logging to ram). But there are already Stretch systems out there with the rc.local restarting in use. We cannot say we will no longer support Jessie, only Stretch in order to only support no rc.local setups. To support all existing Stretch images we need to rollout an upgrade or maintain multiple images. By supporting existing Stretch images, one way or another, we also support Jessie, theres nothing different (on this front) between Jessie or Stretch thus far. Only from here on will the stretch images be different so you cannot split it strictly Jessie vs Stretch.

(Dave Howorth) #38

Well, losing the logs is the status quo, so we haven’t lost anything. I agree log2ram looks interesting (specifically ) but in the main cases of a system crash or a power failure (we had one this morning, they happen regularly) it won’t help. And in any case, as I said, saving service logs over reboot is not as useful as saving system logs.

The point is that journald does low-write ram-buffering out of the box. So you set it to persist to disk and it manages the writes every five minutes by default. I wouldn’t want to trust my system logs to log2ram until I have a lot more experience of it. It’s just not standard.

(Paul) #39

I’m not sure why “it won’t help” it’s miles better than what we have, only the logs since the last “persist to disk” would be lost, currently everything is lost.

The system logs will be saved, if you create /var/log/journal then all the system logs will be “persisted” (as systemd see’s it) to RAM and then log2ram will treat those systemd logs as any other, persisting them to disk at our chosen frequency and rotating them out of RAM to disk-only at a regular interval as per usual logrotates.

That’s great but it doesn’t fix the non-systemd logging eg emoncms, so we end up with 2 partial solutions to one problem.

That would need to be somewhere other than /var/log as we still need to keep /var/log in ram for the non-systemd logs.

That can be said for what ever solution we may concoct here, but log2ram has been around far longer than anything we haven’t yet come up with!

So run it on a dev system, or in parallel to something you do trust, If we never try it, it will never reach a “trusted” state. It is not possible for it to be worse than we have! It is a starting block, we will never find an established open-source commercial solution for what we need, ramlog has been suggested a few times but it is too complex, log2ram’s appeal is it’s simplicity. Don’t think of it as an application, but a gathering of a few commands in a service to automate logfile management in ram (specifically file and folder “creation”), that way it isn’t a million miles from what you are seeking, you would put trust in a handful of bash and/or systemd commands you/we cobbled together, we would then iorn out any issues that crop up. Using log2ram is just jumping in with a head start based on some ready made code!

(Dave Howorth) #40

You’ve missed the point. journald already does the low write buffering of system logs, but most importantly it writes-through to disk immediately any emergency messages as a system crashes, overriding the low-write policy. That’s exactly what’s needed* and what log2ram doesn’t do.

  • well modulo those rare cases where you have to have a serially-connected console to capture the last kernel message.

log2ram does not persist logs in the case of a crash or power fail. journald does in the case of a crash and [I can think how to implement] could on a power fail. But system crash is the most important case. Power fails don’t usually need investigating, and service fails don’t usually need persistent logs.

(Dave Howorth) #41

I was doing a bit more investigation into how to switch applications to use the journal for logging to see how easy to use that would be. I decided that redis has a suitably small log to try so I investigated how it is set up.

But it clearly isn’t set up according to DigitalOcean’s instructions since that says it should be set up to use systemd to manage it, whilst on the emonSD it uses an init.d script. It’s also set to daemonize, which isn’t the default and isn’t mentioned on that web page. So where has the configuration come from? Has it been copied from jessie or somesuch? There’s also a warning in the log about 32-bit ness, although that doesn’t have any effect AFAICT.

(Paul) #42

You can rest assured, I understand fully what you are saying.

Don’t forget

What you are now proposing doesn’t eliminate rc.local. To eliminate rc.local we need the log file and folder structure to be in place before the services are started, log2ram does that. Transparently for all softwares, systemd or not, included on the emonSD by default or not, Jessie or Stretch, with no tweaking, maintenance or hardcoded lists. log2 ram fixes the issues brought about by having logs in ram. It fixes what is currently broken. By broken I mean what has been lost (from a vanilla OS) by moving the logs into ram to protect the sdcard. We get all our services to start first time without assistance and the normal logs get persisted across boots. We are back were the OS started before OEM moved the log files. Remember, the default position of a Debian based distro is that systemd loses all the journalctl logs on reboot.

So log2ram fixes the root issue that results in us losing all our standard logs and forces us to use rc.local to ensure services start.

With me so far?

What I was explaining above was that, to that fix you could then “just add /var/log/journal” so that we can also persist our system logs across a reboot, this is a bonus, it is a “standard bonus” that is brought about by just adding a folder, just once and it adds so much advantage because system logs are not lost on reboot. This simple advance has zero cost and will be managed by log2ram with no additional settings or coding etc because log2ram works transparently for all logs in /var/log.

Ok? Still with me?

Now, if you want to go one step further and use the inbuilt “low write buffering of system logs” that’s great, I think that would be valuable. It (alone) doesn’t replace rc.local or fix the root issue, log2ram does. But these do not need to be competing idea’s.

Great, all we need to do is allow this to bypass log2ram. either we can change the place it persists to because log2ram is managing /var/log, after all we need to change the config to enable this feature set OR we can maybe just set up a symlink so systemd thinks it is persisting to /var/log/journal when in fact it is actually persisting to /var/log.journal (to fit in with the syslog naming Ive been using) directly on the disc. log2ram would ensure that symlink is always present in the ram held /var/log no problem.

This adds to what is “standard” in Debian based distros and sounds great, it would work well with log2ram. log2ram fixes the root issues and on top of that we can indeed add the advanced system logging that’s not usually present in standard distro’s, so wasn’t really missed on the emonSD.

The addition of log2ram and even persisting system logs beyond a reboot requires no changes to systemd at all, except for adding the ‘/var/log/journal’ folder. everything up to this point remains stock/vanilla. But to add the advanced logging you now suggest would mean deviating from the stock settings. This isn’t a show stopper, it is just putting things into perspective.

Adding log2ram and persisting journal logs fixes the issues without deviating from stock configs. Then to add the additional feature of potentially catching crashes, does involve some deviation, but by the sounds of it, it could be worth doing. but it doesn’t negate the need for log2ram.

Does that make more sense?

(Paul) #43

systemd is totally backwards compatible. Many softwares are still init.d and systemd creates it’s own service units from the init.d script.

Do a sudo ls -la /etc/init.d/ to see all the major players all still use init.d scripts. mysql, redis, mosquitto, apache2 and many others are all still init.d and systemd takes it from there. That’s why I don’t understand this need to convert all our (init.d) services even when they already work perfectly well in systemd (especially when systemd is handicapped by rc.local). We have many bigger fish to fry.

(Brian Orpin) #44


(Dave Howorth) #45

This is all totally new and foreign to me. Could you give me a reference that explains how this behaviour works, please? I ask because suggests something different and because on my emonBase /lib/systemd/system/ is full of service units which according to are supplied by Debian packages.

(Dave Howorth) #46

Indeed, but I’m trying to discover why there’s a difference rather than just edit the file and hope for the best.

[edit]: actually, I wouldn’t edit the file; I’d create a drop-in to modify it in /etc/systemd/system/redis-server.service.d

[edit2]: I was confused. It’s the redis-server.conf I need to change, but that doesn’t work so there’s something more complicated.

(Paul) #47

I’m just passing through so this will be short[edit - so much for that idea!], I will answer more later. But, why don’t you create a simple init script with a unique name and see what happens when you enable it via systemd?

All the Debian packages for the services in question (mysql, redis etc) are still using init.d and a init script is installed by the package manager. systemd then generates the service units for those services as systemd will only start services via service units.

Did you try it? Offhand I do not know if you can override a “generated” service unit the same way as you can override a native service unit. I would hope you could, but I don’t know, perhaps since it is a init.d script it is limited to the config of that init.d script ie /etc/default/servicename?

Regardless, lets pretend all services (mysql, redis etc) are already native systemd just for a moment. The route you are engaging would mean that every service would require a overriding edited service unit file and any new packages installed by the developers or by the users, will require tweaking, new softwares would possibly fail when first installed, until such time as a edited service unit was in play. This might not be obvious to the average user so it could spell out problems.

And again! even if all sytemd services (and init.d services) were edited so as not to use a log file and put everything to jounalctl. We still have non systemd stuff like emoncms etc that need to be catered for. it will be a messy and incomplete solution.

It will also potentially cause other issues, for example, if all logs are going to journalctl, that’s a lot of traffic, will the “persist in emergency” you suggest still work? I know it will still try to work, but how succesful would it be trying to persiste so much data in a crash situation? It could probably handle just the systemd logs no problem. Plus with all these edits to init.d generated systemd service units, what happens when these services are eventually converted to systemd by the package maintainers, will our edits block the upgrades being pulled in or will our edits get overridden and break things?

For example. (as of a couple onf months ago, haven’t checked recently) the mosquitto packages on Debian were all init.d versions, but the latest version at the mosquitto repos was systemd, they were having issues with it so it wasn’t making mainstream release, but one day it will, where would that leave our mosquito service unit edits? we have no control over when a user will do a apt-get dist-upgrade to tie that in with a patch via the emonpi update routine

To avoid all the individual edits the solution needs to be transparent and work behind the scenes so that all packages and softwares can operate without change, this facilitates easy install and easy maintainance, plus there are no surprised eg when a user googles “where’s my mysql error messages”, “in /var/log” is still a true and relative answer. There’s a lot to be said for trying to stick to existing configurations.

Someone recently said something like

which I agree with entirely, but IMO that would also extend to using stock configurations and setups where ever possible. When you start asking for help and googling stuff, it is very useful if helper and helpee are on the same page. Sometimes it’s hard to recall or overlook things are not stock and certain general advice in blogs and wiki’s etc, will not be aimed at the more exotic configurations.

(Brian Orpin) #48

Not strictly true, look at the different output for redis and mosquitto - redis loads service file, and mosquitto does what you say and is running init.d script via sysv generator). I suspect the redis entry in init.d is for backwards compatibility with anything that calls it directly.

[email protected]:~ $ systemctl status redis-server.service
● redis-server.service - Advanced key-value store
   Loaded: loaded (/lib/systemd/system/redis-server.service; disabled; vendor preset: enabled)
   Active: active (running) since Sun 2019-03-17 14:54:53 UTC; 8min ago
 Main PID: 889 (redis-server)
   CGroup: /system.slice/redis-server.service
           └─889 /usr/bin/redis-server

Mar 17 14:54:52 emonpi systemd[1]: Starting Advanced key-value store...
Mar 17 14:54:52 emonpi run-parts[881]: run-parts: executing /etc/redis/redis-server.pre-up.d/00_example
Mar 17 14:54:53 emonpi run-parts[890]: run-parts: executing /etc/redis/
Mar 17 14:54:53 emonpi systemd[1]: Started Advanced key-value store.
Mar 17 15:00:30 emonpi systemd[1]: [/lib/systemd/system/redis-server.service:14] Unknown lvalue 'RunTimeDirectory' in section 'Service'

[email protected]:~ $ systemctl status mosquitto.service
● mosquitto.service - LSB: mosquitto MQTT v3.1 message broker
   Loaded: loaded (/etc/init.d/mosquitto; generated; vendor preset: enabled)
   Active: active (running) since Sun 2019-03-17 14:54:53 UTC; 8min ago
     Docs: man:systemd-sysv-generator(8)
   CGroup: /system.slice/mosquitto.service
           └─916 /usr/sbin/mosquitto -c /etc/mosquitto/mosquitto.conf

Mar 17 14:54:53 emonpi systemd[1]: Starting LSB: mosquitto MQTT v3.1 message broker...
Mar 17 14:54:53 emonpi mosquitto[908]: Starting network daemon:: mosquitto.
Mar 17 14:54:53 emonpi systemd[1]: Started LSB: mosquitto MQTT v3.1 message broker.

[edit] actually looks like mosquitto is the only one.

(Brian Orpin) #49

Strictly, these should all be systemctl commands (not that it appears to make any difference - service must default to systemd as the controller).

I had not noticed Paul had referenced this post;

I think this is an elegant solution and solves the issue of other things being installed on the Pi.

There is an open dev branch and prospective PR