Community
OpenEnergyMonitor

Community

Startup & systemd - a personal view


(Paul) #37

Indeed it is, eliminating rc.local is long overdue and that is the larger part of the root issue (the current method of logging to ram). But there are already Stretch systems out there with the rc.local restarting in use. We cannot say we will no longer support Jessie, only Stretch in order to only support no rc.local setups. To support all existing Stretch images we need to rollout an upgrade or maintain multiple images. By supporting existing Stretch images, one way or another, we also support Jessie, theres nothing different (on this front) between Jessie or Stretch thus far. Only from here on will the stretch images be different so you cannot split it strictly Jessie vs Stretch.


(Dave Howorth) #38

Well, losing the logs is the status quo, so we haven’t lost anything. I agree log2ram looks interesting (specifically https://github.com/azlux/log2ram/blob/master/log2ram ) but in the main cases of a system crash or a power failure (we had one this morning, they happen regularly) it won’t help. And in any case, as I said, saving service logs over reboot is not as useful as saving system logs.

The point is that journald does low-write ram-buffering out of the box. So you set it to persist to disk and it manages the writes every five minutes by default. I wouldn’t want to trust my system logs to log2ram until I have a lot more experience of it. It’s just not standard.


(Paul) #39

I’m not sure why “it won’t help” it’s miles better than what we have, only the logs since the last “persist to disk” would be lost, currently everything is lost.

The system logs will be saved, if you create /var/log/journal then all the system logs will be “persisted” (as systemd see’s it) to RAM and then log2ram will treat those systemd logs as any other, persisting them to disk at our chosen frequency and rotating them out of RAM to disk-only at a regular interval as per usual logrotates.

That’s great but it doesn’t fix the non-systemd logging eg emoncms, so we end up with 2 partial solutions to one problem.

That would need to be somewhere other than /var/log as we still need to keep /var/log in ram for the non-systemd logs.

That can be said for what ever solution we may concoct here, but log2ram has been around far longer than anything we haven’t yet come up with!

So run it on a dev system, or in parallel to something you do trust, If we never try it, it will never reach a “trusted” state. It is not possible for it to be worse than we have! It is a starting block, we will never find an established open-source commercial solution for what we need, ramlog has been suggested a few times but it is too complex, log2ram’s appeal is it’s simplicity. Don’t think of it as an application, but a gathering of a few commands in a service to automate logfile management in ram (specifically file and folder “creation”), that way it isn’t a million miles from what you are seeking, you would put trust in a handful of bash and/or systemd commands you/we cobbled together, we would then iorn out any issues that crop up. Using log2ram is just jumping in with a head start based on some ready made code!


(Dave Howorth) #40

You’ve missed the point. journald already does the low write buffering of system logs, but most importantly it writes-through to disk immediately any emergency messages as a system crashes, overriding the low-write policy. That’s exactly what’s needed* and what log2ram doesn’t do.

  • well modulo those rare cases where you have to have a serially-connected console to capture the last kernel message.

log2ram does not persist logs in the case of a crash or power fail. journald does in the case of a crash and [I can think how to implement] could on a power fail. But system crash is the most important case. Power fails don’t usually need investigating, and service fails don’t usually need persistent logs.


(Dave Howorth) #41

I was doing a bit more investigation into how to switch applications to use the journal for logging to see how easy to use that would be. I decided that redis has a suitably small log to try so I investigated how it is set up.

But it clearly isn’t set up according to DigitalOcean’s instructions https://www.digitalocean.com/community/tutorials/how-to-install-and-secure-redis-on-debian-9 since that says it should be set up to use systemd to manage it, whilst on the emonSD it uses an init.d script. It’s also set to daemonize, which isn’t the default and isn’t mentioned on that web page. So where has the configuration come from? Has it been copied from jessie or somesuch? There’s also a warning in the log about 32-bit ness, although that doesn’t have any effect AFAICT.


(Paul) #42

You can rest assured, I understand fully what you are saying.

Don’t forget

What you are now proposing doesn’t eliminate rc.local. To eliminate rc.local we need the log file and folder structure to be in place before the services are started, log2ram does that. Transparently for all softwares, systemd or not, included on the emonSD by default or not, Jessie or Stretch, with no tweaking, maintenance or hardcoded lists. log2 ram fixes the issues brought about by having logs in ram. It fixes what is currently broken. By broken I mean what has been lost (from a vanilla OS) by moving the logs into ram to protect the sdcard. We get all our services to start first time without assistance and the normal logs get persisted across boots. We are back were the OS started before OEM moved the log files. Remember, the default position of a Debian based distro is that systemd loses all the journalctl logs on reboot.

So log2ram fixes the root issue that results in us losing all our standard logs and forces us to use rc.local to ensure services start.

With me so far?

What I was explaining above was that, to that fix you could then “just add /var/log/journal” so that we can also persist our system logs across a reboot, this is a bonus, it is a “standard bonus” that is brought about by just adding a folder, just once and it adds so much advantage because system logs are not lost on reboot. This simple advance has zero cost and will be managed by log2ram with no additional settings or coding etc because log2ram works transparently for all logs in /var/log.

Ok? Still with me?

Now, if you want to go one step further and use the inbuilt “low write buffering of system logs” that’s great, I think that would be valuable. It (alone) doesn’t replace rc.local or fix the root issue, log2ram does. But these do not need to be competing idea’s.

Great, all we need to do is allow this to bypass log2ram. either we can change the place it persists to because log2ram is managing /var/log, after all we need to change the config to enable this feature set OR we can maybe just set up a symlink so systemd thinks it is persisting to /var/log/journal when in fact it is actually persisting to /var/log.journal (to fit in with the syslog naming Ive been using) directly on the disc. log2ram would ensure that symlink is always present in the ram held /var/log no problem.

This adds to what is “standard” in Debian based distros and sounds great, it would work well with log2ram. log2ram fixes the root issues and on top of that we can indeed add the advanced system logging that’s not usually present in standard distro’s, so wasn’t really missed on the emonSD.

The addition of log2ram and even persisting system logs beyond a reboot requires no changes to systemd at all, except for adding the ‘/var/log/journal’ folder. everything up to this point remains stock/vanilla. But to add the advanced logging you now suggest would mean deviating from the stock settings. This isn’t a show stopper, it is just putting things into perspective.

Adding log2ram and persisting journal logs fixes the issues without deviating from stock configs. Then to add the additional feature of potentially catching crashes, does involve some deviation, but by the sounds of it, it could be worth doing. but it doesn’t negate the need for log2ram.

Does that make more sense?


(Paul) #43

systemd is totally backwards compatible. Many softwares are still init.d and systemd creates it’s own service units from the init.d script.

Do a sudo ls -la /etc/init.d/ to see all the major players all still use init.d scripts. mysql, redis, mosquitto, apache2 and many others are all still init.d and systemd takes it from there. That’s why I don’t understand this need to convert all our (init.d) services even when they already work perfectly well in systemd (especially when systemd is handicapped by rc.local). We have many bigger fish to fry.


(Brian Orpin) #44

TIMTOWTDI rules.


(Dave Howorth) #45

This is all totally new and foreign to me. Could you give me a reference that explains how this behaviour works, please? I ask because https://unix.stackexchange.com/questions/233468/how-does-systemd-use-etc-init-d-scripts/233581 suggests something different and because on my emonBase /lib/systemd/system/ is full of service units which according to https://wiki.debian.org/systemd are supplied by Debian packages.


(Dave Howorth) #46

Indeed, but I’m trying to discover why there’s a difference rather than just edit the file and hope for the best.

[edit]: actually, I wouldn’t edit the file; I’d create a drop-in to modify it in /etc/systemd/system/redis-server.service.d

[edit2]: I was confused. It’s the redis-server.conf I need to change, but that doesn’t work so there’s something more complicated.


(Paul) #47

I’m just passing through so this will be short[edit - so much for that idea!], I will answer more later. But, why don’t you create a simple init script with a unique name and see what happens when you enable it via systemd?

All the Debian packages for the services in question (mysql, redis etc) are still using init.d and a init script is installed by the package manager. systemd then generates the service units for those services as systemd will only start services via service units.

Did you try it? Offhand I do not know if you can override a “generated” service unit the same way as you can override a native service unit. I would hope you could, but I don’t know, perhaps since it is a init.d script it is limited to the config of that init.d script ie /etc/default/servicename?

Regardless, lets pretend all services (mysql, redis etc) are already native systemd just for a moment. The route you are engaging would mean that every service would require a overriding edited service unit file and any new packages installed by the developers or by the users, will require tweaking, new softwares would possibly fail when first installed, until such time as a edited service unit was in play. This might not be obvious to the average user so it could spell out problems.

And again! even if all sytemd services (and init.d services) were edited so as not to use a log file and put everything to jounalctl. We still have non systemd stuff like emoncms etc that need to be catered for. it will be a messy and incomplete solution.

It will also potentially cause other issues, for example, if all logs are going to journalctl, that’s a lot of traffic, will the “persist in emergency” you suggest still work? I know it will still try to work, but how succesful would it be trying to persiste so much data in a crash situation? It could probably handle just the systemd logs no problem. Plus with all these edits to init.d generated systemd service units, what happens when these services are eventually converted to systemd by the package maintainers, will our edits block the upgrades being pulled in or will our edits get overridden and break things?

For example. (as of a couple onf months ago, haven’t checked recently) the mosquitto packages on Debian were all init.d versions, but the latest version at the mosquitto repos was systemd, they were having issues with it so it wasn’t making mainstream release, but one day it will, where would that leave our mosquito service unit edits? we have no control over when a user will do a apt-get dist-upgrade to tie that in with a patch via the emonpi update routine

To avoid all the individual edits the solution needs to be transparent and work behind the scenes so that all packages and softwares can operate without change, this facilitates easy install and easy maintainance, plus there are no surprised eg when a user googles “where’s my mysql error messages”, “in /var/log” is still a true and relative answer. There’s a lot to be said for trying to stick to existing configurations.

Someone recently said something like

which I agree with entirely, but IMO that would also extend to using stock configurations and setups where ever possible. When you start asking for help and googling stuff, it is very useful if helper and helpee are on the same page. Sometimes it’s hard to recall or overlook things are not stock and certain general advice in blogs and wiki’s etc, will not be aimed at the more exotic configurations.


(Brian Orpin) #48

Not strictly true, look at the different output for redis and mosquitto - redis loads service file, and mosquitto does what you say and is running init.d script via sysv generator). I suspect the redis entry in init.d is for backwards compatibility with anything that calls it directly.

[email protected]:~ $ systemctl status redis-server.service
● redis-server.service - Advanced key-value store
   Loaded: loaded (/lib/systemd/system/redis-server.service; disabled; vendor preset: enabled)
   Active: active (running) since Sun 2019-03-17 14:54:53 UTC; 8min ago
     Docs: http://redis.io/documentation,
           man:redis-server(1)
 Main PID: 889 (redis-server)
   CGroup: /system.slice/redis-server.service
           └─889 /usr/bin/redis-server 127.0.0.1:6379

Mar 17 14:54:52 emonpi systemd[1]: Starting Advanced key-value store...
Mar 17 14:54:52 emonpi run-parts[881]: run-parts: executing /etc/redis/redis-server.pre-up.d/00_example
Mar 17 14:54:53 emonpi run-parts[890]: run-parts: executing /etc/redis/redis-server.post-up.d/00_example
Mar 17 14:54:53 emonpi systemd[1]: Started Advanced key-value store.
Mar 17 15:00:30 emonpi systemd[1]: [/lib/systemd/system/redis-server.service:14] Unknown lvalue 'RunTimeDirectory' in section 'Service'

[email protected]:~ $ systemctl status mosquitto.service
● mosquitto.service - LSB: mosquitto MQTT v3.1 message broker
   Loaded: loaded (/etc/init.d/mosquitto; generated; vendor preset: enabled)
   Active: active (running) since Sun 2019-03-17 14:54:53 UTC; 8min ago
     Docs: man:systemd-sysv-generator(8)
   CGroup: /system.slice/mosquitto.service
           └─916 /usr/sbin/mosquitto -c /etc/mosquitto/mosquitto.conf

Mar 17 14:54:53 emonpi systemd[1]: Starting LSB: mosquitto MQTT v3.1 message broker...
Mar 17 14:54:53 emonpi mosquitto[908]: Starting network daemon:: mosquitto.
Mar 17 14:54:53 emonpi systemd[1]: Started LSB: mosquitto MQTT v3.1 message broker.

[edit] actually looks like mosquitto is the only one.


(Brian Orpin) #49

Strictly, these should all be systemctl commands (not that it appears to make any difference - service must default to systemd as the controller).

[edit]
I had not noticed Paul had referenced this post;

I think this is an elegant solution and solves the issue of other things being installed on the Pi.

[edit2]
There is an open dev branch and prospective PR


(Dave Howorth) #50

Because I prefer documentation? Because I don’t believe that’s how it actually works (as it says in the links I posted).

I don’t believe this either. See for example the service files included at https://packages.debian.org/stretch/armhf/mariadb-server-10.1/filelist

And in https://wiki.debian.org/Init it says " In jessie and stretch, the default init system is systemd, but switching to sysvinit is supported."

Hence why I’d like to see where it’s documented that gives you a different idea.

[edit]:

[email protected]:~ $ ls -l /sbin/init
lrwxrwxrwx 1 root root 20 Jun 13  2018 /sbin/init -> /lib/systemd/systemd

(Paul) #51

That makes sense, I did think that the generated units were in /var/run and not in the usual location, the generator creates a temp unit file rather than creating an actual service file per se in the usual places.

That would also answer my pondering on what would happen if both existed.

Just to clarify, you have quoted me, quoting the rc.local. I’ve not said anything to the contrary, although I would say Strictly they should be neither “service” nor “systemctl” as they shouldn’t even be there!

It’s another partial solution.

I’m not sure what everyone has against the solution I put forward it is the simplest solution and the only one that fixes all the points whilst improving the logging position to boot. I think it’s time I reduced the wear and tear on my keyboard and stopped flogging the proverbial horse.

I can’t help you there I do not note every site I visit and there is so much bad info out there, I usually test things to prove one way or the other to establish what is actually fact regardless of what is documented.

There is a lot of indepth info in that post about runtime levels etc, I do not think it actually states specifically that this behaviour is not correct. But even if it does there are a lot of misinformed discussions out there and many discussions between people that are not experts in the matter, much like this one.

Well that’s understandable since it has already been disproved above!

Yep, I’ve read similar, but I have no experience of that at all.

I have not got my info from one place, it is an accumulation of info and an understanding that I have built over the years frequenting many sites and researching/resolving various issues. I have at no time claimed to be an expert, if any of us were experts, this would already of been sorted no doubt!

I was going to offer to convert a emonSD image to use log2ram so it could be assessed, but I can clearly see that no matter how powerful the argument for trying it, there no interest so I’m not going to bother.

My main interest here is to reduce the issues raised on the forum and to have better info available to assist with those issue and move the development forward. I do not use emonSD, my setups are totally stable, I don’t need to find a better solution for my own use, so I’ll leave you guys to it, good luck.


(Brian Orpin) #52

I’m not aginst it. Give me a set of instructions to do, and I’ll do it on my EmonPi and see what happens (especially seeing the other problem I have). Might kill 2 birds…

I’m interested.

Agreed

Me neither for my main setup as I never used the tempfs.


(Trystan Lea) #53

Going back a few posts :slight_smile:

The emonSD-26Oct17 is based on Jessie and I think it was evident from some of the related forum support questions after I missed adding the service restart to rc.local that there are many systems running this image that need to be considered in any upgrade process.

Ideally we can have a solution that does not break the emonSD-26Oct17 image. @glyn.hudson suggested adding a check to the update progress to stop the update on older images, which makes a lot of sense. I felt however in this case that the change required to allow emonSD-26Oct17 users to update was so minor that it would be good to support it. I can imagine however that as we go forward there will be a point where the changes will deviate beyond the point where its possible to run the same emonPi/base update script on emonSD-26Oct17 as on whatever the new image release is at that point.

It will be interesting to reconsider all of this as part of an effort to make the emonPi/base update procedure runnable on a wider variety of systems as we’ve discussed, rather than exclusively an emonSD development target.


(Trystan Lea) #54

Thanks @pb66 I’ve noted this for discussion with @glyn.hudson


(Trystan Lea) #55

Yes interested! Could you document a step by step guide taking the latest image and modifying it to use log2ram? @glyn.hudson and I can then follow along here and try it


(Dave Howorth) #56

And contrary to the impression I may have given, I too am interested. So please do. I will add more in a bit.