Startup & systemd - a personal view

Emoncms
42

You can rest assured, I understand fully what you are saying.

Don’t forget

What you are now proposing doesn’t eliminate rc.local. To eliminate rc.local we need the log file and folder structure to be in place before the services are started, log2ram does that. Transparently for all softwares, systemd or not, included on the emonSD by default or not, Jessie or Stretch, with no tweaking, maintenance or hardcoded lists. log2 ram fixes the issues brought about by having logs in ram. It fixes what is currently broken. By broken I mean what has been lost (from a vanilla OS) by moving the logs into ram to protect the sdcard. We get all our services to start first time without assistance and the normal logs get persisted across boots. We are back were the OS started before OEM moved the log files. Remember, the default position of a Debian based distro is that systemd loses all the journalctl logs on reboot.

So log2ram fixes the root issue that results in us losing all our standard logs and forces us to use rc.local to ensure services start.

With me so far?

What I was explaining above was that, to that fix you could then “just add /var/log/journal” so that we can also persist our system logs across a reboot, this is a bonus, it is a “standard bonus” that is brought about by just adding a folder, just once and it adds so much advantage because system logs are not lost on reboot. This simple advance has zero cost and will be managed by log2ram with no additional settings or coding etc because log2ram works transparently for all logs in /var/log.

Ok? Still with me?

Now, if you want to go one step further and use the inbuilt “low write buffering of system logs” that’s great, I think that would be valuable. It (alone) doesn’t replace rc.local or fix the root issue, log2ram does. But these do not need to be competing idea’s.

Great, all we need to do is allow this to bypass log2ram. either we can change the place it persists to because log2ram is managing /var/log, after all we need to change the config to enable this feature set OR we can maybe just set up a symlink so systemd thinks it is persisting to /var/log/journal when in fact it is actually persisting to /var/log.journal (to fit in with the syslog naming Ive been using) directly on the disc. log2ram would ensure that symlink is always present in the ram held /var/log no problem.

This adds to what is “standard” in Debian based distros and sounds great, it would work well with log2ram. log2ram fixes the root issues and on top of that we can indeed add the advanced system logging that’s not usually present in standard distro’s, so wasn’t really missed on the emonSD.

The addition of log2ram and even persisting system logs beyond a reboot requires no changes to systemd at all, except for adding the ‘/var/log/journal’ folder. everything up to this point remains stock/vanilla. But to add the advanced logging you now suggest would mean deviating from the stock settings. This isn’t a show stopper, it is just putting things into perspective.

Adding log2ram and persisting journal logs fixes the issues without deviating from stock configs. Then to add the additional feature of potentially catching crashes, does involve some deviation, but by the sounds of it, it could be worth doing. but it doesn’t negate the need for log2ram.

Does that make more sense?

43

systemd is totally backwards compatible. Many softwares are still init.d and systemd creates it’s own service units from the init.d script.

Do a sudo ls -la /etc/init.d/ to see all the major players all still use init.d scripts. mysql, redis, mosquitto, apache2 and many others are all still init.d and systemd takes it from there. That’s why I don’t understand this need to convert all our (init.d) services even when they already work perfectly well in systemd (especially when systemd is handicapped by rc.local). We have many bigger fish to fry.

1 Like
44

TIMTOWTDI rules.

45

This is all totally new and foreign to me. Could you give me a reference that explains how this behaviour works, please? I ask because How does systemd use /etc/init.d scripts? - Unix & Linux Stack Exchange suggests something different and because on my emonBase /lib/systemd/system/ is full of service units which according to systemd - Debian Wiki are supplied by Debian packages.

46

Indeed, but I’m trying to discover why there’s a difference rather than just edit the file and hope for the best.

[edit]: actually, I wouldn’t edit the file; I’d create a drop-in to modify it in /etc/systemd/system/redis-server.service.d

[edit2]: I was confused. It’s the redis-server.conf I need to change, but that doesn’t work so there’s something more complicated.

47

I’m just passing through so this will be short[edit - so much for that idea!], I will answer more later. But, why don’t you create a simple init script with a unique name and see what happens when you enable it via systemd?

All the Debian packages for the services in question (mysql, redis etc) are still using init.d and a init script is installed by the package manager. systemd then generates the service units for those services as systemd will only start services via service units.

Did you try it? Offhand I do not know if you can override a “generated” service unit the same way as you can override a native service unit. I would hope you could, but I don’t know, perhaps since it is a init.d script it is limited to the config of that init.d script ie /etc/default/servicename?

Regardless, lets pretend all services (mysql, redis etc) are already native systemd just for a moment. The route you are engaging would mean that every service would require a overriding edited service unit file and any new packages installed by the developers or by the users, will require tweaking, new softwares would possibly fail when first installed, until such time as a edited service unit was in play. This might not be obvious to the average user so it could spell out problems.

And again! even if all sytemd services (and init.d services) were edited so as not to use a log file and put everything to jounalctl. We still have non systemd stuff like emoncms etc that need to be catered for. it will be a messy and incomplete solution.

It will also potentially cause other issues, for example, if all logs are going to journalctl, that’s a lot of traffic, will the “persist in emergency” you suggest still work? I know it will still try to work, but how succesful would it be trying to persiste so much data in a crash situation? It could probably handle just the systemd logs no problem. Plus with all these edits to init.d generated systemd service units, what happens when these services are eventually converted to systemd by the package maintainers, will our edits block the upgrades being pulled in or will our edits get overridden and break things?

For example. (as of a couple onf months ago, haven’t checked recently) the mosquitto packages on Debian were all init.d versions, but the latest version at the mosquitto repos was systemd, they were having issues with it so it wasn’t making mainstream release, but one day it will, where would that leave our mosquito service unit edits? we have no control over when a user will do a apt-get dist-upgrade to tie that in with a patch via the emonpi update routine

To avoid all the individual edits the solution needs to be transparent and work behind the scenes so that all packages and softwares can operate without change, this facilitates easy install and easy maintainance, plus there are no surprised eg when a user googles “where’s my mysql error messages”, “in /var/log” is still a true and relative answer. There’s a lot to be said for trying to stick to existing configurations.

Someone recently said something like

which I agree with entirely, but IMO that would also extend to using stock configurations and setups where ever possible. When you start asking for help and googling stuff, it is very useful if helper and helpee are on the same page. Sometimes it’s hard to recall or overlook things are not stock and certain general advice in blogs and wiki’s etc, will not be aimed at the more exotic configurations.

48

Not strictly true, look at the different output for redis and mosquitto - redis loads service file, and mosquitto does what you say and is running init.d script via sysv generator). I suspect the redis entry in init.d is for backwards compatibility with anything that calls it directly.

pi@emonpi:~ $ systemctl status redis-server.service
● redis-server.service - Advanced key-value store
   Loaded: loaded (/lib/systemd/system/redis-server.service; disabled; vendor preset: enabled)
   Active: active (running) since Sun 2019-03-17 14:54:53 UTC; 8min ago
     Docs: http://redis.io/documentation,
           man:redis-server(1)
 Main PID: 889 (redis-server)
   CGroup: /system.slice/redis-server.service
           └─889 /usr/bin/redis-server 127.0.0.1:6379

Mar 17 14:54:52 emonpi systemd[1]: Starting Advanced key-value store...
Mar 17 14:54:52 emonpi run-parts[881]: run-parts: executing /etc/redis/redis-server.pre-up.d/00_example
Mar 17 14:54:53 emonpi run-parts[890]: run-parts: executing /etc/redis/redis-server.post-up.d/00_example
Mar 17 14:54:53 emonpi systemd[1]: Started Advanced key-value store.
Mar 17 15:00:30 emonpi systemd[1]: [/lib/systemd/system/redis-server.service:14] Unknown lvalue 'RunTimeDirectory' in section 'Service'

pi@emonpi:~ $ systemctl status mosquitto.service
● mosquitto.service - LSB: mosquitto MQTT v3.1 message broker
   Loaded: loaded (/etc/init.d/mosquitto; generated; vendor preset: enabled)
   Active: active (running) since Sun 2019-03-17 14:54:53 UTC; 8min ago
     Docs: man:systemd-sysv-generator(8)
   CGroup: /system.slice/mosquitto.service
           └─916 /usr/sbin/mosquitto -c /etc/mosquitto/mosquitto.conf

Mar 17 14:54:53 emonpi systemd[1]: Starting LSB: mosquitto MQTT v3.1 message broker...
Mar 17 14:54:53 emonpi mosquitto[908]: Starting network daemon:: mosquitto.
Mar 17 14:54:53 emonpi systemd[1]: Started LSB: mosquitto MQTT v3.1 message broker.

[edit] actually looks like mosquitto is the only one.

49

Strictly, these should all be systemctl commands (not that it appears to make any difference - service must default to systemd as the controller).

[edit]
I had not noticed Paul had referenced this post;

I think this is an elegant solution and solves the issue of other things being installed on the Pi.

[edit2]
There is an open dev branch and prospective PR

50

Because I prefer documentation? Because I don’t believe that’s how it actually works (as it says in the links I posted).

I don’t believe this either. See for example the service files included at Debian -- Error

And in Init - Debian Wiki it says " In jessie and stretch, the default init system is systemd, but switching to sysvinit is supported."

Hence why I’d like to see where it’s documented that gives you a different idea.

[edit]:

pi@emonpi:~ $ ls -l /sbin/init
lrwxrwxrwx 1 root root 20 Jun 13  2018 /sbin/init -> /lib/systemd/systemd
51

That makes sense, I did think that the generated units were in /var/run and not in the usual location, the generator creates a temp unit file rather than creating an actual service file per se in the usual places.

That would also answer my pondering on what would happen if both existed.

Just to clarify, you have quoted me, quoting the rc.local. I’ve not said anything to the contrary, although I would say Strictly they should be neither “service” nor “systemctl” as they shouldn’t even be there!

It’s another partial solution.

I’m not sure what everyone has against the solution I put forward it is the simplest solution and the only one that fixes all the points whilst improving the logging position to boot. I think it’s time I reduced the wear and tear on my keyboard and stopped flogging the proverbial horse.

I can’t help you there I do not note every site I visit and there is so much bad info out there, I usually test things to prove one way or the other to establish what is actually fact regardless of what is documented.

There is a lot of indepth info in that post about runtime levels etc, I do not think it actually states specifically that this behaviour is not correct. But even if it does there are a lot of misinformed discussions out there and many discussions between people that are not experts in the matter, much like this one.

Well that’s understandable since it has already been disproved above!

Yep, I’ve read similar, but I have no experience of that at all.

I have not got my info from one place, it is an accumulation of info and an understanding that I have built over the years frequenting many sites and researching/resolving various issues. I have at no time claimed to be an expert, if any of us were experts, this would already of been sorted no doubt!

I was going to offer to convert a emonSD image to use log2ram so it could be assessed, but I can clearly see that no matter how powerful the argument for trying it, there no interest so I’m not going to bother.

My main interest here is to reduce the issues raised on the forum and to have better info available to assist with those issue and move the development forward. I do not use emonSD, my setups are totally stable, I don’t need to find a better solution for my own use, so I’ll leave you guys to it, good luck.

1 Like
52

I’m not aginst it. Give me a set of instructions to do, and I’ll do it on my EmonPi and see what happens (especially seeing the other problem I have). Might kill 2 birds…

I’m interested.

Agreed

Me neither for my main setup as I never used the tempfs.

53

Going back a few posts :slight_smile:

The emonSD-26Oct17 is based on Jessie and I think it was evident from some of the related forum support questions after I missed adding the service restart to rc.local that there are many systems running this image that need to be considered in any upgrade process.

Ideally we can have a solution that does not break the emonSD-26Oct17 image. @glyn.hudson suggested adding a check to the update progress to stop the update on older images, which makes a lot of sense. I felt however in this case that the change required to allow emonSD-26Oct17 users to update was so minor that it would be good to support it. I can imagine however that as we go forward there will be a point where the changes will deviate beyond the point where its possible to run the same emonPi/base update script on emonSD-26Oct17 as on whatever the new image release is at that point.

It will be interesting to reconsider all of this as part of an effort to make the emonPi/base update procedure runnable on a wider variety of systems as we’ve discussed, rather than exclusively an emonSD development target.

54

Thanks @pb66 I’ve noted this for discussion with @glyn.hudson

55

Yes interested! Could you document a step by step guide taking the latest image and modifying it to use log2ram? @glyn.hudson and I can then follow along here and try it

56

And contrary to the impression I may have given, I too am interested. So please do. I will add more in a bit.

57

I’ve discussed this with @glyn.hudson and we had a look at the log2ram repository. The ability to avoid the /var/log setup currently in rc.local as you describe @pb66 is really interesting. Is it possible to turn off the syncing to disk? We are keen to avoid writing the logs to disk, I think this has been discussed before. Is there a way the behaviour could be user selectable? will log2ram work with older distributions, /etc/init.d scripts etc?

Glyn has also just tested the logrotate script in the emoncms repository that we had running on the previous image emoncms/scripts/logger at master · emoncms/emoncms · GitHub (this was mistakenly left out in this current image). The script works well, rotates and deletes the logs but does of course require the rc.local entries.

We’ve also looked again at Better handling of files in tmpfs /var/log - #13 by pb66, the requirement for /var/log/directory/logfile.log structure seems to be a sticking issue. We also need to rotate /var/log/daemon.log and /var/log/syslog.

For the current emonpi image with the existing rc.local, solution 2 will fix the immediate problem of the logs filling up quickly. Glyn’s keen to merge that in to the update process in the interim so that we can then consider a better solution as part of the longer term plan

58

As somebody who wasn’t involved in any previous discussions and whose initial opinion would be the opposite, I’m keen to read these previous discussions, or see a precis of the arguments in favour of discarding logs (which logs, all logs?), if you can supply either.

59

I don’t really see why. We are not suggesting writing on a continual basis to the SDcard (at worst) but rotating the logs from tmpfs to a semi permanent storage location to aid debugging at a later time.

Logrotate can easily do that and keep the logs for a specified period of time.

60

No, that is the core of how it works, I guess you could use a really long sync interval eg daily, monthly etc but then if users install a package from apt-get and there is a powercut or the plug is pulled accidentally, it may fail to reboot properly as the newly created logfiles will not be there. By syncing hourly, as long as power isn’t interrupted within 1 hour max, the new logs and folders will be persisted, as they will at a controlled reboot also. The persistence to disc IS the advantage of using log2ram.

I’m with the others that this is a bad idea IMO. This is the coding and development equivalent of poking your eyes out and sticking you fingers in your ears.

It’s only a few MB every hour, it will not take up space, it will not wear the card and it will not take much resource. But it will provide enormous amount of help towards debugging and development.

Cast you mind back to the issues with rfm2pi/emonpi’s locking out (rfm’s needing re-initialising) and then all the issues with emonhub interfacer threading and emoncms mqtt that could have been fixed so much sooner if we had more logs available sooner.

So many users just instinctively reboot.

This thread (Reboot Required Every 20 Days Or So To Restore Logging - #31 by TrystanLea) is a year old and we cannot help because we have no info, each time a user reboots and then asks what could it have been, we have to guess rather than inspect the logs and give some proper answers.

61

Ok we will consider persisting the logs, we would like to check the write load and see what can be done to reduce the logging level where it is not needed. E.g only logging at warning or error and above. daemon.log is also mirrored in syslog.

@pb66 do you use rsync with log2ram? I see that option in the config. Do you also disable the mail option? seems like that would be sensible.