Startup & systemd - a personal view

(Dave Howorth) #1

As some will know, I’m in process of making emoncms work on an openSUSE system. As such I’ve taken an interest in various threads on this forum concerned with startup issues. I thought I’d write down some of where my head is at, because it might be of interest to others.

I use openSUSE through a historical accident, though I’ve grown to appreciate the responsiveness of some of its support processes. Maybe it just suits my mindset. I’ve previously used Debian and Ubuntu and didn’t get on as well with their support systems. If I was to move from openSUSE, it would probably be to Arch because their documentation often comes up in searches and is excellent.

That’s enough about me, the point here is that openSUSE is well along the path to a systemd world, and Arch is even further, having abandoned the sysVinit-compatibility shims altogether. Debian/Ubuntu/Raspbian is a bit behind the curve here and emoncms is even further behind, IMHO. Long ago I used to object to systemd but it is a fact of life and generally works better than sysVinit, IMHO.

So what I’d like to know is whether there is any interest in cleaning up emoncms and migrating towards systemd? For example, it seems there’s currently an rc.local in production and in that file are obsolete commands. So a first thought would be to clean up the contents of rc.local but then a better thing to do would be to remove it altogether. How do people think about that idea?

For background, people might like to read the ’ Forget about rc.local’ answer to

(Greebo) #2

There has certainly been a move to shift things to systemd. However, it only happens on an ad-hoc basis as those “things” themselves are updated.

Take for example the service-runner, which started as a bash script triggered from crontab.
Due to a problem we encountered during one of the beta’s of the new EmonSD image, I rewrote it in python (to match the language emonhub was using) and moved it from crontab to a systemd “service”. If I hadn’t rewritten it, it would probably still be a cron-triggered bash script.

If you have a willingness to roll your sleeves up and submit PR’s back to the GitHub repo’s, I have no doubt they’d be appreciated. It is after-all, one of the benefits of this all being open source.

(Dave Howorth) #3

I’d like to do some of this but I’m feeling overwhelmed. The problem I’m finding is that the lack of documentation, and the out-of-date-ness and other errors in what does exist, mean that it’s very difficult to grasp exactly what the existing system does before figuring out how to do it in a more portable and future-proof way.

(Greebo) #4

Totally hear you! That’s where most of us started though, and plenty of us are still there!
I have no idea what your background or training is, so some (or all) of this may already be obvious to you, so apologies in advance if it is…

Don’t try and boil the ocean.

Basically, chunk it up into less overwhelming parts…
Lets say the “end game” is just the removal of rc.local, how can that be achieved? Only by removing every line that is in there.

So pick a line and work out how you get rid of that. Undoubtedly that is going to take you down a bit of a rabbit hole working out why that line is there, what else it impacts etc… but just stay focused on that one line. Eventually you’ll work out what the best replacement is for that and the line can be removed.

Now pick another line and repeat. You may even find that some of the things you learned doing the last line apply directly or otherwise speed up the process of removing the next line.

A number of iterations later and rc.local is no more!

Now is that going to have fixed all of the legacy ills of the EmonCMS ecosystem that have accumulated over the years? Very unlikely! Will it have made it easier for everyone else going forward and improved the portability of it all? I’d say so.

If the whole GitHub thing is all too much, I can perfectly understand that too, so just put a great big long post together with all of the details of what you did, how you did it, why you did it etc. attach any relevant files (even a tar.gz of the changed files if that works for you) and post it to the forums for one of the maintainers to push into the appropriate GitHub repository.

If you get stuck down one of those rabbit holes along the way - post a question here to see if folks can point you to the appropriate documentation or can provide the historical context on why that is there in the first place.

In my experience, nobody here will shout you down for wanting to help improve things.

(Paul) #5

Most of the services in the emon family are already systemd so getting rid of the dependency on rc.local is indeed the largest share of the work, believe it or not, it would be quite easy to achieve code wise, the tricky part could be getting the changes adopted, for some reason there has been significant resistance to moving away from using rc.local.

Much of the content of rc.local can be removed just by sorting the logging to ram properly. I have put forward using log2ram, with some subtle changes. I have been using it for some time and it solves all the issues derived from having the log files in RAM, in fact it improves thing significantly as not only does it fix the logfile creation, it also persists logfiles accross reboots and manages the size in RAM better. But using this specific solution or another is for another discussion, the point here is the large part of the rc.local content can be grouped and resolved as one thing, leaving very little else to tackle.

Below is the current rc.local file chopped into 4 sections

Section 1,

the header, just a shebang and prints the ip address. non-essential and easy to drop.

# rc.local

echo" My IP address is":
hostname -I

Section 2,

I’m not sure where this originates from, but it was introduced in this commit it looks like some sort of forced reset by deleting the /var/run copy of wlan0 config? Surely that file is brand new at reboot?

# WLAN fix on RasPi 3B+ Stretch
rm /var/run/wpa_supplicant/wlan0
ifdown wlan0
sleep 1
ifup wlan0

Section 3,

This is the larger portion that can be dealt with in one blow, it all revolves around recreating the log files and folders that were lost when rebooting and changing thier ownership and permissions etc.

# Will only run if /var/log is mounted in tmpfs
if ( mount | grep "on /var/log "| grep -q "^tmpfs " )
  for i in "redis" "apache2" "mysql" "openhab" "logrotate" "mosquitto" "supervisor"; do mkdir /var/log/"$i"; done
  for i in "emoncms.log" "mysql.log" "mqtt_input.log" "redis/redis-server.log" "service-runner.log" "mysql/error.log" "apache2/error.log" "supervisor/supervisord.log" "ntp_update.log"; do touch /var/log/"$i"; done
  for i in "emoncms.log" "mysql.log" "mqtt_input.log" "redis/redis-server.log" "service-runner.log" "mysql/error.log" "apache2/error.log" "supervisor/supervisord.log" "ntp_update.log"; do ""chmod 666"" /var/log/"$i"; done
  chown -R root:adm /var/log/apache2
  chown -R redis:redis /var/log/redis
  chown -R mysql:adm /var/log/mysql
  chown -R openhab:openhab /var/log/openhab
  chown -R pi:pi /var/log/logrotate
  chown -R mosquitto:mosquitto /var/log/mosquitto
  chown -R dataplicity:dataplicity /var/log/supervisor;

  # Restart random seed process now ~/data RW partition has been mounted 
  sudo systemctl restart systemd-random-seed.service

  # Start / Restart services,they should run happy now log dir's are created
  sleep 3
  service mysql restart
  service redis-server restart
  service mosquitto restart
  service emonhub restart
  service emonPiLCD restart
  service apache2 restart
  service supervisor restart
  service feedwriter restart
  service mqtt_input restart
  service lwrfd restart
  service service-runner restart
  service emoncms_mqtt restart


Section 4,

This should be set up as a service that runs on first boot only by simply enabling a service to run a command && then disable itself. Much like the method Raspbian already uses to expand it’s root partition.

# Run emonPi Update of first factory boot as Pi user (run condition > web connection exisits && ~/data/emonpiu$
su pi -c '/home/pi/emonpi/./firstbootupdate'

(note - It looks like the su pi is not exited so the next part is being run as user pi too!)

Section 5,

This looks like it too could/should be a service that is simply enabled/disabled like any other service.

## Start Wifi AP (uncomment if required) see emonpi/wifiAP/

and that leaves just the exit line

exit 0

So if (for example) we implemented log2ram that would be section 3 taken care of, aside from sections 4 and 5 just needing fairly straight forward new services added to replace them (perhaps amending the user and paths to something more universal along the way?) only section 3 needs tackling, possibly by resolving the need rather than re-implementing the workaround in systemd (if it is a workaround that is).

If needed I can go into the history and current requirements of the tmp logfiles in more depth, what’s been tried and suggested etc, this has been discussed in great depth many, many times over the last few years, it’s all on the forum if you do want to read up on it, quite a bit of time has been sunk into different solutions but none adopted or even tested to any real degree, so be wary of investing too much time on this, you could be reinventing the wheel and/or not make it to press.

(Greebo) #6

Can anyone else provide any thoughts on this?

I wonder why none of that was adopted - was it because the argument for it wasn’t strong enough? Or was it simply because not enough testing was done against the EmonSD images?

As I recall it, a lot of the discussions were driven by folks using their own self installs… so could it be that all that was required to get it adopted was someone to spend the time proving it was also a much better option on the EmonSD Pi image?

(Paul) #7

No the discussions, development and trials were firmly aimed at the emonSD. Some of the discussions are quite long and indepth so they can make things sound overly complex. The sticking point seems to be that to make “it simpler” it is believed we should just create new files and folders each boot as it’s not so complex in principle, but that’s actually quite hard to manage (by comparison). By simply keeping a persisted copy of the logfiles held in RAM, every hour and at shutdown, it eliminates all the issues. So what sounds like a complex arrangement (managing 2 copies etc) is actually far, far simpler to setup, manage and it provides the additional benefit of keeping the logs even when users (understandably) react to an issue with an immediate reboot (currently all useful info is lost until it happens again).

My minor mods then also use logrotate to shift rotated logs out of ram and are only held on disc, thus freeing up ram, we do not need rotated and compressed logs in RAM.

Since it is a mirror image that’s persisted each hour, even if you install contraband softwares, the original install instructions will work fine, no amendments needed due to our logfiles being in ram. if a guide says create a folder in /var/log, that’s exactly what you do, even a package manager (apt-get) can install as per usual, as long as the host doesn’t crash before the hourly backup or a graceful reboot, the new logs will be persisted and then copied back to ram on start up, no hardcoded lists, no files to edit with new software, it all works transparently.

Some will raise concern about boot up speed, but all log2ram is doing is reading the logfiles to RAM, not writing anything to disc so it is quite quick, certainly no slower than restarting all the services in rc.local with a few seconds sleep in the midddle :smile:.

I’ve had it running on a test setup for a while with no issue, but I have not stress tested it in great depth with lots of traffic etc. I have my own RO RPi’s that are stable (that do not use rc.local like the emonSD either) so I do not want to change them unnecessarily. We could test it further if there was interest, but I’m just wary of sinking even more time into something I don’t necessarily need to change and no one else wants to change.

[edit - Some bedtime reading material :grin: ]

An earlier, longer more complex discussion

A more recent discussion where Glyn showed some interest, but it did start looking a bit more complex.

The log2ram repo is at and my minor changes were these This could be installed (and uninstalled) with a one liner command with some minor edits, eg the script can be saved as an executable on github, so if we pull and execute rather than downloading a zip and use chmod etc etc.

(Greebo) #8

I guess without @TrystanLea or @glyn.hudson chiming in here with some history or some other reasons for reluctance we’ll never know. Maybe they figured log2ram was no longer necessary since they were no longer making the root partition r/o - except that all that “legacy” stuff is still in rc.local :slight_smile:

Maybe its still worth revisiting all of this now that the R/W version of EmonSD is in the wild and could still be improved…

(Paul) #9

Although the FS is no longer RO, it has been decided the logfiles will remain in RAM, that’s why all the stuff is still needed in rc.local.

(Greebo) #10

So they’re still all lost after a reboot… Then there is definitely room for improvement.

I just thought it was there for the “first start” use case. Serves me right for not taking a closer look!

(Paul) #11

No, systemd is totally undermined at every bootup. None of the services listed in rc.local are started just the once at boot up by systemd, they are always manually restarted in the order they are listed by rc.local.

Well you say that! But it has been commented before that the current situation is “nice because we get to start a fresh with new clean and empty log files”. IMO discarding everything (whether you need it or not) is not the best way of housekeeping.

(Greebo) #12

Sorry, that’s not what I meant… I just meant the logfile and permission creations etc…
I know the services were all restarted because they all previously failed without logfile locations.

There’s always rm -rf *

(Paul) #13

Indeed, although some log files can be quite resilient and require an elevated user level and/or some softwares (eg emoncms) can not recreate the logfile themselves, so I usually echo ‘’ (nothing) instaed. None the less, it would be easy to write a custom command for clearing the logs and for those that insist on periodically clearing the logs they can use cron to execute said command.

It is much easier to retrospectively delete unwanted log files than it is to retrospectively view deleted log files :slight_smile: once the’re gone, the’re gone!

Also, although pointless mentioning this previously (for emonSD) as all the logs are lost at reboot regardless, but journalctl files are (by default) also lost at reboot, but the default setting for Debian based journalctl is “auto” which means if there is a “journal” folder in /var/log it will automatically persist the journal logfile, so you end up with a subfolder for each boot up, then you can compare boot ups and/or find out what services were up to immediately before a paniced reboot because something wasn’t working.

(Dave Howorth) #14

Why do separate logs exist at all? All those processes are run as systemd services, I think, so their output is already logged by journald.

As you say, journald can be configured to write to RAM or to persist to disk/SD. And it will buffer things until the appropriate directories are available. So what are all the manually created log files and directories for?

edit: Just to add that the only need for persistent logs is if there’s some problem causing the whole machine to crash, yes? And in that case, it’s often better to have the log on another machine anyway (or on a serial console in the good old days ITGOD). So why not leave the journal in RAM and then switch the journald mode to persistent if such a problem arises. Or as an additional bonus, if there is a disk (or USB) connected with a partition labelled say JOURNAL-DATA (case-immaterial) or an NFS filesystem available/mounted with a similar name then use that as an auxiliary destination for the journal data?

(Brian Orpin) #15

I suspect because this has grown organically since before systemd was in use. Possibly you are correct and they could be handled a different way. However, I always think switching logging on (efectively) is a bit like shutting the stable door after the horse has bolted. I’d rather a well implemented log rotation but I have no idea how to do that with systemd.

Looks like you have found yourself your first task :slight_smile:

(Dave Howorth) #16

Could be. It seems more promising than other things I’ve looked at.

The journal is always running, so the logs are always available for all problems except those that cause a system crash, which one hopes is rare. I’d be happy to try implementing something that automatically used an external logging device in addition (disk/USB key/NFS) if one is present.

(Paul) #17

The first exception that springs to mind is emoncms, which is pretty much centre stage here.

I think it’s a bit soon to be looking at getting rid of /var/log altogether and it takes a reasonably sized task and makes it a mammoth task.

There are a lot of users that write custom scripts that run from cron triggers and manually started utilities etc. I know I have dozens. Yes it’s possible to convert them all I guess, but are we really going to take that option away from more novice users that don’t want to write a service? There is a lot of scripts and hacks out there to interconnect IoT stuff and perform various monitoring tasks etc, currently users can add stuff like that without having to totally overhaul it, we need to keep that flexibility IMO.

Well no, not entirely. Many issues with front end hardware and firmware has been resolved by going back through logs such as emonhub.log, sometimes when patterns appear a log history is extremely useful. Especially when debugging emoncms as you can spot an anomaly in a emoncms graph and then look at the raw data processed by emonhub. You get to see the data from the front end and the data that was sent to emoncms.

I think that is a good option, but the emonSD is aimed at being self contained, dependency on a 3rd party site for log info cannot be the only option. And if that doesn’t run smoothly, how would we debug that without some local logs?

I think what you suggest is something to be looked into and possibly implemented in time, but the immediate problem(s) of getting shot of rc.local and improving the logging in RAM (that isn’t going to be totally removed anytime soon) could be done swiftly by simply persisting the logs to disk regularly and reading the existing logs (and folders etc) into ram at boot without muddying the waters with other issues and ambitions. It’s a ready made solution that gives us a big leg up, a vast stride forward with very little effort. That’s not to say that development cannot continue on from there and look closer at more dependence on journalctl. I’m not convinced a shift to journalctl ticks all the issues boxes and offers something that will work for everyone in all circumstances.

(Dave Howorth) #18

I’m talking about the processes that are listed in rc.local. emoncms is not one of them.

I don’t think I suggested that.

All my cron jobs are silent when things are going well and send me email if things go wrong. They never write anything to a log. Are yours different?

You can manually start jobs under systemd too.

I didn’t suggest that either. Everybody is free to keep on doing whatever they want. The point is that the emon-world wouldn’t.

The logs would still be available through journalctl, so I’m not sure what the problem is?

I don’t know the system well enough to comment about this. Personally, I’ve been looking at the actual feed data files and the data reported by the graph module in its CSV output. But in any case to repeat, I’m not suggesting doing away with any log data at all, just changing how it is accessed.

I’m not suggesting any third-party involvement at all. If I have a USB disk or USB key or if I have another machine (like the one I’m typing on now) that can make an NFS export available, then I could export the logs if I chose to.

And again, I am not suggesting doing away with local logs, just not storing them twice, once in the journal and once in a log file.

I think what I’ve suggested does exactly what you ask. It keeps the logs in RAM and it optionally stores them to disk too. And it’s a lot more ready-made than what I think you’re suggesting.

(Trystan Lea) #19

I’ve noted this discussion for @glyn.hudson and I to review next week. I dont know the historical detail on this as it was @glyn.hudson’s work however I noted that in the recent thread Services not restarting on upgrade to 9.9.7 I had to add the entries for restarting service-runner & emoncms_mqtt to the jessie rc.local to ensure those services started at boot while these entries are not present on the strech version of rc.local.

The stretch version of rc.local is much reduced in general, @glyn.hudson can Im sure comment.

We are moving in that direction, emonhub.log is now accessed from journalctl

emoncms.log does not use systemd, as it is a mixture of logging coming from the emoncms application running on the apache webserver as well as potentially other scripts.

Im not sure about /var/log/openhab (the final log item listed in the stretch rc.local).

(Paul) #20

That file you linked is NOT the stretch version, it was last updated in 2015. The jessieminimal version is the current version (the one you edited).

Openhab is no longer installed to emonSD by default but for some backward compatibility it may still be needed. The greater issue is moving forward, implementing any per service edits tie us into a life of always tweaking any new softwares to fit what ever solution emonSD is using, the aim has to be to accommodate packages and softwares without individually modifying them so that users can add stuff without having to come to us to tweak it for them.