Services not restarting on upgrade to 9.9.7

Running those command gives me

pi@emonpi(ro):~$ ls -l /etc/systemd/system/service-runner*
ls: cannot access /etc/systemd/system/service-runner*: No such file or directory
pi@emonpi(ro):~$ ls -l /lib/systemd/system/service-runner*
lrwxrwxrwx 1 root root 71 Feb 27 20:43 /lib/systemd/system/service-runner.service -> /var/www/emoncms/scripts/services/service-runner/service-runner.service
pi@emonpi(ro):~$

If I do need to run that script to correct as I have not done that for several months can I have a reminder how to do it from the PuTTY console?

Many thanks

Could you try this please.

cd ~/emonpi
git status

When you tested this, did you not discover systemd on Jessie did not follow symbolic links?

I suddenly realised I was having Deja Vu - Symlinking services in Jessie doesn't work

I think the solution may be to use a hard link instead.

I get this…

pi@emonpi(ro):~$ cd ~/emonpi
pi@emonpi(ro):emonpi$ git status
On branch master
Your branch is up-to-date with 'origin/master'.
Untracked files:
  (use "git add <file>..." to include in what will be committed)

        1

nothing added to commit but untracked files present (use "git add" to track)
pi@emonpi(ro):emonpi$

Hi @IM35461, sorry about this. It seems it is possibly an oddity of the Jessie systemd although I’ve searched out a Jessie system I have and I can see symlinks in the /lib/ systemd folder so do not know why these are not working. As it is a running system (doing other things) I really do not want to play with it!

The solution might be a slight change to the enable command. Can you try this and see if it creates an error message.

sudo systemctl enable /lib/systemd/system/service-runner.service

If no error, could you reboot and then do a status on the service please.

Alas it does not like that command :frowning:

pi@emonpi(ro):~$ sudo systemctl enable /lib/systemd/system/service-runner.service
Failed to execute operation: Too many levels of symbolic links
pi@emonpi(ro):~$

This is very perplexing as the feedwriter service is identical.

Could you possibly reboot and then do:

sudo systemctl status emoncms_mqtt service-runner feedwriter

If you then do a

sudo systemctl restart emoncms_mqtt
sudo systemctl restart service-runner

Everything will run until you next boot but that isn’t really a solution.

That seems to work and I was wondering if it may just be best to have a fresh start with a new MicroSD card…

pi@emonpi(ro):~$ sudo systemctl status emoncms_mqtt service-runner feedwriter
● emoncms_mqtt.service - Emoncms emoncms_mqtt script
   Loaded: loaded (/var/www/emoncms/scripts/services/emoncms_mqtt/emoncms_mqtt.service; disabled)
   Active: inactive (dead)
     Docs: https://github.com/emoncms/emoncms/blob/master/docs/RaspberryPi/MQTT.md

● service-runner.service - Emoncms service-runner Input Script
   Loaded: loaded (/var/www/emoncms/scripts/services/service-runner/service-runner.service; disabled)
   Active: inactive (dead)
     Docs: https://github.com/emoncms/emoncms/blob/master/scripts/services/install-service-runner-update.md

Feb 28 16:02:24 emonpi systemd[1]: [/var/www/emoncms/scripts/services/servi...t'
Feb 28 16:04:04 emonpi systemd[1]: [/var/www/emoncms/scripts/services/servi...t'

● feedwriter.service - Emoncms feedwriter script
   Loaded: loaded (/var/www/emoncms/scripts/services/feedwriter/feedwriter.service; disabled)
   Active: active (running) since Thu 2019-02-28 16:02:02 UTC; 2min 2s ago
     Docs: https://github.com/emoncms/emoncms/blob/master/scripts/services/install-service-feedwriter.md
 Main PID: 2918 (php)
   CGroup: /system.slice/feedwriter.service
           └─2918 /usr/bin/php /var/www/emoncms/scripts/feedwriter.php

Feb 28 16:02:02 emonpi systemd[1]: Started Emoncms feedwriter script.
Feb 28 16:02:03 emonpi feedwriter[2918]: Buffered feed writer daemon started....
Hint: Some lines were ellipsized, use -l to show in full.
pi@emonpi(ro):~$ sudo systemctl restart emoncms_mqtt
pi@emonpi(ro):~$ sudo systemctl restart service-runner
pi@emonpi(ro):~$

Could this have anything to do with the shenanigans that happens in the rc.local file? The feedwriter service is the only one of the 3 services that gets restarted in rc.local.

If this is still Jessie based, it is still a read-only filesystem, could that be causing any issue with the new services?

I know that the services do not have thier own log files, but are they dependent on other services that do require a log file (eg redis) therefore failing to start as the required service(s) have failed to start due to a missing log file and then only the feedwriter gets restarted once the log files are created, leaving the other 2 not running.

[edit]

As a quick test you could edit the service mqtt_input restart to service emoncms_mqtt restart or add it to the bottom of the list. Any services that cannot start (eg mqtt_input?) should be removed as rc.local will just abort on the first error and not continue with the rest of the commands.

Interesting point. @glyn.hudson I’m not sure this was changed as part of the update script.

Ah yes, very possible - obvious really. The enable command would be unable to create the symlinks in /etc/systemd/system/. Doesn’t explain why the script would fail though as it is in RW mode.

@IM35461, my mistake.

Could you try

rpi-rw
sudo systemctl enable service-runner.service
sudo systemctl start service-runner.service
rpi-ro

If the enable command is successful, reboot and see if it starts.

Well that would get you onto Stretch, which is a good idea.

Looking at my jessie based EmonBase, (which is where I found the issues on jessie mentioned above), I have:

pi@emonpi(ro):~$ ls -l /etc/systemd/system/service-runner*
lrwxrwxrwx 1 root root 42 Nov 13 10:26 /etc/systemd/system/service-runner.service -> /lib/systemd/system/service-runner.service
pi@emonpi(ro):~$ ls -l /lib/systemd/system/service-runner*
lrwxrwxrwx 1 root root 71 Nov 13 10:22 /lib/systemd/system/service-runner.service -> /var/www/emoncms/scripts/services/service-runner/service-runner.service
pi@emonpi(ro):~$
pi@emonpi(ro):~$ systemctl status service-runner
● service-runner.service - Emoncms service-runner Input Script
   Loaded: loaded (/var/www/emoncms/scripts/services/service-runner/service-runner.service; enabled)
   Active: active (running) since Tue 2019-02-19 06:17:34 UTC; 1 weeks 2 days ago
 Main PID: 883 (python)
   CGroup: /system.slice/service-runner.service
           └─883 /usr/bin/python /var/www/emoncms/scripts/services/service-runner/service-runner.py

On this system, service-runner restarts successfully after a reboot.

So it looks like I manually linked /lib/systemd/system/service-runner.service into /etc/systemd/system

Trouble is, if you forget, need to disable it for some reason, then come to enable it again, it won’t work as the original symlink in /etc/systemd/system has been deleted by the disable command and you need to add in the link manually as well.

What is annoying is that it should work. I think the RO issue might be it - needs to be RW when enabled.

Do you have an entry in the rc.local?

Yep, that’s absolutely correct. Its a bug in the systemd version running on jessie and there’s no good way around it while still using systemd.

jessie is running systemd v215, the jessie-backports version is v230, stretch is running v232.
Its possible that the jessie-backports version fixes it but I’ve not tried it.

The instructions for using backports are here: Instructions
For this particular thread, replace stretch with jessie in all references in commands from that link.

Nothing related to service-runner.

pi@emonpi(ro):~$ ls -l /etc/rc.local
lrwxrwxrwx 1 root root 38 Apr 28  2016 /etc/rc.local -> /home/pi/emonpi/rc.local_jessieminimal

That jessieminimal file contains the code to create the log folders in tmpfs and then restart the services that need logs to exist

I just don’t buy that. If you look in /lib/systemd/system/ you will see symlinked units.

Don’t worry about me, I have gone for a fresh install and now seem to have 9.9.8 beta.

System can be restarted with no issues

2 Likes

I’m not trying to sell it to you :wink:
I went through the exact same thought process at the time that you’re going through right now. “But it should work!”… I spent quite a few hours on this only to come to the realisation that there are things less frustrating to spend my time on :smiley:

1 Like

@glyn.hudson @TrystanLea Looks like the upgrade on Jessie is problematic.

Thanks @borpin @Greebo @pb66 @IM35461 we will look into it.

Testing here with the older jessie based emonSD-26Oct17 image and the latest emonpi update I was able to replicate the issue of the services not starting after a reboot and have fixed the issue by adding the relevant entries to the rc.local_jessieminimal:

https://github.com/openenergymonitor/emonpi/commit/ecf1ef301cf8633e3c0000f54efc138678407e3e

Apologies for not catching this issue sooner. Unfortunately users who have updated prior to this update will need to login via SSH and start service-runner manually to initiate the update process to then fix for subsequent updates. The command to do this once logged in is:

sudo servicectl start service-runner

then run emonpi update from the emoncms Administration menu.

Note: It can take 5 minutes for all the services to come online after a reboot on jessie. The new stretch image is a bit faster. This caught me out to start with as I thought none of the services where starting but they did all start up fine after a couple of minutes.

1 Like

One issue here is that there is no way of knowing what state the ‘emonpi’ part of an EmonPi is in as there is no versioning of the emonpi repository. I have opened an issue.

1 Like