Services not restarting on upgrade to 9.9.7

Looks like you’re running jessie, I thought there was an extra manual step to enable service-runner on jessie, something like linking the .service file into a different place?

Can you provide the output of:

ls -l /etc/systemd/system/service-runner*
ls -l /lib/systemd/system/service-runner*

Ah, that will be the problem. I do not have a Jessie based system.

@glyn.hudson, I suspect there is an issue on Jessie based distributions.

The script is here https://github.com/openenergymonitor/emonpi/blob/service_correction/emoncmsupdate and was discussed here Emoncms V9.9.6 Stable merge (pre-merge check).

There is a command to create the link in the script. I think I saw that the script itself may not have updated correctly in the logs above.

Running those command gives me

[email protected](ro):~$ ls -l /etc/systemd/system/service-runner*
ls: cannot access /etc/systemd/system/service-runner*: No such file or directory
[email protected](ro):~$ ls -l /lib/systemd/system/service-runner*
lrwxrwxrwx 1 root root 71 Feb 27 20:43 /lib/systemd/system/service-runner.service -> /var/www/emoncms/scripts/services/service-runner/service-runner.service
[email protected](ro):~$

If I do need to run that script to correct as I have not done that for several months can I have a reminder how to do it from the PuTTY console?

Many thanks

Could you try this please.

cd ~/emonpi
git status

When you tested this, did you not discover systemd on Jessie did not follow symbolic links?

I suddenly realised I was having Deja Vu - Symlinking services in Jessie doesn't work

I think the solution may be to use a hard link instead.

I get this…

[email protected](ro):~$ cd ~/emonpi
[email protected](ro):emonpi$ git status
On branch master
Your branch is up-to-date with 'origin/master'.
Untracked files:
  (use "git add <file>..." to include in what will be committed)

        1

nothing added to commit but untracked files present (use "git add" to track)
[email protected](ro):emonpi$

Hi @IM35461, sorry about this. It seems it is possibly an oddity of the Jessie systemd although I’ve searched out a Jessie system I have and I can see symlinks in the /lib/ systemd folder so do not know why these are not working. As it is a running system (doing other things) I really do not want to play with it!

The solution might be a slight change to the enable command. Can you try this and see if it creates an error message.

sudo systemctl enable /lib/systemd/system/service-runner.service

If no error, could you reboot and then do a status on the service please.

Alas it does not like that command :frowning:

[email protected](ro):~$ sudo systemctl enable /lib/systemd/system/service-runner.service
Failed to execute operation: Too many levels of symbolic links
[email protected](ro):~$

This is very perplexing as the feedwriter service is identical.

Could you possibly reboot and then do:

sudo systemctl status emoncms_mqtt service-runner feedwriter

If you then do a

sudo systemctl restart emoncms_mqtt
sudo systemctl restart service-runner

Everything will run until you next boot but that isn’t really a solution.

That seems to work and I was wondering if it may just be best to have a fresh start with a new MicroSD card…

[email protected](ro):~$ sudo systemctl status emoncms_mqtt service-runner feedwriter
● emoncms_mqtt.service - Emoncms emoncms_mqtt script
   Loaded: loaded (/var/www/emoncms/scripts/services/emoncms_mqtt/emoncms_mqtt.service; disabled)
   Active: inactive (dead)
     Docs: https://github.com/emoncms/emoncms/blob/master/docs/RaspberryPi/MQTT.md

● service-runner.service - Emoncms service-runner Input Script
   Loaded: loaded (/var/www/emoncms/scripts/services/service-runner/service-runner.service; disabled)
   Active: inactive (dead)
     Docs: https://github.com/emoncms/emoncms/blob/master/scripts/services/install-service-runner-update.md

Feb 28 16:02:24 emonpi systemd[1]: [/var/www/emoncms/scripts/services/servi...t'
Feb 28 16:04:04 emonpi systemd[1]: [/var/www/emoncms/scripts/services/servi...t'

● feedwriter.service - Emoncms feedwriter script
   Loaded: loaded (/var/www/emoncms/scripts/services/feedwriter/feedwriter.service; disabled)
   Active: active (running) since Thu 2019-02-28 16:02:02 UTC; 2min 2s ago
     Docs: https://github.com/emoncms/emoncms/blob/master/scripts/services/install-service-feedwriter.md
 Main PID: 2918 (php)
   CGroup: /system.slice/feedwriter.service
           └─2918 /usr/bin/php /var/www/emoncms/scripts/feedwriter.php

Feb 28 16:02:02 emonpi systemd[1]: Started Emoncms feedwriter script.
Feb 28 16:02:03 emonpi feedwriter[2918]: Buffered feed writer daemon started....
Hint: Some lines were ellipsized, use -l to show in full.
[email protected](ro):~$ sudo systemctl restart emoncms_mqtt
[email protected](ro):~$ sudo systemctl restart service-runner
[email protected](ro):~$

Could this have anything to do with the shenanigans that happens in the rc.local file? The feedwriter service is the only one of the 3 services that gets restarted in rc.local.

If this is still Jessie based, it is still a read-only filesystem, could that be causing any issue with the new services?

I know that the services do not have thier own log files, but are they dependent on other services that do require a log file (eg redis) therefore failing to start as the required service(s) have failed to start due to a missing log file and then only the feedwriter gets restarted once the log files are created, leaving the other 2 not running.

[edit]

As a quick test you could edit the service mqtt_input restart to service emoncms_mqtt restart or add it to the bottom of the list. Any services that cannot start (eg mqtt_input?) should be removed as rc.local will just abort on the first error and not continue with the rest of the commands.

Interesting point. @glyn.hudson I’m not sure this was changed as part of the update script.

Ah yes, very possible - obvious really. The enable command would be unable to create the symlinks in /etc/systemd/system/. Doesn’t explain why the script would fail though as it is in RW mode.

@IM35461, my mistake.

Could you try

rpi-rw
sudo systemctl enable service-runner.service
sudo systemctl start service-runner.service
rpi-ro

If the enable command is successful, reboot and see if it starts.

Well that would get you onto Stretch, which is a good idea.

Looking at my jessie based EmonBase, (which is where I found the issues on jessie mentioned above), I have:

[email protected](ro):~$ ls -l /etc/systemd/system/service-runner*
lrwxrwxrwx 1 root root 42 Nov 13 10:26 /etc/systemd/system/service-runner.service -> /lib/systemd/system/service-runner.service
[email protected](ro):~$ ls -l /lib/systemd/system/service-runner*
lrwxrwxrwx 1 root root 71 Nov 13 10:22 /lib/systemd/system/service-runner.service -> /var/www/emoncms/scripts/services/service-runner/service-runner.service
[email protected](ro):~$
[email protected](ro):~$ systemctl status service-runner
● service-runner.service - Emoncms service-runner Input Script
   Loaded: loaded (/var/www/emoncms/scripts/services/service-runner/service-runner.service; enabled)
   Active: active (running) since Tue 2019-02-19 06:17:34 UTC; 1 weeks 2 days ago
 Main PID: 883 (python)
   CGroup: /system.slice/service-runner.service
           └─883 /usr/bin/python /var/www/emoncms/scripts/services/service-runner/service-runner.py

On this system, service-runner restarts successfully after a reboot.

So it looks like I manually linked /lib/systemd/system/service-runner.service into /etc/systemd/system

Trouble is, if you forget, need to disable it for some reason, then come to enable it again, it won’t work as the original symlink in /etc/systemd/system has been deleted by the disable command and you need to add in the link manually as well.

What is annoying is that it should work. I think the RO issue might be it - needs to be RW when enabled.

Do you have an entry in the rc.local?

Yep, that’s absolutely correct. Its a bug in the systemd version running on jessie and there’s no good way around it while still using systemd.

jessie is running systemd v215, the jessie-backports version is v230, stretch is running v232.
Its possible that the jessie-backports version fixes it but I’ve not tried it.

The instructions for using backports are here: Instructions
For this particular thread, replace stretch with jessie in all references in commands from that link.

Nothing related to service-runner.

[email protected](ro):~$ ls -l /etc/rc.local
lrwxrwxrwx 1 root root 38 Apr 28  2016 /etc/rc.local -> /home/pi/emonpi/rc.local_jessieminimal

That jessieminimal file contains the code to create the log folders in tmpfs and then restart the services that need logs to exist

I just don’t buy that. If you look in /lib/systemd/system/ you will see symlinked units.

Don’t worry about me, I have gone for a fresh install and now seem to have 9.9.8 beta.

System can be restarted with no issues

2 Likes

I’m not trying to sell it to you :wink:
I went through the exact same thought process at the time that you’re going through right now. “But it should work!”… I spent quite a few hours on this only to come to the realisation that there are things less frustrating to spend my time on :smiley:

1 Like

@glyn.hudson @TrystanLea Looks like the upgrade on Jessie is problematic.