Services not restarting on upgrade to 9.9.7

IM35461 · 2019-02-28T12:55:59.712Z

Running those command gives me

pi@emonpi(ro):~$ ls -l /etc/systemd/system/service-runner*
ls: cannot access /etc/systemd/system/service-runner*: No such file or directory
pi@emonpi(ro):~$ ls -l /lib/systemd/system/service-runner*
lrwxrwxrwx 1 root root 71 Feb 27 20:43 /lib/systemd/system/service-runner.service -> /var/www/emoncms/scripts/services/service-runner/service-runner.service
pi@emonpi(ro):~$

If I do need to run that script to correct as I have not done that for several months can I have a reminder how to do it from the PuTTY console?

Many thanks

borpin · 2019-02-28T13:01:49.394Z

Could you try this please.

cd ~/emonpi
git status

borpin · 2019-02-28T13:15:02.132Z

Greebo:

Looks like you’re running jessie, I thought there was an extra manual step to enable service-runner on jessie, something like linking the .service file into a different place?

When you tested this, did you not discover systemd on Jessie did not follow symbolic links?

I suddenly realised I was having Deja Vu - Symlinking services in Jessie doesn't work

I think the solution may be to use a hard link instead.

IM35461 · 2019-02-28T13:26:32.271Z

borpin:

cd ~/emonpi

I get this…

pi@emonpi(ro):~$ cd ~/emonpi
pi@emonpi(ro):emonpi$ git status
On branch master
Your branch is up-to-date with 'origin/master'.
Untracked files:
  (use "git add <file>..." to include in what will be committed)

        1

nothing added to commit but untracked files present (use "git add" to track)
pi@emonpi(ro):emonpi$

borpin · 2019-02-28T14:03:34.120Z

Hi @IM35461, sorry about this. It seems it is possibly an oddity of the Jessie systemd although I’ve searched out a Jessie system I have and I can see symlinks in the /lib/ systemd folder so do not know why these are not working. As it is a running system (doing other things) I really do not want to play with it!

The solution might be a slight change to the enable command. Can you try this and see if it creates an error message.

sudo systemctl enable /lib/systemd/system/service-runner.service

If no error, could you reboot and then do a status on the service please.

IM35461 · 2019-02-28T14:55:11.785Z

Alas it does not like that command

pi@emonpi(ro):~$ sudo systemctl enable /lib/systemd/system/service-runner.service
Failed to execute operation: Too many levels of symbolic links
pi@emonpi(ro):~$

borpin · 2019-02-28T15:50:03.047Z

This is very perplexing as the feedwriter service is identical.

Could you possibly reboot and then do:

sudo systemctl status emoncms_mqtt service-runner feedwriter

If you then do a

sudo systemctl restart emoncms_mqtt
sudo systemctl restart service-runner

Everything will run until you next boot but that isn’t really a solution.

IM35461 · 2019-02-28T16:06:21.728Z

That seems to work and I was wondering if it may just be best to have a fresh start with a new MicroSD card…

pi@emonpi(ro):~$ sudo systemctl status emoncms_mqtt service-runner feedwriter
● emoncms_mqtt.service - Emoncms emoncms_mqtt script
   Loaded: loaded (/var/www/emoncms/scripts/services/emoncms_mqtt/emoncms_mqtt.service; disabled)
   Active: inactive (dead)
     Docs: https://github.com/emoncms/emoncms/blob/master/docs/RaspberryPi/MQTT.md

● service-runner.service - Emoncms service-runner Input Script
   Loaded: loaded (/var/www/emoncms/scripts/services/service-runner/service-runner.service; disabled)
   Active: inactive (dead)
     Docs: https://github.com/emoncms/emoncms/blob/master/scripts/services/install-service-runner-update.md

Feb 28 16:02:24 emonpi systemd[1]: [/var/www/emoncms/scripts/services/servi...t'
Feb 28 16:04:04 emonpi systemd[1]: [/var/www/emoncms/scripts/services/servi...t'

● feedwriter.service - Emoncms feedwriter script
   Loaded: loaded (/var/www/emoncms/scripts/services/feedwriter/feedwriter.service; disabled)
   Active: active (running) since Thu 2019-02-28 16:02:02 UTC; 2min 2s ago
     Docs: https://github.com/emoncms/emoncms/blob/master/scripts/services/install-service-feedwriter.md
 Main PID: 2918 (php)
   CGroup: /system.slice/feedwriter.service
           └─2918 /usr/bin/php /var/www/emoncms/scripts/feedwriter.php

Feb 28 16:02:02 emonpi systemd[1]: Started Emoncms feedwriter script.
Feb 28 16:02:03 emonpi feedwriter[2918]: Buffered feed writer daemon started....
Hint: Some lines were ellipsized, use -l to show in full.
pi@emonpi(ro):~$ sudo systemctl restart emoncms_mqtt
pi@emonpi(ro):~$ sudo systemctl restart service-runner
pi@emonpi(ro):~$

pb66 · 2019-02-28T18:42:13.746Z

Could this have anything to do with the shenanigans that happens in the rc.local file? The feedwriter service is the only one of the 3 services that gets restarted in rc.local.

github.com

openenergymonitor/emonpi/blob/master/rc.local_jessieminimal#L40

    
      
          touch /home/pi/data/emonpiupdate.log
          ln -s /home/pi/data/emonpiupdate.log /var/log/emoncms/emonpiupdate.log
          
          
# Restart random seed process now ~/data RW partition has been mounted 
          sudo systemctl restart systemd-random-seed.service
          
          
# Start / Restart services,they should run happy now log dir's are created
          sleep 3
          service mysql restart
          service redis-server restart
          service mosquitto restart
          service emonhub restart
          service emonPiLCD restart
          service apache2 restart
          service supervisor restart
          service feedwriter restart
          service mqtt_input restart
          service lwrfd restart
          service service-runner restart
          service emoncms_mqtt restart

If this is still Jessie based, it is still a read-only filesystem, could that be causing any issue with the new services?

I know that the services do not have thier own log files, but are they dependent on other services that do require a log file (eg redis) therefore failing to start as the required service(s) have failed to start due to a missing log file and then only the feedwriter gets restarted once the log files are created, leaving the other 2 not running.

[edit]

As a quick test you could edit the service mqtt_input restart to service emoncms_mqtt restart or add it to the bottom of the list. Any services that cannot start (eg mqtt_input?) should be removed as rc.local will just abort on the first error and not continue with the rest of the commands.

borpin · 2019-02-28T19:13:13.813Z

pb66:

The feedwriter service is the only one of the 3 services that gets restarted in rc.local

Interesting point. @glyn.hudson I’m not sure this was changed as part of the update script.

pb66:

If this is still Jessie based, it is still a read-only filesystem, could that be causing any issue with the new services?

Ah yes, very possible - obvious really. The enable command would be unable to create the symlinks in /etc/systemd/system/. Doesn’t explain why the script would fail though as it is in RW mode.

@IM35461, my mistake.

Could you try

rpi-rw
sudo systemctl enable service-runner.service
sudo systemctl start service-runner.service
rpi-ro

If the enable command is successful, reboot and see if it starts.

IM35461:

That seems to work and I was wondering if it may just be best to have a fresh start with a new MicroSD card

Well that would get you onto Stretch, which is a good idea.

Greebo · 2019-02-28T20:06:48.026Z

Looking at my jessie based EmonBase, (which is where I found the issues on jessie mentioned above), I have:

pi@emonpi(ro):~$ ls -l /etc/systemd/system/service-runner*
lrwxrwxrwx 1 root root 42 Nov 13 10:26 /etc/systemd/system/service-runner.service -> /lib/systemd/system/service-runner.service
pi@emonpi(ro):~$ ls -l /lib/systemd/system/service-runner*
lrwxrwxrwx 1 root root 71 Nov 13 10:22 /lib/systemd/system/service-runner.service -> /var/www/emoncms/scripts/services/service-runner/service-runner.service
pi@emonpi(ro):~$
pi@emonpi(ro):~$ systemctl status service-runner
● service-runner.service - Emoncms service-runner Input Script
   Loaded: loaded (/var/www/emoncms/scripts/services/service-runner/service-runner.service; enabled)
   Active: active (running) since Tue 2019-02-19 06:17:34 UTC; 1 weeks 2 days ago
 Main PID: 883 (python)
   CGroup: /system.slice/service-runner.service
           └─883 /usr/bin/python /var/www/emoncms/scripts/services/service-runner/service-runner.py

On this system, service-runner restarts successfully after a reboot.

So it looks like I manually linked /lib/systemd/system/service-runner.service into /etc/systemd/system

borpin · 2019-02-28T20:12:07.884Z

Greebo:

So it looks like I manually linked /lib/systemd/system/service-runner.service into /etc/systemd/system

Trouble is, if you forget, need to disable it for some reason, then come to enable it again, it won’t work as the original symlink in /etc/systemd/system has been deleted by the disable command and you need to add in the link manually as well.

What is annoying is that it should work. I think the RO issue might be it - needs to be RW when enabled.

Do you have an entry in the rc.local?

Greebo · 2019-02-28T20:22:26.045Z

borpin:

Trouble is, if you forget, need to disable it for some reason, then come to enable it again, it won’t work as the original symlink in /etc/systemd/system has been deleted by the disable command and you need to add in the link manually as well.

Yep, that’s absolutely correct. Its a bug in the systemd version running on jessie and there’s no good way around it while still using systemd.

borpin:

What is annoying is that it should work.

jessie is running systemd v215, the jessie-backports version is v230, stretch is running v232.
Its possible that the jessie-backports version fixes it but I’ve not tried it.

The instructions for using backports are here: Instructions
For this particular thread, replace stretch with jessie in all references in commands from that link.

borpin:

Do you have an entry in the rc.local?

Nothing related to service-runner.

pi@emonpi(ro):~$ ls -l /etc/rc.local
lrwxrwxrwx 1 root root 38 Apr 28  2016 /etc/rc.local -> /home/pi/emonpi/rc.local_jessieminimal

That jessieminimal file contains the code to create the log folders in tmpfs and then restart the services that need logs to exist

borpin · 2019-02-28T20:25:49.508Z

Greebo:

Yep, that’s absolutely correct. Its a bug in the systemd version running on jessie and there’s no good way around it while still using systemd.

I just don’t buy that. If you look in /lib/systemd/system/ you will see symlinked units.

IM35461 · 2019-02-28T20:26:05.193Z

Don’t worry about me, I have gone for a fresh install and now seem to have 9.9.8 beta.

System can be restarted with no issues

Screenshot_2019-02-28-20-23-23-3311600×2560 653 KB

Greebo · 2019-02-28T20:29:12.016Z

borpin:

I just don’t buy that.

I’m not trying to sell it to you
I went through the exact same thought process at the time that you’re going through right now. “But it should work!”… I spent quite a few hours on this only to come to the realisation that there are things less frustrating to spend my time on

borpin · 2019-02-28T20:36:43.479Z

@glyn.hudson @TrystanLea Looks like the upgrade on Jessie is problematic.

TrystanLea · 2019-02-28T20:41:25.452Z

Thanks @borpin @Greebo @pb66 @IM35461 we will look into it.

TrystanLea · 2019-03-04T11:22:35.023Z

Testing here with the older jessie based emonSD-26Oct17 image and the latest emonpi update I was able to replicate the issue of the services not starting after a reboot and have fixed the issue by adding the relevant entries to the rc.local_jessieminimal:

https://github.com/openenergymonitor/emonpi/commit/ecf1ef301cf8633e3c0000f54efc138678407e3e

Apologies for not catching this issue sooner. Unfortunately users who have updated prior to this update will need to login via SSH and start service-runner manually to initiate the update process to then fix for subsequent updates. The command to do this once logged in is:

sudo servicectl start service-runner

then run emonpi update from the emoncms Administration menu.

Note: It can take 5 minutes for all the services to come online after a reboot on jessie. The new stretch image is a bit faster. This caught me out to start with as I thought none of the services where starting but they did all start up fine after a couple of minutes.

borpin · 2019-03-04T11:45:38.227Z

TrystanLea:

Unfortunately users who have updated prior to this update

One issue here is that there is no way of knowing what state the ‘emonpi’ part of an EmonPi is in as there is no versioning of the emonpi repository. I have opened an issue.