Looks like you’re running jessie, I thought there was an extra manual step to enable service-runner on jessie, something like linking the .service file into a different place?
Can you provide the output of:
ls -l /etc/systemd/system/service-runner*
ls -l /lib/systemd/system/service-runner*
Ah, that will be the problem. I do not have a Jessie based system.
@glyn.hudson, I suspect there is an issue on Jessie based distributions.
The script is here https://github.com/openenergymonitor/emonpi/blob/service_correction/emoncmsupdate and was discussed here Emoncms V9.9.6 Stable merge (pre-merge check).
There is a command to create the link in the script. I think I saw that the script itself may not have updated correctly in the logs above.
Running those command gives me
pi@emonpi(ro):~$ ls -l /etc/systemd/system/service-runner*
ls: cannot access /etc/systemd/system/service-runner*: No such file or directory
pi@emonpi(ro):~$ ls -l /lib/systemd/system/service-runner*
lrwxrwxrwx 1 root root 71 Feb 27 20:43 /lib/systemd/system/service-runner.service -> /var/www/emoncms/scripts/services/service-runner/service-runner.service
pi@emonpi(ro):~$
If I do need to run that script to correct as I have not done that for several months can I have a reminder how to do it from the PuTTY console?
Many thanks
Could you try this please.
cd ~/emonpi
git status
When you tested this, did you not discover systemd on Jessie did not follow symbolic links?
I suddenly realised I was having Deja Vu - Symlinking services in Jessie doesn't work
I think the solution may be to use a hard link instead.
I get this…
pi@emonpi(ro):~$ cd ~/emonpi
pi@emonpi(ro):emonpi$ git status
On branch master
Your branch is up-to-date with 'origin/master'.
Untracked files:
(use "git add <file>..." to include in what will be committed)
1
nothing added to commit but untracked files present (use "git add" to track)
pi@emonpi(ro):emonpi$
Hi @IM35461, sorry about this. It seems it is possibly an oddity of the Jessie systemd although I’ve searched out a Jessie system I have and I can see symlinks in the /lib/ systemd folder so do not know why these are not working. As it is a running system (doing other things) I really do not want to play with it!
The solution might be a slight change to the enable command. Can you try this and see if it creates an error message.
sudo systemctl enable /lib/systemd/system/service-runner.service
If no error, could you reboot and then do a status on the service please.
Alas it does not like that command 
pi@emonpi(ro):~$ sudo systemctl enable /lib/systemd/system/service-runner.service
Failed to execute operation: Too many levels of symbolic links
pi@emonpi(ro):~$
This is very perplexing as the feedwriter service is identical.
Could you possibly reboot and then do:
sudo systemctl status emoncms_mqtt service-runner feedwriter
If you then do a
sudo systemctl restart emoncms_mqtt
sudo systemctl restart service-runner
Everything will run until you next boot but that isn’t really a solution.
That seems to work and I was wondering if it may just be best to have a fresh start with a new MicroSD card…
pi@emonpi(ro):~$ sudo systemctl status emoncms_mqtt service-runner feedwriter
● emoncms_mqtt.service - Emoncms emoncms_mqtt script
Loaded: loaded (/var/www/emoncms/scripts/services/emoncms_mqtt/emoncms_mqtt.service; disabled)
Active: inactive (dead)
Docs: https://github.com/emoncms/emoncms/blob/master/docs/RaspberryPi/MQTT.md
● service-runner.service - Emoncms service-runner Input Script
Loaded: loaded (/var/www/emoncms/scripts/services/service-runner/service-runner.service; disabled)
Active: inactive (dead)
Docs: https://github.com/emoncms/emoncms/blob/master/scripts/services/install-service-runner-update.md
Feb 28 16:02:24 emonpi systemd[1]: [/var/www/emoncms/scripts/services/servi...t'
Feb 28 16:04:04 emonpi systemd[1]: [/var/www/emoncms/scripts/services/servi...t'
● feedwriter.service - Emoncms feedwriter script
Loaded: loaded (/var/www/emoncms/scripts/services/feedwriter/feedwriter.service; disabled)
Active: active (running) since Thu 2019-02-28 16:02:02 UTC; 2min 2s ago
Docs: https://github.com/emoncms/emoncms/blob/master/scripts/services/install-service-feedwriter.md
Main PID: 2918 (php)
CGroup: /system.slice/feedwriter.service
└─2918 /usr/bin/php /var/www/emoncms/scripts/feedwriter.php
Feb 28 16:02:02 emonpi systemd[1]: Started Emoncms feedwriter script.
Feb 28 16:02:03 emonpi feedwriter[2918]: Buffered feed writer daemon started....
Hint: Some lines were ellipsized, use -l to show in full.
pi@emonpi(ro):~$ sudo systemctl restart emoncms_mqtt
pi@emonpi(ro):~$ sudo systemctl restart service-runner
pi@emonpi(ro):~$
Could this have anything to do with the shenanigans that happens in the rc.local file? The feedwriter service is the only one of the 3 services that gets restarted in rc.local.
If this is still Jessie based, it is still a read-only filesystem, could that be causing any issue with the new services?
I know that the services do not have thier own log files, but are they dependent on other services that do require a log file (eg redis) therefore failing to start as the required service(s) have failed to start due to a missing log file and then only the feedwriter gets restarted once the log files are created, leaving the other 2 not running.
[edit]
As a quick test you could edit the service mqtt_input restart to service emoncms_mqtt restart or add it to the bottom of the list. Any services that cannot start (eg mqtt_input?) should be removed as rc.local will just abort on the first error and not continue with the rest of the commands.
Interesting point. @glyn.hudson I’m not sure this was changed as part of the update script.
Ah yes, very possible - obvious really. The enable command would be unable to create the symlinks in /etc/systemd/system/. Doesn’t explain why the script would fail though as it is in RW mode.
@IM35461, my mistake.
Could you try
rpi-rw
sudo systemctl enable service-runner.service
sudo systemctl start service-runner.service
rpi-ro
If the enable command is successful, reboot and see if it starts.
Well that would get you onto Stretch, which is a good idea.
Looking at my jessie based EmonBase, (which is where I found the issues on jessie mentioned above), I have:
pi@emonpi(ro):~$ ls -l /etc/systemd/system/service-runner*
lrwxrwxrwx 1 root root 42 Nov 13 10:26 /etc/systemd/system/service-runner.service -> /lib/systemd/system/service-runner.service
pi@emonpi(ro):~$ ls -l /lib/systemd/system/service-runner*
lrwxrwxrwx 1 root root 71 Nov 13 10:22 /lib/systemd/system/service-runner.service -> /var/www/emoncms/scripts/services/service-runner/service-runner.service
pi@emonpi(ro):~$
pi@emonpi(ro):~$ systemctl status service-runner
● service-runner.service - Emoncms service-runner Input Script
Loaded: loaded (/var/www/emoncms/scripts/services/service-runner/service-runner.service; enabled)
Active: active (running) since Tue 2019-02-19 06:17:34 UTC; 1 weeks 2 days ago
Main PID: 883 (python)
CGroup: /system.slice/service-runner.service
└─883 /usr/bin/python /var/www/emoncms/scripts/services/service-runner/service-runner.py
On this system, service-runner restarts successfully after a reboot.
So it looks like I manually linked /lib/systemd/system/service-runner.service into /etc/systemd/system
Trouble is, if you forget, need to disable it for some reason, then come to enable it again, it won’t work as the original symlink in /etc/systemd/system has been deleted by the disable command and you need to add in the link manually as well.
What is annoying is that it should work. I think the RO issue might be it - needs to be RW when enabled.
Do you have an entry in the rc.local?
Yep, that’s absolutely correct. Its a bug in the systemd version running on jessie and there’s no good way around it while still using systemd.
jessie is running systemd v215, the jessie-backports version is v230, stretch is running v232.
Its possible that the jessie-backports version fixes it but I’ve not tried it.
The instructions for using backports are here: Instructions
For this particular thread, replace stretch with jessie in all references in commands from that link.
Nothing related to service-runner.
pi@emonpi(ro):~$ ls -l /etc/rc.local
lrwxrwxrwx 1 root root 38 Apr 28 2016 /etc/rc.local -> /home/pi/emonpi/rc.local_jessieminimal
That jessieminimal file contains the code to create the log folders in tmpfs and then restart the services that need logs to exist
I just don’t buy that. If you look in /lib/systemd/system/ you will see symlinked units.
Don’t worry about me, I have gone for a fresh install and now seem to have 9.9.8 beta.
System can be restarted with no issues
I’m not trying to sell it to you 
I went through the exact same thought process at the time that you’re going through right now. “But it should work!”… I spent quite a few hours on this only to come to the realisation that there are things less frustrating to spend my time on 
