Service-runner shuts down periodically

GLe1 · 3 October 2026 09:57

When I view the Admin/System Info page, I see that service-runner is disabled. When I click “Start,” sometimes it restarts, but more often than not it doesn’t—it’s very unpredictable. If I go to the Backup page, there’s also a message about this. Sometimes refreshing the page shows the service as active, but more often than not, it doesn’t.
Then, after a while, without doing anything, the service comes back on its own.
So far, this hasn’t affected the automatic nightly backups.

I have three Raspberry Pi units installed with EmonCMS on SSDs using the EmonCMS Installation Scripts.

Raspberry 1: Pi 4 SSD/USB / EmonCMS 11.15.17
Raspberry 2: Pi 5 NVMe SSD / EmonCMS 11.18.0
Raspberry 3: Pi 5 NVMe SSD / EmonCMS 11.19.8

Raspberry 1 does not have this issue.
Only Raspberry 2 and 3 are affected.

It seemed like this might be caused by the browser (Firefox), but testing on Edge and on a smartphone yields the same result.

So, I ran some checks via SSH using a command to verify the status of service-runner, and the command’s output confirms what I’m seeing in Firefox.

Is this an issue that might prevent automatic backups from running properly?

TrystanLea · 5 October 2026 16:12

Hello @GLe1, Thanks for the detail. The Start button on the admin page sends its command through service-runner itself, so it can’t start service-runner when it is down. When it appears to work, systemd has restarted it on its own; it retries every 30 seconds after a crash. This suggests the service is crashing repeatedly on the two Pi 5 systems.

Could you post the output of the following from one of the affected Pis?

systemctl status service-runner --no-pager -n 50
systemctl show service-runner -p NRestarts
sudo journalctl -u service-runner --since "1 day ago" --no-pager | tail -n 100
python3 -c "import redis; print(redis.__version__)"
cat /etc/os-release | head -n 4

The journal should show a Python traceback if the script is crashing.

Scheduled nightly backups don’t use service-runner, so they are not affected. Only actions started from the web interface (manual backup, updates, service buttons) need it.

GLe1 · 5 October 2026 17:28

Hello Trystan,
I started the scripts one by one whilst service-runner was down, and by the time I’d finished, it was up and running again.
Below the output from emonpi2 :

pi@emonpi2:\~ $ systemctl status service-runner --no-pager -n 50
● service-runner.service - Emoncms service-runner Input Script
Loaded: loaded (/etc/systemd/system/service-runner.service; enabled; preset: enabled)
Active: active (running) since Mon 2026-10-05 19:07:39 CEST; 24s ago
Docs: https://github.com/emoncms/emoncms/blob/master/scripts/services/install-service-runner-update.md
Main PID: 177571 (python3)
Tasks: 1 (limit: 2321)
CPU: 169ms
CGroup: /system.slice/service-runner.service
└─177571 /usr/bin/python3 /var/www/emoncms/scripts/services/servic…

Oct 05 19:07:39 emonpi2 systemd\[1\]: Started service-runner.service - Emoncm…ipt.
Oct 05 19:07:40 emonpi2 service-runner\[177571\]: Starting service-runner
Oct 05 19:07:40 emonpi2 service-runner\[177571\]: Connected to redis server
Hint: Some lines were ellipsized, use -l to show in full.

pi@emonpi2:\~ $ systemctl show service-runner -p NRestarts
NRestarts=2970

pi@emonpi2:\~ $ sudo journalctl -u service-runner --since "1 day ago" --no-pager | tail -n 100
Oct 05 19:10:06 emonpi2 service-runner\[177930\]:   File "/home/pi/.local/lib/python3.11/site-packages/redis/commands/core.py", line 4862, in blpop
Oct 05 19:10:06 emonpi2 service-runner\[177930\]:     return self.execute_command("BLPOP", \*keys)
Oct 05 19:10:06 emonpi2 service-runner\[177930\]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Oct 05 19:10:06 emonpi2 service-runner\[177930\]:   File "/home/pi/.local/lib/python3.11/site-packages/redis/client.py", line 867, in execute_command
Oct 05 19:10:06 emonpi2 service-runner\[177930\]:     return self.\_execute_command(\*args, \*\*options)
Oct 05 19:10:06 emonpi2 service-runner\[177930\]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Oct 05 19:10:06 emonpi2 service-runner\[177930\]:   File "/home/pi/.local/lib/python3.11/site-packages/redis/client.py", line 889, in \_execute_command
Oct 05 19:10:06 emonpi2 service-runner\[177930\]:     result = conn.retry.call_with_retry(
Oct 05 19:10:06 emonpi2 service-runner\[177930\]:              ^^^^^^^^^^^^^^^^^^^^^^^^^^^
Oct 05 19:10:06 emonpi2 service-runner\[177930\]:   File "/home/pi/.local/lib/python3.11/site-packages/redis/retry.py", line 132, in call_with_retry
Oct 05 19:10:06 emonpi2 service-runner\[177930\]:     raise error
Oct 05 19:10:06 emonpi2 service-runner\[177930\]:   File "/home/pi/.local/lib/python3.11/site-packages/redis/retry.py", line 120, in call_with_retry
Oct 05 19:10:06 emonpi2 service-runner\[177930\]:     return do()
Oct 05 19:10:06 emonpi2 service-runner\[177930\]:            ^^^^
Oct 05 19:10:06 emonpi2 service-runner\[177930\]:   File "/home/pi/.local/lib/python3.11/site-packages/redis/client.py", line 890, in
Oct 05 19:10:06 emonpi2 service-runner\[177930\]:     lambda: self.\_send_command_parse_response(
Oct 05 19:10:06 emonpi2 service-runner\[177930\]:             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Oct 05 19:10:06 emonpi2 service-runner\[177930\]:   File "/home/pi/.local/lib/python3.11/site-packages/redis/client.py", line 819, in \_send_command_parse_response
Oct 05 19:10:06 emonpi2 service-runner\[177930\]:     return self.parse_response(conn, command_name, \*\*options)
Oct 05 19:10:06 emonpi2 service-runner\[177930\]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Oct 05 19:10:06 emonpi2 service-runner\[177930\]:   File "/home/pi/.local/lib/python3.11/site-packages/redis/client.py", line 935, in parse_response
Oct 05 19:10:06 emonpi2 service-runner\[177930\]:     response = connection.read_response()
Oct 05 19:10:06 emonpi2 service-runner\[177930\]:                ^^^^^^^^^^^^^^^^^^^^^^^^^^
Oct 05 19:10:06 emonpi2 service-runner\[177930\]:   File "/home/pi/.local/lib/python3.11/site-packages/redis/connection.py", line 1407, in read_response
Oct 05 19:10:06 emonpi2 service-runner\[177930\]:     response = self.\_parser.read_response(
Oct 05 19:10:06 emonpi2 service-runner\[177930\]:                ^^^^^^^^^^^^^^^^^^^^^^^^^^^
Oct 05 19:10:06 emonpi2 service-runner\[177930\]:   File "/home/pi/.local/lib/python3.11/site-packages/redis/\_parsers/resp3.py", line 40, in read_response
Oct 05 19:10:06 emonpi2 service-runner\[177930\]:     result = self.\_read_response(
Oct 05 19:10:06 emonpi2 service-runner\[177930\]:              ^^^^^^^^^^^^^^^^^^^^
Oct 05 19:10:06 emonpi2 service-runner\[177930\]:   File "/home/pi/.local/lib/python3.11/site-packages/redis/\_parsers/resp3.py", line 65, in \_read_response
Oct 05 19:10:06 emonpi2 service-runner\[177930\]:     raw = self.\_buffer.readline(timeout=timeout)
Oct 05 19:10:06 emonpi2 service-runner\[177930\]:           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Oct 05 19:10:06 emonpi2 service-runner\[177930\]:   File "/home/pi/.local/lib/python3.11/site-packages/redis/\_parsers/socket.py", line 117, in readline
Oct 05 19:10:06 emonpi2 service-runner\[177930\]:     self.\_read_from_socket(timeout=timeout)
Oct 05 19:10:06 emonpi2 service-runner\[177930\]:   File "/home/pi/.local/lib/python3.11/site-packages/redis/\_parsers/socket.py", line 77, in \_read_from_socket
Oct 05 19:10:06 emonpi2 service-runner\[177930\]:     raise TimeoutError("Timeout reading from socket")
Oct 05 19:10:06 emonpi2 service-runner\[177930\]: redis.exceptions.TimeoutError: Timeout reading from socket
Oct 05 19:10:06 emonpi2 systemd\[1\]: service-runner.service: Main process exited, code=exited, status=1/FAILURE
Oct 05 19:10:06 emonpi2 systemd\[1\]: service-runner.service: Failed with result 'exit-code'.
Oct 05 19:10:37 emonpi2 systemd\[1\]: service-runner.service: Scheduled restart job, restart counter is at 2971.
Oct 05 19:10:37 emonpi2 systemd\[1\]: Stopped service-runner.service - Emoncms service-runner Input Script.
Oct 05 19:10:37 emonpi2 systemd\[1\]: Started service-runner.service - Emoncms service-runner Input Script.
Oct 05 19:10:37 emonpi2 service-runner\[178021\]: Starting service-runner
Oct 05 19:10:37 emonpi2 service-runner\[178021\]: Connected to redis server
Oct 05 19:11:36 emonpi2 service-runner\[178021\]: Traceback (most recent call last):
Oct 05 19:11:36 emonpi2 service-runner\[178021\]:   File "/home/pi/.local/lib/python3.11/site-packages/redis/\_parsers/socket.py", line 64, in \_read_from_socket
Oct 05 19:11:36 emonpi2 service-runner\[178021\]:     data = sock.recv(socket_read_size)
Oct 05 19:11:36 emonpi2 service-runner\[178021\]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
Oct 05 19:11:36 emonpi2 service-runner\[178021\]: TimeoutError: timed out
Oct 05 19:11:36 emonpi2 service-runner\[178021\]: During handling of the above exception, another exception occurred:
Oct 05 19:11:36 emonpi2 service-runner\[178021\]: Traceback (most recent call last):
Oct 05 19:11:36 emonpi2 service-runner\[178021\]:   File "/var/www/emoncms/scripts/services/service-runner/service-runner.py", line 210, in
Oct 05 19:11:36 emonpi2 service-runner\[178021\]:     main()
Oct 05 19:11:36 emonpi2 service-runner\[178021\]:   File "/var/www/emoncms/scripts/services/service-runner/service-runner.py", line 146, in main
Oct 05 19:11:36 emonpi2 service-runner\[178021\]:     packed = server.blpop(KEYS)
Oct 05 19:11:36 emonpi2 service-runner\[178021\]:              ^^^^^^^^^^^^^^^^^^
Oct 05 19:11:36 emonpi2 service-runner\[178021\]:   File "/home/pi/.local/lib/python3.11/site-packages/redis/commands/core.py", line 4862, in blpop
Oct 05 19:11:36 emonpi2 service-runner\[178021\]:     return self.execute_command("BLPOP", \*keys)
Oct 05 19:11:36 emonpi2 service-runner\[178021\]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Oct 05 19:11:36 emonpi2 service-runner\[178021\]:   File "/home/pi/.local/lib/python3.11/site-packages/redis/client.py", line 867, in execute_command
Oct 05 19:11:36 emonpi2 service-runner\[178021\]:     return self.\_execute_command(\*args, \*\*options)
Oct 05 19:11:36 emonpi2 service-runner\[178021\]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Oct 05 19:11:36 emonpi2 service-runner\[178021\]:   File "/home/pi/.local/lib/python3.11/site-packages/redis/client.py", line 889, in \_execute_command
Oct 05 19:11:36 emonpi2 service-runner\[178021\]:     result = conn.retry.call_with_retry(
Oct 05 19:11:36 emonpi2 service-runner\[178021\]:              ^^^^^^^^^^^^^^^^^^^^^^^^^^^
Oct 05 19:11:36 emonpi2 service-runner\[178021\]:   File "/home/pi/.local/lib/python3.11/site-packages/redis/retry.py", line 132, in call_with_retry
Oct 05 19:11:36 emonpi2 service-runner\[178021\]:     raise error
Oct 05 19:11:36 emonpi2 service-runner\[178021\]:   File "/home/pi/.local/lib/python3.11/site-packages/redis/retry.py", line 120, in call_with_retry
Oct 05 19:11:36 emonpi2 service-runner\[178021\]:     return do()
Oct 05 19:11:36 emonpi2 service-runner\[178021\]:            ^^^^
Oct 05 19:11:36 emonpi2 service-runner\[178021\]:   File "/home/pi/.local/lib/python3.11/site-packages/redis/client.py", line 890, in
Oct 05 19:11:36 emonpi2 service-runner\[178021\]:     lambda: self.\_send_command_parse_response(
Oct 05 19:11:36 emonpi2 service-runner\[178021\]:             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Oct 05 19:11:36 emonpi2 service-runner\[178021\]:   File "/home/pi/.local/lib/python3.11/site-packages/redis/client.py", line 819, in \_send_command_parse_response
Oct 05 19:11:36 emonpi2 service-runner\[178021\]:     return self.parse_response(conn, command_name, \*\*options)
Oct 05 19:11:36 emonpi2 service-runner\[178021\]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Oct 05 19:11:36 emonpi2 service-runner\[178021\]:   File "/home/pi/.local/lib/python3.11/site-packages/redis/client.py", line 935, in parse_response
Oct 05 19:11:36 emonpi2 service-runner\[178021\]:     response = connection.read_response()
Oct 05 19:11:36 emonpi2 service-runner\[178021\]:                ^^^^^^^^^^^^^^^^^^^^^^^^^^
Oct 05 19:11:36 emonpi2 service-runner\[178021\]:   File "/home/pi/.local/lib/python3.11/site-packages/redis/connection.py", line 1407, in read_response
Oct 05 19:11:36 emonpi2 service-runner\[178021\]:     response = self.\_parser.read_response(
Oct 05 19:11:36 emonpi2 service-runner\[178021\]:                ^^^^^^^^^^^^^^^^^^^^^^^^^^^
Oct 05 19:11:36 emonpi2 service-runner\[178021\]:   File "/home/pi/.local/lib/python3.11/site-packages/redis/\_parsers/resp3.py", line 40, in read_response
Oct 05 19:11:36 emonpi2 service-runner\[178021\]:     result = self.\_read_response(
Oct 05 19:11:36 emonpi2 service-runner\[178021\]:              ^^^^^^^^^^^^^^^^^^^^
Oct 05 19:11:36 emonpi2 service-runner\[178021\]:   File "/home/pi/.local/lib/python3.11/site-packages/redis/\_parsers/resp3.py", line 65, in \_read_response
Oct 05 19:11:36 emonpi2 service-runner\[178021\]:     raw = self.\_buffer.readline(timeout=timeout)
Oct 05 19:11:36 emonpi2 service-runner\[178021\]:           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Oct 05 19:11:36 emonpi2 service-runner\[178021\]:   File "/home/pi/.local/lib/python3.11/site-packages/redis/\_parsers/socket.py", line 117, in readline
Oct 05 19:11:36 emonpi2 service-runner\[178021\]:     self.\_read_from_socket(timeout=timeout)
Oct 05 19:11:36 emonpi2 service-runner\[178021\]:   File "/home/pi/.local/lib/python3.11/site-packages/redis/\_parsers/socket.py", line 77, in \_read_from_socket
Oct 05 19:11:36 emonpi2 service-runner\[178021\]:     raise TimeoutError("Timeout reading from socket")
Oct 05 19:11:36 emonpi2 service-runner\[178021\]: redis.exceptions.TimeoutError: Timeout reading from socket
Oct 05 19:11:36 emonpi2 systemd\[1\]: service-runner.service: Main process exited, code=exited, status=1/FAILURE
Oct 05 19:11:36 emonpi2 systemd\[1\]: service-runner.service: Failed with result 'exit-code'.
Oct 05 19:12:06 emonpi2 systemd\[1\]: service-runner.service: Scheduled restart job, restart counter is at 2972.
Oct 05 19:12:06 emonpi2 systemd\[1\]: Stopped service-runner.service - Emoncms service-runner Input Script.
Oct 05 19:12:06 emonpi2 systemd\[1\]: Started service-runner.service - Emoncms service-runner Input Script.
Oct 05 19:12:06 emonpi2 service-runner\[178089\]: Starting service-runner
Oct 05 19:12:06 emonpi2 service-runner\[178089\]: Connected to redis server

pi@emonpi2:\~ $ python3 -c "import redis; print(redis.**version**)"
8.1.0

pi@emonpi2:\~ $ cat /etc/os-release | head -n 4
PRETTY_NAME="Debian GNU/Linux 12 (bookworm)"
NAME="Debian GNU/Linux"
VERSION_ID="12"
VERSION="12 (bookworm)"

TrystanLea · 5 October 2026 19:16

Hello @GLe1 looks like this is likely a default timeout settings change in redis-py 8.1.0. My own system is running redis-py 4.3.4, I will check what we are running on the latest images, i guess we might just not be running this latest version yet and so haven’t encountered this issue..

Could you try this patch on one of the Pi 5s:

cd /var/www/emoncms
cat > /tmp/service-runner.patch <<'EOF'
diff --git a/scripts/services/service-runner/service-runner.py b/scripts/services/service-runner/service-runner.py
--- a/scripts/services/service-runner/service-runner.py
+++ b/scripts/services/service-runner/service-runner.py
@@ -142,12 +142,12 @@ def main():
     server = connect_redis()
     while True:
         try:
-            # Get the next item from the queue, blocking until one exists
-            packed = server.blpop(KEYS)
+            # Block for up to 30s, below the redis-py socket timeout
+            packed = server.blpop(KEYS, timeout=30)
             if not packed:
                 continue
             raw = packed[1].decode()
-        except redis.exceptions.ConnectionError:
+        except (redis.exceptions.ConnectionError, redis.exceptions.TimeoutError):
             print("Connection to redis server lost, attempting to reconnect", flush=True)
             server = connect_redis()
             continue
EOF
git apply /tmp/service-runner.patch
sudo systemctl restart service-runner

After 10 minutes or so, please post the output of:

systemctl show service-runner -p NRestarts
sudo journalctl -u service-runner --since "15 min ago" --no-pager | tail -n 30

If this works and after I get a chance to test and push this as a proper fix to stable you will need to undo the patch. The steps to do this as a reference for later are:

cd /var/www/emoncms
git checkout scripts/services/service-runner/service-runner.py
sudo systemctl restart service-runner

GLe1 · 5 October 2026 21:46

I remember that at first I had some issues with service-runner, and on Gemini’s advice, I updated Redis, but that didn’t change anything.
I didn’t quite understand the patch script—I’m a beginner.
I ran it, and here’s the log output:

pi@emonpi3:/var/www/emoncms $ systemctl show service-runner -p NRestarts
sudo journalctl -u service-runner --since "15 min ago" --no-pager | tail -n 30
NRestarts=0
Oct 05 23:19:37 emonpi3 service-runner\[186968\]: Connection to redis server lost, attempting to reconnect
Oct 05 23:19:37 emonpi3 service-runner\[186968\]: Connected to redis server
Oct 05 23:20:36 emonpi3 service-runner\[186968\]: Connection to redis server lost, attempting to reconnect
Oct 05 23:20:36 emonpi3 service-runner\[186968\]: Connected to redis server
Oct 05 23:21:36 emonpi3 service-runner\[186968\]: Connection to redis server lost, attempting to reconnect
Oct 05 23:21:36 emonpi3 service-runner\[186968\]: Connected to redis server
Oct 05 23:22:35 emonpi3 service-runner\[186968\]: Connection to redis server lost, attempting to reconnect
Oct 05 23:22:35 emonpi3 service-runner\[186968\]: Connected to redis server
Oct 05 23:23:34 emonpi3 service-runner\[186968\]: Connection to redis server lost, attempting to reconnect
Oct 05 23:23:34 emonpi3 service-runner\[186968\]: Connected to redis server
Oct 05 23:24:33 emonpi3 service-runner\[186968\]: Connection to redis server lost, attempting to reconnect
Oct 05 23:24:33 emonpi3 service-runner\[186968\]: Connected to redis server
Oct 05 23:25:32 emonpi3 service-runner\[186968\]: Connection to redis server lost, attempting to reconnect
Oct 05 23:25:32 emonpi3 service-runner\[186968\]: Connected to redis server
Oct 05 23:26:32 emonpi3 service-runner\[186968\]: Connection to redis server lost, attempting to reconnect
Oct 05 23:26:32 emonpi3 service-runner\[186968\]: Connected to redis server
Oct 05 23:27:31 emonpi3 service-runner\[186968\]: Connection to redis server lost, attempting to reconnect
Oct 05 23:27:31 emonpi3 service-runner\[186968\]: Connected to redis server
Oct 05 23:28:30 emonpi3 service-runner\[186968\]: Connection to redis server lost, attempting to reconnect
Oct 05 23:28:30 emonpi3 service-runner\[186968\]: Connected to redis server
Oct 05 23:29:30 emonpi3 service-runner\[186968\]: Connection to redis server lost, attempting to reconnect
Oct 05 23:29:30 emonpi3 service-runner\[186968\]: Connected to redis server
Oct 05 23:30:29 emonpi3 service-runner\[186968\]: Connection to redis server lost, attempting to reconnect
Oct 05 23:30:29 emonpi3 service-runner\[186968\]: Connected to redis server
Oct 05 23:31:28 emonpi3 service-runner\[186968\]: Connection to redis server lost, attempting to reconnect
Oct 05 23:31:28 emonpi3 service-runner\[186968\]: Connected to redis server
Oct 05 23:32:27 emonpi3 service-runner\[186968\]: Connection to redis server lost, attempting to reconnect
Oct 05 23:32:27 emonpi3 service-runner\[186968\]: Connected to redis server
Oct 05 23:33:25 emonpi3 service-runner\[186968\]: Connection to redis server lost, attempting to reconnect
Oct 05 23:33:25 emonpi3 service-runner\[186968\]: Connected to redis server

It seems like service-runner isn’t shutting down anymore, but I need to monitor it for longer before drawing any conclusions. I’ll monitor it for a full day to confirm.

[Edit by Mod - please use Ctl-E so code/log output is formatted]

GLe1 · 6 October 2026 12:15

Since yesterday, I’ve been monitoring the Pi 5 that I patched with your script, and there are no longer any issues with service-runner. The logs show that the service hasn’t stopped at all.
The other Pi 5 that hasn’t been patched is still generating just as many errors.
The logs confirm this, so your script is really effective.
Just let me know when an update is available and I’ll need to undo the patch.
Thanks

TrystanLea · 6 October 2026 13:44

Thanks @GLe1 great to hear! will let you know once a fix is live.