Mqtt_input script run as systemd instead of initd

Andy_Taylor · 2017-01-06T11:03:18.964Z

Argh Systemd - where to start with that…

It’s supposed to be the fix-all solution to everything, and to some extent it is, but we’re at the point where there are some stupid dependancies relating to desktop type services that now rely on systemd, crazy IMO.

As for logging - there are usually two way to specify the log, in the unit file or in the config for the application / process.
Looking at the old init.d init script - you were setting the log file there previously;

NAME=mqtt_input
DESC="Daemon for the emoncms MQTT script"
PIDFILE="/var/run/${NAME}.pid"
LOGFILE="/var/log/${NAME}.log"

DAEMON="/usr/bin/php"
DAEMON_OPTS="/var/www/emoncms/scripts/phpmqtt_input.php"

Since this is the case, we should be doing the same in the systemd unit file, and this is where you run into problems.

Using the init.d - you were actually redirecting the output from the application to STDOUT to a log file, you can replicate that in systemd - kinda…

I have modified my version of the unit file as follows;

# Systemd unit file for mqtt input script

# INSTALL:
# sudo cp /var/www/emoncms/scripts/mqtt_input.service /etc/systemd/system/mqtt_input.service

# RUN AT STARTUP
# sudo systemctl daemon-reload
# sudo systemctl enable mqtt_service

# START / STOP With:
# sudo systemctl start mqtt_input
# sudo systemctl stop mqtt_input    

# VIEW STATUS / LOG
# sudo systemctl status mqtt_input -n50
# where -nX is the number of log lines to view 

###
#
#  All Emoncms code is released under the GNU Affero General Public License.
#  See COPYRIGHT.txt and LICENSE.txt.
#
#  ---------------------------------------------------------------------
#  Emoncms - open source energy visualisation
#  Part of the OpenEnergyMonitor project:
#  http://openenergymonitor.org
###

[Unit]
Description=Emoncms MQTT Input Script
After=mosquitto.service mysql.service redis.service

[Service]
Type=forking
ExecStart=/bin/sh -c '/usr/bin/php /var/www/emoncms/scripts/phpmqtt_input.php 2>&1 > /var/log/mqtt_input.log &'

# Restart script if stopped
#Restart=always
Restart=on-failure
# Wait 60s before restart
RestartSec=60

[Install]
WantedBy=multi-user.target

One other note - running the update failed on my Pi - the init.d script was removed, but the systemd unit file was not added to /etc/systemd/system - check that you have the file system writable at that point in the script…

Also because you are modifying systemd - make sure you call “systemctl daemon-reload” after adding the file, and then make it active with “systemctl enable mqtt_input”.

I have not checked your update script yet to see where these errors come from yet…

glyn.hudson · 2017-01-06T11:33:25.515Z

jimmy:

Having the ability to grep, awk and other text manipulation commands is vital for us for debugging

Yup, totally agree. The systemd logs can be outputted as text using journalctl then the usual text manipulation tools can be applied.

sudo journalctl -f -u mqtt_input -o cat | grep emontx

I’ve just updated the unit file to tag the systemd log entries with mqtt_input, this makes them easier to search.

github.com/emoncms/emoncms

update mqtt_service systemd unit file to tag logs and give other logging options

Commit by

glynhudson in update mqtt_service systemd unit file to tag logs and give other logg…

emoncms:stable ← emoncms:master

Andy_Taylor · 2017-01-06T11:52:25.679Z

I should point out that all my changes do - is revert to using the /var/log/mqtt_input.log rather than using /var/log/daemon.log as systemd will do by default.

You / We still need to work out why when I updated the init.d script was removed but the systemd script was not added.

pb66 · 2017-01-06T12:39:03.403Z

I tried sudo journalctl -f -u mqtt_input -o cat | grep emontx ( and some similar searches) before changing anything and found it worked ok prior to the tag being added, also there is no time and date printed can that be set in command line?

Since updating emonpi I am unable to restart the service and have an error message

pi@emonpi(rw):~$ sudo systemctl restart mqtt_input.service
Failed to restart mqtt_input.service: Unit mqtt_input.service failed to load: Bad message. See system logs and 'systemctl status mqtt_input.service' for details.

pi@emonpi(rw):~$ systemctl status mqtt_input.service
● mqtt_input.service - Emoncms MQTT Input Script
   Loaded: error (Reason: Bad message)
   Active: active (running) since Thu 2017-01-05 13:34:59 UTC; 22h ago
     Docs: https://github.com/emoncms/emoncms/blob/master/docs/RaspberryPi/MQTT.md
 Main PID: 1832 (php)
   CGroup: /system.slice/mqtt_input.service
           └─1832 /usr/bin/php /var/www/emoncms/scripts/phpmqtt_input.php

I have had a look and see no obvious (to me) reason, the update was done after the typo corrections to the comment lines.

Since the above search worked prior to the update and yet the “Docs” url has been added to the status output I’m unclear if the new unit is in service or not, but the daemon.log has not changed, I expected to see the tag there or is the daemon.log an additional log to a seperate systemd internal log? if it is, then great we can redirect what is going to the daemon.log to the mqtt_input.log and the systemd/systemctl/journalctl internal log will remain intact perhaps?

I notice @glyn.hudson that you have also added a commented out option to log to mqtt_input.log, but won’t that get overwritten at every emonpi update?

glyn.hudson · 2017-01-06T12:58:57.660Z

Andy_Taylor:

I should point out that all my changes do - is revert to using the /var/log/mqtt_input.log rather than using /var/log/daemon.log as systemd will do by default.

What changes are these? Sorry if I missed something. As I’m typing this, your order has just been shipped

pb66:

Since updating emonpi I am unable to restart the service and have an error message

Woops, my fault. I noticed a typo in the unit file which a corrected a few seconds later on the master branch. I created a PR to merge into stable but forgot to merge it…until now. Please try and update again

github.com/emoncms/emoncms

comment typo

emoncms:stable ← emoncms:master

opened 11:40AM - 06 Jan 17 UTC

glynhudson

+1 -1

Andy_Taylor · 2017-01-06T13:10:29.382Z

The changes are the ones a few posts up where I pasted a copy of the systemd unit file I modified to make mutt_input log to the correct file again.

pb66 · 2017-01-06T13:28:40.318Z

@glyn.hudson thanks the correction fixed the unit issue, unfortunately your alternative log option doesn’t seem to work, even after reloading and restarting the unit the logs still go to the daemon.log, although I can now see the “mqtt_inpu” tag in place of the “php” tag.

pi@emonpi(rw):~$ tail  /var/log/daemon.log
Jan  6 13:06:35 emonpi mqtt_input[22099]: emon/emonpi/3 0
Jan  6 13:06:35 emonpi mqtt_input[22099]: emon/emonpi/4 0
Jan  6 13:06:35 emonpi mqtt_input[22099]: emon/emonpi/5 0
Jan  6 13:06:35 emonpi mqtt_input[22099]: emon/emonpi/6 0
Jan  6 13:06:35 emonpi mqtt_input[22099]: emon/emonpi/7 0
Jan  6 13:06:35 emonpi mqtt_input[22099]: emon/emonpi/8 0
Jan  6 13:06:35 emonpi mqtt_input[22099]: emon/emonpi/9 0
Jan  6 13:06:35 emonpi mqtt_input[22099]: emon/emonpi/10 0
Jan  6 13:06:35 emonpi mqtt_input[22099]: emon/emonpi/11 0
Jan  6 13:06:35 emonpi mqtt_input[22099]: emon/emonpi/rssi 0

I tried @Andy_Taylor’s code

Type=forking
ExecStart=/bin/sh -c '/usr/bin/php /var/www/emoncms/scripts/phpmqtt_input.php 2>&1 > /var/log/mqtt_input.log &'

and that does send the logs to mqtt_input,log or more importantly, does not send them to ‘daemon.log’, (thanks Andy) however there is no time and date, I have not yet checked to see if this is due to the actual script not attaching it or if it is getting removed en-route.

pi@emonpi(rw):~$ tail  /var/log/mqtt_input.log
emon/emonpi/9 0
emon/emonpi/10 0
emon/emonpi/11 0
emon/emonpi/rssi 0
emon/emontx/Vrms 238.13
emon/emontx/power1 121
emon/emontx/power2 166
emon/emontx/power3 3
emon/emontx/temp 5.25
emon/emontx/rssi -69

For now I have also blindly carried over the “Type=forking” without really knowing which type is required.

glyn.hudson · 2017-01-06T13:38:47.254Z

Apologies, I missed your post above. Got it now, thanks.

Is possible I would like to move to using the systemd journal log, this seems to be the preferred way to go ‘all-in’ with systemd. This is how nodeRED runs. I have been looking at the nodeRED unit file as an example.

However, it would be good to give the option for users to log to mqtt_input.log if they prefer as a commented out option in the unit file which can easily be enabled.

pb66:

unfortunately your alternative log option doesn’t seem to work,

I have updated the unit file to use @Andy_Taylor logging code as an alternative commented out option. I don’t quite understand what is meant by forking. I will leaf type=forking commented out.

from: [Solved] systemd - To fork or not to fork when writing services / Applications & Desktop Environments / Arch Linux Forums

If you have a choice then use Type=simple for your systemd services. It is simpler and systemd will take care or proper process deamonizing, restarts and shutdowns.

Type=forking is for historical services that cannot be run in foreground. In this case process itself will take care of its deamonizing, but systemd should know the child pid somehow so you need to provide PIDFile= option as well.

I choose type=idle to ensure the script starts after all other services have started. Here is a full explanation of the different service types:

https://wiki.archlinux.org/index.php/Systemd#Service_types

I will keep these changes in the master branch until they have been more tested:

github.com

emoncms/emoncms/blob/master/scripts/mqtt_input.service#L39


      
          
          [Unit]
          Description=Emoncms MQTT Input Script
          After=mosquitto.service mysql.service redis-server.service
          Documentation=https://github.com/emoncms/emoncms/blob/master/docs/RaspberryPi/MQTT.md
          
          [Service]
          Type=idle
          ExecStart=/usr/bin/php /var/www/emoncms/scripts/phpmqtt_input.php
          
          # Uncomment instead of above to use standard log file, else use systemd log
          # Type=forking
          # ExecStart=/bin/sh -c '/usr/bin/php /var/www/emoncms/scripts/phpmqtt_input.php 2>&1 > /var/log/mqtt_input.log &'
          
          # Restart script if stopped
          Restart=always
          # Wait 60s before restart
          RestartSec=60
          
          # Tag things in the log
          # View with: sudo journalctl -f -u mqtt_input -o cat

glyn.hudson · 2017-01-06T13:47:07.199Z

pb66:

I tried sudo journalctl -f -u mqtt_input -o cat | grep emontx ( and some similar searches) before changing anything and found it worked ok prior to the tag being added, also there is no time and date printed can that be set in command line?

Yes adding -o short instead of -o cat will display the date

sudo journalctl -f -u mqtt_input -o short
-- Logs begin at Fri 2017-01-06 08:35:09 UTC. --
Jan 06 13:44:13 emonpi mqtt_input[18518]: emon/emonpi/t5 0
Jan 06 13:44:13 emonpi mqtt_input[18518]: emon/emonpi/t6 0
Jan 06 13:44:13 emonpi mqtt_input[18518]: emon/emonpi/pulsecount 1032
Jan 06 13:44:13 emonpi mqtt_input[18518]: emon/emonpi/rssi 0
Jan 06 13:44:13 emonpi mqtt_input[18518]: emon/emonth5/temperature 13
Jan 06 13:44:13 emonpi mqtt_input[18518]: emon/emonth5/external temperature 13
Jan 06 13:44:13 emonpi mqtt_input[18518]: emon/emonth5/humidity 64.9
Jan 06 13:44:13 emonpi mqtt_input[18518]: emon/emonth5/battery 2.7
Jan 06 13:44:13 emonpi mqtt_input[18518]: emon/emonth5/pulsecount 1
Jan 06 13:44:13 emonpi mqtt_input[18518]: emon/emonth5/rssi -67
Jan 06 13:44:18 emonpi mqtt_input[18518]: emon/emonpi/power1 40
Jan 06 13:44:18 emonpi mqtt_input[18518]: emon/emonpi/power2 13
Jan 06 13:44:18 emonpi mqtt_input[18518]: emon/emonpi/power1pluspower2 53

There are lots of -o output options:

short, short-iso, short-precise, short-monotonic, verbose,export, json, json-pretty, json-sse, cat

glyn.hudson · 2017-01-06T14:34:53.549Z

Andy_Taylor:

One other note - running the update failed on my Pi - the init.d script was removed, but the systemd unit file was not added to /etc/systemd/system - check that you have the file system writable at that point in the script…

I’ve just tested again on a fresh emonPi and the update script worked. Could it be that you have un-committed changes to either ~/emonpi or /var/www/emoncms folders on your pi? Try git pulling these folder manually.

Yes the update script does a reload:

github.com

openenergymonitor/emonpi/blob/master/emoncmsupdate#L165


      
          #!/bin/bash
          # REDIRECT FOR UPDATE TO NEW UPDATE PATH LOCATIONS
          # KEEP THIS FILE HERE FOR ALL UPDATED FROM OCT17 and OCT18 IMAGES
          DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"
          usrdir=${DIR/\/emonpi/}
          $usrdir/emonpi/service-runner-update.sh all custom

Andy_Taylor · 2017-01-06T17:46:43.320Z

Glyn - anything is possible - my Pi gets quite messed with while I am working on things - I’ll re-image it in the next few days and try it again - if your update works - its almost certainly an issue with mine.

I will also take a look at the log mqtt logs before / after the update and see if I can bottom that out.

pb66 · 2017-01-06T20:06:35.665Z

Having played with this a bit this afternoon, I have discovered the “Type=forking” is required for Andy’s code to work, however I have also figured out that the mqtt_script is both “logging” and “echoing” messages and the messages we are trying to redirect are std out not “log messages” as I expected. eg

github.com

emoncms/emoncms/blob/master/scripts/phpmqtt_input.php#L113-L114

    
      
              $device = new Device($mysqli,$redis);
          }

I have found that the actual “log messages” are directed to emoncms.log if the loglevel is set correctly in settings.php and that the mqtt_input.log is superfluous.

This is why there is no timestamp when redirected.

I have run though phpmqtt_input.php and commented out most of (but not all) of the echo lines and returned the unit to it’s standard form and now the logging works as expected.

Paul · 2017-01-08T20:48:25.994Z

Just deleted my init.d script to start mqtt_input, and got the following results on a ‘self-build’ emoncms (up to date with git).

sudo cp /var/www/emoncms/scripts/mqtt_input.service /etc/systemd/system/mqtt_input.service
sudo systemctl daemon-reload
…everythink ok so far, but when I try and enable the service;
sudo systemctl enable mqtt_service
I get;
Failed to execute operation: No such file or directory

If I look in the /etc/systemd/system directory, I can see the mqtt_input.service file is present.

Checking the log, I see;
Loaded: loaded (/etc/systemd/system/mqtt_input.service; disabled) Active: inactive (dead)

I can manually start the service OK;
sudo systemctl start mqtt_input

…but as the service is not enabled, it won’t restart after a reboot.

Paul

pb66 · 2017-01-08T20:52:12.189Z

Paul:

…everythink ok so far, but when I try and enable the service;sudo systemctl enable mqtt_service

That should be sudo systemctl enable mqtt_input.service

Paul · 2017-01-08T20:59:23.494Z

pb66:

sudo systemctl enable mqtt_input.service

Cheers Paul. I’ve amended the command on the script fie, do you want to add this to your read.me git PR?

Paul

github.com/emoncms/emoncms

Amend enable command typo

committed 08:58PM - 08 Jan 17 UTC

Paul-Reed

+1 -1

pb66 · 2017-01-08T21:19:36.516Z

I have corrected the typo in the readme.md, but I think that section needs a re-write and the old instructions resurrecting as per my comment on the commit.

github.com/emoncms/emoncms

update install instructions for systemd mqtt_input

committed 10:05AM - 05 Jan 17 UTC

glynhudson

+19 -5

recomend using systemd instead of init.d for mqtt_input

Paul · 2017-01-08T22:42:02.810Z

I’ve done a quick re-write to add back the section which I wrote in Oct 2015, which uses init.d instead of systemd as an alternative solution for older OS’s.
…For consideration.

Paul

github.com/emoncms/emoncms

MQTT script support for non systemd systems

emoncms:master ← Paul-Reed:master

opened 10:21PM - 08 Jan 17 UTC

Paul-Reed

+14 -3

As raised by pb66, removing the readme section about init.d systems, stops non s…ystemd users from running the mqtt_input script.

glyn.hudson · 2017-01-09T14:16:01.246Z

Thanks a lot guys, the PR’s fixes and doc updates have been tested and merged (into master & stable).

pb66 · 2017-01-11T21:53:54.226Z

6 posts were split to a new topic: Unable to use certain symbols in MQTT topics with emonCMS mqtt_input

pb66 · 2018-01-15T20:42:09.024Z

4 posts were split to a new topic: Installing Mosquitto for emoncms