Mosquitto startup issues

emonSD
,
1

Continuing the discussion from Mosquitto won't start on boot after raspbian and emonsd update:

This issue has been in the back of my mind for a while, so I did a little more digging. I do wonder if it is in some way related to some of the other issues that have been seen with MQTT.

In any case, @jsroques came up with a solution for the mosquitto service, that is create a unit file that checks for the presence of the log file location before starting.

[Unit]
Description=Mosquitto MQTT Broker
Documentation=man:mosquitto(8) https://mosquitto.org/

[Service]
ExecStartPre=-/bin/mkdir -p /var/log/mosquitto
ExecStartPre=/bin/chown -R mosquitto:mosquitto /var/log/mosquitto
ExecStart=/usr/sbin/mosquitto -c /etc/mosquitto/mosquitto.conf
Restart=always

[Install]
WantedBy=multi-user.target

The issue raised with this, was the undesirable nature of modifying stock files.

A little more research for me; (every day a learning day) reveals that unit files for systemd can be placed in one of 3 places and each has a different precedence. Files in /etc followed by /run followed by /lib are used in that order. You can also add in snippets, but I’d suggest a complete unit file is the better solution. reference.

Therefore, by putting this replacement service file in /etc/systemd/system, systemd will use that over the stock file in /lib/systemd/system/

However, I am surprised that the stock mosquitto service file does not include a After=syslog.target directive as that may well prevent the issues (rather than check for and create the file paths) - just a thought.

The other related issue the above reference exposes, is relationship of the mqtt_input.service to the mosquiito.service. Implicitly, the mqtt service requires the mosquitto service so an After=: directive has been included (though the syntax should be .target rather than .service I believe). However, it seems this directive does not implicitly mean it waits until the mosquitto service has started as there is no actual dependency.

To ensure that dependency I think there is also a requirement for a Requires=mosquitto directive in the mqtt_input service file. Again though, I think this still allows systemd to start the 2 in parallel. To prevent that, I suggest a Before=mqtt_input.target would be required in the new mosquitto.service unit file.

A note of caution; whilst this may be fine for the emonpi, self installs might, for instance, not run the mosquitto broker on the same machine as the emoncms instance. Adding in a stricter dependency requirement, might just break the mqtt_input service for them.

On my part, this is all theory as I do not have an emonpi to test it on!

2

You are quite right and I agree that IF a replacement service is required, then the best way to implement that is to create a custom over-riding unit as you describe.

The reasoning behind the modified service unit was to ensure the log file was there on read-only OS’s when the log files are held in RAM and lost on a reboot. The method of using the service script was taken from emonhub, it has always created it’s own log file if it doesn’t already exist to make it robust, self-dependent and more suited to the read-only OS scenario.

However MQTT isn’t the only service that falls foul of the “missing log file” issue and I doubt I/we will convince all software providers to adopt this more robust route, that is why the thread you linked has evolved to explore ways of ensure log files are always there for all installed software, despite the RO OS and without resorting to writing a custom service unit for each and every software.

Although my own preference is that all services are robust and self-dependent, the use of a log file manager on a RO system should theoretically remove any need to use custom service units for this particular reason.

I also agree that it might be useful if the mosquitto and mqtt_input services were started in order and that editing the mosquitto service to start before the mqtt_input service might be a better approach than editing the mqtt_service to start after the mosquitto service so that the mqtt_input service can be used without a local broker, especially IF we had already been doing a custom service unit for the log file issue…

However, with that consideration in mind, this would not help a local emoncms that is connecting to a remote mosquitto broker, so it would be of limited use. The reality is that any MQTT device will at some point have to handle there being no available broker, whether it be a network issue interrupting a remote broker connection or a software update or local problem causing the local broker to be restarted etc.

The emoncms mqtt_input must be resilient to failed connections in it’s own right.

I’m not against making the mosquitto service more robust, but IMO the custom service unit should not be mandatory for the system wide log file issue and it should not matter what order the mqtt services start in, as any mqtt device should be able to cope with no broker available, especially for the potential few seconds between different services being started.

If the emoncms mqtt_input was robust in this way (and we globally resolved the RO logfiles issue), both the mosquitto broker and the emoncms mqtt input services could work in exactly the same way local or remote, RO OS or not, with no need to edit any service units.

3

I will agree to a point, but the installation of emoncms does involve the modification of standard files (the additions for getting REDIS to work for instance), so that should not be a block. If the install is scripted, you could use the systemctl edit function to create a suitable snippet in the script.

If all the log files need sorted, I came across systemd-tmpfiles during my research. Only problem is that every service that needs them might have to wait for that service to start…

4

I wouldn’t expect it to be a “block”, I’m just not a fan of change for the sake of change, I try to run as much stuff with default settings and configurations as possible for simplicity of installing, debugging and consistency across platforms, users and devices. Every time someone has an issue we need to find out what era, what guide, what differences etc. If it’s essential, I’m in, but I do prefer “stock”.

What are the Redis changes you refer to? I’ve never changed anything to install Redis. Do you mean the changes in the emonpi guide to redis.conf for using redis in a RO environment? If so I’m not convinced they are needed or that they improve anything, if the redis tables were persisted at shutdown it would make the whaccumulator type processes in emoncms less erroneous, besides that is part of the RO “optimizations” rather than something that is required to “run Redis”. A simple symlink to the data partition would be a far better solution, easier to “move and symlink” than to explain editing conf’s and deleting files/folders, plus you get better performance, but more importantly, consistency as Redis would work the same way whether it was a RO OS or not.

5

There is a line that adds something to a config file is there not? Amending a service file is no different really as that is the config file for the service (note I’m not suggesting actually amending the stock file as explained above).

Totally agree, but when you are putting a script in rc.local to fudge the issue by adding a delay, a more robust solution should be sought.

I agree the MQTT implemntation should be robust, but configuring the services in the best possible way is one of the steps to them being robust.

Still, this is a solution that may work if someone finds they are having issues.