emonPi service-runner now uses redis (core, wifi, sync & postprocess module updates required)

In order to get WIFI scanning to work on the new RaspberryPi 3 B+ I had to move the scanning for WIFI networks to a background script triggered by the emonpi service-runner.

In the process I realised that a simpler way of triggering these background scripts would be for the service-runner script to use a redis queue rather than file based flag. A file based flag required a set directory (/tmp) which I have not been able to write to with the www-data user on RPi3B+, so rather than have this directory location dependency I realised it would be easier to move to using redis.

A standard emonPi update will pull in all the updates and restart the service-runner script at the end so that its using the new version, but I thought it best to flag up this change as it might catch some people out.

If your post processes dont run, or sync module downloads or wifi network scans it may be related. If this is indeed a issue you see. Try updating the module in question, the /home/pi/emonpi repository and restarting service-runner or rebooting the emonpi/emonbase/self-hosted install.

For those interested in the code, hereā€™s the new service-runner script:
https://github.com/openenergymonitor/emonpi/blob/master/service-runner

and this is an example service-runner trigger in the wifi module:
https://github.com/emoncms/wifi/blob/master/wifi_controller.php#L90

1 Like

Do you think it might be a good idea to move the service-runner from the emonPi repo to the emoncms repo?

The service-runner has no real connection with the ā€œemonPiā€ per se, it is a service hook for the emoncms application, that preforms other tasks (eg the sync module) that may not be installed to an emonPi or emonSD.

The ā€œemonPiā€ connection is only that an ā€œupdate emonPiā€ specific script can be invoked when a button in emoncms is clicked. That script could actually be anything, a generic updater perhaps, so that all self-hosted users have the option to update via a button on the admin page (for example).

It can still be installed as part of the emonSD image (via the emonPi repo and updater) so emonPi /SD users would not even detect any difference. but much like the mqtt_input or the feedwriter scripts, it should be available to a wider audience. IMO it should be part of emoncms not emonPi.

1 Like

How does the service-runner work in conjunction with multiple emoncms instances?

If there are 2 or more emoncmsā€™s installed and running, there is likely to be only one service-runner service running, does it use a redis prefix? Are the paths passed from emoncms to the service runner? eg will the right post-process module get called if triggered from within emoncms?

1 Like

The service-runner Python script connects to the default instance of Redis:

It then just runs any command that it pops off the ā€œservice-runnerā€ key. In my testing, those commands included full paths, @TrystanLea is that always the case?

How do multiple instances of emonCMS work with a single instance of Redis anyway?

You set a prefix for each instance

I havenā€™t looked into the code in depth, but Iā€™m guessing the service-runner doesnā€™t use a prefix so anyone setting the redis prefix is going to lose the use of the service-runner as the key will become prefix:service:runner, correct?

The new python version doesnā€™t use a prefix, correct. But neither did the cron triggered shell script version. Is anyone able to confirm if the shell script version worked with a prefixed Redis? I suspect it didnā€™t/doesnā€™t work either.

@glyn.hudson, or @TrystanLea, are you able to provide any extra detail on how this works under the hood? I presume the prefix is all internal to EmonCMS and thereā€™s only a single backend Redis ā€œdatabaseā€ being used by all EmonCMS instances?

Looking at the documentation for REDIS SELECT at SELECT ā€“ Redis, it appears you can only select a different ā€œdatabaseā€ by index and thereā€™s no mention of prefix etc.

Correct, no it didnā€™t. The service runner was developed for the emonPi which would only have a single Emoncms instance

Yes, I beleive this is the case.

In a multi-instance EmonCMS, what sort of text gets pushed into Redis for the service-runner? Normally its the full path to a script followed by the full path to a log file like this:
/home/pi/data/usefulscript/runthisscript>/var/tmp/logfileforthisscript.log

I presume it would be something like this in a multi-instance case:
/root/folder/of/instance1/usefulescript>/var/tmp/logfileforthisscript.log
or
/root/folder/of/instance2/usefulescript>/var/tmp/logfileforthisscript.log

I think the main question is probably around how things like the log file are differentiated between instances. In the use cases Iā€™ve seen, that log file is displayed in the EmonCMS web interface - can that be per instance too? Is that handled within EmonCMS already?

@pb66, are you able to give me a sample of what gets pushed into the ā€œprefix:service-runnerā€ Redis key by any of the modules that use service-runner on your multi-instance systems?

I can probably update the python version to enable this functionality if I know what it might look like.

Understood, just to be sure you know, I am not criticizing your work, I think youā€™re doing a great job and I understand you have just ported what was there previously and Iā€™m very grateful you have undertaken that task.

Unfortunately not, as Glyn points out this was exclusively a emonpi thing until more recently where Trystan has used it for post-process and sync, but the documentation of the use of service-runner was to install the emonpi repo just to get the service-runner regardless of platform, I do not think any thought was put into multi-instance. Itā€™s only with moving it into a more global space (ie emoncms main repo) that these points like user and instances will now arise.

I havenā€™t checked but I suspect the /home/pi/ part of the path is taken from the settings.php so that might be better utilized to point to different instance locations.

There are also several discussions about emoncms structure and the need to have a single dedicated folder location for installing all non-www path emoncms modules (and usefulscripts etc) to. IMO that location should be explicitly defined in the settings.php (as per data paths and log file etc) for multiple reasons discussed elsewhere.

That would allow paths like

/path/to/emoncms-modules/usefulscripts/somescript.sh

Iā€™m not so sure on exactly how logfiles and their paths should be defined, I have some ideaā€™s but not really given it too much thought. I have personally used 2 schemes in the past, either putting a custom name to the logfile in settings.php eg

/var/log/emoncms/instanceA.log

or more recently that has changed to using an instance folder and reverting to ā€œemoncms.logā€ filename for all instances eg

/var/log/emoncms/instanceA/emoncms.log

I guess the latter is better suited to also holding per instance service-runner.logs etc (but the path is perhaps getting a bit long) but it is much tidier than having all the log files loose in the /var/log folder, especially after logrotate has done itā€™s thing a few times.

The emoncms.log is displayed on the Admin page using the logfile path from settings.php, so yes that already works ok. I guess the different modules eg backup, will have to follow the same path formulation to display as it does to pass to the service-runner flag so whilst it might currently be a hardcoded path, a new $backup_logfile_path variable formulated and set on page load could serve both the flag and the display so once the ā€œformulaā€ is defined eg $logpathfromsettings + ā€œ/backup.logā€ and $logpathfromsettings + ā€œ/service-runner.logā€ it could put all log files relating to anyone (or the only one) instance in a single folder in /var/log

All good, it wasnā€™t taken that way. :slight_smile:

I suspect weā€™ll need to revisit this problem as part of those changes then. Is the best way to keep this on the radar to get an issue raised in the emoncms repo?

Linked post Development Roadmap 2019 - Robust Backup Process - #33 by borpin

@Greebo @pb66, digging around in the Redis docs, could you use the SCAN command with a pattern match so it will pick up any prefix? Iā€™m not clear on what the SCAN command returns or how to use it so I am stuck at this point.

Yep, thatā€™s pretty much where I ended up too!

Iā€™m not sure if SCAN is suitable for normal running or if it is just for locating a particular queue to watch rather than watching multiple queues.

Iā€™m certainly happy to revisit this, I think the main question I need an answers for isnā€™t to do with redis per-se but was asked previously:

@borpin, can you include a copy of what ended up in redis when you ran your backup (or a copy of what ended up in your syslog)? Never mind - I now see that you already did in the post in that other thread!

[EDIT] Iā€™ve had a play around with the redis SCAN operation and it does appear that it is the best option for pattern matching. Iā€™ll update the script and submit a PR and weā€™ll see where we end up :slight_smile:

1 Like

It was that which gave me the clue and I thought ā€˜I wonderā€¦ā€™!