emonPi service-runner now uses redis (core, wifi, sync & postprocess module updates required)

TrystanLea · 7 June 2018 17:39

In order to get WIFI scanning to work on the new RaspberryPi 3 B+ I had to move the scanning for WIFI networks to a background script triggered by the emonpi service-runner.

In the process I realised that a simpler way of triggering these background scripts would be for the service-runner script to use a redis queue rather than file based flag. A file based flag required a set directory (/tmp) which I have not been able to write to with the www-data user on RPi3B+, so rather than have this directory location dependency I realised it would be easier to move to using redis.

A standard emonPi update will pull in all the updates and restart the service-runner script at the end so that its using the new version, but I thought it best to flag up this change as it might catch some people out.

If your post processes dont run, or sync module downloads or wifi network scans it may be related. If this is indeed a issue you see. Try updating the module in question, the /home/pi/emonpi repository and restarting service-runner or rebooting the emonpi/emonbase/self-hosted install.

For those interested in the code, here’s the new service-runner script:
https://github.com/openenergymonitor/emonpi/blob/master/service-runner

and this is an example service-runner trigger in the wifi module:
https://github.com/emoncms/wifi/blob/master/wifi_controller.php#L90

pb66 · 7 June 2018 19:24

Do you think it might be a good idea to move the service-runner from the emonPi repo to the emoncms repo?

The service-runner has no real connection with the “emonPi” per se, it is a service hook for the emoncms application, that preforms other tasks (eg the sync module) that may not be installed to an emonPi or emonSD.

The “emonPi” connection is only that an “update emonPi” specific script can be invoked when a button in emoncms is clicked. That script could actually be anything, a generic updater perhaps, so that all self-hosted users have the option to update via a button on the admin page (for example).

It can still be installed as part of the emonSD image (via the emonPi repo and updater) so emonPi /SD users would not even detect any difference. but much like the mqtt_input or the feedwriter scripts, it should be available to a wider audience. IMO it should be part of emoncms not emonPi.

pb66 · 15 October 2018 13:27

How does the service-runner work in conjunction with multiple emoncms instances?

If there are 2 or more emoncms’s installed and running, there is likely to be only one service-runner service running, does it use a redis prefix? Are the paths passed from emoncms to the service runner? eg will the right post-process module get called if triggered from within emoncms?

Greebo · 16 October 2018 00:23

The service-runner Python script connects to the default instance of Redis:

github.com

emoncms/emoncms/blob/e6da270246c7981210e668c81d448d9f052f5a20/scripts/services/service-runner/service-runner.py#L27

    
      
            print("Got Termination signal, exiting")
            sys.exit(0)
          
          
# Setup the signal handler to gracefully exit
          signal.signal(signal.SIGTERM, handle_sigterm)
          signal.signal(signal.SIGINT, handle_sigterm)
          
          
def connect_redis():
            while True:
              try:
                server = redis.Redis()
                if server.ping():
                  print("Connected to redis-server")
                  sys.stdout.flush()
                  return server
              except redis.exceptions.ConnectionError:
                print("Unable to connect to redis-server, sleeping for 30s")
                sys.stdout.flush()
              time.sleep(30)
          
          
print("Starting service-runner")

It then just runs any command that it pops off the “service-runner” key. In my testing, those commands included full paths, @TrystanLea is that always the case?

How do multiple instances of emonCMS work with a single instance of Redis anyway?

pb66 · 16 October 2018 09:44

You set a prefix for each instance

github.com

emoncms/emoncms/blob/9.8.31/default.emonpi.settings.php#L18


      
              $port     = "3306";
              // Skip database setup test - set to false once database has been setup.
              $dbtest = true;
          
          
          //2 #### Redis
              $redis_enabled = true;
              $redis_server = array( 'host'   => 'localhost',
                                     'port'   => 6379,
                                     'auth'   => '',
                                     'prefix' => '');
          
          
          //3 #### MQTT
              // The 'subscriber' topic format is rx/* - where * is the emoncms input node number.
              // The 'publisher' topic format is user selectable from the 'Publish to MQTT' input process, for example power/solar
              $mqtt_enabled = true;            // Activate MQTT by changing to true
              $mqtt_server = array( 'host'     => 'localhost',
                                    'port'     => 1883,
                                    'user'     => 'emonpi',
                                    'password' => 'emonpimqtt2016',

I haven’t looked into the code in depth, but I’m guessing the service-runner doesn’t use a prefix so anyone setting the redis prefix is going to lose the use of the service-runner as the key will become prefix:service:runner, correct?

Greebo · 16 October 2018 21:54

The new python version doesn’t use a prefix, correct. But neither did the cron triggered shell script version. Is anyone able to confirm if the shell script version worked with a prefixed Redis? I suspect it didn’t/doesn’t work either.

@glyn.hudson, or @TrystanLea, are you able to provide any extra detail on how this works under the hood? I presume the prefix is all internal to EmonCMS and there’s only a single backend Redis “database” being used by all EmonCMS instances?

Looking at the documentation for REDIS SELECT at SELECT – Redis, it appears you can only select a different “database” by index and there’s no mention of prefix etc.

glyn.hudson · 17 October 2018 01:53

Correct, no it didn’t. The service runner was developed for the emonPi which would only have a single Emoncms instance

Yes, I beleive this is the case.

Greebo · 17 October 2018 05:12

In a multi-instance EmonCMS, what sort of text gets pushed into Redis for the service-runner? Normally its the full path to a script followed by the full path to a log file like this:
/home/pi/data/usefulscript/runthisscript>/var/tmp/logfileforthisscript.log

I presume it would be something like this in a multi-instance case:
/root/folder/of/instance1/usefulescript>/var/tmp/logfileforthisscript.log
or
/root/folder/of/instance2/usefulescript>/var/tmp/logfileforthisscript.log

I think the main question is probably around how things like the log file are differentiated between instances. In the use cases I’ve seen, that log file is displayed in the EmonCMS web interface - can that be per instance too? Is that handled within EmonCMS already?

@pb66, are you able to give me a sample of what gets pushed into the “prefix:service-runner” Redis key by any of the modules that use service-runner on your multi-instance systems?

I can probably update the python version to enable this functionality if I know what it might look like.

pb66 · 18 October 2018 19:09

Understood, just to be sure you know, I am not criticizing your work, I think you’re doing a great job and I understand you have just ported what was there previously and I’m very grateful you have undertaken that task.

Unfortunately not, as Glyn points out this was exclusively a emonpi thing until more recently where Trystan has used it for post-process and sync, but the documentation of the use of service-runner was to install the emonpi repo just to get the service-runner regardless of platform, I do not think any thought was put into multi-instance. It’s only with moving it into a more global space (ie emoncms main repo) that these points like user and instances will now arise.

I haven’t checked but I suspect the /home/pi/ part of the path is taken from the settings.php so that might be better utilized to point to different instance locations.

There are also several discussions about emoncms structure and the need to have a single dedicated folder location for installing all non-www path emoncms modules (and usefulscripts etc) to. IMO that location should be explicitly defined in the settings.php (as per data paths and log file etc) for multiple reasons discussed elsewhere.

That would allow paths like

/path/to/emoncms-modules/usefulscripts/somescript.sh

I’m not so sure on exactly how logfiles and their paths should be defined, I have some idea’s but not really given it too much thought. I have personally used 2 schemes in the past, either putting a custom name to the logfile in settings.php eg

/var/log/emoncms/instanceA.log

or more recently that has changed to using an instance folder and reverting to “emoncms.log” filename for all instances eg

/var/log/emoncms/instanceA/emoncms.log

I guess the latter is better suited to also holding per instance service-runner.logs etc (but the path is perhaps getting a bit long) but it is much tidier than having all the log files loose in the /var/log folder, especially after logrotate has done it’s thing a few times.

The emoncms.log is displayed on the Admin page using the logfile path from settings.php, so yes that already works ok. I guess the different modules eg backup, will have to follow the same path formulation to display as it does to pass to the service-runner flag so whilst it might currently be a hardcoded path, a new $backup_logfile_path variable formulated and set on page load could serve both the flag and the display so once the “formula” is defined eg $logpathfromsettings + “/backup.log” and $logpathfromsettings + “/service-runner.log” it could put all log files relating to anyone (or the only one) instance in a single folder in /var/log

Greebo · 18 October 2018 20:46

All good, it wasn’t taken that way.

I suspect we’ll need to revisit this problem as part of those changes then. Is the best way to keep this on the radar to get an issue raised in the emoncms repo?

borpin · 20 January 2019 20:12

Linked post Development Roadmap 2019 - Robust Backup Process - #33 by borpin

@Greebo @pb66, digging around in the Redis docs, could you use the SCAN command with a pattern match so it will pick up any prefix? I’m not clear on what the SCAN command returns or how to use it so I am stuck at this point.

pb66 · 20 January 2019 20:19

Yep, that’s pretty much where I ended up too!

I’m not sure if SCAN is suitable for normal running or if it is just for locating a particular queue to watch rather than watching multiple queues.

Greebo · 20 January 2019 20:55

I’m certainly happy to revisit this, I think the main question I need an answers for isn’t to do with redis per-se but was asked previously:

@borpin, ~~can you include a copy of what ended up in redis when you ran your backup (or a copy of what ended up in your syslog)?~~ Never mind - I now see that you already did in the post in that other thread!

[EDIT] I’ve had a play around with the redis SCAN operation and it does appear that it is the best option for pattern matching. I’ll update the script and submit a PR and we’ll see where we end up

borpin · 20 January 2019 22:45

It was that which gave me the clue and I thought ‘I wonder…’!