Emoncms V9.9.6 Stable merge (pre-merge check)

Emoncms
1

Its time for another emoncms master → stable merge :slight_smile: targetting version v9.9.6.
I think its all ready to merge, let me know if there is anything we have missed? @Paul @borpin @pb66

There has been a lot of emoncms development activity over the last few months. Thankyou to everyone who has contributed! I’ve tried to summarize and link to commits below, sorry if I have missed any contributions:

2

Great, the number of changes were starting to build up.

Before it goes can the Tags be setup as @glyn.hudson and I discussed a while back please?

Does the update script, update the feedwriter setup? If not could we include the manual steps required in the release notes?

Will a new SD image be created?

Final thing. Could the announcement be pinned and locked so any queries on the update go onto new conversations please.

Looking good :smile:

1 Like
3

Thanks @borpin

I’ve updated the master branch tag to 9.9.6-beta: https://github.com/emoncms/emoncms/blob/master/version.txt and will remove the -beta part on merge.

I will review the update script. I was seeing this as a seperate step to the update script and image creation, but it would be a good time to apply the new feedwriter updates.

4

I assume you mean the emonPI/SD updater rather than the usefulscripts update script? The one in usefulscripts only pulls in changes via git, no installation type actions are done, ie only symlinked files in the filesystem outside the repo would get updated, those copied/moved into place are not touched.

5

I’m meaning the actual git tags. Needs to be done from the command line as per Align output of 'git describe' with released version number · Issue #1115 · emoncms/emoncms · GitHub as in;

git tag 9.9.6 -m "short message here"
git push origin 9.9.6

Last few git tags have just been the number without a ‘v’.

Yes I did and understand the 2 are different.

1 Like
6

I’ve made the changes to the emonpi update process to switch over to the feedwriter.service installed in lib/systemd/system, testing in isolation so far of the whole emonpi update it seems to work fine:
See my proposed change here: https://github.com/openenergymonitor/emonpi/compare/service_correction

I’ve taken the oppertunity to remove the copied mqtt_input service file located in /etc/systemd and replace with a symlink in /lib/systemd. The location of mqtt_input.service is however not in /var/www/emoncms/scripts/services.

To avoid breaking existing installations that may have symlinked this service I could copy the service file to /var/www/emoncms/scripts/services inline with feedwriter and service-runner and at least take this opportunity to apply a more consistent service configuration, any votes/recommendations?

7

I am not sure a sudo systemctl mqtt_input stop is enough as that does not remove any of the links that would have been created by an enable. I think there should be a disable as well.

I am not sure what scenario you are painting here. If you have done a rm from /etc/systemd/ that will remove a symlink or a file. If you mean there may have been a copy in /lib/systemd/ you could just do an rm first to make sure there is nothing there before creating the symlink.

      sudo systemctl enable mqtt_input.service
      sudo systemctl stop mqtt_input.service
      sudo systemctl start mqtt_input.service

Did you mean the stop between the enable and start?

I also think we should bite the bullet and rename the mqtt service and move the service file.

Inevitably we will break some non-pi or old pi systems but with plenty of warnings and good instructions on how to fix we should be OK?

Or else hold back the mqtt service changes, introduce them as a separate upgrade?

That is a bit hurried response I’m afraid - probably missed something.

8

Thanks @borpin

I realise I was also removing the systemctl created symlink to /etc/systemd/system as well, created as part of enable. Here’s my updated script for this section, including itteration through services at the top:

#!/bin/bash

for service in "mqtt_input" "feedwriter" "service-runner"; do

  if [ -d /etc/systemd ] && [ -f /var/www/emoncms/scripts/services/$service/$service.service ]; then
    if [ -f /etc/init.d/$service ]; then
      echo "removing initd $service service"
      sudo /etc/init.d/$service stop
      sudo rm /etc/init.d/$service
    fi
    
    # service will be symlinked to /etc/systemd/system by "systemctl enable"
    if [ -f /etc/systemd/system/$service.service ]; then
      if ! [ -L /etc/systemd/system/$service.service ]; then
        echo "Removing hard copy of $service.service from /etc/systemd/system (should be symlink)"
        sudo systemctl stop $service.service
        sudo systemctl disable $service.service
        sudo rm /etc/systemd/system/$service.service
        sudo systemctl daemon-reload
      fi
    fi
    
    if [ -f /lib/systemd/system/$service.service ]; then
      if ! [ -L /lib/systemd/system/$service.service ]; then
        echo "Removing hard copy of $service.service in /lib/systemd/system (should be symlink)"
        sudo systemctl stop $service.service
        sudo systemctl disable $service.service
        sudo rm /lib/systemd/system/$service.service
        sudo systemctl daemon-reload
      fi
    fi
    
    if [ ! -f /lib/systemd/system/$service.service ]; then
      echo "Installing $service.service in /lib/systemd/system (creating symlink)"
      sudo ln -s /var/www/emoncms/scripts/services/$service/$service.service /lib/systemd/system
      sudo systemctl enable $service.service
      sudo systemctl start $service.service
    else 
      echo "$service.service already installed"
    fi
  fi
done

It requires mqtt_input to be in services, commit: add mqtt_input to services folder for consistency · emoncms/emoncms@a8cbd78 · GitHub

Noting issue here: rename phpmqtt_input.php · Issue #915 · emoncms/emoncms · GitHub did we discuss this anywhere else?
here it is https://github.com/emoncms/emoncms/pull/905 :slight_smile:

9

It all looks good to me. I discovered recently a hard copy of service-runner in the lib folder (wondered why it wasn’t doing what I expected) so this check is good.

Should it check and see if the service is already installed and running otherwise it may install a service that is not required.

On style, I would usually always put in an else with a log output saying that it didn’t do anything.

I would also suggest a warning (especially if non-emonpi) as there may be local changes.

The only other issue I think is that the old service-runner was cron based (if I am not getting confused again).

Not that I can remember.

Can this script be available for non emonpi users to update the service file setup?

10

In order to rename mqtt_input.

The following code works ok for removing the mqtt_input service and could be ran above the generic service installation check code above:

# Remove service
service="mqtt_input"
if [ -f /etc/init.d/$service ]; then
  echo "removing initd $service service"
  sudo /etc/init.d/$service stop
  sudo rm /etc/init.d/$service
fi

if [ -L /lib/systemd/system/$service.service ] || [ -f /lib/systemd/system/$service.service ]; then
  echo "Removing $service.service"
  sudo systemctl stop $service.service
  sudo systemctl disable $service.service
  sudo rm /lib/systemd/system/$service.service
  sudo systemctl daemon-reload
fi

# if the service still exists in /etc then remove it
if [ -L /etc/systemd/system/$service.service ] || [ -f /etc/systemd/system/$service.service ]; then
  echo "Removing $service.service from /etc/systemd/system"
  sudo rm /etc/systemd/system/$service.service
fi

then we can run the code above to install “emoncms_mqtt” with the emoncms_mqtt service in the emoncms services folder.

Im a little concerned that we will add another item of confusion by renaming the mqtt_input script, given all references in the forum & documentation but it seems relatively straight forward to make the change in the update script and emoncms repo.

11

Yes there is a risk there, but at some point it needs to be done.

12

Ok I will go forward with it

13

While looking at the emonpi/emoncmsupdate script I thought it could be a good idea to place many of the repetitive items in for loops so that the same procedure is called for each module and service.

I’ve also made the emoncmsupdate script less pi specific. I can run it on my laptop installation with a different user folder and it works great (once the username is changed at the top of the script)

One key difference for consideration is that the branches of each git repository are not switched back to stable on update if the user has changed the branch to be something else. I found this to be a pain and usually means I manually update my pi, rather than hitting update. Let me know if anyone can see a big issue with that.

https://github.com/openenergymonitor/emonpi/blob/service_correction/emoncmsupdate

14

I’ve renamed the mqtt_input service script in the newly created emoncms/scripts/services/emoncms_mqtt location to be emoncms_mqtt:
https://github.com/emoncms/emoncms/commit/897c67b3182fafb175431aaf014ac897c94bd657

This wont break existing installations as the original is available in emoncms/scripts. I havent changed the actual php script name yet which I think would be better done as a second stage

I’ve also updated the docs in emoncms master to mention emoncms_mqtt and added a check for emoncms_mqtt in the emoncms admin view:
https://github.com/emoncms/emoncms/commit/6f289516c745ba0d2dae30822529002cda6109e6

1 Like
15

I think that is a good idea.

Couple of comments :grin: ,

  • Only specific Modules get updated rather than any that are installed.
  • Could there be an output similar to the script in the useful scripts folder that give a git status and describe after the update (will help any debugging)
  • Could the messages be a bit more descriptive :wink:.
  • This line is a big assumption. I’d prefer a variable at the top for $homedirectory if it cannot get it from settings.php.
  • What does userpi=0 and env_pi=1 actually mean?
  • A setup may not have a /home/$username folder. As above, I would change that to check for the $homedirectory value.
  • The database is only updated if it is an emonpi. There should be a warning on this.

One other thought, can any logfile created not overwrite an old log file? Sometimes users try a repeated update if there is an issue and sometimes the first logfile will be the most useful. Perhaps an append?

1 Like
16

Thanks @borpin, in reply to your points:

  1. I have modified the script to update all modules with a .git folder in the emoncms/Modules folder in the same way as that neat update script in usefulscripts
  2. Yes, done
  3. Yes, in line with usefulscripts script
  4. Amended as suggested
  5. Improved check for pi user and variable is now called $emonSD_pi_env
  6. amended as in 4
  7. I’ve removed the check for an emonSD pi environment, database update now runs in all.

I will come back to your last question on the logfile

1 Like
17

Rather than clear the log completely what do you think of appending the older log to a backup log? e.g:

cat /home/pi/data/emonpiupdate.log >> /home/pi/data/emonpiupdate_backup.log
cat /dev/null >  /home/pi/data/emonpiupdate.log
1 Like
18

Testing this on an emonpi, for some reason emonhub was not starting after an update. I notice its still running using /etc/init.d and so I have added the service correction code to the emonpi/emonhubupdate script as well
https://github.com/openenergymonitor/emonpi/blob/service_correction/emonhubupdate

It runs fine and emonhub starts up fine after the update

1 Like
19

@TrystanLea, I suggest we also need to fix issue #1167 as currentlyservice-runner will not pick up commands from non-emonpi based builds because the default settings are different.

However, if the script cannot be triggered in the first place…

20

I think that is probably the best solution although will it be subject to being rotated?