Emoncms install and update script/module

Emoncms
1

There are several discussions around this topic. I started this thread to try and pull the various bits of discussion together.

I have always used a “emoncms installer” script to install emoncms from one command, previously I have not made that publicly available because it doesn’d fully comply with the current documentation and the way things are done on the emonSD image.

More recently I wrote a simple bash script for updating emoncms and the emoncms-modules that was shared and it was the precurser to the way the emoncms modules are currently listed on the admin page in emoncms (see Admin display currently installed Emoncms modules), That script can also be found in emoncms/usefulscripts.

That script has evolved quite a bit since then and now checks for updates rather than doing a blind pull, it is also branch and remote specific, this means it will determine the existing remote url for the repo and check there, for updates to the branch currently in use and update from there if updates are available (or blindly pull in IF it cannot find a “version file” so this can work on repos that have not yet had a “version file” created).

There is a discussion on github about implementing the “version file”.

Much of this work has stalled due to recent discussions about the emoncms installation docs, emoncms dependencies and the best file structure for emoncms to facilitate easy installation, upgrading, support, documentation, debugging and at the same time hopefully make the installation more IFS compliant and also uniform across various linux distros, incl RO Raspbian like the emonSD image.

My intention was to arrive at 2 scripts, the first would install emoncms (and defined modules), this script will ideally would be “multi run safe” so that any failed/interrupted installs could be completed by simply running the installer again, another benefit of this is that it could be used to “fix” issues, such as missing (accidentally deleted) symlinks, re-enabling php and apathe2 modules eg mod-rewrite and correcting file permissions such as the phpfina folders and the log files etc. The second script would be for updating the existing installation without having to hardcode the remote repos or branches, in fact I have kept this so generic (so far) that it would work for all git repos if located in a known place, here I’m thinking about all the FW repos and emonpi and emonhub etc that are not emoncms modules. even personal bespoke repos could be updated by this one updater.

Eventually I intended the updater to be triggered from within emoncms pretty much like the existing emonpi update is and more recently I thought the installer could also be triggered from within emoncms, obviously it cannot install emoncms from within emoncms, but it can fix things like permissions etc and if the “defined modules” list had been edited it would add any additional modules so it is useful from within emoncms.

I have primarily focused on external scripts, with an aim to trigger from within emoncms, purely because my php coding skills are not up to doing something more elaborate on my own.

@cagabi appears to be doing something quite similar but from a different angle, in the form of a emoncms module. So rather than contaminate his thread with all of the above I have written this to outline current development in hope we can collaborate rather than duplicating work or possibly trying to lead the project in different directions.

The issues discussed in that thread relating to permissions and ownership could be addressed by triggering (my?) external scripts and the approach I have taken tackles the backend of honoring existing branch and remote repo urls, but a module could allow a way of selecting branches (as suggested by @glyn.hudson) and defining the remote url plus maintain a list of “defined modules” for the installer script.

I like the idea of putting this all in a emoncms module and it should be possible to reuse the code I have written from within emoncms either via a flag and cron check like the current emonPi updater or possibly via an elevated user command like the “shutdown” in emoncms for the emonPi.

It would be quite simple to separate my updater checks from the updates themselves so that the checks run more frequently (automated) and the user can be kept informed and elect to update or not, there could also be an optional “keep fully updated” function that automatically installs updates as they become available.

Te able to do the complete job I would like to see an admin user only level api to update the database when there’s a schema change, currently the emonpi does this via a backdoor using php, that could be replicated, but I think a proper api call might be better.

2 Likes
2

Great idea. I’ve seen a similar mechanism for updates within HomeAssistant and Hass.io (although that does not delve into branches etc) and to a lesser extent (it just tells you there is an update) in PiHole. Perhaps there is something in those projects that could help.

3

I don’t think you’ll find anything that does, it is considered bad practice to use github for production deployment and updating, but I think we are fooling ourselves if we think we will ever change that aspect. may as well embrace it. This will probably exclude shared-hosting instances, but perhaps once we have the mechanisuim for checking and controlling the updates, we could look to change to using downloaded zips rather than git pulls?

4

Perhaps consider a ‘simple’ update mechanism for the majority initially (like the functionality provided by the EmonPi update), but designed to be extendable to the wider use cases.

1 Like
5

Pb thank you very much for this super informative post.

I was not aware of all your work and thinking about this matter :roll_eyes: I find difficult to follow the forum and miss many things, so thanks for the call.

When I looked around before starting with the Updates module I only found very old information about how to do manual updates. What I remember finding was the service-runner-update.sh but that didn’t work for me as we are not using emonPi.

I need to start using the version file for the modules I do! It makes a lot of sense for automating updates. I have read something in one of the links you have sent but can’t remember: do you check if the branch is behind even if the version has not changed? How do check the version in the remote repo?

That first script is a very good idea. I guess it makes more sense for the the whole emonPi but it would be good to sill have it accessible for emonCMS only installations.
About the second script, that is what the Updates module :slight_smile: despite it only looks in the root emonCMS directory, Modules and Theme directories for the .git one. It could also look in the other parts of an emonPi adding new code or reusing your one

In the Updates module, the check if there is an update available only happens when the user logs into the UI. For the “keep fully updated” function the cron job is needed

There is a parallel discussion that we could have here. I see your emonpiupdate uses a cron job, I know the postproccess module has it’s own cron job and the task module i made not long ago uses its own one. I think a hooking system needs to be divised, a cron job that is set up once and can be used from a module or a script

What the Updates module does is to run a “git status” command before the “git pull”, then if the schema file has changed it will prompt a message saying the database has to be updated. It wouldn’t be difficult to add a button that uses the admin_model.php to do the update.

I see the functionality to select the branches as very useful more than for updates, for checking out branches without having to go to the command line.
The functionality of defining the remote URL I think makes lots of sense when you are installing a module. But I wonder if it is too much to try to make a git UI in emonCMS, if somebody needs higher git control I think the command line is the option.

I’ll follow up this thread, I am very keen on finding out the way to make the most of all our work!

6

It would be great to agree on just one version of the script, instead of 2 scripts tackling the same issues via different routes by different authors.
Especially if the ultimate aim is to integrate an update process into emoncms, as only one would be integrated, and the other author will have wasted a lot of time in development on a script that would be redundant.

Have you both considered having sight of each other’s scripts, and hopefully jointly agree a unified way forward (I’m pretty sure you must both have considered this anyway ?)

Paul

7

This is essentially why I posted. if there’s a common goal and @cagabi’s strengths are in coding for emoncms and I have a lot of the code sorted for the install and update procedures, it might make sense to work together. At the very least I wanted to make @cagabi aware of the potential changes in the emoncms file structure before he invested a lot of time in a accommodating a structure that may change,

1 Like
8

I understand entirely, I too find it very difficult with various discussions going on in github “issues” across many repos, lost in the depths of the forum and of course those that go on behind closed doors in PM’s and emails because that’s the only way to have a proper discussion about these things, If only there was a place set aside for OEM developers to talk shop eg a “developers corner”.

It would be a generic (linux) emoncms installer/fixer, hopefully it would become part of the emonPi/Sd update routine, but it this script would be specific to installing/fixing emoncms, not the wider emonPi stack.

The second part is currently a generic “git repos updater”, you can list folders eg /var/www/html, /home/pi and /var/www/html/emoncms/Modules and it will look in each of those locations for folders with the .git folder. Those found are shortlisted and each is checked for a remote url and branch, that info is used to check for a version file in the root of that remote repo/branch.

There was a discussion about a version file verses a modules file, currently I’m drawn towards a version text file that only contains the version number alone, because the module.json has to be in the sub-folder for the “Modules” part of a module rather than the root. I also think the renaming the modules (using modules.json) is superfluous, we do not need pretty names on the admin page, what we need is to know “absolutely” what is installed, not what someone has decided would be the “friendliest name”.

For the purpose of demonstration. Here is a link to the “version” of the emoncms device module, master branch and to the device-integration branch of the same repo. Alternatively if I was running my own variant of that the remote could be my own repo and no that isn’t the nodes module it is still the device module, I just changed the name in the modules.json file to demonstrate the point I made above,

This is most useful when (for example) you are developing the groups module and ask for testers, I could switch to your repo and any thing you push could be pulled in by my install without having to avoid using any in-built “fixed” url/repo/branch updating methods.

Nothing to do with me! I have had no involvement with the emonPi updater and I would do things quite differently if I was to do it.

For completness here are a couple of links about the emonpi updater Update EmonPi Button or Update EmonBase Button? and Errors not reported in emonpiupdate.log file - #9 by pb66. I believe the flashing of an attached device should be a separate operation, triggered by the user. or at least a user setting whether it is included or not.

I have pondered on this before, and I believe there are too many moving parts, moving at thier own pace. yes there are multiple reasons to use a cronjob, there are also scripts for the mqtt_input and for buffered write, not to mention the many calls for a way to notify when feeds are not updating, plus there are many good reasons to include processing that doesn’t depend on an input to trigger it, eg summing the power of 3 phases from 3 devices, how do you ensure that processing continues if ANY one phase goes down? Sod law says the phase that goes down will be the one that you chose to put your processing in and is no longer being triggered. I wonder if the time is here for a single emoncms daemon that handles all of the above.

No not currently, It is dependent on the version file content only. I am one the fence about local changes too. On one hand I’m thinking the script should just raise an error for the user to stash the changes (easiest) or automatically stash and reapply changes (quite difficult and may transfer old issues to new updated code) or just ignore and reset changes (perhaps stashing them in case the user wants to reapply them). none is ideal, there could be a user setting to define what happens. At the end of the day, any other (non-git pull) method would just overwrite changes without checking so any improvement on that could be a bonus.

I see your point, but I think we have 2 choices, we embrace choice and give users the ability to make selections or we hardcode everything, removing a lot of choice and almost adopting a “do it all our way or figure it out for yourself” type attitude, because if a user wants to make changes to a module and run his own repo, he might not be able to use the updater for the other repos. The result is many users just stick with whats easier and don’t explore or contribute because straying away from the norm is too involved, eg the emonPi/SD.

9

I think at this stage we are ready to define what is all the functionality we want, with that we could see how we make it according to what there is already there from Paul and me, and finally coordinate any new development needed to make the most of our skills.

So I have thoughts for the specification but they are focused exclusively in emonCMS, it would be good to add what we could fit to interact with the rest of emonPi. So these are my ideas:

  • A script to install/fix emonCMS,
  • A UI to allow the user:
    • Trigger the fix emonCMS script
    • Install new modules from a git repo
    • Install new themes from a git repo
    • What other things do we want to install from UI?
    • See all the repos installed (by searching all the folders) with
      • info about current version (if updatable), remote and branch
      • options to update, add remote and change remote/branch of a module/theme.
    • Option to tell emonCMS to autoupdate and define frequency
  • Auto check for updates available and tell user via UI about availability of updates

Apart of this, some comments about your post

But ate least the cron job is calling your script: service-runner-update.sh? I think this will be a discussion: how to trigger the auto update

The task module (which uses its own cron job) can do that for you be checking at a specific frequency :wink:

10

I think the installer/fixer could install additional modules (themes/templates etc) if a list is maintained in emoncms somewhere, adding “newmodule” to the list and hitting install/fix would install any modules on the list that are not yet installed.

  • A script to install/fix emonCMS,
  • A UI to allow the user:
    • Trigger the fix emonCMS script [Agreed]
    • Install new modules from a git repo [Agreed, as part of the install/fix script ?]
    • Install new themes from a git repo [not entirely sold on this yet, but not ruling it out]
    • What other things do we want to install from UI? [potentially device module and app module templates?]
    • See all the repos installed (by searching all the folders) with
      • info about current version (if updatable), remote and branch
      • options to update, add remote and change remote/branch of a module/theme.
    • Option to tell emonCMS to autoupdate and define frequency
  • Auto check for updates available and tell user via UI about availability of updates

As we are potentially trying to configure emoncms to automatically notify the user or perform the updates autonomously, perhaps we are looking at a “modules manager” module rather than an “update module”, a module listing available modules/themes/apps templates/device templates with fields for address(url/repo) and branch/version(some users may want to stick with a certain version) and an enable/disable button would work.

The buttons to update and/or fix/install (and update DB) could all reside on the admin page.

Just off the top of my head, I would prefer to see a “emoncms_commandline.sh” type script that is able to be called by the www-data user from within emoncms eg a button, but with a switch eg “-u” or “–update” to run the update script, the permisions can be set from within that script for certain commands and keep the available tasks to a defined list that can be called instantly rather than allowing emoncms to pass a full command out to the commandline via a cron triggered service-runner. I have concerns about security/safety of potentially exposing an unrestricted root level commandline via the service runner.

11

Hi,

I agree with the install/fix script. You seem to have its functionality very clear (like dealing with file permissions when triggered by www-data, where I got stuck).

About the list of installed modules, what info would be kept there? I am asking thinking about displaying to the user the extra info like current repo and branch. Maybe there is no need to search the folders and just fetching the info from the list can provide everything.

If we go down the route of the script we need to remember the rest of the git functionality (add remote, checkout branch). Do you think that should go in the install/fix script or another one?

I am not very sure about what would the enable/disable module button actually do? I guess it would make emonCMS behave as if it was not installed. My first thought is that it would require touching emonCMS in many places (when calling the controller, loading the menu items, checking the database for updates and probably more).

Yes to the “modules manager” instead of “update module”!

Finally, about the auto update. I’m still not clear how to trigger it if it is not using a cron job.

Cheers

12

So 3 years later, no modules manager yet? :thinking: