emonSD next steps

emonSD
37

Ahhh! I was tricked into thinking he was referring to a emonSD image due to the emonBase reference. Yes you do not get it by default except on emonSD installs.

But you can still enable $allow_ emonpi _admin = true; even on a hosted server, I have it enabled on a test server, all I need is to install service-runner and then make sure there is something to run at the target path /home/pi/emonpi/emonpiupdate (IIRC).

Then I would have the 2 buttons “update emonbase” and “update emonpi” even though this is a Debian9 VPS.

That is fine, It might be a possibility. But it adds another layer of config, perhaps as a one use “build time” config file maybe eg to specify local emoncms or not, attached HW and other settings used to create an image and then deleted could work.

I believe that the shop would NOT edit the file, it would be set to emonpi for all because it’s time consuming, they do not always know what it’s going to be used with, there needs to be a default “as downloaded” setting and the emonpi is the 1st focus whilst the emonbase could be either rfm,12 or rfm69 as explained above, if all emonpi/emonbase users used the (default?) emonpi setting no damage is done and no recent updates are missed.

You are definitely not a minority in that respect. I really do not like the FW being constantly overwritten for no good reason, it’s unlikely we will hit the 10k writes but it’s still undue risk and wear. If it was done in a more orderly fashon and checked the FW version and only updated IF there was a newer version, that would be different. BUT even then, I would expect there to be an override, that’s why I suggested the method I have, there is the option for no firmware updates.

Exactly, however, whatever your views the decision is in the hands of G&T and they want it the way it is, hence my suggestion to keep it close to the way it is by default. It is IMO the best we will achieve.

I do understand their opinion of just having one button to click to do it all for simplicity. There are (arguably) valid reasons this might not be entirely great. But I do not see that opinion changing any time soon, plus users that are capable of self-building an image or installing their own firmware are capable of changing a setting in emoncms.php.

38

Ah OK, I’d suggested moving that update to the EmonHub page but then it got caught up in the fact that I’d forgotten there were 2 parts to that; FW and SW.

I suppose it is dependency. EmonCMS, EmonHub & an RFM are actually all reasonably independent (can run independently) with the only reasonably hard dependency is that the RFM card needs EmonHub (in an OEM scenario) to extract the transmitted data.

Perhaps in the same way that the Database update has a separate page, perhaps these should also have a separate page (with the EmonPi being a special case - all in one update).

Perhaps also then, the EmonHub management should not be a top level menu item but perhaps a ‘tab’ (or menu item) within the main Admin Page. The current grouping/menu has evolved rather than been designed.

39

The emonhub config menu item comes from the fact it is a seperate emoncms module.

Whilst I see no reason why there couldn’t be a separate “update” page, perhaps to host a single update button and have the console and download buttons. But I see no reason to have a separate “update db” button if we manage to get a global update (at least) emoncms or image button etc. Why would anyone want to not update the db’s following an update of emoncms?

I would prefer that the updater just does the db update automatically, preferably using an api call rather than a separate php script as per the emonpi updater. If no update command is defined in settings.php. The one button could fall back to being an update db button I suppose, just to mimic existing arrangements.

The relationship between emonhub and “the firmware” are closer than the relationship between emoncms and the firmware. As discussed elsewhere, the original course for the emonhub module in emoncms was to be able to have multiple emonHubs, that would suggest multiple firmwares (even one emonPi/base could have an additional jeelink) so being able to trigger remote updaters for multiple emonhub/base/pi’s would be good, but the aim here is for the local emonhub and firmware, which may not even exist if on a remote server or dedicated Pi+hdd server etc. So the option to not update/overwrite software/firmware that doesn’t even exist is IMO essential.

Then adding one (or more?) update emonhub (inc/exc FW’s) buttons to the emonhub module is a another project, but once done, there is no reason why a user that prefers to separate their emonpi’s emonhub+fw from the software stack updates, can use a “server-only” type update from the admin page and setup the local emonhub+FW update on the emonhub page. It’s all about the options! and the only way to get the options is via acceptable defaults. If G&T are getting the behaviour they desire and we are getting the flexibility we need, surely everyone is happy?

40

I don’t understand exactly what is meant by a ‘self install’ but what I have seems probably well described as an ‘emonSD image’. I bought an emonBase from the shop. It has the software that came pre-installed, with the exception that I switched branch to ‘origin/fix_padding_spikes’ and I have modified some files in Lib/flot/

41

Sorry I can’t place what that is,

That may block a git pull, I would expect not though unless those files have been edited since your edits.

This isn’t a new feature so I’m not sure why you wouldn’t have it. despite it’s “emonpi” themed naming it is a all emonPi/emonBase/emonSD wide feature.

Do you have the setting I pointed too in settings.php?

Can you post the server info from the button on the admin page?

[edit]

Found it!

This has now been merged into stable so you could switch back to origin/stable now to get the other updates, (but you may need to stash your flot changes to do so, check the update log to see if it was successful).

But still, I think the update buttons pre-date all this so I’m still not sure why you don’t have them. How old is your set up?

42

I’m pretty sure they were on board - especially since the firmware has been static for so long. I’ll try and find the post…
[EDIT] Found it…

1 Like
43

I saw that the first time round - I think the real problems are there isn’t a bullet-proof way of detecting non-original firmware as well as the type/age of the module.
(And I think it should have read “… there have not been any changes to the RFM69Pi firmware …”)

2 Likes
44

Maybe some clarification is needed, I read that to mean “yes divide the updating up in to separate bits” but got the impression all those bits would still be triggered from one button.

The comment you linked is 7days old but in this thread 2days ago Trystan said

I see no other reason to try and auto-detect the hardware unless you are planning to update the FW.

Don’t get me wrong, I would love things to change, but after 4years of objection (since the emonpi launch) I get the impression it’s here to stay. Again, That’s why I suggest the optional command approach so we can have what we already have (by default) PLUS options to have something else without breaking anything.

45

I’ve actively told people who have a calibration problem with the emonPi - especially USA users who must have different c.t’s from the ‘Shop’ model, that there’s no realistic way to calibrate it, because they need to edit the front end sketch to change the phase calibration, and like as not the change will be destroyed if they ever “update” emonCMS.

For me that’s a very good reason to split the update process until such time as the front end can be fully calibrated via the browser window.

2 Likes
46

Discussing with @glyn.hudson, we are happy to split out the firmware update from the rest of the update process.

I think it would be good to keep the update actions available in the UI on the same page. There could be an update firmware section just below the update software (emoncms & emonhub) section. Perhaps with a dropdown menu to select the different hardware variants: emonpi, rfm69pi, rfm12pi…

Im trying to understand your proposal @pb66 for the multiple scripts, it seems we dont really need update_rfm69pi or update_emonpi if we just made it a two step update process. Unless we do feel that a one step update process is needed…

I’ve been working on a refactored update process here:
https://github.com/openenergymonitor/emonpi/tree/update_refactor/update

1. The update process starts here:
https://github.com/openenergymonitor/emonpi/blob/update_refactor/service-runner-update.sh this does a git pull on the emonpi repo to pull in the latest emoni/update/main.sh update script.

2. main.sh (run from service-runner-update.sh)
https://github.com/openenergymonitor/emonpi/blob/update_refactor/update/main.sh

  • The first part of main.sh detects the environment. Some or all of this could be saved in a configuration file.
  • Then there are a number of git pulls e.g usefulscripts, RFM2Pi, perhaps usefulscripts make sense to be installed in /usr/emoncms. Im not sure about the others.
  • The firmware section is commented out for now (to be updated separately as discussed)

3. Emonhub update (run from main.sh)
https://github.com/openenergymonitor/emonpi/blob/update_refactor/update/emonhub.sh

  • git pull
  • install/fix service (calls split out script for this)
  • automatic node addition (this would be a good to make configurable)

4. Emoncms update (run from main.sh)
https://github.com/openenergymonitor/emonpi/blob/update_refactor/update/emoncms.sh

  • check if default settings have been changed
  • git pull on emoncms
  • replace settings if they have changed in the update and are not user modified
  • emoncms modules update
  • symlinked emoncms modules update
  • installation of emoncms modules if not present
  • installation of symlinked emoncms modules if not present
  • sudoers and pi related access permissions
  • removal and installation of emoncms services (calls split out script for this)
  • database update
  • service restart

The update script finishes by restarting the emonpiLCD service and service runner.

I have made a start on allowing for custom paths e.g https://github.com/openenergymonitor/emonpi/blob/update_refactor/update/emoncms.sh#L12 the next step is to break this out to a config file.

I need to think a bit more through how this relates to your 5/6 commands idea @pb66. Is there a substantial difference between update_emoncms and update_emonsd? There are obvious path differences but if this is specified in a config file, what do we have left?..

I think I need to work through running this on my ubuntu laptop, to see how much makes sense to run and if anything needs leaving out. The wifi module and wifi sudoers is one pi specific component that would not make sense to run…

47

Not using an SDcard but self installing EmonCMS on a different system using the install instructions.

48 
$allow_emonpi_admin = true;

Sorry, this will seem very dumb but what is ‘the admin page’ and how do I get to it?

Sorry again, but how do I do that?

49

I think we have jumped ahead here. I’m not sure we have planned out the process, UI etc first. Do that, then work out how to implement it after it has been designed.

High risk of selecting the wrong one. Once set via a setting, it stays fixed. Again let’s get the functional design and the UI sorted out then look at implementation.

Backup backup backup

Assumes emonhub is installed. This should clearly be optional.

Assumptions. Plenty of installs with no LCD.

Feedwriter running all this time? Restart?

Hierarchical Settings mechanism would help here.

All of these bits will benefit from the various repositories being versioned with releases (e.g. EmonPi scripts & Useful scripts).

50

You need to read up on git. I’ve used this tutorial extensively git stash - Saving Changes | Atlassian Git Tutorial @Greebo had another suggestion in a different thread.

51

And when would that be shown? On just all RPi based servers? regardless of whether it has attached hardware or not? Or permanently on all servers, even hosted?

What about when there are multiple hardware devices attached eg 3 emonTx’s for a 3ph setup?

So a list needs to be maintained and users are restricted to just those options?

Possibly not, if that’s the route you take, but instead of forcing all users to update their FW everytime you are now taking that option away from them and forcing a 2 step process even when they might be running a stock emonpi with stock FW etc etc. The U-turn might be an improvement on the existing process, but it still isn’t the best option. Let the users decide what they want!

Yes, update_emoncms would fully update emoncms and all it’s modules, it’s the choice of server only setups, if emoncms.org was “stock” it’s the choice you would use there when there is no FW, low-write or emonhub etc to consider.

update_emonsd would call update_emoncms! Plus update_emonHub and also update low-write stuff and the core Raspbian packages too.

Nooooooooooooooooooooooooooo! not another config file please. Some objection to my proposal is based on having to edit a single line in settings.php, another config file is a step too far I think. I’m not against a “recipe config” for the building of a image, but once it’s built and settings.php has had it’s “update_command” set there is no need for another config file.

I’m not trying to muddy the waters here. I would indeed prefer not to have the FW overwritten at every update and I think that users that want to seperate the FW updates from the server updates should have that option (as should those that want to do it all in one hit).

But, are we going to insist that users use both buttons together?

When FW is updated with new payloads, will emonhub.conf get updated at the same time?

Will the FW repos get updated as part of the FW update or the SW stack update? What about the emonpi FW will that be broken out to a new repo? Will the emonPiLCD
code get updated as part of the emonpi (FW?) updates or as part of the emonpi software stack updates? So will there be 2 emonpi repos or will the same repo be updated twice if users click both buttons?

What if they do them in the wrong order?

Pre-U-turn this option was the best route (IMO) and if implemented, different update strategies could have been easily tested by just adding a update_??? file and editing settings.php, in fact doing the u-turn would just involve changing the update command and (and adding a button to finish the (FW) update?), so future changes in stratagy could also be just a change of target file as would custom routines, eg my 3x USB emonTx based 3ph monitors, I could just write my own script to use all the existing bits, plus update each emonTx (if required, I have version based selection) and use the same button as everyone else, from what you propose, I doubt the USB emonTx option will be hardcoded into emoncms and there won’t be 3 additional update FW buttons.

I get that it’s an edge case, but many users are operating thier own edge cases, that’s why we need a fully configurable “open” option. Whatever you decide the default should be today, tomorrow or in years to come, it should be easy to break away from the default, we should not try to limit imagination and/or progress by trying to guess what everyone wants or worse, by telling them what they should have without option. What do we have against flexibility?

52

Fully aware of those, as I said, work in progress, other elements in there e.g wifi module. I think its useful to consider the current state of the update process and what it does in order to evaluate what config options are needed etc.

53

Its visibility could be selected in settings.php

Do you mean by serial connection directly?

What is the alternative? no UI for firmware update in emoncms?

What is your proposal for this? Can you see a way of making this work without requiring SSH access?

this was not one of my suggestions. Im trying to respond to you guys!

54

I think you may be misunderstanding my intentions. Im trying my best to work on this, reflecting on your feedback and input. Im not making U-turns or assumptions, I am prototyping and seeking feedback and acting in good faith

55

Just throwing an idea out there before i go out (so not totally thought though yet).

What if (based on my suggestion) the $update_command was an array? (possibly an array of one item in some cases) and emoncms converts each array member to a button

eg

$update_commands = {/home/pi/emonpi/update_emonsd, /home/pi/emonpi/update_rfm69pi}

would result in 2 buttons “update emonsd” and “update rfm69pi”.

This would allow as many buttons a user wanted, the simplest form would allow the buttons to be labelled from the target filename, but it could be nested arrays so the label could be defined and possibly even a note displayed.

56

No via serialusb adaptors, it was deliberately an edge case.

See above for a quick (half baked) idea.

Well now you mention it, I find it quite perculiar that the emonhub config is editable in emoncms, but the emoncms config is not. Things like this should be accesible, as should setting the email stuff and changing default pages etc etc.

I understand some settings may need to be blocked from access via the browser in case of breakage, but why not allow users to set the update_command via a emoncms config page? Or add thier gmail settings to get email reports and password resets etc

[edit] eg using a remote server for mqtt, users can change emonhub via emoncms, but not emoncms.

You could have a full or safe option so that more experienced users can change potentially breaking options via the gui, but to enable the “full” setting you do need to ssh in to edit settings.php to reveal all. the “safe” option would not display the safe v full setting. obviously “safe” would be the default position.