emonSD next steps

Sorry I can’t place what that is,

That may block a git pull, I would expect not though unless those files have been edited since your edits.

This isn’t a new feature so I’m not sure why you wouldn’t have it. despite it’s “emonpi” themed naming it is a all emonPi/emonBase/emonSD wide feature.

Do you have the setting I pointed too in settings.php?

Can you post the server info from the button on the admin page?

[edit]

Found it!

This has now been merged into stable so you could switch back to origin/stable now to get the other updates, (but you may need to stash your flot changes to do so, check the update log to see if it was successful).

But still, I think the update buttons pre-date all this so I’m still not sure why you don’t have them. How old is your set up?

I’m pretty sure they were on board - especially since the firmware has been static for so long. I’ll try and find the post…
[EDIT] Found it…

1 Like

I saw that the first time round - I think the real problems are there isn’t a bullet-proof way of detecting non-original firmware as well as the type/age of the module.
(And I think it should have read “… there have not been any changes to the RFM69Pi firmware …”)

2 Likes

Maybe some clarification is needed, I read that to mean “yes divide the updating up in to separate bits” but got the impression all those bits would still be triggered from one button.

The comment you linked is 7days old but in this thread 2days ago Trystan said

I see no other reason to try and auto-detect the hardware unless you are planning to update the FW.

Don’t get me wrong, I would love things to change, but after 4years of objection (since the emonpi launch) I get the impression it’s here to stay. Again, That’s why I suggest the optional command approach so we can have what we already have (by default) PLUS options to have something else without breaking anything.

I’ve actively told people who have a calibration problem with the emonPi - especially USA users who must have different c.t’s from the ‘Shop’ model, that there’s no realistic way to calibrate it, because they need to edit the front end sketch to change the phase calibration, and like as not the change will be destroyed if they ever “update” emonCMS.

For me that’s a very good reason to split the update process until such time as the front end can be fully calibrated via the browser window.

2 Likes

Discussing with @glyn.hudson, we are happy to split out the firmware update from the rest of the update process.

I think it would be good to keep the update actions available in the UI on the same page. There could be an update firmware section just below the update software (emoncms & emonhub) section. Perhaps with a dropdown menu to select the different hardware variants: emonpi, rfm69pi, rfm12pi…

Im trying to understand your proposal @pb66 for the multiple scripts, it seems we dont really need update_rfm69pi or update_emonpi if we just made it a two step update process. Unless we do feel that a one step update process is needed…

I’ve been working on a refactored update process here:
https://github.com/openenergymonitor/emonpi/tree/update_refactor/update

1. The update process starts here:
https://github.com/openenergymonitor/emonpi/blob/update_refactor/service-runner-update.sh this does a git pull on the emonpi repo to pull in the latest emoni/update/main.sh update script.

2. main.sh (run from service-runner-update.sh)
https://github.com/openenergymonitor/emonpi/blob/update_refactor/update/main.sh

  • The first part of main.sh detects the environment. Some or all of this could be saved in a configuration file.
  • Then there are a number of git pulls e.g usefulscripts, RFM2Pi, perhaps usefulscripts make sense to be installed in /usr/emoncms. Im not sure about the others.
  • The firmware section is commented out for now (to be updated separately as discussed)

3. Emonhub update (run from main.sh)
https://github.com/openenergymonitor/emonpi/blob/update_refactor/update/emonhub.sh

  • git pull
  • install/fix service (calls split out script for this)
  • automatic node addition (this would be a good to make configurable)

4. Emoncms update (run from main.sh)
https://github.com/openenergymonitor/emonpi/blob/update_refactor/update/emoncms.sh

  • check if default settings have been changed
  • git pull on emoncms
  • replace settings if they have changed in the update and are not user modified
  • emoncms modules update
  • symlinked emoncms modules update
  • installation of emoncms modules if not present
  • installation of symlinked emoncms modules if not present
  • sudoers and pi related access permissions
  • removal and installation of emoncms services (calls split out script for this)
  • database update
  • service restart

The update script finishes by restarting the emonpiLCD service and service runner.

I have made a start on allowing for custom paths e.g https://github.com/openenergymonitor/emonpi/blob/update_refactor/update/emoncms.sh#L12 the next step is to break this out to a config file.

I need to think a bit more through how this relates to your 5/6 commands idea @pb66. Is there a substantial difference between update_emoncms and update_emonsd? There are obvious path differences but if this is specified in a config file, what do we have left?..

I think I need to work through running this on my ubuntu laptop, to see how much makes sense to run and if anything needs leaving out. The wifi module and wifi sudoers is one pi specific component that would not make sense to run…

Not using an SDcard but self installing EmonCMS on a different system using the install instructions.

$allow_emonpi_admin = true;

Sorry, this will seem very dumb but what is ‘the admin page’ and how do I get to it?

Sorry again, but how do I do that?

I think we have jumped ahead here. I’m not sure we have planned out the process, UI etc first. Do that, then work out how to implement it after it has been designed.

High risk of selecting the wrong one. Once set via a setting, it stays fixed. Again let’s get the functional design and the UI sorted out then look at implementation.

Backup backup backup

Assumes emonhub is installed. This should clearly be optional.

Assumptions. Plenty of installs with no LCD.

Feedwriter running all this time? Restart?

Hierarchical Settings mechanism would help here.

All of these bits will benefit from the various repositories being versioned with releases (e.g. EmonPi scripts & Useful scripts).

You need to read up on git. I’ve used this tutorial extensively git stash - Saving Changes | Atlassian Git Tutorial @Greebo had another suggestion in a different thread.

And when would that be shown? On just all RPi based servers? regardless of whether it has attached hardware or not? Or permanently on all servers, even hosted?

What about when there are multiple hardware devices attached eg 3 emonTx’s for a 3ph setup?

So a list needs to be maintained and users are restricted to just those options?

Possibly not, if that’s the route you take, but instead of forcing all users to update their FW everytime you are now taking that option away from them and forcing a 2 step process even when they might be running a stock emonpi with stock FW etc etc. The U-turn might be an improvement on the existing process, but it still isn’t the best option. Let the users decide what they want!

Yes, update_emoncms would fully update emoncms and all it’s modules, it’s the choice of server only setups, if emoncms.org was “stock” it’s the choice you would use there when there is no FW, low-write or emonhub etc to consider.

update_emonsd would call update_emoncms! Plus update_emonHub and also update low-write stuff and the core Raspbian packages too.

Nooooooooooooooooooooooooooo! not another config file please. Some objection to my proposal is based on having to edit a single line in settings.php, another config file is a step too far I think. I’m not against a “recipe config” for the building of a image, but once it’s built and settings.php has had it’s “update_command” set there is no need for another config file.

I’m not trying to muddy the waters here. I would indeed prefer not to have the FW overwritten at every update and I think that users that want to seperate the FW updates from the server updates should have that option (as should those that want to do it all in one hit).

But, are we going to insist that users use both buttons together?

When FW is updated with new payloads, will emonhub.conf get updated at the same time?

Will the FW repos get updated as part of the FW update or the SW stack update? What about the emonpi FW will that be broken out to a new repo? Will the emonPiLCD
code get updated as part of the emonpi (FW?) updates or as part of the emonpi software stack updates? So will there be 2 emonpi repos or will the same repo be updated twice if users click both buttons?

What if they do them in the wrong order?

Pre-U-turn this option was the best route (IMO) and if implemented, different update strategies could have been easily tested by just adding a update_??? file and editing settings.php, in fact doing the u-turn would just involve changing the update command and (and adding a button to finish the (FW) update?), so future changes in stratagy could also be just a change of target file as would custom routines, eg my 3x USB emonTx based 3ph monitors, I could just write my own script to use all the existing bits, plus update each emonTx (if required, I have version based selection) and use the same button as everyone else, from what you propose, I doubt the USB emonTx option will be hardcoded into emoncms and there won’t be 3 additional update FW buttons.

I get that it’s an edge case, but many users are operating thier own edge cases, that’s why we need a fully configurable “open” option. Whatever you decide the default should be today, tomorrow or in years to come, it should be easy to break away from the default, we should not try to limit imagination and/or progress by trying to guess what everyone wants or worse, by telling them what they should have without option. What do we have against flexibility?

Fully aware of those, as I said, work in progress, other elements in there e.g wifi module. I think its useful to consider the current state of the update process and what it does in order to evaluate what config options are needed etc.

Its visibility could be selected in settings.php

Do you mean by serial connection directly?

What is the alternative? no UI for firmware update in emoncms?

What is your proposal for this? Can you see a way of making this work without requiring SSH access?

this was not one of my suggestions. Im trying to respond to you guys!

I think you may be misunderstanding my intentions. Im trying my best to work on this, reflecting on your feedback and input. Im not making U-turns or assumptions, I am prototyping and seeking feedback and acting in good faith

Just throwing an idea out there before i go out (so not totally thought though yet).

What if (based on my suggestion) the $update_command was an array? (possibly an array of one item in some cases) and emoncms converts each array member to a button

eg

$update_commands = {/home/pi/emonpi/update_emonsd, /home/pi/emonpi/update_rfm69pi}

would result in 2 buttons “update emonsd” and “update rfm69pi”.

This would allow as many buttons a user wanted, the simplest form would allow the buttons to be labelled from the target filename, but it could be nested arrays so the label could be defined and possibly even a note displayed.

No via serialusb adaptors, it was deliberately an edge case.

See above for a quick (half baked) idea.

Well now you mention it, I find it quite perculiar that the emonhub config is editable in emoncms, but the emoncms config is not. Things like this should be accesible, as should setting the email stuff and changing default pages etc etc.

I understand some settings may need to be blocked from access via the browser in case of breakage, but why not allow users to set the update_command via a emoncms config page? Or add thier gmail settings to get email reports and password resets etc

[edit] eg using a remote server for mqtt, users can change emonhub via emoncms, but not emoncms.

You could have a full or safe option so that more experienced users can change potentially breaking options via the gui, but to enable the “full” setting you do need to ssh in to edit settings.php to reveal all. the “safe” option would not display the safe v full setting. obviously “safe” would be the default position.

No, I think you are misunderstanding my intentions. When i say “u-turn” I am just referencing the fact you were all for the single button approach at one point and now there is a possible change. It is not intended in a derogatory way, just pointing to a point in the time line.

Nobody has doubted your intentions or your hard work. We are floating idea’s too. And once again I feel the need to point out this doesn’t need to be all your workload. We are happy to help, but we need to agree a direction, it seems unless you are shouldering the full burden of the development, there is no discussion. I have already said I would do this. I also offered you a full build script before the emonpi was launched in 2015 and have continuously offered assistance, all you need to do is talk to us about stuff and we can help you. Just because you are involved in the discussion, it doesn’t mean only you can implement it.

1 Like

I did not mean to give the impression that I was closing down discussion, that was definitely not my intention. I’ve re-read the posts above and tried to list the main points:

  • firmware upload approach needs to take into account custom firmware as well as the standard emonpi/rfm69pi options.
  • update command in settings.php vs making firmware type configurable from the UI.
  • suggestion: default to emonpi and remove rfm69pi update options?
  • update emonSD : includes e.g sudo apt-get update, updates to dependencies
  • update emoncms : emoncms, emoncms modules and services.
  • update_emonsd calling update_emoncms is how it works now. (update_emonsd is currently called service-runner-update and update_emoncms: emoncmsupdate)
  • while the shop could set the firmware type, we ideally need a more flexible approach that allows for changing of attached hardware or firmware type.
  • the update process does already run db update, but it can be useful to run db update manually, e.g manual install of a module.
  • consideration needs to be given to backup as part of the update process
  • emonhub is optional
  • LCD is optional
  • repo versioning

I agree with the general direction of this discussion towards providing a better solution for firmware update including catering for custom firmware. I think we can find a good solution that combines the benefits of a one button update with the configurability that we are after.

I like the idea of extending the update section to allow seperate update of: emonhub, emoncms, firmware and database. It would also be good to have a button to run all updates. We could:

  1. Make the firmware configurable from the UI with a dropdown, this setting could be saved. Subsequent calls to ‘Update All’ would then upload firmware according to the last saved user setting.
  2. If the firmware selected is emonpi or rfm69pi the update process could do an md5 check on the firmware for changes and only upload if there are changes.
  3. The Update All process could provide a popup to say that a new firmware version is available, would you like to install it?

The main issue I see with a settings.php entry is that it is not configurable from the UI at present. My concern with making settings.php configurable from the UI is that it could expose sensitive settings such as the mysql database password. Traditionally we have kept emoncms settings that can be configured from the UI in either the mysql database or in separate config files to settings.php.

Although the above is focused on the UI, also happy for command line options in parallel, that would be great.

Im in agreement regarding optional components: e.g emonhub, lcd, wifi.

Thinking about backup and versioning also makes sense.

@pb66 thankyou for the offer of help with this. Lets figure what is best for each of us to work on.
If we can build from https://github.com/openenergymonitor/emonpi/tree/update_refactor/update and https://github.com/emoncms/usefulscripts/blob/master/emonSD_build_test.sh that would be great. I think the outline is there and similar to your suggestions, we just need to work through the broader design points as above and then the detail in the scripts.

1 Like

Not on my horizon at the moment… Is this assuming a serial connected emontx posting in jeelib payload? Don’t serial connected emontx’s generally use the serial interfacer?

I can see that this would need to be part of the FW update process.

At the moment it would be emonSD as its changing a python script and service file on the pi itself. But I can see that that script would not need updating on an emonbase for example.

I dont think we have got that far.

Include the relevant code to pull in what is needed for firmware update only. On a related point, @glyn.hudson also mentioned that it would be good to use the releases binary approach used for the rfm69pi update for the emonpi.

We’ve got a real risk here of getting bogged down in a hard to follow discussion, are we trying to tackle too many things at the same time? What is the first item to tackle?

  1. What should the user be able to do via the update UI?
  2. Location of configuration for firmware type?
1 Like