emonSD next steps

Good point, we need an identifier, some way of reading what the board is over serial.

I agree with the several smaller files approach. I will start splitting the large install script up soon.

I was also wondering about this, what would you suggest? a repository called emonSD to hold the scripts? but then that would not cover non pi/SD installations… Its more than just emoncms but then it is also primarily an extended emoncms installation. usefulscripts is a bit vague…

Alternatively have a setting and force it to be set in the settings.php file for SDCard downloads. EmonPi could have it pre-set (or see below).

Now you’ve got me. ‘EmonScripts’ perhaps?

The scripts will ultimately be a number of building blocks to build or update an Emon system depending on the configuration.

Could possibly do with a matrix of the permutations. Perhaps the start of the script asks the questions to set the options (if this is a move away from a prebuilt card except for EmonPis perhaps).

Along those lines… I was thinking that as part of the install & update process a main script could first auto detect a module e.g emoncms/wifi and then run the install or update script relating to that module located in the modules github repository. The same would apply well & already happens to a degree with emonhub. There would need to be a naming convention e.g if an install.sh or update.sh script is detected in the repo then that script is run…

./configure
make
make install

And whether it has a standard of custom sketch in it!

BTW, I was looking at log2ram on github and looked at Issues · azlux/log2ram · GitHub I think it’s worth reading the suggestions there before commiting to log2ram.

The other issue is overwriting users calibrated/modified/custom sketches, just because it has a emonGLCD doesn’t mean it is mandatory to run the stock sketch. What if users have something else attached to the GPIO serial port? Possibly a emonTx? Possibly something non-OEM perhaps?

I made a suggestion over in the emonSD first boot - #19 by pb66 thread to have a “update command” defined in the settings.php that could default to an “update emonpi” command to carry forward the existing emonpi update behaviour.

Removing the “update emonBase” button and just having one “update” button that defaults to an “update emonpi” command removes any possibility of a user “bricking” their rfm12pi accidentally as no rfm2pi devices will get updated due to the wrong upload baud. emonBase users will have to edit settings.php to update either their rfm12pi or rfm69pi by defining the appropriate update command (apparently it would be no biggy if they neglect to do so since the rfm??pi FW has not changed in quite sometime).

Possibly there will be (currently) a choice of 5 or 6 update commands update_emonpi(default), update_rfm69pi*, update_rfm12pi*, update_emonsd update_emoncms and update_custom.

[(*) - although even with a choice between rfm12pi and rfm69pi some notes will be needed as the rfm12pi’s came in 2 flavours rfm12b and then later, rfm69cw. This was way before the rfm69pi arrived with the additional IO broken out.]

The names can be agreed later but, these options can be either commented out or just explained in the notes. Users can always add their own, but we can add an update_custom file to .gitignore, and advise users that’s where they can write their own update scripts without fear of it being overwritten or included in any PR’s they may submit.

The update_emoncms option would be just that, just updates emoncms and all it’s modules and update_emonsd would just update the emonSD image completely, but without any firmware updates being done. Handy for users with custom FW that do not want to write their own update script or lose the updates to emonhub etc by using the update_emoncms command.

This is simple and easy to use but very flexible. It removes one of the 2 update buttons whilst increasing the options and removing the accidental “wrong button” usage. It will include a much wider audience rather than just the emonSD/Pi/base users. The 12v69 warning can be removed from the gui for a cleaner more professional looking all inclusive solution. With no additional config files, identifiers or assumptions forcing a mandatory overwrite of users hard work (customising or calibrating sketches).

The 5/6 commands simply call one of the 3/4 update script wrappers (update_emoncms and update_emonsd will be full scripts, in fact update_emoncms would even be part of update_emonsd too) eg for the update_rfm69pi

#!/bin/bash
# Update script for the emonSD on emonBase with rfm69cw based RFM2Pi module.

# First we update the rest of the image inc emonhub, emoncms and the FW repo's etc
source update_emonsd.sh

# Then we update the RFM2Pi with the correct firmware.
echo "Updating rfm69pi firmware"
avrdude -v -c arduino -p ATMEGA328P -P /dev/ttyAMA0 -b 38400 -U flash:w:/home/pi/RFM2Pi/firmware/RFM69CW_RF_Demo_ATmega328/RFM12_Demo_ATmega328.cpp.hex

whilst for the update_emonpi

#!/bin/bash
# Update script for the emonSD on emonPi.

# First we update the rest of the image inc emonhub, emoncms and the FW repo's etc
source update_emonsd.sh

# Then we update the emonpi add-on boards firmware.
echo "Updating emonpi firmware"
avrdude -v -c arduino -p ATMEGA328P -P /dev/ttyAMA0 -b 115200 -U flash:w:/home/pi/emonpi/firmware/compiled/latest.hex

These can be fleshed out with better log messages and/or comments etc, but you get the gist hopefully. The bulk of the updating is done via the same script, included in any number of simple wrappers that offer options to the various user groups, easily defined in settings.php and easily customised without becoming excluded through fear of overwrites and/or refused git pulls etc.

I’d be happy to implement this, this isn’t a demand for someone else to do it, just floating the idea, if it’s well received I will do the work, I just don’t want to spend hours doing it if it’s a non-starter.

2 Likes

Simple for “standard” users, easy to customise, versatile for those who have something different, and by the sound of it, easy enough to maintain. It looks as if it will tick all the boxes for me.

As an emonBase user who can no longer even find an update button (is there one? I seem to recall seeing one once upon a time) why am I being picked on to have to edit something? And how would I know whether I have an rfm12/69pi?

1 Like

Without any further info I can only guess you might have edited the settings.php file and set “$allow_emonpi_admin = false;” perhaps? If you look at the emoncms/default.emonpi.settings.php (lines 149-150) it is/was “true” by default. This setting displays (or not) the emonSD/Pi/base specific section on the admin page.

There are soooo many answers I could offer to this question . . .

  1. You already are “picked on” for being a emonBase user, 2 examples in bold above show how emonpi is the primary target, take a look at your command prompt and hostname for 2 more. I am actually really trying to bring things closer together, but this will need to be done in small steps to avoid over complicating and therefore blocking any progress.

  2. As pointed out, there is no need for you to edit from the default “emonpi” setting, the rfm??pi FW has not changed in years, I doubt it will change in the immediate future, so leave it as it is. That’s one of the main reasons this (initial) idea works.

  3. Point 2. actually does you a favour. Since the FW hasn’t changed and is not likely to any time soon, why would you want the FW to be force fed to the RFM69Pi board at every image update? It increases the chance of a failure, it burns up the finite amount of write the chip has. It is unnecessarily fiddling with something that is not broken for no possible gain (at this time).

  4. Since we are potentially moving towards a install script it is totally possible that we might have a different approach to the default settings, maybe a default.emonbase.settings.php file (or even rfm69pi v rfm12pi versions) or maybe at install time the settings.php will be written directly from the install script with the options passed. We already need to edit other stuff in the settings.php during install, why not this too.

  5. Original emonbase users that have rfm12 based boards are “picked on” to an even greater degree as it stands since they will temporarily brick thier device by using the existing “update emonbase” button as it force feeds the rfm69 FW to the rfm12 based device, leaving users to find their own way to debug and recover.

  6. Strictly speaking, an emonpi IS an emonBase, so it is only some emonbase users that are discriminated against per se. What I’m trying to do here is remove the distinction by allowing ALL users to choose their update routine, whilst attempting to keep the powers that be on side by keeping the default to emonpi so hopefully this might be an acceptable solution.

  7. As a long time user of the emon system, why am I heavily discriminated against because I do not use either an emonPi or the emonSD? I don’t have to resort to editing settings.php, no I have to write my own update scripts from scratch that do not include much of the stuff that is emonPi (aka emonbase) centric. You might arguably think an emonBase user is a second class user, but if you are not using the emonSD, you are very much an outsider that is not really catered for at all currently.

  8. If you were using a “serial-direct” emonTx attached to the GPIO of a emonSD RPi you play this Russian roulette each time you update (via the buttons) too since neither option works for you but one will overwrite your firmware (if you have the reset line connected).

I know some of these points do not apply to you and that they do not explain “why” you might be “picked on” the short answer is “that’s how it is” currently and this suggestion actually goes a long way to correcting that imbalance.

. . . and here lies the big problem! The fact you are asking that question, which button would you click (if they were displaying?) update emonpi or update emonbase? If you have an rfm12 based device and you click update emonbase, it’s game over until you read up how to reinstall your rfm12 FW assuming you manage to work out that’s why your data has stopped!

There needs to be a note and some links to the info needed to make this decision and once it’s researched and decided, set (and forgotten?), users do not want to be faced by the same perilous question at each update, even if you know which button to use, you might just click the wrong one one day, there is currently no confirmation needed.

There is a page in learn that shows you pictures of each module so you can id the exact module, but the easy thing to check first is whether you have a double row of conn points at the opposite edge to the GPIO connector. If you do, it’s a bona fida RFM69Pi with a rfm69cw module fitted. If there is no double row of IO then you have an “RFM12Pi” and it could be either an rfm12b or rfm69cw module fitted, it will need identifying visually unless you know when you purchased it (and it is obvious from that date that it is safely one or the other) OR maybe you can tell by the serial baud setting in emonhub.conf, if it is 9600, it’s rfm12b, if it’s 57200 it’s rfm69cw but if it’s 38400, it could be either. Plus on that last point, any RFM2Pi you update correctly, will then be 38400, so if you had a 57200 or 9600 device, it will stop for another reason, until you edit emonhub.conf.

This is why it MUST be done by a user input somewhere along the line, history has already removed the option to automate it. it’s not that I believe that emonBase users are any less important, to the contrary, I believe they are important enough NOT to get a second class service and get force fed the current rfm69pi FW whether it’s right or not, whether they want it or not. Even if a decision was made to automate it from here on in via a new flag in the FW that can be read over serial, what new FW would you install to what devices? You cannot install the same FW to all RFM2Pi’s, you need the user to identify and declare what device they have for a successful FW update to be possible and until they do that, you are better off avoiding an automated FW update.

1 Like

Thanks @Robert.Wall, another box it ticks is recovering from using the wrong update button. In the case of an rfm69 FW being written to a rfm12 device, currently the user will need to ssh in and issue avrdude commands to recover. With this proposal, they can just edit settings.php and hit update again and hey presto!

Actually, as expalined above, it wouldn’t matter quite so much, try rfm69, if it stops the data try rfm12. It saves stripping the case down and trying to id the module. Worse case scenario is that you have to edit the baud in emonhub.conf if you have a really early rfm12pi, but the upside is you now have the latest FW which if you are needing to change baud, you definately wern’t on the latest FW previously.

If you do a self install, then no, there is no update button.

Look at your invoice - my (old now) invoice tells me it is a 12. Simple rule of thumb will be purchase date otherwise.

I think I agree with this. However, I think I’d like something in the boot partition so it can be set before first boot. Easy enough then for the shop (or a user) to set it as an EmonPi image or an EmonBase image.

I’m still not a fan of updating the FW and the EmonCMS in one go, especially as the FW changes so infrequently, IME updating FW is one of the easiest ways to muck up a working system! However, it seems I am in a minority of one here :smile:. BTW, is there a write limit to the flash on these cards?

I’d also like to bring forward again, that making a backup should be integral to the process but that of course only applies to EmonCMS.

I think there was already general consensus that the firmware and software stacks would be separated, so there’s no minority here (unless its silent on the matter!)…

Yes, there is a write limit on the ATMega’s that are on the emonBase, emonPi, emonTX and any Arduino that uses them. According to the datasheet for the 328:
image

10,000 reflash cycles is still quite a lot :slight_smile:

Ahhh! I was tricked into thinking he was referring to a emonSD image due to the emonBase reference. Yes you do not get it by default except on emonSD installs.

But you can still enable $allow_ emonpi _admin = true; even on a hosted server, I have it enabled on a test server, all I need is to install service-runner and then make sure there is something to run at the target path /home/pi/emonpi/emonpiupdate (IIRC).

Then I would have the 2 buttons “update emonbase” and “update emonpi” even though this is a Debian9 VPS.

That is fine, It might be a possibility. But it adds another layer of config, perhaps as a one use “build time” config file maybe eg to specify local emoncms or not, attached HW and other settings used to create an image and then deleted could work.

I believe that the shop would NOT edit the file, it would be set to emonpi for all because it’s time consuming, they do not always know what it’s going to be used with, there needs to be a default “as downloaded” setting and the emonpi is the 1st focus whilst the emonbase could be either rfm,12 or rfm69 as explained above, if all emonpi/emonbase users used the (default?) emonpi setting no damage is done and no recent updates are missed.

You are definitely not a minority in that respect. I really do not like the FW being constantly overwritten for no good reason, it’s unlikely we will hit the 10k writes but it’s still undue risk and wear. If it was done in a more orderly fashon and checked the FW version and only updated IF there was a newer version, that would be different. BUT even then, I would expect there to be an override, that’s why I suggested the method I have, there is the option for no firmware updates.

Exactly, however, whatever your views the decision is in the hands of G&T and they want it the way it is, hence my suggestion to keep it close to the way it is by default. It is IMO the best we will achieve.

I do understand their opinion of just having one button to click to do it all for simplicity. There are (arguably) valid reasons this might not be entirely great. But I do not see that opinion changing any time soon, plus users that are capable of self-building an image or installing their own firmware are capable of changing a setting in emoncms.php.

Ah OK, I’d suggested moving that update to the EmonHub page but then it got caught up in the fact that I’d forgotten there were 2 parts to that; FW and SW.

I suppose it is dependency. EmonCMS, EmonHub & an RFM are actually all reasonably independent (can run independently) with the only reasonably hard dependency is that the RFM card needs EmonHub (in an OEM scenario) to extract the transmitted data.

Perhaps in the same way that the Database update has a separate page, perhaps these should also have a separate page (with the EmonPi being a special case - all in one update).

Perhaps also then, the EmonHub management should not be a top level menu item but perhaps a ‘tab’ (or menu item) within the main Admin Page. The current grouping/menu has evolved rather than been designed.

The emonhub config menu item comes from the fact it is a seperate emoncms module.

Whilst I see no reason why there couldn’t be a separate “update” page, perhaps to host a single update button and have the console and download buttons. But I see no reason to have a separate “update db” button if we manage to get a global update (at least) emoncms or image button etc. Why would anyone want to not update the db’s following an update of emoncms?

I would prefer that the updater just does the db update automatically, preferably using an api call rather than a separate php script as per the emonpi updater. If no update command is defined in settings.php. The one button could fall back to being an update db button I suppose, just to mimic existing arrangements.

The relationship between emonhub and “the firmware” are closer than the relationship between emoncms and the firmware. As discussed elsewhere, the original course for the emonhub module in emoncms was to be able to have multiple emonHubs, that would suggest multiple firmwares (even one emonPi/base could have an additional jeelink) so being able to trigger remote updaters for multiple emonhub/base/pi’s would be good, but the aim here is for the local emonhub and firmware, which may not even exist if on a remote server or dedicated Pi+hdd server etc. So the option to not update/overwrite software/firmware that doesn’t even exist is IMO essential.

Then adding one (or more?) update emonhub (inc/exc FW’s) buttons to the emonhub module is a another project, but once done, there is no reason why a user that prefers to separate their emonpi’s emonhub+fw from the software stack updates, can use a “server-only” type update from the admin page and setup the local emonhub+FW update on the emonhub page. It’s all about the options! and the only way to get the options is via acceptable defaults. If G&T are getting the behaviour they desire and we are getting the flexibility we need, surely everyone is happy?

I don’t understand exactly what is meant by a ‘self install’ but what I have seems probably well described as an ‘emonSD image’. I bought an emonBase from the shop. It has the software that came pre-installed, with the exception that I switched branch to ‘origin/fix_padding_spikes’ and I have modified some files in Lib/flot/

Sorry I can’t place what that is,

That may block a git pull, I would expect not though unless those files have been edited since your edits.

This isn’t a new feature so I’m not sure why you wouldn’t have it. despite it’s “emonpi” themed naming it is a all emonPi/emonBase/emonSD wide feature.

Do you have the setting I pointed too in settings.php?

Can you post the server info from the button on the admin page?

[edit]

Found it!

This has now been merged into stable so you could switch back to origin/stable now to get the other updates, (but you may need to stash your flot changes to do so, check the update log to see if it was successful).

But still, I think the update buttons pre-date all this so I’m still not sure why you don’t have them. How old is your set up?

I’m pretty sure they were on board - especially since the firmware has been static for so long. I’ll try and find the post…
[EDIT] Found it…

1 Like

I saw that the first time round - I think the real problems are there isn’t a bullet-proof way of detecting non-original firmware as well as the type/age of the module.
(And I think it should have read “… there have not been any changes to the RFM69Pi firmware …”)

2 Likes