emonSD next steps

We are aiming to have an emoncms container or containers which can run on it yes. So we will look at the more recent work that has been done on this. I would also like to look at porting hardware support for emonPi to home assisstant as another proof of concept exercise. We are likely to also be running home assisstant .

Speaking of HA, we were looking at the new 0.90 version this morning. The pace of change with HA is a bit dizzying at the moment. They have just added support for the nabu casa remote access service (which itself is open source) which has some similarities to experiments @TrystanLea has been conducting with a socket.io interface for local emoncms (?). Their new GUI is also coming along quickly. My comments above still stand though - we have seen a lot of stability issues and breaking changes with HA/HassOS so I think it is a gamble to base stuff on it right now. Theoretically it seems like a great fit for moving away from emonSD.

Thanks @beaylott that sounds interesting, the built in remote access sounds great.

Iā€™ve been continuing here with the emonSD build script, currently stuck on Wifi setup I cant seem to write the wpa_supplicant.conf configuration to /tmp/wifidata as part of saving the wifi configurationā€¦ Or emoncms / php sees the changes to the file but they are not then visible directlyā€¦ im sure it will become clear.

The /usr/emoncms/modules location for symlinked modules is working well. The sync and postprocess modules work out of the box ($homedir in settings.php needs to be set to /usr/emoncms/modules, the backup module requires a similar configuration change (tested for creation of backup so far)
https://github.com/emoncms/usefulscripts/blob/master/emonSD_build_test.sh#L191
I can see what changes are needed to make the installation process standard across the modules

We have also in parallel been looking at the emonpi update process in more detail, discussing with @glyn.hudson weā€™ve reused the existing code for detecting if the connected hardware is an rfm69pi or an emonpi board by detecting if the LCD id present on the i2c bus.

The installation and update scripts have a lot in common, it may be possible to bring them together and use flags or a configuration file to select between modes.

1 Like

The real issue is if it is a rfm12pi board.

That should be possible, but as has been said, several smaller files that do individual bits are easier to maintain that one large script.

I also wonder about moving all these scripts out of the EmonPi repo as that in itself can be confusing and also being a little more consistent in naming convention (so it does what it says on the tin).

Good point, we need an identifier, some way of reading what the board is over serial.

I agree with the several smaller files approach. I will start splitting the large install script up soon.

I was also wondering about this, what would you suggest? a repository called emonSD to hold the scripts? but then that would not cover non pi/SD installationsā€¦ Its more than just emoncms but then it is also primarily an extended emoncms installation. usefulscripts is a bit vagueā€¦

Alternatively have a setting and force it to be set in the settings.php file for SDCard downloads. EmonPi could have it pre-set (or see below).

Now youā€™ve got me. ā€˜EmonScriptsā€™ perhaps?

The scripts will ultimately be a number of building blocks to build or update an Emon system depending on the configuration.

Could possibly do with a matrix of the permutations. Perhaps the start of the script asks the questions to set the options (if this is a move away from a prebuilt card except for EmonPis perhaps).

Along those linesā€¦ I was thinking that as part of the install & update process a main script could first auto detect a module e.g emoncms/wifi and then run the install or update script relating to that module located in the modules github repository. The same would apply well & already happens to a degree with emonhub. There would need to be a naming convention e.g if an install.sh or update.sh script is detected in the repo then that script is runā€¦

./configure
make
make install

And whether it has a standard of custom sketch in it!

BTW, I was looking at log2ram on github and looked at Issues Ā· azlux/log2ram Ā· GitHub I think itā€™s worth reading the suggestions there before commiting to log2ram.

The other issue is overwriting users calibrated/modified/custom sketches, just because it has a emonGLCD doesnā€™t mean it is mandatory to run the stock sketch. What if users have something else attached to the GPIO serial port? Possibly a emonTx? Possibly something non-OEM perhaps?

I made a suggestion over in the emonSD first boot - #19 by pb66 thread to have a ā€œupdate commandā€ defined in the settings.php that could default to an ā€œupdate emonpiā€ command to carry forward the existing emonpi update behaviour.

Removing the ā€œupdate emonBaseā€ button and just having one ā€œupdateā€ button that defaults to an ā€œupdate emonpiā€ command removes any possibility of a user ā€œbrickingā€ their rfm12pi accidentally as no rfm2pi devices will get updated due to the wrong upload baud. emonBase users will have to edit settings.php to update either their rfm12pi or rfm69pi by defining the appropriate update command (apparently it would be no biggy if they neglect to do so since the rfm??pi FW has not changed in quite sometime).

Possibly there will be (currently) a choice of 5 or 6 update commands update_emonpi(default), update_rfm69pi*, update_rfm12pi*, update_emonsd update_emoncms and update_custom.

[(*) - although even with a choice between rfm12pi and rfm69pi some notes will be needed as the rfm12piā€™s came in 2 flavours rfm12b and then later, rfm69cw. This was way before the rfm69pi arrived with the additional IO broken out.]

The names can be agreed later but, these options can be either commented out or just explained in the notes. Users can always add their own, but we can add an update_custom file to .gitignore, and advise users thatā€™s where they can write their own update scripts without fear of it being overwritten or included in any PRā€™s they may submit.

The update_emoncms option would be just that, just updates emoncms and all itā€™s modules and update_emonsd would just update the emonSD image completely, but without any firmware updates being done. Handy for users with custom FW that do not want to write their own update script or lose the updates to emonhub etc by using the update_emoncms command.

This is simple and easy to use but very flexible. It removes one of the 2 update buttons whilst increasing the options and removing the accidental ā€œwrong buttonā€ usage. It will include a much wider audience rather than just the emonSD/Pi/base users. The 12v69 warning can be removed from the gui for a cleaner more professional looking all inclusive solution. With no additional config files, identifiers or assumptions forcing a mandatory overwrite of users hard work (customising or calibrating sketches).

The 5/6 commands simply call one of the 3/4 update script wrappers (update_emoncms and update_emonsd will be full scripts, in fact update_emoncms would even be part of update_emonsd too) eg for the update_rfm69pi

#!/bin/bash
# Update script for the emonSD on emonBase with rfm69cw based RFM2Pi module.

# First we update the rest of the image inc emonhub, emoncms and the FW repo's etc
source update_emonsd.sh

# Then we update the RFM2Pi with the correct firmware.
echo "Updating rfm69pi firmware"
avrdude -v -c arduino -p ATMEGA328P -P /dev/ttyAMA0 -b 38400 -U flash:w:/home/pi/RFM2Pi/firmware/RFM69CW_RF_Demo_ATmega328/RFM12_Demo_ATmega328.cpp.hex

whilst for the update_emonpi

#!/bin/bash
# Update script for the emonSD on emonPi.

# First we update the rest of the image inc emonhub, emoncms and the FW repo's etc
source update_emonsd.sh

# Then we update the emonpi add-on boards firmware.
echo "Updating emonpi firmware"
avrdude -v -c arduino -p ATMEGA328P -P /dev/ttyAMA0 -b 115200 -U flash:w:/home/pi/emonpi/firmware/compiled/latest.hex

These can be fleshed out with better log messages and/or comments etc, but you get the gist hopefully. The bulk of the updating is done via the same script, included in any number of simple wrappers that offer options to the various user groups, easily defined in settings.php and easily customised without becoming excluded through fear of overwrites and/or refused git pulls etc.

Iā€™d be happy to implement this, this isnā€™t a demand for someone else to do it, just floating the idea, if itā€™s well received I will do the work, I just donā€™t want to spend hours doing it if itā€™s a non-starter.

2 Likes

Simple for ā€œstandardā€ users, easy to customise, versatile for those who have something different, and by the sound of it, easy enough to maintain. It looks as if it will tick all the boxes for me.

As an emonBase user who can no longer even find an update button (is there one? I seem to recall seeing one once upon a time) why am I being picked on to have to edit something? And how would I know whether I have an rfm12/69pi?

1 Like

Without any further info I can only guess you might have edited the settings.php file and set ā€œ$allow_emonpi_admin = false;ā€ perhaps? If you look at the emoncms/default.emonpi.settings.php (lines 149-150) it is/was ā€œtrueā€ by default. This setting displays (or not) the emonSD/Pi/base specific section on the admin page.

There are soooo many answers I could offer to this question . . .

  1. You already are ā€œpicked onā€ for being a emonBase user, 2 examples in bold above show how emonpi is the primary target, take a look at your command prompt and hostname for 2 more. I am actually really trying to bring things closer together, but this will need to be done in small steps to avoid over complicating and therefore blocking any progress.

  2. As pointed out, there is no need for you to edit from the default ā€œemonpiā€ setting, the rfm??pi FW has not changed in years, I doubt it will change in the immediate future, so leave it as it is. Thatā€™s one of the main reasons this (initial) idea works.

  3. Point 2. actually does you a favour. Since the FW hasnā€™t changed and is not likely to any time soon, why would you want the FW to be force fed to the RFM69Pi board at every image update? It increases the chance of a failure, it burns up the finite amount of write the chip has. It is unnecessarily fiddling with something that is not broken for no possible gain (at this time).

  4. Since we are potentially moving towards a install script it is totally possible that we might have a different approach to the default settings, maybe a default.emonbase.settings.php file (or even rfm69pi v rfm12pi versions) or maybe at install time the settings.php will be written directly from the install script with the options passed. We already need to edit other stuff in the settings.php during install, why not this too.

  5. Original emonbase users that have rfm12 based boards are ā€œpicked onā€ to an even greater degree as it stands since they will temporarily brick thier device by using the existing ā€œupdate emonbaseā€ button as it force feeds the rfm69 FW to the rfm12 based device, leaving users to find their own way to debug and recover.

  6. Strictly speaking, an emonpi IS an emonBase, so it is only some emonbase users that are discriminated against per se. What Iā€™m trying to do here is remove the distinction by allowing ALL users to choose their update routine, whilst attempting to keep the powers that be on side by keeping the default to emonpi so hopefully this might be an acceptable solution.

  7. As a long time user of the emon system, why am I heavily discriminated against because I do not use either an emonPi or the emonSD? I donā€™t have to resort to editing settings.php, no I have to write my own update scripts from scratch that do not include much of the stuff that is emonPi (aka emonbase) centric. You might arguably think an emonBase user is a second class user, but if you are not using the emonSD, you are very much an outsider that is not really catered for at all currently.

  8. If you were using a ā€œserial-directā€ emonTx attached to the GPIO of a emonSD RPi you play this Russian roulette each time you update (via the buttons) too since neither option works for you but one will overwrite your firmware (if you have the reset line connected).

I know some of these points do not apply to you and that they do not explain ā€œwhyā€ you might be ā€œpicked onā€ the short answer is ā€œthatā€™s how it isā€ currently and this suggestion actually goes a long way to correcting that imbalance.

. . . and here lies the big problem! The fact you are asking that question, which button would you click (if they were displaying?) update emonpi or update emonbase? If you have an rfm12 based device and you click update emonbase, itā€™s game over until you read up how to reinstall your rfm12 FW assuming you manage to work out thatā€™s why your data has stopped!

There needs to be a note and some links to the info needed to make this decision and once itā€™s researched and decided, set (and forgotten?), users do not want to be faced by the same perilous question at each update, even if you know which button to use, you might just click the wrong one one day, there is currently no confirmation needed.

There is a page in learn that shows you pictures of each module so you can id the exact module, but the easy thing to check first is whether you have a double row of conn points at the opposite edge to the GPIO connector. If you do, itā€™s a bona fida RFM69Pi with a rfm69cw module fitted. If there is no double row of IO then you have an ā€œRFM12Piā€ and it could be either an rfm12b or rfm69cw module fitted, it will need identifying visually unless you know when you purchased it (and it is obvious from that date that it is safely one or the other) OR maybe you can tell by the serial baud setting in emonhub.conf, if it is 9600, itā€™s rfm12b, if itā€™s 57200 itā€™s rfm69cw but if itā€™s 38400, it could be either. Plus on that last point, any RFM2Pi you update correctly, will then be 38400, so if you had a 57200 or 9600 device, it will stop for another reason, until you edit emonhub.conf.

This is why it MUST be done by a user input somewhere along the line, history has already removed the option to automate it. itā€™s not that I believe that emonBase users are any less important, to the contrary, I believe they are important enough NOT to get a second class service and get force fed the current rfm69pi FW whether itā€™s right or not, whether they want it or not. Even if a decision was made to automate it from here on in via a new flag in the FW that can be read over serial, what new FW would you install to what devices? You cannot install the same FW to all RFM2Piā€™s, you need the user to identify and declare what device they have for a successful FW update to be possible and until they do that, you are better off avoiding an automated FW update.

1 Like

Thanks @Robert.Wall, another box it ticks is recovering from using the wrong update button. In the case of an rfm69 FW being written to a rfm12 device, currently the user will need to ssh in and issue avrdude commands to recover. With this proposal, they can just edit settings.php and hit update again and hey presto!

Actually, as expalined above, it wouldnā€™t matter quite so much, try rfm69, if it stops the data try rfm12. It saves stripping the case down and trying to id the module. Worse case scenario is that you have to edit the baud in emonhub.conf if you have a really early rfm12pi, but the upside is you now have the latest FW which if you are needing to change baud, you definately wernā€™t on the latest FW previously.

If you do a self install, then no, there is no update button.

Look at your invoice - my (old now) invoice tells me it is a 12. Simple rule of thumb will be purchase date otherwise.

I think I agree with this. However, I think Iā€™d like something in the boot partition so it can be set before first boot. Easy enough then for the shop (or a user) to set it as an EmonPi image or an EmonBase image.

Iā€™m still not a fan of updating the FW and the EmonCMS in one go, especially as the FW changes so infrequently, IME updating FW is one of the easiest ways to muck up a working system! However, it seems I am in a minority of one here :smile:. BTW, is there a write limit to the flash on these cards?

Iā€™d also like to bring forward again, that making a backup should be integral to the process but that of course only applies to EmonCMS.

I think there was already general consensus that the firmware and software stacks would be separated, so thereā€™s no minority here (unless its silent on the matter!)ā€¦

Yes, there is a write limit on the ATMegaā€™s that are on the emonBase, emonPi, emonTX and any Arduino that uses them. According to the datasheet for the 328:
image

10,000 reflash cycles is still quite a lot :slight_smile:

Ahhh! I was tricked into thinking he was referring to a emonSD image due to the emonBase reference. Yes you do not get it by default except on emonSD installs.

But you can still enable $allow_ emonpi _admin = true; even on a hosted server, I have it enabled on a test server, all I need is to install service-runner and then make sure there is something to run at the target path /home/pi/emonpi/emonpiupdate (IIRC).

Then I would have the 2 buttons ā€œupdate emonbaseā€ and ā€œupdate emonpiā€ even though this is a Debian9 VPS.

That is fine, It might be a possibility. But it adds another layer of config, perhaps as a one use ā€œbuild timeā€ config file maybe eg to specify local emoncms or not, attached HW and other settings used to create an image and then deleted could work.

I believe that the shop would NOT edit the file, it would be set to emonpi for all because itā€™s time consuming, they do not always know what itā€™s going to be used with, there needs to be a default ā€œas downloadedā€ setting and the emonpi is the 1st focus whilst the emonbase could be either rfm,12 or rfm69 as explained above, if all emonpi/emonbase users used the (default?) emonpi setting no damage is done and no recent updates are missed.

You are definitely not a minority in that respect. I really do not like the FW being constantly overwritten for no good reason, itā€™s unlikely we will hit the 10k writes but itā€™s still undue risk and wear. If it was done in a more orderly fashon and checked the FW version and only updated IF there was a newer version, that would be different. BUT even then, I would expect there to be an override, thatā€™s why I suggested the method I have, there is the option for no firmware updates.

Exactly, however, whatever your views the decision is in the hands of G&T and they want it the way it is, hence my suggestion to keep it close to the way it is by default. It is IMO the best we will achieve.

I do understand their opinion of just having one button to click to do it all for simplicity. There are (arguably) valid reasons this might not be entirely great. But I do not see that opinion changing any time soon, plus users that are capable of self-building an image or installing their own firmware are capable of changing a setting in emoncms.php.

Ah OK, Iā€™d suggested moving that update to the EmonHub page but then it got caught up in the fact that Iā€™d forgotten there were 2 parts to that; FW and SW.

I suppose it is dependency. EmonCMS, EmonHub & an RFM are actually all reasonably independent (can run independently) with the only reasonably hard dependency is that the RFM card needs EmonHub (in an OEM scenario) to extract the transmitted data.

Perhaps in the same way that the Database update has a separate page, perhaps these should also have a separate page (with the EmonPi being a special case - all in one update).

Perhaps also then, the EmonHub management should not be a top level menu item but perhaps a ā€˜tabā€™ (or menu item) within the main Admin Page. The current grouping/menu has evolved rather than been designed.

The emonhub config menu item comes from the fact it is a seperate emoncms module.

Whilst I see no reason why there couldnā€™t be a separate ā€œupdateā€ page, perhaps to host a single update button and have the console and download buttons. But I see no reason to have a separate ā€œupdate dbā€ button if we manage to get a global update (at least) emoncms or image button etc. Why would anyone want to not update the dbā€™s following an update of emoncms?

I would prefer that the updater just does the db update automatically, preferably using an api call rather than a separate php script as per the emonpi updater. If no update command is defined in settings.php. The one button could fall back to being an update db button I suppose, just to mimic existing arrangements.

The relationship between emonhub and ā€œthe firmwareā€ are closer than the relationship between emoncms and the firmware. As discussed elsewhere, the original course for the emonhub module in emoncms was to be able to have multiple emonHubs, that would suggest multiple firmwares (even one emonPi/base could have an additional jeelink) so being able to trigger remote updaters for multiple emonhub/base/piā€™s would be good, but the aim here is for the local emonhub and firmware, which may not even exist if on a remote server or dedicated Pi+hdd server etc. So the option to not update/overwrite software/firmware that doesnā€™t even exist is IMO essential.

Then adding one (or more?) update emonhub (inc/exc FWā€™s) buttons to the emonhub module is a another project, but once done, there is no reason why a user that prefers to separate their emonpiā€™s emonhub+fw from the software stack updates, can use a ā€œserver-onlyā€ type update from the admin page and setup the local emonhub+FW update on the emonhub page. Itā€™s all about the options! and the only way to get the options is via acceptable defaults. If G&T are getting the behaviour they desire and we are getting the flexibility we need, surely everyone is happy?

I donā€™t understand exactly what is meant by a ā€˜self installā€™ but what I have seems probably well described as an ā€˜emonSD imageā€™. I bought an emonBase from the shop. It has the software that came pre-installed, with the exception that I switched branch to ā€˜origin/fix_padding_spikesā€™ and I have modified some files in Lib/flot/