emonSD next steps

I imagined moving all paths users and hostname etc to a config file so the script is universal.

I already have many setup scripts like this and have offered more than once to do this. As I mentioned above, I think each subsection needs to be a smaller script rather than piling it all in one script.

A test fior attached hardware is not required if it’s defined, issue will occur if it’s auto detected, eg not attached at first boot, rfm12 based rfm69pi etc etc.

Yes possibly, but we don’t want to block progress with too many edge cases, this is the most interest we have drummed up in a install script in years. Once we have a basic one in play, we can build on it. until then we effectively have nothing.

Thankyou @pb66 for the detailed run through. I have added your suggestions (some of them as comments to review and work on) to the script, here’s the commit: mostly changes recommended by @pb66 · emoncms/usefulscripts@4aa3158 · GitHub

If you search in general for the work ‘review’ in that script you can find all the things identified so far to look into further.

Sure happy to look into that and the symlinking part. Any suggestions for the folder name? EmoncmsModules? Should it be in /home/user?

Your documentation idea is nice, I was thinking it read much like the build guide as it is, but happy to extend on that.

Your modifications to log2ram sound useful I will switch it to your branch.

Yes happy for there to be a config file for all the key options and possibility of splitting the script up as discussed.

Im testing this on raspbian lite and a Pi3 b+, but I can imagine that we can make it fairly generic? debian based as you suggested? Will it be mostly the same?

I think so, but it would be easy to have the SDcard only stuff in a separate script so the main script is more generic.

The mane is fairly immaterial, I already use emoncms_modules but it could be anything.

It could be anywhere, I use /opt to install git repo based stuff like this. It really should NOT be in the user folder, but the path can/will be configurable so I would want to fallout over where it is if you guys are wanting to stick to putting everything in /home/pi.

Please understand I have not tested this to destruction as there was very little interest beyond my own. I see this as a starting point. If emonSD was to use it, it would possibly double the current user base in one hit, thus testing it beyond where it has been tested previously and the project is small enough that we can get our changes implemented or we can fork it in our own direction.

I will look through the changes you’ve made a bit later, I’m about to cook, but this sounds really promising, I hope we can make it work :slight_smile:

1 Like

There was some info from someone before about the standard Unix file layout - ah found it FHS.

Should it be /usr something?

We could fix the data file structure at the same time :smile:

2 Likes

I’ve converted most of the remaining steps I had listed as manual to automated commands, mostly using sed to change setting files:

https://github.com/emoncms/usefulscripts/blob/master/emonSD_build_test.sh

Testing on a fresh image, completes to a working install.


I will look at the emoncms modules currently in the home folder next. I will test with /usr/emoncms/modules to start with and make it configurable so that we can change the naming as we see fit.

I would like to make use of the code recently written for the emonpi update for the installation and update of emoncms modules and services. I think that code could be broken out and made more modular.
https://github.com/openenergymonitor/emonpi/blob/master/emoncmsupdate

I have a number of other things to attend to today, so it will likely be tomorrow or friday before this progresses much further.

2 Likes

We are aiming to have an emoncms container or containers which can run on it yes. So we will look at the more recent work that has been done on this. I would also like to look at porting hardware support for emonPi to home assisstant as another proof of concept exercise. We are likely to also be running home assisstant .

Speaking of HA, we were looking at the new 0.90 version this morning. The pace of change with HA is a bit dizzying at the moment. They have just added support for the nabu casa remote access service (which itself is open source) which has some similarities to experiments @TrystanLea has been conducting with a socket.io interface for local emoncms (?). Their new GUI is also coming along quickly. My comments above still stand though - we have seen a lot of stability issues and breaking changes with HA/HassOS so I think it is a gamble to base stuff on it right now. Theoretically it seems like a great fit for moving away from emonSD.

Thanks @beaylott that sounds interesting, the built in remote access sounds great.

I’ve been continuing here with the emonSD build script, currently stuck on Wifi setup I cant seem to write the wpa_supplicant.conf configuration to /tmp/wifidata as part of saving the wifi configuration… Or emoncms / php sees the changes to the file but they are not then visible directly… im sure it will become clear.

The /usr/emoncms/modules location for symlinked modules is working well. The sync and postprocess modules work out of the box ($homedir in settings.php needs to be set to /usr/emoncms/modules, the backup module requires a similar configuration change (tested for creation of backup so far)
https://github.com/emoncms/usefulscripts/blob/master/emonSD_build_test.sh#L191
I can see what changes are needed to make the installation process standard across the modules

We have also in parallel been looking at the emonpi update process in more detail, discussing with @glyn.hudson we’ve reused the existing code for detecting if the connected hardware is an rfm69pi or an emonpi board by detecting if the LCD id present on the i2c bus.

The installation and update scripts have a lot in common, it may be possible to bring them together and use flags or a configuration file to select between modes.

1 Like

The real issue is if it is a rfm12pi board.

That should be possible, but as has been said, several smaller files that do individual bits are easier to maintain that one large script.

I also wonder about moving all these scripts out of the EmonPi repo as that in itself can be confusing and also being a little more consistent in naming convention (so it does what it says on the tin).

Good point, we need an identifier, some way of reading what the board is over serial.

I agree with the several smaller files approach. I will start splitting the large install script up soon.

I was also wondering about this, what would you suggest? a repository called emonSD to hold the scripts? but then that would not cover non pi/SD installations… Its more than just emoncms but then it is also primarily an extended emoncms installation. usefulscripts is a bit vague…

Alternatively have a setting and force it to be set in the settings.php file for SDCard downloads. EmonPi could have it pre-set (or see below).

Now you’ve got me. ‘EmonScripts’ perhaps?

The scripts will ultimately be a number of building blocks to build or update an Emon system depending on the configuration.

Could possibly do with a matrix of the permutations. Perhaps the start of the script asks the questions to set the options (if this is a move away from a prebuilt card except for EmonPis perhaps).

Along those lines… I was thinking that as part of the install & update process a main script could first auto detect a module e.g emoncms/wifi and then run the install or update script relating to that module located in the modules github repository. The same would apply well & already happens to a degree with emonhub. There would need to be a naming convention e.g if an install.sh or update.sh script is detected in the repo then that script is run…

./configure
make
make install

And whether it has a standard of custom sketch in it!

BTW, I was looking at log2ram on github and looked at Issues · azlux/log2ram · GitHub I think it’s worth reading the suggestions there before commiting to log2ram.

The other issue is overwriting users calibrated/modified/custom sketches, just because it has a emonGLCD doesn’t mean it is mandatory to run the stock sketch. What if users have something else attached to the GPIO serial port? Possibly a emonTx? Possibly something non-OEM perhaps?

I made a suggestion over in the emonSD first boot - #19 by pb66 thread to have a “update command” defined in the settings.php that could default to an “update emonpi” command to carry forward the existing emonpi update behaviour.

Removing the “update emonBase” button and just having one “update” button that defaults to an “update emonpi” command removes any possibility of a user “bricking” their rfm12pi accidentally as no rfm2pi devices will get updated due to the wrong upload baud. emonBase users will have to edit settings.php to update either their rfm12pi or rfm69pi by defining the appropriate update command (apparently it would be no biggy if they neglect to do so since the rfm??pi FW has not changed in quite sometime).

Possibly there will be (currently) a choice of 5 or 6 update commands update_emonpi(default), update_rfm69pi*, update_rfm12pi*, update_emonsd update_emoncms and update_custom.

[(*) - although even with a choice between rfm12pi and rfm69pi some notes will be needed as the rfm12pi’s came in 2 flavours rfm12b and then later, rfm69cw. This was way before the rfm69pi arrived with the additional IO broken out.]

The names can be agreed later but, these options can be either commented out or just explained in the notes. Users can always add their own, but we can add an update_custom file to .gitignore, and advise users that’s where they can write their own update scripts without fear of it being overwritten or included in any PR’s they may submit.

The update_emoncms option would be just that, just updates emoncms and all it’s modules and update_emonsd would just update the emonSD image completely, but without any firmware updates being done. Handy for users with custom FW that do not want to write their own update script or lose the updates to emonhub etc by using the update_emoncms command.

This is simple and easy to use but very flexible. It removes one of the 2 update buttons whilst increasing the options and removing the accidental “wrong button” usage. It will include a much wider audience rather than just the emonSD/Pi/base users. The 12v69 warning can be removed from the gui for a cleaner more professional looking all inclusive solution. With no additional config files, identifiers or assumptions forcing a mandatory overwrite of users hard work (customising or calibrating sketches).

The 5/6 commands simply call one of the 3/4 update script wrappers (update_emoncms and update_emonsd will be full scripts, in fact update_emoncms would even be part of update_emonsd too) eg for the update_rfm69pi

#!/bin/bash
# Update script for the emonSD on emonBase with rfm69cw based RFM2Pi module.

# First we update the rest of the image inc emonhub, emoncms and the FW repo's etc
source update_emonsd.sh

# Then we update the RFM2Pi with the correct firmware.
echo "Updating rfm69pi firmware"
avrdude -v -c arduino -p ATMEGA328P -P /dev/ttyAMA0 -b 38400 -U flash:w:/home/pi/RFM2Pi/firmware/RFM69CW_RF_Demo_ATmega328/RFM12_Demo_ATmega328.cpp.hex

whilst for the update_emonpi

#!/bin/bash
# Update script for the emonSD on emonPi.

# First we update the rest of the image inc emonhub, emoncms and the FW repo's etc
source update_emonsd.sh

# Then we update the emonpi add-on boards firmware.
echo "Updating emonpi firmware"
avrdude -v -c arduino -p ATMEGA328P -P /dev/ttyAMA0 -b 115200 -U flash:w:/home/pi/emonpi/firmware/compiled/latest.hex

These can be fleshed out with better log messages and/or comments etc, but you get the gist hopefully. The bulk of the updating is done via the same script, included in any number of simple wrappers that offer options to the various user groups, easily defined in settings.php and easily customised without becoming excluded through fear of overwrites and/or refused git pulls etc.

I’d be happy to implement this, this isn’t a demand for someone else to do it, just floating the idea, if it’s well received I will do the work, I just don’t want to spend hours doing it if it’s a non-starter.

2 Likes

Simple for “standard” users, easy to customise, versatile for those who have something different, and by the sound of it, easy enough to maintain. It looks as if it will tick all the boxes for me.

As an emonBase user who can no longer even find an update button (is there one? I seem to recall seeing one once upon a time) why am I being picked on to have to edit something? And how would I know whether I have an rfm12/69pi?

1 Like

Without any further info I can only guess you might have edited the settings.php file and set “$allow_emonpi_admin = false;” perhaps? If you look at the emoncms/default.emonpi.settings.php (lines 149-150) it is/was “true” by default. This setting displays (or not) the emonSD/Pi/base specific section on the admin page.

There are soooo many answers I could offer to this question . . .

  1. You already are “picked on” for being a emonBase user, 2 examples in bold above show how emonpi is the primary target, take a look at your command prompt and hostname for 2 more. I am actually really trying to bring things closer together, but this will need to be done in small steps to avoid over complicating and therefore blocking any progress.

  2. As pointed out, there is no need for you to edit from the default “emonpi” setting, the rfm??pi FW has not changed in years, I doubt it will change in the immediate future, so leave it as it is. That’s one of the main reasons this (initial) idea works.

  3. Point 2. actually does you a favour. Since the FW hasn’t changed and is not likely to any time soon, why would you want the FW to be force fed to the RFM69Pi board at every image update? It increases the chance of a failure, it burns up the finite amount of write the chip has. It is unnecessarily fiddling with something that is not broken for no possible gain (at this time).

  4. Since we are potentially moving towards a install script it is totally possible that we might have a different approach to the default settings, maybe a default.emonbase.settings.php file (or even rfm69pi v rfm12pi versions) or maybe at install time the settings.php will be written directly from the install script with the options passed. We already need to edit other stuff in the settings.php during install, why not this too.

  5. Original emonbase users that have rfm12 based boards are “picked on” to an even greater degree as it stands since they will temporarily brick thier device by using the existing “update emonbase” button as it force feeds the rfm69 FW to the rfm12 based device, leaving users to find their own way to debug and recover.

  6. Strictly speaking, an emonpi IS an emonBase, so it is only some emonbase users that are discriminated against per se. What I’m trying to do here is remove the distinction by allowing ALL users to choose their update routine, whilst attempting to keep the powers that be on side by keeping the default to emonpi so hopefully this might be an acceptable solution.

  7. As a long time user of the emon system, why am I heavily discriminated against because I do not use either an emonPi or the emonSD? I don’t have to resort to editing settings.php, no I have to write my own update scripts from scratch that do not include much of the stuff that is emonPi (aka emonbase) centric. You might arguably think an emonBase user is a second class user, but if you are not using the emonSD, you are very much an outsider that is not really catered for at all currently.

  8. If you were using a “serial-direct” emonTx attached to the GPIO of a emonSD RPi you play this Russian roulette each time you update (via the buttons) too since neither option works for you but one will overwrite your firmware (if you have the reset line connected).

I know some of these points do not apply to you and that they do not explain “why” you might be “picked on” the short answer is “that’s how it is” currently and this suggestion actually goes a long way to correcting that imbalance.

. . . and here lies the big problem! The fact you are asking that question, which button would you click (if they were displaying?) update emonpi or update emonbase? If you have an rfm12 based device and you click update emonbase, it’s game over until you read up how to reinstall your rfm12 FW assuming you manage to work out that’s why your data has stopped!

There needs to be a note and some links to the info needed to make this decision and once it’s researched and decided, set (and forgotten?), users do not want to be faced by the same perilous question at each update, even if you know which button to use, you might just click the wrong one one day, there is currently no confirmation needed.

There is a page in learn that shows you pictures of each module so you can id the exact module, but the easy thing to check first is whether you have a double row of conn points at the opposite edge to the GPIO connector. If you do, it’s a bona fida RFM69Pi with a rfm69cw module fitted. If there is no double row of IO then you have an “RFM12Pi” and it could be either an rfm12b or rfm69cw module fitted, it will need identifying visually unless you know when you purchased it (and it is obvious from that date that it is safely one or the other) OR maybe you can tell by the serial baud setting in emonhub.conf, if it is 9600, it’s rfm12b, if it’s 57200 it’s rfm69cw but if it’s 38400, it could be either. Plus on that last point, any RFM2Pi you update correctly, will then be 38400, so if you had a 57200 or 9600 device, it will stop for another reason, until you edit emonhub.conf.

This is why it MUST be done by a user input somewhere along the line, history has already removed the option to automate it. it’s not that I believe that emonBase users are any less important, to the contrary, I believe they are important enough NOT to get a second class service and get force fed the current rfm69pi FW whether it’s right or not, whether they want it or not. Even if a decision was made to automate it from here on in via a new flag in the FW that can be read over serial, what new FW would you install to what devices? You cannot install the same FW to all RFM2Pi’s, you need the user to identify and declare what device they have for a successful FW update to be possible and until they do that, you are better off avoiding an automated FW update.

1 Like

Thanks @Robert.Wall, another box it ticks is recovering from using the wrong update button. In the case of an rfm69 FW being written to a rfm12 device, currently the user will need to ssh in and issue avrdude commands to recover. With this proposal, they can just edit settings.php and hit update again and hey presto!

Actually, as expalined above, it wouldn’t matter quite so much, try rfm69, if it stops the data try rfm12. It saves stripping the case down and trying to id the module. Worse case scenario is that you have to edit the baud in emonhub.conf if you have a really early rfm12pi, but the upside is you now have the latest FW which if you are needing to change baud, you definately wern’t on the latest FW previously.