emonSD next steps

emonSD
69

I’ve added a brief readme to give an overview of how to run the current state of the update scripts in the update_refactor branch: https://github.com/openenergymonitor/emonpi/tree/update_refactor/update They run well from both the emoncms interface and manually via command line

70
  • What should the user be able to do via the update UI?
  • Location of configuration for firmware type?

Going back to these points, I think my preference for saving the firmware type is as a user preference in the mysql database. Alongside this there could be an attempt to detect the emonpi hardware by looking for the LCD, it would then give a first suggestion as to the relevant hardware, but allow the user to change the selection if its not right (e.g rfm12pi or custom). If the user does upload incorrect firmware, the dropdown selection for firmware type allows for then updating the correct version, which is a step forward from today. What do you both think of this approach?

71

I was thinking along those lines too. So the script/page lists all the available options for a particular item, and adds “Based on [past preference | hardware detected] you should probably choose [the sensible default].” and make that the default choice.

1 Like
72

If the EmonPi is regarded more as a ‘fire & forget’ system, I still think having something in the boot partition that can be read and used to fix the type could be a better first step, if not found then determine type by other means.

73

Thanks @borpin While I think that would be a good, the main issue I see is that we produce one image that is used for both emonpi and emonbase systems that leave our shop. We could use the LCD detection method at first boot update to create such a file automatically but then is that any better than detection later?

1 Like
74

Not wishing to hammer the point home too much, here’s the problem rearing its head again: Compile calibration changes - #2 by Robert.Wall

1 Like
75

In the short term, if we have an entry called ‘custom - do not overwrite’ in the dropdown menu, the ‘UpdateAll’ button could then skip the upload step.

In the longer term the emonhub based calibration approach using your unitless code @Robert.Wall will be the best solution for this.

76

I don’t have an emonPi so I can’t really test much of this, but going over the calibration process in my head, it feels like it should be possible to create an interface in the emonCMS admin page that lets you change the various “calibration” portions of the sketch, then compiles and uploads that to the ATMega add-on board.

The current calibration method for the emonTX requires the calibrating user to edit a handful of places in the code based on their own measurements, recompile and repeat. If those could be sed’d into the .ino file by a script, it should be possible to fully calibrate the front-end firmware all from the web interface.

Obviously that still doesn’t fix the “I’ve written my own custom firmware” situation… but for those users, they already know how to upload their own firmware using avrdude, pio or whatever.

Or… have I seriously under-thought this?
I guess I could test the theory just with an Arduino hooked up to the appropriate Pi GPIO pins?

If I spent some time making something like this work, would there be any interest in it?

77

The problem with the emonTx is, it doesn’t have OTA input of any sort back from the base - it is transmit only. It doesn’t have to be that way, but there are a lot of factors to consider if we do move in that direction. I do know Paul B has it on his long-range radar, but whether it will ever come to fruition is another story.

In terms of the emonPi front end, it’s a whole lot easier because the serial interconnection already exists, and when I get the chance, I’m actually working on combining on-line (via emonhub.conf) calibration and continuous monitoring for the front end.

The real problem for the user who wants to accurately calibrate his emonPi is the sketch’s editing and uploading is a fiendishly difficult procedure - I can’t find the full instructions any more, so a new user has no chance, and when they’ve done it, it will always be overwritten unless they choose the wrong update method.

2 Likes
78

I think your unitless approach is the way to proceed here . It makes calibration much easier as its all in the emoncms interface and doesn’t require compilation and upload of code.

1 Like
79

I still don’t understand what that is? Are we still talking about updating the nodes section of emonhub.conf with scale factors, or something entirely different?

80

Re-reading my post, I can see how it was confusing…

To clarify, I was only ever suggesting fixing the calibration for the emonPi (as the emonTX is never in danger of being overwritten by hitting an update button). I merely used the emonTX calibration process as an example that I was aware of…

81

Ah, I see. I thought you envisaged including in the calibration regime, emonTx’s that fed the emonPi. It has indeed been thought about. In simple terms, there are two ways of doing it: send the calibration instruction over the air from the emonPi to the emonTx, or send uncalibrated data from the emonTx to the emonPi. The latter is technically simpler, but it significantly increases the payload on the radio channel, especially if energy values are included, as seems to be the trend.

Yes we are, but quite how they would be applied is still very much undecided. The same two basic methods can still be used, both are significantly easier in practical terms because of the duplex nature of the connection, and the final method adopted might well incorporate elements of both.

This:

could be misleading. “Unitless” refers to sending uncalibrated data, which, when done properly, means the measurement code should never require updating. There’s an alternative that doesn’t require compilation and upload, and that is to send the calibration instruction from emonHub (or as in the 3-phase PLL sketch, the user directly) to either the front end or an emonTx, and then save the calibration data in EEPROM.

82

Thanks Robert, “unitless” was indeed what confused me the most.
I’ll not spend any more time or energy on it :slight_smile:

83

Its kind of you to offer anyway @Greebo, thanks a lot.

1 Like
84

I’ve spent much of the day testing and tweaking the update scripts in the update_refactor branch. They are working fine on the target images: emonSD-30Oct18 & emonSD-26Oct17 as well as my Ubuntu Laptop. So far we have:

  • Much improved multi-environment support, the update process is no longer dependent on the pi username or /home/pi directory. Its possible to use any location, the update scripts detects the script folder location automatically. There is a corresponding update to emoncms to tell emoncms where to find the update scripts (if they are not in /home/pi) https://github.com/emoncms/emoncms/commit/bf96300178fd4e9ebaf2c4bef362155dc3933a4f

  • Modularised service install & update scripts, this helps reuse code and improve the readability of the main scripts.

  • The image version check fetches a list of image versions that the update supports (from the remote git repo via a curl request) prior to starting the update. Fixing a bug with the previous implementation.

  • Updates can be run from command line, see https://github.com/openenergymonitor/emonpi/tree/update_refactor/update

Im happy with the current state as an interim release (with a longer term effort in mind to improve things further) I’ve created a pull request, listing the above changes here:
https://github.com/openenergymonitor/emonpi/pull/89

The admin interface concept changes are also in a development branch here:
https://github.com/emoncms/emoncms/tree/update_concept

The emonpi update_refactor branch works fine with the original interface, but also works with the new interface that allows for the separate update options.

I would very much appreciate input on these scripts and any ideas on how to improve them further. It would probably make sense to move items such as the logrotate section in the emoncms update script into a emonsd dedicated section in main.sh or another specific emonsd.sh script to keep things cleaner. I haven’t yet added automatic iteration through /usr/emoncms/modules (if modules are installed there). Any code examples or suggestions on how to do things better would be great.

2 Likes
85

Not had a chance to test but commented on the PR re installing every package at once.

I think we should split out a config file (things like user and directory) create a default and setup a default/modified mechanism that git will ignore (becomes a pain later if not done now).

Make the settings hierarchical as well (i.e. load default then overwrite with any user changes).

Perhaps also make what is installed (list of modules) plus target directories, LAMP, Mosquitto etc. also RFM / EmonBase items configurable.

Simple enough to setup as a default config but makes it easier for a user to modify the required install as they wish without commenting out code.

By default, should the script install the EmonPi LCD stuff?

Still has hardcoded /home/pi locations.

[edit] - should this be in the EmonPi repository?

[edit2] - Am I right or am I seeing things - is the script emonSD_build_test.sh in both usefulscripts and emonpi repositories?

86

When I come to consider doing an update, the first place I’ll look is at the documentation of the process, and specifically I’ll look to see whether the process has a -n / --dry-run option that will run all the scripts and show me what it will do but not change anything on my machine.

How close am I to being able to consider doing an update?

87

Thanks @borpin

Yes I was thinking of this one as well, I agree.

I will get back to you on the config files and other install script related items, I’m going to spend some time on the install script today rather than the update process and will think about and refresh on this.

1 Like
88

Ok, I’ve made some progress on this.

I’ve split out the install script into separate components and created a config file. The scripts are all here, there are a lot of components! I’ve moved some of packages to the relevant scripts, but there is still a lot loaded at the top. Work in progress of course…
https://github.com/openenergymonitor/emonpi/tree/master/install

main.sh calls the different scripts: https://github.com/openenergymonitor/emonpi/blob/master/install/main.sh

The installation on a stock raspbian stretch image can be started with:

wget https://raw.githubusercontent.com/openenergymonitor/emonpi/master/install/init.sh
sh init.sh

What do you think of the overall concept @borpin? I feel its getting there, I would like to add more installation verification as part of the scripts, the scripts are currently safe to run multiple times but some parts of the install are not checked fully on subsequent runs which I would like to improve on.

The config file is called config.ini (see install folder), it is loaded at the top of each script.

1 Like