emonSD next steps

Ah, I see. I thought you envisaged including in the calibration regime, emonTx’s that fed the emonPi. It has indeed been thought about. In simple terms, there are two ways of doing it: send the calibration instruction over the air from the emonPi to the emonTx, or send uncalibrated data from the emonTx to the emonPi. The latter is technically simpler, but it significantly increases the payload on the radio channel, especially if energy values are included, as seems to be the trend.

Yes we are, but quite how they would be applied is still very much undecided. The same two basic methods can still be used, both are significantly easier in practical terms because of the duplex nature of the connection, and the final method adopted might well incorporate elements of both.

This:

could be misleading. “Unitless” refers to sending uncalibrated data, which, when done properly, means the measurement code should never require updating. There’s an alternative that doesn’t require compilation and upload, and that is to send the calibration instruction from emonHub (or as in the 3-phase PLL sketch, the user directly) to either the front end or an emonTx, and then save the calibration data in EEPROM.

Thanks Robert, “unitless” was indeed what confused me the most.
I’ll not spend any more time or energy on it :slight_smile:

Its kind of you to offer anyway @Greebo, thanks a lot.

1 Like

I’ve spent much of the day testing and tweaking the update scripts in the update_refactor branch. They are working fine on the target images: emonSD-30Oct18 & emonSD-26Oct17 as well as my Ubuntu Laptop. So far we have:

  • Much improved multi-environment support, the update process is no longer dependent on the pi username or /home/pi directory. Its possible to use any location, the update scripts detects the script folder location automatically. There is a corresponding update to emoncms to tell emoncms where to find the update scripts (if they are not in /home/pi) https://github.com/emoncms/emoncms/commit/bf96300178fd4e9ebaf2c4bef362155dc3933a4f

  • Modularised service install & update scripts, this helps reuse code and improve the readability of the main scripts.

  • The image version check fetches a list of image versions that the update supports (from the remote git repo via a curl request) prior to starting the update. Fixing a bug with the previous implementation.

  • Updates can be run from command line, see https://github.com/openenergymonitor/emonpi/tree/update_refactor/update

Im happy with the current state as an interim release (with a longer term effort in mind to improve things further) I’ve created a pull request, listing the above changes here:
https://github.com/openenergymonitor/emonpi/pull/89


The admin interface concept changes are also in a development branch here:
https://github.com/emoncms/emoncms/tree/update_concept

The emonpi update_refactor branch works fine with the original interface, but also works with the new interface that allows for the separate update options.


I would very much appreciate input on these scripts and any ideas on how to improve them further. It would probably make sense to move items such as the logrotate section in the emoncms update script into a emonsd dedicated section in main.sh or another specific emonsd.sh script to keep things cleaner. I haven’t yet added automatic iteration through /usr/emoncms/modules (if modules are installed there). Any code examples or suggestions on how to do things better would be great.

2 Likes

Not had a chance to test but commented on the PR re installing every package at once.

I think we should split out a config file (things like user and directory) create a default and setup a default/modified mechanism that git will ignore (becomes a pain later if not done now).

Make the settings hierarchical as well (i.e. load default then overwrite with any user changes).

Perhaps also make what is installed (list of modules) plus target directories, LAMP, Mosquitto etc. also RFM / EmonBase items configurable.

Simple enough to setup as a default config but makes it easier for a user to modify the required install as they wish without commenting out code.

By default, should the script install the EmonPi LCD stuff?

Still has hardcoded /home/pi locations.

[edit] - should this be in the EmonPi repository?

[edit2] - Am I right or am I seeing things - is the script emonSD_build_test.sh in both usefulscripts and emonpi repositories?

When I come to consider doing an update, the first place I’ll look is at the documentation of the process, and specifically I’ll look to see whether the process has a -n / --dry-run option that will run all the scripts and show me what it will do but not change anything on my machine.

How close am I to being able to consider doing an update?

Thanks @borpin

Yes I was thinking of this one as well, I agree.

I will get back to you on the config files and other install script related items, I’m going to spend some time on the install script today rather than the update process and will think about and refresh on this.

1 Like

Ok, I’ve made some progress on this.

I’ve split out the install script into separate components and created a config file. The scripts are all here, there are a lot of components! I’ve moved some of packages to the relevant scripts, but there is still a lot loaded at the top. Work in progress of course…
https://github.com/openenergymonitor/emonpi/tree/master/install

main.sh calls the different scripts: https://github.com/openenergymonitor/emonpi/blob/master/install/main.sh

The installation on a stock raspbian stretch image can be started with:

wget https://raw.githubusercontent.com/openenergymonitor/emonpi/master/install/init.sh
sh init.sh

What do you think of the overall concept @borpin? I feel its getting there, I would like to add more installation verification as part of the scripts, the scripts are currently safe to run multiple times but some parts of the install are not checked fully on subsequent runs which I would like to improve on.

The config file is called config.ini (see install folder), it is loaded at the top of each script.

1 Like

Great progress. I was having a look over it earlier.

First thought was that we could do with a default.config.ini as well, with any config.ini in the .gitignore. This then allows for user changes, but will not stop a git pull. I think you can then just load the default then check for a config.ini and if there use it. My Bash is a little thin but my guess is that it will just overwrite the values.

Make it clear that install command works on raspbian and some instructions on how to clone the repo before running the script, so the config file can be edited and then the script run on a non-Raspbian system.

I was going to burn a Rasbian card and give it a whirl, then put it in my EmonPi and see how it goes.

I’m not convinced about this being in the EmonPi repo especially if the update script is in both places. I’d feel more comfortable with it in an EmonScripts repo and move the useful stuff out of the usefulscripts repo.

Next enhancement is to be able to select which components to install especially EmonHub & Mosquitto and also the firmware part allow for EmonBase (i.e. no LCD). Suggest via the config.ini simple true/false flag.

You need to mask the service. disable simply stops it starting at boot but does not stop another service wanting and so starting it.

(thought I was done :smile:)

Why is emonhub (and emonpilcd) still going in the user directory and linking to /usr/share/? Is there a reason more than “we’ve always done it that way”? :smile:

I think as much as possible should be kept out of the user directory but that is me.

Final thought (yes really for now any rate), things like the emonhub.sh should stand on their own. The ‘apt-get’ builds on other installs I guess; it should have everything needed to install it I feel.

Still, looks really good progress to me.

1 Like

Thanks @borpin

Good points on the config.ini and install instructions.

Yes I also see the EmonPi name as confusing in this instance, definitely one to think about. I think its useful having the install and update scripts in the same repo, Im starting to re-use component in the install scripts that I wrote for the update scripts (the service install/update script in particular) and can see more opportunity for this.

I have removed the emonSD build script from usefulscripts for the time being to avoid confusion.

Great, I will get that added.

thanks, will look into that.

No reason in particular. I will make these paths configurable as a starting point.

I agree and can see that working.

Thanks for the input @borpin

1 Like

Another step forwards:

  • true/false flags for each install component
  • further checks to see if components have already been installed
  • automated service script install for symlinked modules, looks for module/module.service
  • emonPiLCD systemd service script
  • added mask command

commit compare:
https://github.com/openenergymonitor/emonpi/compare/7a21a2ef7386ddd9bffd040fbb86e5bebb29f934…master


Interested to know what you think of the following @borpin

I’ve been testing so far with the /usr/emoncms base directory. While the naming makes sense for

/usr/emoncms/modules

Im not sure that it makes as much sense for

/usr/emoncms/emonpi 
/usr/emoncms/avr-dude
/usr/emoncms/RFM2Pi
/usr/emoncms/emonhub

Would the following make more sense?

/usr/emon/emonpi
/usr/emon/emoncms_modules

Having a single base directory e.g /usr/emon available via the $usrdir variable where the above is all installed helps I think rather than having say emonhub installed at /usr/emonhub and then /usr/RFM2Pi etc

You mentioned the question re /usr/share location for symlinked copy of emonhub location do you think we should just switch this over to /usr/emon or whatever we decide?

I’ve also noted the following items that need further work in the emoncms_core.sh script:

  • fix flexible emoncms_core install location (currently /var/www/emoncms symlinked to /var/www/html)
  • review emoncms logfile location
  • review /var/www/html/emoncms symlink
  • review rebootbutton and emoncms-setup sudoers

and emoncms_modules.sh script:

  • emoncms module branch options
  • review wifi sudoers
  • emoncms-sync.log written to $usrdir/modules/data
  • demandshaper service has hardcoded emonpi path
  • fix backup module hard-coded paths and merge multienv branch
1 Like

I’d thought of it but didn’t want to say :laughing: :laughing:

Yes I think that is much more intuitive.

Yes - keep it all together.

Really coming on :smile:.

1 Like

thanks @borpin, a couple more steps:

  • fixed permissions issue with /var/www/html/emoncms symlink
  • fixed missing python redis client installation
  • checks for existing installation in emonpilcd script
  • changed backup module multienv branch to use backup/backup-module naming convention.
  • changed backup module config to use placeholders for easier setting of paths defined by install script
  • created backup module install.sh script that handles config of php.ini, script is then called from installation process

One of the issues I have is that the systemd service scripts all have hard-coded Exec paths e.g:

demandshaper:

ExecStart=/usr/bin/php /home/pi/demandshaper/run.php

emonhub

ExecStart=/usr/share/emonhub/emonhub.py --config-file=/home/pi/data/emonhub.conf

These could all be changed to /usr/emon but then they are still hard-coded, is there a better way to do it?
In the emonhub init.d service file the defaults were loaded from /etc/defaults, should we return to something like that? emonhub/emonhub at emon-pi · openenergymonitor/emonhub · GitHub

The only way I have seen it done that solves this, is for the systemd file to be created by the install script rather than a predefined one copied / symlinked.

1 Like

@borpin

Brian …
If you go that route then the install script could have the service script section in 3 parts - common emonhub/cms, emonhub only and emoncms only?

Lateral thought - all the sudo mkdir’s in the install script also - crazy?? - cf: a building … steelwork first then the cladding??

Sorry John, not sure I am picking up what you mean or in relation to which comment. The idea is that the install script will enable you to select the components you need so if you just want emonhub, then that is what will get installed. This bit about the services files is slightly aside to that.

Again, not really understanding your point. Trystan has largely written the scripts themselves. If you’ve got a specific point in the code, go into github and, where the line number is, click on that and copy a permalink to paste here.

Cheers

@borpin
It sounds like the ability to easily select components is covered - thx

Thanks @borpin, happy to do it this way, I guess it means we loose the ability to symlink straight, we’d need to include a mechanism to update overwrite the service file on update if applicable.

Can you load a configuration file from a systemd service file? I guess that would still rely on a fixed path e.g /etc/emonhub/default or other, perhaps modifying the service file during install is cleaner overall?

Looking around, there is a discussion on a similar line here: RFE: Add template specifier for path to unit file · Issue #6308 · systemd/systemd · GitHub although more to do with launching a service from the working directory… which looks not currently possible with the requested relative path suggestion.

Happy to go with service script modification - it’s straightforward to implement.

Discussing with @glyn.hudson here it looks like the /usr directory may not be the best location for the installation.

Reading:
http://www.pathname.com/fhs/pub/fhs-2.3.html#THEUSRHIERARCHY

/usr is the second major section of the filesystem. /usr is shareable, read-only data

Large software packages must not use a direct subdirectory under the /usr hierarchy.

It looks like /usr/local is best suited for applications that fit the /usr/local/bin, /usr/local/src /usr/local/etc hierarchy.

/opt/ looks like a better install location /opt/emon/

http://www.pathname.com/fhs/pub/fhs-2.3.html#OPTADDONAPPLICATIONSOFTWAREPACKAGES

/opt is reserved for the installation of add-on application software packages.

A package to be installed in /opt must locate its static files in a separate /opt/<package> or /opt/<provider> directory tree, where <package> is a name that describes the software package and <provider> is the provider’s LANANA registered name.

The structure of the directories below /opt/<provider> is left up to the packager of the software, though it is recommended that packages are installed in /opt/<provider>/<package> and follow a similar structure to the guidelines for /opt/package.

any views?

we should be able to make this flexible, set in the installation configuration file, so its not an urgent question.