emonSD next steps

In light of the recent discussion on emonSD logs, earlier discussions on the emonSD update process, the variety of solutions and whether we should support a more generic install/build process this thread is a starting point for thinking about the next step.

The next image is also in our development plan, published last November following the release of the October 18 image. OpenEnergyMonitor Development Plan Outline We noted the following for the emonSD section:


EmonSD Image & Hub software stack

We have recently released the new image, full details and release notes can be read here: emonSD-30Oct18 (Raspbian Stretch Pi3B+ compatible) .

Looking ahead we are interested in exploring a different approach for future image releases. Compiling an image update is currently very time consuming and the upgrade process across major image versions is overly complicated. We want to find a way to upgrade systems without asking to install the latest image and copy data across.

@beaylott at CarbonCoop has been working on an promising approach using Docker. A super lightweight base image is first installed which is then used to launch a docker container containing the OS, Apache, MySQL, Redis, Mosquitto and Emoncms. The data is stored separately to the software stack contained in the docker container, which can be easily changed in its entirety.

Examples of lightweight base images onto which a docker container/s can be installed:

Good example (thanks to jakezp) of an Emoncms self contained docker container with everything running in a single container. I tested this on a synology disk drive and it worked really well:

Alternatively, its possible to have multiple containers for the different system services:


From recent discussions it would also be good to consider:

  • systemd support
  • logging approach: e.g log2ram
  • does the docker approach fit wider community needs? e.g weā€™ve discussed a generic update script that is independent of emonSD
  • installation locations of all the different components
  • detection of hardware type: emonbase vs emonpi

What else do we need to consider?

2 Likes

In the pre-merge emoncms v9.9.6 thread we discussed making the update process compatible with non emonSD systems Emoncms V9.9.6 Stable merge (pre-merge check) - #33 by TrystanLea I made the emoncmsupdate script less emonSD specific and you asked @borpin for it to be included in the usefulscripts repository.

This line of development seems separate from the docker side of things. Do we need a generic image build script as well? e.g for DietPi?

Discussing with Glyn, the advantage of docker is that we can specify the versions of all the dependencies and have a common environment. Having a script that can run on multiple distributions might fall foul of this, with issues resulting from different dependency versions etc. What do you think @borpin?

We (Carbon Co-op) are just about to embark on a project using balena (formerly resin) so we can at least give some feedback on how the general approach. The use case is a bit different though as we dont anticipate users having a huge amount of control over the OS itself - we are mainly using it for the maintenance/operations of the device fleet. But you could conceivably base something like emonSD on it. You can grant specific containers access to the supervisor daemon API which would allow wifi and other configuration through a web GUI.

balena is mostly open source now but there is still a sense that you are tying yourselves into someones eco-system. A lot of engineering has gone into balenaOS and it is generally a more stable and better supported experience than HassOS, which in my recent experience struggles with essentials like getting on to wifi networks and stability and performance problems. I really hope HassOS improves because there is a lot to like about the architecture. For the way most people are using emonPiā€™s it could have a lot of benefits (of course if you are using it more like a Debian server then it would make sense to carry on using a Debian server as the base).

So it is a tough one. I would be interested in what other people think about these IoT docker distros.

1 Like

Thanks for the input @beaylott that sounds interesting. Are you building the emoncms stack in there? are you placing that stack in a single container or splitting it out into multiple containers? I had more luck with the former, I guess it handled closer to what I am used too when setting up a server e.g: GitHub - jakezp/emoncms

I think a good backup and restore system trumps the work involved in maintaining backward compatibility. I can backup my Unifi Controller, create a fresh install, restore the backup and Iā€™m back in business. Is a huge relief to know it works as well. A reliable backup / restore would make moving to a new image easy (assuming the image does nothing else :smile:).

yes we did

yes, but it is not available except from the scripts i.e. not from the admin screen.

I think a Build script aimed at Debian OS is a good idea.

Diet Pi is slightly odd (I said it) as when I install, I use their system for installing some of the dependencies (notably not Redis). For DietPi it would be something more specific for the parts not covered by their own install mechanism. The current available install on DietPi is just EmonHub (though it says emoncms).

I have had bad experiences with Docker. Getting an image for your underlying hardware can be a nightmare. Getting to the underlying files is likewise difficult IME. I do use a docker for the UniFi controller and for a HassIO version of Home Assistant. For the latter Iā€™d like to change some of the CSS but cannot get to the underlying source files (or so it seems).

I think a good docker is quite a specialist task. There needs to also be a consideration on the EOL of some of the core dependencies (PHP 7.0, Python 2.7 spring to mind).

Iā€™m only really suggesting a Debian based script. The user base for any other distro is so small as to not really be viable to ā€˜officiallyā€™ support. If there is a keen user of that distro then support may be available while they are around.

1 Like

Thanks @borpin, Good point re backup and restore. Glyn suggested disabling the update feature on older images rather than attempting a backward compatibility update process which would fit in with this approach. That would mean that updates would become unavailable on say the older jessie image as soon as the stretch image came out, regardless of the updated state of the jessie image. Having a backup & restore via USB disk is on my wish list, the backup could happen periodically and would not be dependent on there being enough space on the SD card to create the backup.

The potential of docker is that you could just swap out the docker container/s rather than writing a new image, there would be no need to move data etc. But that of course depends on pulling off a good docker solution, which as you and @beaylott allude can have its own challenges.

Perhaps its worth doing this regardless of what we do on the docker front and evaluating both approaches over time. I think the readonly nature of the earlier image made a build script quite complicated, now that we dropped that approach, it should be easier.

Developing a build script could also clarify the build process and configuration requirements for a docker approach.

1 Like

I thought I would make a start on a build script, it is not yet complete but it does get all the way to a working emonbase setup (after completing a few manual steps described at the end of the script), with:

  • emoncms,
  • emonhub reading from an rfmpi,
  • emoncms_mqtt, feedwriter, service-runner
  • associated dependencies for the above

https://github.com/emoncms/usefulscripts/blob/master/emonSD_build_test.sh

Its based on the raspberrypi emoncms instructions here (and associated pages e.g mqtt & low write readme but without the read only partition):

https://github.com/emoncms/emoncms/blob/master/docs/RaspberryPi/readme.md
and
https://github.com/openenergymonitor/emonpi/blob/master/docs/SD-card-build.md

Iā€™ve also updated and fixed a couple of things in the emoncms repo raspberrypi guides as I have been going along. It made it clear that the reference to the old image build guide and differing configuration from emonSD is confusing.

I can also see the oppertunity for thinking about script and documentation locations. Should they be in the emonpi repo or emoncms repo, or usefulscripts or perhaps somewhere elseā€¦

Im not fixing what the file paths should be etc at this stage, e.g emonhub.conf location, it should probably be /etc/emonhub/emonhub.conf but I have it as /home/pi/data/emonhub.conf as thatā€™s the current service path location.

Iā€™m open to input on all of that. Let me know what you think.

Iā€™ve added log2ram as @pb66 suggested, its at the end of the build process, installation was very straightforward and I can see the logs are now showing as log2ram:

image

I also reduced the logging level in emonhub.conf to warning in order to reduce the logging volume in the first place (thereā€™s a note in the manual steps section of the build script).

to set password

printf "raspberry\nemonpi2016\nemonpi2016" | passwd

to set hostname, wonā€™t get applied til reboot.

hostname=emonpi
sudo sed -i "s/$HOSTNAME/$hostname/g" /etc/hosts
printf $hostname | sudo tee /etc/hostname > /dev/null

https://github.com/emoncms/usefulscripts/blob/master/emonSD_build_test.sh#L16-L16

should really be

sudo apt-get -y dist-upgrade
sudo apt-get clean

https://github.com/emoncms/usefulscripts/blob/master/emonSD_build_test.sh#L23-L23

need to remove php-redis to avoid conflicting packages since you are also installing via PECL.

https://github.com/emoncms/usefulscripts/blob/master/emonSD_build_test.sh#L27-L27

should be

printf "\n" | sudo pecl install redis Mosquitto-beta

so that you get v0.4.0 (2017-03-13) rather than v0.2.2 (2014-05-12)

I would also seize this opportunity to put all the non-www emoncms modules in a dedicated folder so emoncms can be updated easily using 2 ā€œfor each inā€ loops one for the www path and the other for the non-www, the symlinked folders and core folders would be ignored if updater tests for being a git repo (like the one in usefulscripts).

The installation of the modules could be a simple loop rather than verbose, as could installing and enabling the emoncms services. But thatā€™s just tidier, it still works just fine.

# Symlink emoncms module folders here...

that bitā€™s not going to be easy unless you make sure there is a common theme across modules, eg each non-www module has a subfolder of the same name for the www bit.

The emoncms module branches could do with mirroring the ā€œstableā€ and ā€œmasterā€ (I know some do already) so that ALL are stable, currently all installed modules would be ā€œmasterā€.

A good start, Iā€™ve been doing it this way for many years, if you recall I offered you an image build script leading up to the emonPi launch. I also use a heredoc to set up a proper vhost rather than use the 000-default and set up UFW and few other bits if you are interested.

Personally I would split this into seperate bits eg an emoncms installer and a emonhub installer so that different images (eg hdd or non-Rpi) can be created using the same parts, just a custom main script.

The script should be written with more verbose notes and comments so that the script is the documentation, done right it can also be a webpage.

Take a look at https://emonsd.duckdns.org, I just setup a quick dynamicdns to show it can be linked to an oem site (instead of duckdns).

The code for the ā€œbuildguideā€ linked from that page is

    #!/bin/bash

##### Although this file is written in markdown and has an `.md` extension, it is an executable bash script.
##### The first line of this file is a bash shebang and that is mandatory for the script to execute correctly.
##### The sheband and any other evecutable code or command lines are all indented by 4 spacesfor better reading when viewing on GitHub or as a webpage.
##### All comments like this are written as a series of individual notes for each point  explaination of each step and 


    echo "this is code"

##### To avoid the need to harcode various path and details we have the option to include a seperate configuration file

    source settings.cnf
    echo $test_setting
    

If you clone that repo and run emonSD/build.md it still runs like any other .sh script, the .md just lets it be used directly as a webpage.

pb66@test2:~/emonSD$ ./build.md
this is code
test123
pb66@test2:~/emonSD$

The website, docs and script and there image cannot ever be out of sync as they are the same thing, only one place to edit!

all comments are directly linkable eg emonSD/build.md at master Ā· pb66/emonSD Ā· GitHub since they are headings not just text, this is useful for discussions without linking to the lines in the github repo. The hashes of the heading make the comments invisible at runtime as usual.

The 4 space indents do not effect the running of the code, but they do make the code looked fenced on the webpage.

Great, but you haveā€™nt used the version I made changes to, so the rotated logs will stay in RAM, with my version, the rotated logs are moved out of the ram&sync cycle, they are held only on disc, this reduces the size in RAM and the effort required to sync and load at startup.

See the changes here Comparing azlux:master...pb66:pb66 Ā· azlux/log2ram Ā· GitHub

Might the use of hostname be a bad choice for a Bash variable name?
I ask because there is a command called hostname, that will set a systemā€™s hostname
active as soon as itā€™s run. Food for thoughtā€¦

indeed, it could be anything as itā€™s just a local var used in the next couple of lines.

Comes from reading ā€œnot a good idea to name your variables the same as a commandā€ lots of times. :wink:
(programming in gereral vice just Bash)

Can we provide an option of ā€œDo you have an RFM Card installedā€ and if not it just does a base emoncms install?

Is there an assumed starting point of a Rasbian Lite image? What hardware (if any) is assumed?

Again, Iā€™d offer this as a user input. I already have this host name in use.

Offer a static IP setting?

Perhaps there needs to be a config file option and or an unattended setting that allows it to just run through with defaults.

I imagined moving all paths users and hostname etc to a config file so the script is universal.

I already have many setup scripts like this and have offered more than once to do this. As I mentioned above, I think each subsection needs to be a smaller script rather than piling it all in one script.

A test fior attached hardware is not required if itā€™s defined, issue will occur if itā€™s auto detected, eg not attached at first boot, rfm12 based rfm69pi etc etc.

Yes possibly, but we donā€™t want to block progress with too many edge cases, this is the most interest we have drummed up in a install script in years. Once we have a basic one in play, we can build on it. until then we effectively have nothing.

Thankyou @pb66 for the detailed run through. I have added your suggestions (some of them as comments to review and work on) to the script, hereā€™s the commit: mostly changes recommended by @pb66 Ā· emoncms/usefulscripts@4aa3158 Ā· GitHub

If you search in general for the work ā€˜reviewā€™ in that script you can find all the things identified so far to look into further.

Sure happy to look into that and the symlinking part. Any suggestions for the folder name? EmoncmsModules? Should it be in /home/user?

Your documentation idea is nice, I was thinking it read much like the build guide as it is, but happy to extend on that.

Your modifications to log2ram sound useful I will switch it to your branch.

Yes happy for there to be a config file for all the key options and possibility of splitting the script up as discussed.

Im testing this on raspbian lite and a Pi3 b+, but I can imagine that we can make it fairly generic? debian based as you suggested? Will it be mostly the same?

I think so, but it would be easy to have the SDcard only stuff in a separate script so the main script is more generic.

The mane is fairly immaterial, I already use emoncms_modules but it could be anything.

It could be anywhere, I use /opt to install git repo based stuff like this. It really should NOT be in the user folder, but the path can/will be configurable so I would want to fallout over where it is if you guys are wanting to stick to putting everything in /home/pi.

Please understand I have not tested this to destruction as there was very little interest beyond my own. I see this as a starting point. If emonSD was to use it, it would possibly double the current user base in one hit, thus testing it beyond where it has been tested previously and the project is small enough that we can get our changes implemented or we can fork it in our own direction.

I will look through the changes youā€™ve made a bit later, Iā€™m about to cook, but this sounds really promising, I hope we can make it work :slight_smile:

1 Like

There was some info from someone before about the standard Unix file layout - ah found it FHS.

Should it be /usr something?

We could fix the data file structure at the same time :smile:ā€¦

2 Likes

Iā€™ve converted most of the remaining steps I had listed as manual to automated commands, mostly using sed to change setting files:

https://github.com/emoncms/usefulscripts/blob/master/emonSD_build_test.sh

Testing on a fresh image, completes to a working install.


I will look at the emoncms modules currently in the home folder next. I will test with /usr/emoncms/modules to start with and make it configurable so that we can change the naming as we see fit.

I would like to make use of the code recently written for the emonpi update for the installation and update of emoncms modules and services. I think that code could be broken out and made more modular.
https://github.com/openenergymonitor/emonpi/blob/master/emoncmsupdate

I have a number of other things to attend to today, so it will likely be tomorrow or friday before this progresses much further.

2 Likes