EmonSD build script progress update and beta release

For beta release details see post below.

EmonSD build script progress update and alpha release

Back in March of this year we started a discussion on the next steps for emonSD, discussing both docker and a more generic install/build process. An initial goal of a build script was decided upon that could then potentially be adapted for a docker solution later on. Over the last three months this build script has been pieced together with long running discussions here on the forums, thanks to @borpin @pb66 @djh @Robert.Wall @Greebo @beaylott @Bill.Thomson @alexandrecuer @johnbanks for your contributions.

Much of the discussion is contained in the following topics:

Development topics

Alpha release

While not complete yet, the current state of the build script warrants an alpha release as a marker of the current state of development. The build script completes a full image build, the core emonhub, emoncms and associated dependencies all work. All using the standard master branches of the emoncms and emoncms module repositories.

There are known issues with Wifi AP setup and work needed on the update scripts, there are also ongoing discussions on the logrotate, log2ram and service installation path configurations.


The latest version can be found here:
https://github.com/openenergymonitor/EmonScripts


Features

Configurable build script, enable/disable components as required
The build script is configurable via the config.ini configuration file included in the install script directory. It is possible to set paths, database credentials, repository branches and enable/disable installation components via this configuration file:
https://github.com/openenergymonitor/EmonScripts/blob/master/install/config.ini

The installation scripts repository currently contains 14 scripts, these are launched by first init.sh and then main.sh which runs each respective component installation script:
https://github.com/openenergymonitor/EmonScripts/blob/master/install/main.sh

Log2ram and logrotate
Much of the forum discussion has centered around the log2ram and logrotate configuration, the current implementation in the build script is a basic implementation that keeps most of the default package settings the same, these are installed using the emonsd.sh component script:
https://github.com/openenergymonitor/EmonScripts/blob/master/install/emonsd.sh

This script installs the rsync_mods branch of log2ram cloned to openenergymonitor. Log2ram handles logging to tmpfs with a periodic rsync to disk for persistance, this reduces disk write load while maintaining the benefits of log persistance.

The following basic logrotate configuration is installed:

We didn’t get to a point of full agreement and clarity on what the final logrotate configuration should be, so the above implementation is just a work in progress first step, that tried to incorporate several of the recommendations (using defaults as far as possible, custom emoncms and emonhub logrotate configuration, using olddir logrotation in combination with log2ram, a global maxsize override) … to be evolved into a better final solution.

General Emoncms and Emonhub improvements
As part of the process of developing these scripts a number of changes to the way emoncms and emonhub handles logging and script paths have been implemented. Including the use of /var/log/emoncms for emoncms logs and a return to using /var/log/emonhub/emonhub.log and /etc/emonhub/emonhub.conf for emonhub. These are all live in the main branches of emoncms and emonhub and are changes that have been incorporated in the existing emonSD images via the update script. The discussed environment file approach to setting emonhub paths has not been adopted yet - as it requires further discussion in the context of wider development on how emoncms handles settings.

Improved installation locations for openenergymonitor and emoncms components
The installation paths for the main components have been moved from /home/pi to /opt/openenenergymonitor and /opt/emoncms. There are associated changes that have been applied to emoncms to make this possible.

Emoncms Apache VirtualHost
The build script introduces an emoncms specific virtual host rather than symlinking /var/www/html. See virtual host configuration here: EmonScripts/emoncms.conf at master · openenergymonitor/EmonScripts · GitHub


Testing and next steps

Im now running my home system using a build built using this script and will be testing the write load again, WiFi and working on the update process. This will also give me an opportunity to assess the logrotate implementation in practice and flag up any other issues.

I also need to shift my focus on to other work for a period of time and I’ve written the above partly as a reference point so that I can read back and pick up from where we left off, maybe it will be useful to others who have been contributing & following along too.

If you are otherwise considering building your own image from scratch rather than using a prebuilt emonSD image you might wish to consider using the build script and helping with testing as it represents a more up to date configuration than that described by the manual emoncms installation guide.

Thankyou everyone for your help with this so far.

1 Like

Wifi AP setup and switching to client mode are now working as expected. A number of changes to the wifi module and setup module have been made to improve functionality.

The update process is working.

The build script has been tested to work with the RaspberryPi 4 & Buster.
https://community.openenergymonitor.org/t/raspberry-pi-4/11242/50

My personal test of a system built using the build script is running along nicely. My OS partition and logs write load is 12 sectors/minute (0.1 kb/s) which is similar to what I reported in earlier testing. The data partition write rate is ~40 sectors/minute (0.33 kb/s) that’s with a buffered write to disk at 5 mins interval.

I’ve created an issue on the EmonScripts repository with my logrotate results:
Rotated logs review · Issue #6 · openenergymonitor/EmonScripts · GitHub

The EmonScripts automated emoncms build scripts have now reached beta release!:
https://github.com/openenergymonitor/EmonScripts/releases/tag/v1.0-beta

Changes include:

Special thanks to @borpin for all his help and guidance throughout the development process.

There are quite a few things we would still like to improve further to make the installation process less emonpi/SD specific for VM or other environments and improve the way the configuration works, but the current state is an important milestone, warranting a beta release. :tada:

For ongoing issues see: Issues · openenergymonitor/EmonScripts · GitHub

1 Like

A definite step forward and lots of work by you @TrystanLea!

1 Like

I’ve now retired the old guides, they have been removed from the installation guide list here: https://github.com/emoncms/emoncms and while the archived guides are still available in the emoncms repository, I’ve added a note to the top of the guides to point to the new installation scripts e.g: https://github.com/emoncms/emoncms/tree/master/docs/RaspberryPi

1 Like