For beta release details see post below.
EmonSD build script progress update and alpha release
Back in March of this year we started a discussion on the next steps for emonSD, discussing both docker and a more generic install/build process. An initial goal of a build script was decided upon that could then potentially be adapted for a docker solution later on. Over the last three months this build script has been pieced together with long running discussions here on the forums, thanks to @borpin @pb66 @djh @Robert.Wall @Greebo @beaylott @Bill.Thomson @alexandrecuer @johnbanks for your contributions.
Much of the discussion is contained in the following topics:
Development topics
- emonSD next steps
- emonSD next steps: filesystem & logrotate
- emonSD next steps: log2ram
- emonSD next steps: emonhub logging
- emonSD issues created using new scripts + emonhub environment file
- Switching emonpi/emoncms logfile locations to /var/log/emoncms
Alpha release
While not complete yet, the current state of the build script warrants an alpha release as a marker of the current state of development. The build script completes a full image build, the core emonhub, emoncms and associated dependencies all work. All using the standard master branches of the emoncms and emoncms module repositories.
There are known issues with Wifi AP setup and work needed on the update scripts, there are also ongoing discussions on the logrotate, log2ram and service installation path configurations.
The latest version can be found here:
https://github.com/openenergymonitor/EmonScripts
Features
Configurable build script, enable/disable components as required
The build script is configurable via the config.ini configuration file included in the install script directory. It is possible to set paths, database credentials, repository branches and enable/disable installation components via this configuration file:
https://github.com/openenergymonitor/EmonScripts/blob/master/install/config.ini
The installation scripts repository currently contains 14 scripts, these are launched by first init.sh and then main.sh which runs each respective component installation script:
https://github.com/openenergymonitor/EmonScripts/blob/master/install/main.sh
Log2ram and logrotate
Much of the forum discussion has centered around the log2ram and logrotate configuration, the current implementation in the build script is a basic implementation that keeps most of the default package settings the same, these are installed using the emonsd.sh component script:
https://github.com/openenergymonitor/EmonScripts/blob/master/install/emonsd.sh
This script installs the rsync_mods branch of log2ram cloned to openenergymonitor. Log2ram handles logging to tmpfs with a periodic rsync to disk for persistance, this reduces disk write load while maintaining the benefits of log persistance.
The following basic logrotate configuration is installed:
- The logrotate.d/emoncms configuration only directs rotated emoncms logs to a specific emoncms olddir directory.
- The logrotate.d/emonhub configuration currently specifies copytruncate, compress and also directs rotated logs to a specific emonhub olddir directory. @pb66’s proposed a likely better appraoch here that utilises emonhub’s inbuilt logrotation for the first rotation.
- logrotate.d/00_defaults sets a maxsize global override of 100k
- logrotate is called from the hourly log2ram cron.hourly entry.
We didn’t get to a point of full agreement and clarity on what the final logrotate configuration should be, so the above implementation is just a work in progress first step, that tried to incorporate several of the recommendations (using defaults as far as possible, custom emoncms and emonhub logrotate configuration, using olddir logrotation in combination with log2ram, a global maxsize override) … to be evolved into a better final solution.
General Emoncms and Emonhub improvements
As part of the process of developing these scripts a number of changes to the way emoncms and emonhub handles logging and script paths have been implemented. Including the use of /var/log/emoncms for emoncms logs and a return to using /var/log/emonhub/emonhub.log and /etc/emonhub/emonhub.conf for emonhub. These are all live in the main branches of emoncms and emonhub and are changes that have been incorporated in the existing emonSD images via the update script. The discussed environment file approach to setting emonhub paths has not been adopted yet - as it requires further discussion in the context of wider development on how emoncms handles settings.
Improved installation locations for openenergymonitor and emoncms components
The installation paths for the main components have been moved from /home/pi to /opt/openenenergymonitor and /opt/emoncms. There are associated changes that have been applied to emoncms to make this possible.
Emoncms Apache VirtualHost
The build script introduces an emoncms specific virtual host rather than symlinking /var/www/html. See virtual host configuration here: EmonScripts/emoncms.conf at master · openenergymonitor/EmonScripts · GitHub
Testing and next steps
Im now running my home system using a build built using this script and will be testing the write load again, WiFi and working on the update process. This will also give me an opportunity to assess the logrotate implementation in practice and flag up any other issues.
I also need to shift my focus on to other work for a period of time and I’ve written the above partly as a reference point so that I can read back and pick up from where we left off, maybe it will be useful to others who have been contributing & following along too.
If you are otherwise considering building your own image from scratch rather than using a prebuilt emonSD image you might wish to consider using the build script and helping with testing as it represents a more up to date configuration than that described by the manual emoncms installation guide.
Thankyou everyone for your help with this so far.
