# EmonSD build script progress update and beta release **URL:** \ **Category:** emonSD\ **Created:** [20 June 2019 12:56 UTC](https://community.openenergymonitor.org/t/emonsd-build-script-progress-update-and-beta-release/11222 "2019-06-20T12:56:49Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![TrystanLea](https://community.openenergymonitor.org/user_avatar/community.openenergymonitor.org/trystanlea/32/31_2.png) [@TrystanLea](https://community.openenergymonitor.org/u/TrystanLea)\ **Post date:** [20 June 2019 12:56 UTC](https://community.openenergymonitor.org/t/emonsd-build-script-progress-update-and-beta-release/11222/1 "2019-06-20T12:56:49Z")
For beta release details see post below. ### EmonSD build script progress update and alpha release Back in March of this year we started a discussion on the next steps for emonSD, discussing both docker and a more generic install/build process. An initial goal of a build script was decided upon that could then potentially be adapted for a docker solution later on. Over the last three months this build script has been pieced together with long running discussions here on the forums, thanks to @borpin @pb66 @djh@Robert.Wall@Greebo@beaylott @Bill.Thomson @alexandrecuer @johnbanks for your contributions. Much of the discussion is contained in the following topics: ### Development topics - [emonSD next steps](https://community.openenergymonitor.org/t/emonsd-next-steps/10426) - [emonSD next steps: filesystem & logrotate](https://community.openenergymonitor.org/t/emonsd-next-steps-filesystem-logrotate) - [emonSD next steps: log2ram](https://community.openenergymonitor.org/t/emonsd-next-steps-log2ram) - [emonSD next steps: emonhub logging](https://community.openenergymonitor.org/t/emonsd-next-steps-emonhub-logging) - [emonSD issues created using new scripts + emonhub environment file](https://community.openenergymonitor.org/t/emonsd-issues-created-using-new-scripts-emonhub-environment-file) - [Switching emonpi/emoncms logfile locations to /var/log/emoncms](https://community.openenergymonitor.org/t/switching-emonpi-emoncms-logfile-locations-to-var-log-emoncms/11052) ### Alpha release While not complete yet, the current state of the build script warrants an [alpha release](https://en.wikipedia.org/wiki/Software_release_life_cycle) as a marker of the current state of development. The build script completes a full image build, the core emonhub, emoncms and associated dependencies all work. All using the standard master branches of the emoncms and emoncms module repositories. There are known issues with Wifi AP setup and work needed on the update scripts, there are also ongoing discussions on the logrotate, log2ram and service installation path configurations. * * * **The latest version can be found here: [https://github.com/openenergymonitor/EmonScripts](https://github.com/openenergymonitor/EmonScripts)** * * * ### Features **Configurable build script, enable/disable components as required** The build script is configurable via the config.ini configuration file included in the install script directory. It is possible to set paths, database credentials, repository branches and enable/disable installation components via this configuration file: [https://github.com/openenergymonitor/EmonScripts/blob/master/install/config.ini](https://github.com/openenergymonitor/EmonScripts/blob/master/install/config.ini) The installation scripts repository currently contains 14 scripts, these are launched by first init.sh and then main.sh which runs each respective component installation script: [https://github.com/openenergymonitor/EmonScripts/blob/master/install/main.sh](https://github.com/openenergymonitor/EmonScripts/blob/master/install/main.sh) **Log2ram and logrotate** Much of the forum discussion has centered around the log2ram and logrotate configuration, the current implementation in the build script is a basic implementation that keeps most of the default package settings the same, these are installed using the emonsd.sh component script: [https://github.com/openenergymonitor/EmonScripts/blob/master/install/emonsd.sh](https://github.com/openenergymonitor/EmonScripts/blob/master/install/emonsd.sh) This script installs the rsync\_mods branch of log2ram [cloned to openenergymonitor](http://github.com/openenergymonitor/log2ram). Log2ram handles logging to tmpfs with a periodic rsync to disk for persistance, this reduces disk write load while maintaining the benefits of log persistance. The following basic logrotate configuration is installed: - The [logrotate.d/emoncms](https://github.com/openenergymonitor/EmonScripts/blob/master/defaults/etc/logrotate.d/emoncms) configuration only directs rotated emoncms logs to a specific emoncms olddir directory. - The [logrotate.d/emonhub](https://github.com/openenergymonitor/EmonScripts/blob/master/defaults/etc/logrotate.d/emonhub) configuration currently specifies copytruncate, compress and also directs rotated logs to a specific emonhub olddir directory. @pb66’s proposed a likely better appraoch here that [utilises emonhub’s inbuilt logrotation for the first rotation](https://community.openenergymonitor.org/t/emonsd-next-steps-filesystem-logrotate/10693/173). - [logrotate.d/00\_defaults](https://github.com/openenergymonitor/EmonScripts/blob/master/defaults/etc/logrotate.d/00_defaults) sets a maxsize global override of 100k - logrotate is called from the hourly [log2ram cron.hourly entry](https://github.com/openenergymonitor/EmonScripts/blob/master/defaults/etc/cron.hourly/log2ram). We didn’t get to a point of full agreement and clarity on what the final logrotate configuration should be, so the above implementation is just a work in progress first step, that tried to incorporate several of the recommendations (using defaults as far as possible, custom emoncms and emonhub logrotate configuration, using olddir logrotation in combination with log2ram, a global maxsize override) … to be evolved into a better final solution. **General Emoncms and Emonhub improvements** As part of the process of developing these scripts a number of changes to the way emoncms and emonhub handles logging and script paths have been implemented. Including the use of /var/log/emoncms for emoncms logs and a return to using /var/log/emonhub/emonhub.log and /etc/emonhub/emonhub.conf for emonhub. These are all live in the main branches of emoncms and emonhub and are changes that have been incorporated in the existing emonSD images via the update script. The discussed environment file approach to setting emonhub paths has not been adopted yet - as it requires further discussion in the context of wider development on how emoncms handles settings. **Improved installation locations for openenergymonitor and emoncms components** The installation paths for the main components have been moved from /home/pi to /opt/openenenergymonitor and /opt/emoncms. There are associated changes that have been applied to emoncms to make this possible. **Emoncms Apache VirtualHost** The build script introduces an emoncms specific virtual host rather than symlinking /var/www/html. See virtual host configuration here: [EmonScripts/defaults/apache2/emoncms.conf at master · openenergymonitor/EmonScripts · GitHub](https://github.com/openenergymonitor/EmonScripts/blob/master/defaults/apache2/emoncms.conf) * * * ### Testing and next steps Im now running my home system using a build built using this script and will be testing the write load again, WiFi and working on the update process. This will also give me an opportunity to assess the logrotate implementation in practice and flag up any other issues. I also need to shift my focus on to other work for a period of time and I’ve written the above partly as a reference point so that I can read back and pick up from where we left off, maybe it will be useful to others who have been contributing & following along too. If you are otherwise considering building your own image from scratch rather than using a prebuilt emonSD image you might wish to consider using the build script and helping with testing as it represents a more up to date configuration than that described by the manual emoncms installation guide. Thankyou everyone for your help with this so far. ---
**Author:** ![TrystanLea](https://community.openenergymonitor.org/user_avatar/community.openenergymonitor.org/trystanlea/32/31_2.png) [@TrystanLea](https://community.openenergymonitor.org/u/TrystanLea)\ **Post date:** [20 June 2019 14:23 UTC](https://community.openenergymonitor.org/t/emonsd-build-script-progress-update-and-beta-release/11222/2 "2019-06-20T14:23:07Z")
---
**Author:** ![TrystanLea](https://community.openenergymonitor.org/user_avatar/community.openenergymonitor.org/trystanlea/32/31_2.png) [@TrystanLea](https://community.openenergymonitor.org/u/TrystanLea)\ **Post date:** [11 August 2019 08:49 UTC](https://community.openenergymonitor.org/t/emonsd-build-script-progress-update-and-beta-release/11222/3 "2019-08-11T08:49:58Z")
Wifi AP setup and switching to client mode are now working as expected. A number of changes to the wifi module and setup module have been made to improve functionality. The update process is working. The build script has been tested to work with the RaspberryPi 4 & Buster. [https://community.openenergymonitor.org/t/raspberry-pi-4/11242/50](https://community.openenergymonitor.org/t/raspberry-pi-4/11242/50) My personal test of a system built using the build script is running along nicely. My OS partition and logs write load is 12 sectors/minute (0.1 kb/s) which is similar to what I reported in [earlier testing](https://community.openenergymonitor.org/t/emonsd-next-steps-filesystem-logrotate/10693/77). The data partition write rate is ~40 sectors/minute (0.33 kb/s) that’s with a buffered write to disk at 5 mins interval. I’ve created an issue on the EmonScripts repository with my logrotate results: [Rotated logs review · Issue #6 · openenergymonitor/EmonScripts · GitHub](https://github.com/openenergymonitor/EmonScripts/issues/6) ---
**Author:** ![TrystanLea](https://community.openenergymonitor.org/user_avatar/community.openenergymonitor.org/trystanlea/32/31_2.png) [@TrystanLea](https://community.openenergymonitor.org/u/TrystanLea)\ **Post date:** [3 October 2019 13:13 UTC](https://community.openenergymonitor.org/t/emonsd-build-script-progress-update-and-beta-release/11222/4 "2019-10-03T13:13:08Z")
The EmonScripts automated emoncms build scripts have now reached beta release!: [https://github.com/openenergymonitor/EmonScripts/releases/tag/v1.0-beta](https://github.com/openenergymonitor/EmonScripts/releases/tag/v1.0-beta) Changes include: - support for new hierarchical emoncms settings - automated data partition resize as part of the initial setup process thanks to @borpin - improved documentation thanks to @alexandrecuer - improved configuration file loader - improved handling of services installation, including drop-ins for non pi users - ability to set git repository URL’s @alexandrecuer - wifi AP working - user input check to allow review of configuration - first boot update script included - testing on Ubuntu VM @borpin @TrystanLea, see: - [Installation on a DigitalOcean droplet](https://github.com/openenergymonitor/EmonScripts/issues/44) - [Install EmonCMS on Ubuntu 18LTS with scripts](https://community.openenergymonitor.org/t/install-emoncms-on-ubuntu-18lts-with-scripts/11920) - lots of raspberrypi based testing Pi4 and Pi3 B+ Special thanks to @borpin for all his help and guidance throughout the development process. There are quite a few things we would still like to improve further to make the installation process less emonpi/SD specific for VM or other environments and improve the way the configuration works, but the current state is an important milestone, warranting a beta release. 🎉 For ongoing issues see: [Issues · openenergymonitor/EmonScripts · GitHub](https://github.com/openenergymonitor/EmonScripts/issues) ---
**Author:** ![borpin](https://community.openenergymonitor.org/user_avatar/community.openenergymonitor.org/borpin/32/193_2.png) [@borpin](https://community.openenergymonitor.org/u/borpin)\ **Post date:** [3 October 2019 13:23 UTC](https://community.openenergymonitor.org/t/emonsd-build-script-progress-update-and-beta-release/11222/5 "2019-10-03T13:23:55Z")
A definite step forward and lots of work by you @TrystanLea! ---
**Author:** ![TrystanLea](https://community.openenergymonitor.org/user_avatar/community.openenergymonitor.org/trystanlea/32/31_2.png) [@TrystanLea](https://community.openenergymonitor.org/u/TrystanLea)\ **Post date:** [15 October 2019 16:45 UTC](https://community.openenergymonitor.org/t/emonsd-build-script-progress-update-and-beta-release/11222/6 "2019-10-15T16:45:12Z")
> [@Visualization: Cannot read property '0' of undefined](https://community.openenergymonitor.org/t/visualization-cannot-read-property-0-of-undefined/12171/4): > > @glyn.hudson @TrystanLea retiring these install guides really is a must ASAP! I’ve now retired the old guides, they have been removed from the installation guide list here: [https://github.com/emoncms/emoncms](https://github.com/emoncms/emoncms) and while the archived guides are still available in the emoncms repository, I’ve added a note to the top of the guides to point to the new installation scripts e.g: [https://github.com/emoncms/emoncms/tree/master/docs/RaspberryPi](https://github.com/emoncms/emoncms/tree/master/docs/RaspberryPi)