# emonSD issues created using new scripts + emonhub environment file **URL:** \ **Category:** emonSD\ **Created:** [11 May 2019 09:45 UTC](https://community.openenergymonitor.org/t/emonsd-issues-created-using-new-scripts-emonhub-environment-file/10918 "2019-05-11T09:45:32Z")\ **Posts on this page:** 20\ **Page:** 3
**Author:** ![djh](https://community.openenergymonitor.org/letter_avatar_proxy/v4/letter/d/e36b37/32.png) [@djh](https://community.openenergymonitor.org/u/djh)\ **Post date:** [23 May 2019 11:56 UTC](https://community.openenergymonitor.org/t/emonsd-issues-created-using-new-scripts-emonhub-environment-file/10918/41 "2019-05-23T11:56:26Z")
> [@pb66](#): > > I sort of panicked a bit and made a hasty call because these changes are going into the live repo’s, I’m really not comfortable with that. It means we must get it right at every commit because we don’t know when anyone might update. It seems to me that this is a separate but extremely important issue. Surely it would make more sense to have the classical three repos: development, testing and production? So all this stuff would be going on in the development repo, which people who wish could use on test installation, but nobody would use for their actual energy monitoring. Then when there was an apparently stable release candidate, it could be cloned to testing and more people would use it, including me if we resolve this paths issue, and a few brave souls might actually use it for live monitoring. Finally, when its known to work, it’s cloned to a production repo and downloaded by the great unwashed and used for producing factory product etc. ---
**Author:** ![djh](https://community.openenergymonitor.org/letter_avatar_proxy/v4/letter/d/e36b37/32.png) [@djh](https://community.openenergymonitor.org/u/djh)\ **Post date:** [23 May 2019 11:57 UTC](https://community.openenergymonitor.org/t/emonsd-issues-created-using-new-scripts-emonhub-environment-file/10918/42 "2019-05-23T11:57:38Z")
> [@pb66](#): > > Can we please try and separate “fixing immediate issues with the current image” and “developing the new installer and updater” and get it right, before rolling it out? +99 - see previous post ---
**Author:** ![djh](https://community.openenergymonitor.org/letter_avatar_proxy/v4/letter/d/e36b37/32.png) [@djh](https://community.openenergymonitor.org/u/djh)\ **Post date:** [23 May 2019 12:05 UTC](https://community.openenergymonitor.org/t/emonsd-issues-created-using-new-scripts-emonhub-environment-file/10918/43 "2019-05-23T12:05:29Z")
> [@pb66](#): > > My reasons for doing the “clone&symlink” is that conf files are found where conf files live and the executables or binaries are found where the executables/binaries live etc etc tec. I part company with you here. /opt is a sensible place for binaries and /etc/opt is a sensible place for their config files etc. So there’s no need to symlink them to other places. But again, the first step is an environment file. Then the actual paths are moot. I can install it on my openSUSE machine. I could submit an OBS ( [https://build.opensuse.org/](https://build.opensuse.org/) ) job to build packaged versions for openSUSE and Debian etc. Note that /var/www is not where FHS says to put web data - that’s /srv/www - and that’s where openSUSE puts it. ---
**Author:** ![borpin](https://community.openenergymonitor.org/user_avatar/community.openenergymonitor.org/borpin/32/193_2.png) [@borpin](https://community.openenergymonitor.org/u/borpin)\ **Post date:** [23 May 2019 12:06 UTC](https://community.openenergymonitor.org/t/emonsd-issues-created-using-new-scripts-emonhub-environment-file/10918/44 "2019-05-23T12:06:20Z")
> [@djh](#): > > It seems to me that this is a separate but extremely important issue. Surely it would make more sense to have the classical three repos: development, testing and production? There is a bit of history in play and how it has ‘developed’ over time. For some of the repos, there is a ‘stable’ branch which is effectively a release branch. However, some repos do not have a ‘stable’ branch. ---
**Author:** ![djh](https://community.openenergymonitor.org/letter_avatar_proxy/v4/letter/d/e36b37/32.png) [@djh](https://community.openenergymonitor.org/u/djh)\ **Post date:** [23 May 2019 12:14 UTC](https://community.openenergymonitor.org/t/emonsd-issues-created-using-new-scripts-emonhub-environment-file/10918/45 "2019-05-23T12:14:08Z")
> [@pb66](#): > > In theory you should be able to move /var/www/emoncms to /opt/emon/emoncms and edit the symlink in /var/www/html and everything work as it does now! I have some heavily customised wiki software on my PC that I use for my diary and making notes etc. It’s installed in /opt and runs happily from there. It isn’t symlinked below /srv/www; I just tell apache where to find it with an apache config file - I need that anyway because there are specials like rewrite rules that the wiki software needs. ---
**Author:** ![djh](https://community.openenergymonitor.org/letter_avatar_proxy/v4/letter/d/e36b37/32.png) [@djh](https://community.openenergymonitor.org/u/djh)\ **Post date:** [23 May 2019 12:22 UTC](https://community.openenergymonitor.org/t/emonsd-issues-created-using-new-scripts-emonhub-environment-file/10918/46 "2019-05-23T12:22:05Z")
> [@pb66](#): > > These are 2 separate things IMO, no we shouldn’t necessarily “stick to using /opt”, regardless of whether we use EnvironmentFile, in fact if you stick to using /opt, there is no need to use EnvironmentFile as the path is never likely to change, it’s no different to using /home/pi where every setup _does_ have a user pi’s homedir to install to. > > Using EnvironmentFile is not entirely a bad idea IMO, but it is a very “init.d” way of doing things, it’s how the emonhub init.d service has worked for years with the default paths in /etc/default/emonhub. The “systemd” way of doing things is to use edit files. But which ever way you get the path into the service unit, it is only necessary if you want to be able to vary the path, if all installs are going to be in /opt why edit the unit at all? Because this isn’t just about the path to the binary, it’s also about the apache path. It’s essential for ALL paths to be in the Environment File (and that is a systemd concept just as much as a sysV one). There are very few runtime environments that can’t read environment variables, and that’s for a very good reason. > [@pb66](#): > > Only on more exotic installs that don’t use FHS will that path need to be overridden by a drop-in file, it seems pointless to add the confusion of mandatory edits, environment files or non-fhs paths just to serve those very few users when there is a perfectly good other way. As I pointed out elsewhere, Debian and Raspbian don’t follow FHS, so you’re arguing against yourself. One reason there aren’t more installs on other distros is precisely because paths are hardcoded to Debian and other distros use other paths. IMHO, one Environment File is a whole lot simpler and cleaner than a bunch of symlinks. ---
**Author:** ![djh](https://community.openenergymonitor.org/letter_avatar_proxy/v4/letter/d/e36b37/32.png) [@djh](https://community.openenergymonitor.org/u/djh)\ **Post date:** [23 May 2019 16:23 UTC](https://community.openenergymonitor.org/t/emonsd-issues-created-using-new-scripts-emonhub-environment-file/10918/47 "2019-05-23T16:23:35Z")
> [@borpin](#): > > There is a bit of history in play and how it has ‘developed’ over time. For some of the repos, there is a ‘stable’ branch which is effectively a release branch. However, some repos do not have a ‘stable’ branch. My immediate reaction to that was to file a bug for it. But then I thought that you know a whole lot more than me about git and the best ways to use it, so perhaps you’d like to write a bug that proposes a sensible setup? Pretty please 🙂 ---
**Author:** ![borpin](https://community.openenergymonitor.org/user_avatar/community.openenergymonitor.org/borpin/32/193_2.png) [@borpin](https://community.openenergymonitor.org/u/borpin)\ **Post date:** [23 May 2019 16:52 UTC](https://community.openenergymonitor.org/t/emonsd-issues-created-using-new-scripts-emonhub-environment-file/10918/48 "2019-05-23T16:52:16Z")
> [@djh](#): > > I thought that you know a whole lot more than me about git and the best ways to use it I definitely would not go that far! It isn’t that simple. The releases for the main repo now use tags so that is an improvement but for now we need to live with how it is and just recognize the limitations. ---
**Author:** ![TrystanLea](https://community.openenergymonitor.org/user_avatar/community.openenergymonitor.org/trystanlea/32/31_2.png) [@TrystanLea](https://community.openenergymonitor.org/u/TrystanLea)\ **Post date:** [25 May 2019 14:55 UTC](https://community.openenergymonitor.org/t/emonsd-issues-created-using-new-scripts-emonhub-environment-file/10918/49 "2019-05-25T14:55:47Z")
I think Im edging towards the Environment file solution for setting the service paths and using /opt/emon/emonhub directly. We discussed the use of /opt over /usr here: [https://community.openenergymonitor.org/t/emonsd-next-steps/10426/106](https://community.openenergymonitor.org/t/emonsd-next-steps/10426/106) @djh said: > [@emonSD next steps](https://community.openenergymonitor.org/t/emonsd-next-steps/10426/106): > > /opt/package-name for code and move data directories to /var/opt/package-name for data after installation are reasonable places for external products to be placed. If a distro packages a product and releases it, then it will typically put it under /usr with data under /var/lib Given that in this case the distro is not packaging the ‘product’ is it not therefore correct to use /opt directly? The precise location aside, one of the goals of this installation script is to provide flexible installation paths rather than nailing everything down. I know we are still a number of steps away from that (e.g /var/www discussion), but perhaps using the environment file to allow emonhub to either be installed by our installer or perhaps if someone wants to do so be packaged independently and installed by the distro into /usr/local or other relevant location? I haven’t tried using an environment file with systemd yet so I’m assuming this is possible and not bad practice ---
**Author:** ![TrystanLea](https://community.openenergymonitor.org/user_avatar/community.openenergymonitor.org/trystanlea/32/31_2.png) [@TrystanLea](https://community.openenergymonitor.org/u/TrystanLea)\ **Post date:** [25 May 2019 14:57 UTC](https://community.openenergymonitor.org/t/emonsd-issues-created-using-new-scripts-emonhub-environment-file/10918/50 "2019-05-25T14:57:51Z")
> [@pb66](#): > > it should be served from /var/www/emoncms by a dedicated emoncms.conf vhost Lets do this, seems like a sensible step. ---
**Author:** ![djh](https://community.openenergymonitor.org/letter_avatar_proxy/v4/letter/d/e36b37/32.png) [@djh](https://community.openenergymonitor.org/u/djh)\ **Post date:** [25 May 2019 16:30 UTC](https://community.openenergymonitor.org/t/emonsd-issues-created-using-new-scripts-emonhub-environment-file/10918/51 "2019-05-25T16:30:57Z")
Both ideas sound sensible to me, FWIW @TrystanLea Note that the environment file should include the `/var/www/emoncms` path and all other paths. You can read the file into the environment in a systemd unit with the EnvironmentFile directive. It looks like the best way to read them into a PHP environment is with the dotenv package as described at [How to Use Environment Variables ― Scotch.io](https://scotch.io/tutorials/how-to-use-environment-variables#toc-getting-environment-variables-with-php) . There’s also a python-dotenv package that does something similar. I think apache requires special handling, since it only accepts environment variables from the process that starts it, and the startup mechanism for apache varies between distros and is changing over time. It may be simplest to make the apache vhost conf file use Define statements within it for the paths. Then in order to install it in a non-default place it will be necessary to edit that file as well as the environment file before copying/moving it to wherever it is needed on that system. A bit more hassle but I don’t see a simpler way and it shouldn’t be a problem as long as it is documented. If anybody can think of a better way, I’m all ears! I suggest using an `EMON_` prefix for all the variables. ---
**Author:** ![TrystanLea](https://community.openenergymonitor.org/user_avatar/community.openenergymonitor.org/trystanlea/32/31_2.png) [@TrystanLea](https://community.openenergymonitor.org/u/TrystanLea)\ **Post date:** [26 May 2019 18:25 UTC](https://community.openenergymonitor.org/t/emonsd-issues-created-using-new-scripts-emonhub-environment-file/10918/52 "2019-05-26T18:25:00Z")
> [@djh](#): > > Note that the environment file should include the `/var/www/emoncms` path and all other paths. With a view to emonhub being able to be installed seperatly, ideally the environment file would be specific to emonhub in this case rather than one file for all aspects of the emonSD installation. Can you see any issue with this? ---
**Author:** ![djh](https://community.openenergymonitor.org/letter_avatar_proxy/v4/letter/d/e36b37/32.png) [@djh](https://community.openenergymonitor.org/u/djh)\ **Post date:** [26 May 2019 18:39 UTC](https://community.openenergymonitor.org/t/emonsd-issues-created-using-new-scripts-emonhub-environment-file/10918/53 "2019-05-26T18:39:01Z")
> [@TrystanLea](#): > > With a view to emonhub being able to be installed seperatly, ideally the environment file would be specific to emonhub in this case rather than one file for all aspects of the emonSD installation. Can you see any issue with this? Not off the top of my head. It obviously complicates things with multiple files then necessary for the complete system. Equally, having extra environment variables defined in the file wouldn’t have a negative effect on anything if they weren’t used in an emonhub-only installation, so it might be easier to keep everything in one file. [edit]: perhaps the best idea depends on the directory structure. I think I would have one `.env` file in each top-level directory. So if the plan is to have emonhub as a package in `/opt/emonhub` and emoncms (or whatever) in `/opt/emoncms` then I would have an environment file in each. Say `emonhub.env` and `emoncms.env` ---
**Author:** ![borpin](https://community.openenergymonitor.org/user_avatar/community.openenergymonitor.org/borpin/32/193_2.png) [@borpin](https://community.openenergymonitor.org/u/borpin)\ **Post date:** [27 May 2019 09:52 UTC](https://community.openenergymonitor.org/t/emonsd-issues-created-using-new-scripts-emonhub-environment-file/10918/54 "2019-05-27T09:52:07Z")
> [@TrystanLea](#): > > The precise location aside, one of the goals of this installation script is to provide flexible installation paths rather than nailing everything down. Are you going to write the Environment file on the fly rather than have one in the git repo(s)? How will you handle updates (where a user may have edited it)? This takes us back to the hierarchical settings issue IMHO. You could employ the general principle of drop-ins - effectively the same as my hierarchical settings proposal from way back. ---
**Author:** ![djh](https://community.openenergymonitor.org/letter_avatar_proxy/v4/letter/d/e36b37/32.png) [@djh](https://community.openenergymonitor.org/u/djh)\ **Post date:** [27 May 2019 10:12 UTC](https://community.openenergymonitor.org/t/emonsd-issues-created-using-new-scripts-emonhub-environment-file/10918/55 "2019-05-27T10:12:57Z")
> [@borpin](#): > > Are you going to write the Environment file on the fly rather than have one in the git repo(s)? How will you handle updates (where a user may have edited it)? The standard way to handle such issues is to have a file named `emonhub-defaults.env` or similar in the repo, so that is what gets overwritten in an update. But the running system looks for `emonhub.env` to get its paths so it can stop with an error message if there is no such file, or perhaps automatically copy the defaults file to the operational file location if it doesn’t exist. That way any customisation is never overwritten accidently. ---
**Author:** ![borpin](https://community.openenergymonitor.org/user_avatar/community.openenergymonitor.org/borpin/32/193_2.png) [@borpin](https://community.openenergymonitor.org/u/borpin)\ **Post date:** [27 May 2019 10:15 UTC](https://community.openenergymonitor.org/t/emonsd-issues-created-using-new-scripts-emonhub-environment-file/10918/56 "2019-05-27T10:15:00Z")
> [@djh](#): > > The standard way to handle such issues Not a standard that is currently used here - which is my point. Oh and the problem with that is, that, unless you load both (unclear if you meant that), if a new variable is added, the file in use does not have that variable. The emoncms settings file is an example of the issues. ---
**Author:** ![djh](https://community.openenergymonitor.org/letter_avatar_proxy/v4/letter/d/e36b37/32.png) [@djh](https://community.openenergymonitor.org/u/djh)\ **Post date:** [27 May 2019 10:31 UTC](https://community.openenergymonitor.org/t/emonsd-issues-created-using-new-scripts-emonhub-environment-file/10918/57 "2019-05-27T10:31:31Z")
> [@borpin](#): > > Oh and the problem with that is, that, unless you load both (unclear if you meant that), if a new variable is added, the file in use does not have that variable. I don’t think that’s a serious difficulty. Firstly, I think it will be fairly rare for a new path to be added to the environment; most changes are within existing trees. Secondly, the update can check whether the file in use was modified and if not use the new default. Thirdly, if it was modified then the user clearly understands enough to do that, and likely expects future maintenance and a clear and comprehensive prompt about what needs doing should be enough to get them going. ---
**Author:** ![borpin](https://community.openenergymonitor.org/user_avatar/community.openenergymonitor.org/borpin/32/193_2.png) [@borpin](https://community.openenergymonitor.org/u/borpin)\ **Post date:** [27 May 2019 10:33 UTC](https://community.openenergymonitor.org/t/emonsd-issues-created-using-new-scripts-emonhub-environment-file/10918/58 "2019-05-27T10:33:16Z")
> [@djh](#): > > I don’t think that’s a serious difficulty. You have never had to rewrite your settings.php file because a new setting has been added then. ---
**Author:** ![TrystanLea](https://community.openenergymonitor.org/user_avatar/community.openenergymonitor.org/trystanlea/32/31_2.png) [@TrystanLea](https://community.openenergymonitor.org/u/TrystanLea)\ **Post date:** [27 May 2019 11:19 UTC](https://community.openenergymonitor.org/t/emonsd-issues-created-using-new-scripts-emonhub-environment-file/10918/59 "2019-05-27T11:19:41Z")
> [@TrystanLea](#): > > - finalise emoncms log locations i.e: /var/log/emon/emoncms/sync.log > - update emoncms & modules with configurable log directory paths for sync/update etc. I now have a solution to this here [Flexible log\_location for emoncms · Issue #1294 · emoncms/emoncms · GitHub](https://github.com/emoncms/emoncms/issues/1294). Feel free to discuss on this issue so as not to split the discussion on systemd environment variables. ---
**Author:** ![pb66](https://community.openenergymonitor.org/user_avatar/community.openenergymonitor.org/pb66/32/27_2.png) [@pb66](https://community.openenergymonitor.org/u/pb66)\ **Post date:** [27 May 2019 12:03 UTC](https://community.openenergymonitor.org/t/emonsd-issues-created-using-new-scripts-emonhub-environment-file/10918/60 "2019-05-27T12:03:58Z")
> [@TrystanLea](#): > > Thinking more about this, I think I will go with your recommendations @pb66 re emonhub installation procedure: > > - Using /usr/share/emonhub for the emonhub.py symlink > - Using /etc/emonhub/emonhub.conf > - We’ve already agreed on emonhub.log > - Creating the logfile and folder using the service file rather than in the install scripts > > I’ve got this all working on my test SD card build and Im happy with it./ (I know the first point about emonhub.py path is still under discussion, but) In connection with your recently released fix ([Fix for full /var/log issue, causing inputs to stop updating is now available](https://community.openenergymonitor.org/t/fix-for-full-var-log-issue-causing-inputs-to-stop-updating-is-now-available/11030)) and the comments in the [above mentioned issue](https://github.com/emoncms/emoncms/issues/1294), the forth point about the emonhub service creating it’s own log folder/file would, if adopted for the emoncms services too, negate any need for entries in the rc.local for those services, and since feedwriter, emoncms\_mqtt and service-runner would all check and create /var/log/emon if required, the folder and emoncms.log would already exist before any webpage is served so /var/log/emon would not need to be created in rc.local in the interim. (as described in [/var/log still filling up - #27 by pb66](https://community.openenergymonitor.org/t/var-log-still-filling-up/10921/27)) Both the emonhub.conf and emonhub.log paths may well be configurable, but the ability to manage it’s own log folder/file is a good addition regardless, it’s more robust, especially when there’s a possibility of it being used with any form of RAM based log partition. The pending decision on paths, should not prevent the unit’s from creating their own log folder/files With regards to the postprocess,backup,sync module logs and the update logs, these can only work with service-runner active so there is no chance of them being triggered without a log in place. It’s only if emoncms is installed with no services at all that a provision to create the log folder (and file) is needed, that’s not the position on a emonSD so emoncms.log would’nt need creating in rc.local at all. [Previous page](https://community.openenergymonitor.org/t/emonsd-issues-created-using-new-scripts-emonhub-environment-file/10918.md?page=2) [Next page](https://community.openenergymonitor.org/t/emonsd-issues-created-using-new-scripts-emonhub-environment-file/10918.md?page=4)