emonSD issues created using new scripts + emonhub environment file

@glyn.hudson @TrystanLea Just ran an update ‘all’ from an EmonPi system created with the new script and emonhub will not restart

Ah, it is because the path to the log file is hardcoded in the emonhub.service file to the home/pi/ folder and the emonhub service itself seems to be in the wrong place.

Not got time to debug further right now. Just though you should know.

Cheers

[edit]
this as well

[edit2]
EXEC on service file needs to be

/opt/emon/emonhub/src/emonhub.py --config-file=/var/opt/emon/emonhub.conf no still not right. Need to start from scratch again.

There was some discussion about building the services files (and other config files) on the fly to account for different paths - I think this is going to have to be adopted as we transition from the SD Images to the install scripts as both data and system files are moved out of the home folder.

Alternatively there will need for a checking mechanism to either delete existing installed locations and reinstall or use existing locations (less desireable I suggest). The risk there is local modifications would be wiped. Needs more discussion and experimentation I suggest.

I’ve moved this to a new topic since it’s not related to Emocnms V10.

The new emonSD build scripts are still in development and not yet being used to build any production images. @TrystanLea has been working on these scripts, he’s away for another week, I’ll work on this with him when he’s back in the office.

Thanks for testing.

1 Like

This is where the init scripts were much simpler as they had config files to hold such paths. When i was originally looking into whether there was a similar ability with the systemd service units, the common answer seems to be “No, they do not need a config file as they ARE the config file!”.

IMO, there should be a default “usual path” included in the service unit and that default file should be symlinked to the repo so that changes to the service unit args and options etc can be done just by pulling the repo because we are not experts in systemd yet, and I’m sure we will be changing things. When someone uses a less mainstream linux distro and needs to use a different path, a drop-in “edit file” can be used to override the “usual path”, I think this will be rarely used as most images and installs are biased towards Debian, but it is a good way of providing for the more exotic distro’s (providing they are systemd!).

As much as I like using heredoc’s written into place as part of an installation, I do not think we are ready to “fit and forget” any of the OEM service unit’s yet, so holding them in the repo and symlinking, gives us an easy way to make changes, I don’t think trading that feature for a “on the fly” command line edit is worth it when 90%+ installs will use the same path and the correct way to “edit” a stock service unit is via an additional file rather than editing the unit directly (even IMO if that edit is done as part of the install).

Also worth noting, I do not think that any files or scripts should be used directly from the /opt path, that is just a convienient place to put stuff, eg emonhub.py should still be symlinked to somewhere in the PATH eg /usr/local/bin/ (like log2ram). The current emonhub path (/usr/share/emonhub/emonhub.py) was based on an expectation for emonhub to be a debian package one day, the original emonhub install script I wrote was called “dev-emonhub” as it installed the repo and symlinked to the correct debian package location’s for the development of emonhub, the content of that script was taken into the emonpi variant of emonhub’s repo. Although I think the concept is right, the paths may need tweaking inline with the linux FHS if not going to be debian packaged.

eg emonhub.conf should always be found at /etc/emonhub/emonhub.conf (even if it is symlinked somewhere else eg a data partition) it certainly should not be located in /opt/*! There should be no user editable files in /opt, that is solely where the repo’s are cloned (and left untouched!).

So the service unit should use

execStart=/usr/share/emonhub/emonhub.py --config-file /etc/emonhub/emonhub.conf --logfile /var/log/emonhub/emonhub.log

(until/unless /usr/share/ is changed to a more appropriate path)
and symlink(s) used so that the actual files can be located in the cloned repos (/opt/*) or on a (RW?) data partition to keep all the data and conf files in one location for easy backup/recovery (or to allow the rootfs to be RO?). This gives maximum flexibility and an easy way to update source files whilst also adhering to FHS recommendations too.

I have no issues with all that you say, and TBH I’m not too fussed as to how, the issue is that currently there are various paths so one way or another we need to build the scripts & service files to create consistency.

Hi @pb66, I had a look at the FHS 3.0 reference document, mostly agree.

I think though that rather than /etc/emon it should be /etc/opt/emon to be strictly correct. That fits with the statement, para 3.13.2 “Host-specific configuration files must be installed in /etc/opt. See the section on /etc for more information.”

Not sure about this bit. Same part of the ref document implies that they can as there is an option for a bin in the structure. However, I’m not fussed either way, just want to get to a consistent approach that makes sense!

I’ve tried to put a table together of some of the paths currently used/created by the install scripts as a result of another discussion on this. I think we are close but not quite there.

Script Variable & Source Value Creates sub folder Links? Comment
init.sh Defines its own $usrdir /opt/emon emonpi/ Folder created by git clone.
main.sh From config.ini $emoncms_datadir /var/opt/emon phpfina/ phpfiwa/ phptimeseries/ Main data directories
emoncms_core.sh From config.ini $emoncms_www /var/www/emoncms Symlinks from services in scripts folder Folder created by git clone
emoncms_modules.sh From config.ini $usrdir /opt/emon modules/ modules/data Modules installed here are symlinked back to the $emoncms_www/Modules folder 1. Folders created by git clone – one for each module. 2. $userdir passed to module install script. 3.Config should be in /etc/opt/emon/
From config.ini $emoncms_www /var/www/emoncms Folders created in Modules/ by git for each module.
emonhub/install.sh Input variable $usrdir /opt/emon data/ Creates data/emonhub.conf should be in /etc/opt/emon/
Hardcoded $emonhub_logfile_path /var/log/emonhub (creates if not existing) Creates service file by copying file and editing paths.
backup/install.sh Input variable $usrdir /opt/emon /data/uploads should be in /var/opt/emon/ as changes as part of normal operation.

I think the bits to be reviewed are the use of /opt/emon/ for config files (might be /etc/opt/emon) and the use of /opt/emon/ for data that will change during the normal course of operation (might be /var/opt/emon/).

@TrystanLea thoughts?

Does it need to be strictly correct? The goal here is to make things more consistent and easier to find and use, not to meet some regulation at any cost. For a user that has some linux exposure, a good guess at where the emonhub.conf might be is /etc likewise with the data files, mysql, mariadb and redis all have thier data files in /var/lib and that is traditionally where the php??? folders have resided to (until they started getting accessed directly on the data partition of the emonSD images without symlinking).

I see no advantage to adding /opt to the various paths just to conform strictly to FHS, when I (started using and) suggested /opt to install the repo’s to, it was to locate the cloned repo’s to a single point somewhere out of the mainstream FS haunts as just a place to clone and manage the repo’s rather than having them scattered across the FS or all lumped into /home/pi. The concept of symlinking the various bits into their final positions is so that it operates and appears as if installed “correctly”, by “correctly” in this context I mean as per other softwares, ie the conf is in /etc, the log is in /var/log, the data is in /var/lib and the executable is in /usr/local/bin/ (or similar!) this could of course have been done whilst the repo’s were in /home/pi too, but that location wasn’t right either, hence the move to /opt, but if the move to /opt means all the paths need to include /opt to be strictly correct then perhaps /opt is the wrong place!

Going way back to v8 emoncms and when it was being packaged for Debian, a very good case was put forward around why we should not be operating software directly from cloned repo’s. To be strictly correct we should be downloading the source (usually via a zip, but cloning can work too) and moving, coping installing things to their final locations, disconnecting the software from the source repo’s. Git is a version control system not a package manager. Whilst I understand why git shouldn’t strictly be used for updating and upgrading deployed software, I also recognise that OEM is dependent on doing so and that to change that would be a whole new level of headache, plus since the OEM project is pretty much in constant a permanent state of development, it’s not entirely “wrong” as such, we just need to accept that the way OEM is installed, including the emonSD, is “structured for development” (even if SSH is discouraged and released “stable” branches are in play). SO, in my mind, when we are cloning all the repo’s to /opt we are not installing them! The “installation” should/could consist of moving all the required stuff into place (which would then render the /opt location irrelevant) BUT since we use git as an update/upgrade tool, instead we symlink the various bits to “where they should be”, which in my mind is not relative to the fact we are (arguably) incorrectly using the cloned repo’s directly rather than moving stuff out of where ever the cloned repo might be placed (just temporarily during a conventional install) ie /opt in this case.

Long story short, I just don’t see that typing a more obscure path for every component to include /opt to highlight the source is in /opt brings anything but confusion (and extra keyboard wear) to the party when the concept of symlinking the various bits is to essentially to emulate NOT being installed in /opt.

Just to be clear here, I’m not advocating we change the way OEM installs and updates to not use cloned repo’s (I think it’s essential we use cloned repo’s to rapidly deploy fixes as the releases are rarely fully tested and bug free). But the fact we DO use cloned repo’s could be done in such a way that it isn’t set in concrete. If/when the OEM stuff reaches maturity and ends up formally packaged (for example) emonhub.conf will still be at /etc/emonhub regardless of whether it is Debian packaged, cloned and symlinked or downloaded and installed in a conventional manner. All the paths needn’t have /opt in them and if they do not contain /opt it facilitates a move to a package/conventional installer by just replacing the symlinking with move/copy/install actions and the existing paths will remain valid, no further upheaval/reeducation of new paths then without /opt is required.

In operation /opt path would only be used “behind the scenes” for cloning&symlinking during install or upgrades etc as it is a convenient place to group the cloned repo’s. All standard operation and path’s in configuration files etc etc should use “non-opt” paths, there is no reason for a regular user to even be aware of /opt as everything will be accessible via an alternative “non-opt” path, either due to symlinks to the source or because user data and config files are not located in /opt at all. /opt/emon should IMO be just clean cloned repo’s, no edits or user files at all!!

Also, I think this discussion should be part of the emonSD next steps: filesystem & logrotate discussion rather than here (or at least linked?), these are development considerations rather than “issues created by new scripts” (strictly speaking).

As examples of what I say above, look at log2ram/install.sh and dev-emonhub/install, not because I think either is perfect, purely because they are examples I know demonstrate the 2 approaches. Log2ram attempts to install conventionally by downloading the zip from github (via curl) and then installing each component to it’s final location. You could delete the L2R repo after it’s installed with no problems arising. In contrast to that, the dev-emonhub install script tries to emulate a packaged install by using symlinks to the cloned repo so that quick and easy changes can be made by pushing and pulling the repo to github, for the purpose of development.

I think L2R is possibly “more correct” than the emonhub installer but the way the emonhub installer works is more aligned with the rest of the OEM installs and update routines etc. I will (when I get a moment) be changing my fork of L2R to use symlinks rather than install (even if only temporarily) as I think that will assist developing L2R further, eg when I changed the rsync command recently, that could easily be pulled in via git if “cloned&symlinked”, but as it stands, L2R needs to be uninstalled and reinstalled to get that minor edit. I might however do it in such a way that passing a --dev switch to the installer results in a symlinked clone and the original install method is also retained. This “switch” clearly demonstrates the difference between a “conventional” and a “development” install and from that we can see that all OEM stuff is generally installed using a “development” method. The /opt folder is nothing more than a convenient place to clone the repo’s (IMO!).

There was some discussion about building the services files (and other config files) on the fly to account for different paths - I think this is going to have to be adopted as we transition from the SD Images to the install scripts as both data and system files are moved out of the home folder.

I have implemented this approach in the install scripts, e.g:

Emonhub: emonhub/install.sh at emon-pi · openenergymonitor/emonhub · GitHub
EmonPiLCD service: https://github.com/openenergymonitor/emonpi/blob/master/install/emonpilcd.sh#L39

service-runner, emoncms_mqtt and feedwriter are all ran from /var/www/emoncms/scripts which is currently a fixed path and so there are no related modifications yet: emoncms/scripts/services/service-runner/service-runner.service at master · emoncms/emoncms · GitHub

Im happy with this approach. We could change the default to reflect our choosen standard install path of course but at present we need to support both /home/pi/emonhub and /opt/emon/emonhub.

Rather than symlinking the proposal here is to overwrite on update, both ways allow the service file to be updated easily, but the overwrite approach allows for a modification step.

The /opt as clone path + symlinks to relevant locations, is an interesting point of view @pb66 I will have to give it some thought. My immediate reaction is that is seems like an over-complication. I can see the logic for /etc/emonhub/emonhub.conf Im not 100% sure about /usr/share/emonhub/emonhub.py.

What other paths do we need to consider? Currently some of the emoncms modules are symlinked from /opt/emon/modules to /var/www/emoncms/Modules, emoncms itself is installed directly in /var/www/emoncms (we have discussed changing the folder structure here to symlink the www part to /var/www, backwards compatibility would beed to be managed carefully).

Would emoncms_mqtt service be symlinked to /usr/share/emoncms/emoncms_mqtt.php?

@TrystanLea, can we develop these in a separate repo please? It will make amending and testing far easier if it is split out from the (live) emonpi repo?

Sure, I will look into that next. Otherwise it looks like the remaining items on the todo list for the emonSD install script first release are:

  • finalise emoncms log locations i.e: /var/log/emon/emoncms/sync.log
  • update emoncms & modules with configurable log directory paths for sync/update etc.
  • finalise logrotate config, my latest post on this is here: emonSD next steps: filesystem & logrotate - #144 by TrystanLea
  • Review @pb66’s points above, re installation location and symlinks
1 Like

I’m afraid I disagree with both points. Firstly the writing into place is NOT easily updated in comparison to the symlink method as it requires further steps, be it cp, mv or sed etc, with the symlink method, once the changes are pulled in (git pull is required for either method) the changes are complete there are no “steps” at all.

Secondly, overwriting using sed (during updates) is potentially an ever increasing string of updates, if we set direction x to abc, then decide it should be def, any update script will need to sed “abc” to “def”, but then if it needs to be changed to xyz, maybe not all setups have the first update yet, so moving forward you need to either keep adding edits “abc” to “def” then also “def” to “xyz” or add some form of “abc or def” to “xyz”, this method could get real messy and cause issues, alternately you might rewrite the whole service unit, would that be on every update like the emonpi FW or just when there are changes, how would you test for changes when the content is potentially different for each setup (paths etc).

Plus, the “correct” way to “edit” systemd unit files is to add an edit file, it frequently comes up that we should not be editing service units. So I would understand that to imply that all (say) emoncms_mqtt.service units to be identical across all installs and then any local changes to be in a separate edit file, you could argue that when a service unit source code is modified (ie global upgrade and improvements to the overall service unit) that is not an “edit” and quite rightly that revised unit should replace the current unit (not be an additional edit file). By using the symlink method, changes made at install time to path’s would stick beyond any replaced service unit as the changes are in an external edit file, the new incoming service file doesn’t need customising for each install (eg paths).

Quite the opposite, it really simplifies things, the concept might be tough to get your head around at first, but it is simpler to install, simpler to update and simpler to navigate.

I assumed that to be non-negotiable, or at least a hard sell so I have avoided commenting about the core emoncms folder. I already symlink that into place myself. The main advantage to this (aside from all repos in one place) is for development purposes, I can have several emoncms repo folders and several vhosts and chose/change which versions of emoncms are served by each vhost purely through symlinks, to do this I needed to make a minor edit so that emoncms(s) look for settings.php in it’s parent folder (usually the vhost root folder), I don’t think there’s any demand for this ability, I’m just outlining the flexibility and the fact I already do it that way, but I did not expect the location to change on the emonSD.

No, I don’t think so. emoncms_mqtt.php is a emoncms helper script, it has no purpose if emoncms isn’t running. emonhub can be installed alone, besides, /usr/share was the proposed location for when emonhub was to be Debian packaged, I think /usr/local/bin/ might be more suitable for emonhub. It seems you already adopt this path and symlinking method to the wifi-check

pi@emonpi:~ $ ls -la /usr/local/bin/
total 164
drwxrwsr-x  2 root staff  4096 Apr 23 11:57 .
drwxrwsr-x 10 root staff  4096 Jun 27  2018 ..
-rwxr-xr-x  1 root staff  2060 Apr 23 11:57 log2ram
-rwxr-xr-x  1 root staff 13796 Oct  4  2018 lwrfmqtt
-rwxr-xr-x  1 root staff 31988 Oct  4  2018 paho_c_pub
-rwxr-xr-x  1 root staff 27012 Oct  4  2018 paho_cs_pub
-rwxr-xr-x  1 root staff 27088 Oct  4  2018 paho_cs_sub
-rwxr-xr-x  1 root staff 27788 Oct  4  2018 paho_c_sub
-rwxr-xr-x  1 root staff 13396 Oct  4  2018 paho_c_version
-rw-r--r--  1 root staff   675 Apr 23 11:57 uninstall-log2ram.sh
lrwxrwxrwx  1 root staff    26 Oct 22  2018 wifi-check -> /home/pi/emonpi/wifi-check

I’m certainly open to ideas regards the emonhub.py symlink (location), if it’s going to be the single oddball then perhaps for consistency it should be called direct in /opt? Where do you plan on installing L2R? Are you going to use the existing L2R “download zip and install” or are you interested in the “clone&symlink” mods I proposed to make? I see emonhub more akin to log2ram than to the other emon services. They are both Python, both run independent of emoncms and are essentially the only 2 things you need to install to a Raspbian image make a “rock solid gateway” style emonBase.

Agreed, can we try and keep some distinction between the new installer, the new updater, fixes that need rolling out to all (eg sorting the /var/log filling) and improvements that maybe rolled out to previous images (eg L2R), the lines seem to get a bit blurred. and (for example) logrotation not being looked at on the existing images because L2R is being discussed and/or fixes to problems that won’t exist in the new installs being included in the new updater.

@TrystanLea can you also look at Permanent "Raspberry Pi Booting..." after failed emonPi Update

Maybe with the new install method looming, we should consider starting a fresh, potentially (but not deliberately) breaking backward-compatibility to get the best solution moving forward, ie fix it once and for all, rather than being restricted by whats happened in the past. In contrast to your 10-20 year sdcard lifespan, Glyn seems to think users should be starting a new image and new sdcard at a much faster frequency (from github Mariadb will not start after a dist-upgrade · Issue #1202 · emoncms/emoncms · GitHub)

if we were to use upgrade in place of dist-upgrade it would limit the operational life of an emonPi (image) to whatever the LTS is for security updates and we might all have still been running wheezy! Even stretch LTS expires in 2022, if you do not use dist-upgrade, it will remain “stretch” for ever, even when support expires.

I would recommend that in 2022 the user would be best to back up their data and move to a fresh SD card with the latest emonSD card build with a vanilla build of Debian Buster.

I’m not convinced that users that buy an emonSD at the end of 2021 would be happy with that approach but if the intended lifespan is going to be shorter and changing base image regularly is the recommended route to keeping an image up to date, then supporting older images becomes a little less important. Currently the “safe-update list” is just 2 significantly different images Jessie RO and Stretch RW (the latter buildguide wasn’t documented) that span less than 2years (Oct2017 ->)

I’m just pointing out that currently the “I recommend downloading the latest image and installing to a new sdcard” line is used so frequently for relatively minor issues, that backwards compatibility is clearly not as big a concern as it should be. Whilst we are currently in a “just start again” era, lets at least mull over the option of totally starting again now, once and for all rather than perhaps rushing out another short lived solution.

I’ve created the dedicated repo for the installer and updater, I have not yet tested it for building the image: GitHub - openenergymonitor/EmonScripts: Emoncms Stack Installation and Update scripts

I will consider and give your other points some thought @pb66, thanks.

2 Likes

Unless you simply overwrite on each occasion the script is run (with paths created on the fly). Requires the running of a script rather than a manual git pull (depending on what has changed). For emon service files that is OK, for other packages the edit mechanism is better.

Pretty sure you cannot systemctl edit symlinked files.

See below. The emonhub.service unit is a symlink, I change the config file path by creating the edit folder and file on the fly and then restart emonhub to use the new config path, then I remove the “edit” again using systemctl revert. The status outputs show the different conf paths actively in use and also that the correctly formed drop-in is being used, as does systemctl cat.

The quirk of overwriting ExecStart requires it to be cleared with an empty ExecStart= before being set again, this is true of all service units (except Type=oneshot) and nothing to do with the symlink.

pi@emonpi:~ $ ls -la /lib/systemd/system/emonhub.service
lrwxrwxrwx 1 root root 40 Apr 23 10:20 /lib/systemd/system/emonhub.service -> /home/pi/emonhub/service/emonhub.service
pi@emonpi:~ $ cat /lib/systemd/system/emonhub.service
[Unit]
Description=emonHub service description
DefaultDependencies=no
Before=shutdown.target
Conflicts=shutdown.target
Requires=local-fs.target
After=sysinit.target syslog.target local-fs.target

[Service]
User=emonhub
#PIDFile=/var/run/emonhub.pid
ExecStart=/usr/share/emonhub/emonhub.py --config-file=/home/pi/data/emonhub.conf --logfile=/var/log/emonhub/emonhub.log
PermissionsStartOnly=true
ExecStartPre=/bin/mkdir -p /var/log/emonhub/
ExecStartPre=/bin/chgrp -R emonhub /var/log/emonhub/
ExecStartPre=/bin/chmod -R 775 /var/log/emonhub/
Type=simple
Restart=always

[Install]
WantedBy=multi-user.target
pi@emonpi:~ $ sudo systemctl status emonhub
â emonhub.service - emonHub service description
   Loaded: loaded (/home/pi/emonhub/service/emonhub.service; enabled; vendor preset: enabled)
   Active: active (running) since Tue 2019-05-21 14:03:51 UTC; 2min 17s ago
 Main PID: 3849 (python)
   CGroup: /system.slice/emonhub.service
           ââ3849 python /usr/share/emonhub/emonhub.py --config-file=/etc/emonhub/emonhub.conf --logfile=/var/log/emonhub/emonhub.log

May 21 14:03:51 emonpi systemd[1]: Starting emonHub service description...
May 21 14:03:51 emonpi systemd[1]: Started emonHub service description.
pi@emonpi:~ $ sudo mkdir -p /etc/systemd/system/emonhub.service.d/
pi@emonpi:~ $ printf "[Service]\nExecStart=\nExecStart=/usr/share/emonhub/emonhub.py --config-file=/etc/emonhub/emonhub.conf --logfile=/var/log/emonhub/emonhub.log" | sudo tee /etc/systemd/system/emonhub.service.d/override.conf > /dev/null
pi@emonpi:~ $ sudo systemctl daemon-reload
pi@emonpi:~ $ sudo systemctl restart emonhub
pi@emonpi:~ $ sudo systemctl status emonhub
â emonhub.service - emonHub service description
   Loaded: loaded (/home/pi/emonhub/service/emonhub.service; enabled; vendor preset: enabled)
  Drop-In: /etc/systemd/system/emonhub.service.d
           ââoverride.conf
   Active: active (running) since Tue 2019-05-21 14:13:41 UTC; 3s ago
  Process: 4150 ExecStartPre=/bin/chmod -R 775 /var/log/emonhub/ (code=exited, status=0/SUCCESS)
  Process: 4147 ExecStartPre=/bin/chgrp -R emonhub /var/log/emonhub/ (code=exited, status=0/SUCCESS)
  Process: 4144 ExecStartPre=/bin/mkdir -p /var/log/emonhub/ (code=exited, status=0/SUCCESS)
 Main PID: 4153 (python)
   CGroup: /system.slice/emonhub.service
           ââ4153 python /usr/share/emonhub/emonhub.py --config-file=/etc/emonhub/emonhub.conf --logfile=/var/log/emonhub/emonhub.log

May 21 14:13:41 emonpi systemd[1]: Starting emonHub service description...
May 21 14:13:41 emonpi systemd[1]: Started emonHub service description.
pi@emonpi:~ $ sudo systemctl cat emonhub
# /home/pi/emonhub/service/emonhub.service
[Unit]
Description=emonHub service description
DefaultDependencies=no
Before=shutdown.target
Conflicts=shutdown.target
Requires=local-fs.target
After=sysinit.target syslog.target local-fs.target

[Service]
User=emonhub
#PIDFile=/var/run/emonhub.pid
ExecStart=/usr/share/emonhub/emonhub.py --config-file=/home/pi/data/emonhub.conf --logfile=/var/log/emonhub/emonhub.log
PermissionsStartOnly=true
ExecStartPre=/bin/mkdir -p /var/log/emonhub/
ExecStartPre=/bin/chgrp -R emonhub /var/log/emonhub/
ExecStartPre=/bin/chmod -R 775 /var/log/emonhub/
Type=simple
Restart=always

[Install]
WantedBy=multi-user.target

# /etc/systemd/system/emonhub.service.d/override.conf
[Service]
ExecStart=
ExecStart=/usr/share/emonhub/emonhub.py --config-file=/etc/emonhub/emonhub.conf --logfile=/var/log/emonhub/emonhub.log
pi@emonpi:~ $ sudo systemctl revert emonhub
Removed /etc/systemd/system/emonhub.service.d/override.conf.
Removed /etc/systemd/system/emonhub.service.d.
pi@emonpi:~ $ sudo systemctl daemon-reload
pi@emonpi:~ $ sudo systemctl status emonhub
â emonhub.service - emonHub service description
   Loaded: loaded (/home/pi/emonhub/service/emonhub.service; enabled; vendor preset: enabled)
   Active: active (running) since Tue 2019-05-21 14:13:41 UTC; 1min 5s ago
 Main PID: 4153 (python)
   CGroup: /system.slice/emonhub.service
           ââ4153 python /usr/share/emonhub/emonhub.py --config-file=/etc/emonhub/emonhub.conf --logfile=/var/log/emonhub/emonhub.log

May 21 14:13:41 emonpi systemd[1]: Starting emonHub service description...
May 21 14:13:41 emonpi systemd[1]: Started emonHub service description.
pi@emonpi:~ $ sudo systemctl restart emonhub
pi@emonpi:~ $ sudo systemctl status emonhub
â emonhub.service - emonHub service description
   Loaded: loaded (/home/pi/emonhub/service/emonhub.service; enabled; vendor preset: enabled)
   Active: active (running) since Tue 2019-05-21 14:14:56 UTC; 2s ago
  Process: 4245 ExecStartPre=/bin/chmod -R 775 /var/log/emonhub/ (code=exited, status=0/SUCCESS)
  Process: 4242 ExecStartPre=/bin/chgrp -R emonhub /var/log/emonhub/ (code=exited, status=0/SUCCESS)
  Process: 4239 ExecStartPre=/bin/mkdir -p /var/log/emonhub/ (code=exited, status=0/SUCCESS)
 Main PID: 4248 (python)
   CGroup: /system.slice/emonhub.service
           ââ4248 python /usr/share/emonhub/emonhub.py --config-file=/home/pi/data/emonhub.conf --logfile=/var/log/emonhub/emonhub.log

May 21 14:14:56 emonpi systemd[1]: Starting emonHub service description...
May 21 14:14:56 emonpi systemd[1]: Started emonHub service description.
pi@emonpi:~ $

systemctl edit specifically, also works fine, i just did it this way as non-ineractive is what we would need and using systemctl edit (and/or nano) isn’t as easily demonstrated.

Yes, you need to create the additional folder for it work. I meant you cannot just use systemctl edit.

Can you create the dropin folder under /lib/? never tried personally.

I still think this is overly complex. If you are going to script to do that, you might as well script to overwrite a service file generated on the fly. Amounts to the same thing - a script has to be run - a git pull is not sufficient.

Also note that major packages do not do it this way. They create a file directly in the folder rather than symlink.

YMMV as always :smile:.

No, systemctl creates the folder, it’s only because I was manually writing the override file that I needed to create the folder first.

Not at all.

rather than edit on the fly at install with sed you would just create a folder and write to a blank file (if it needs editing at all) and when it comes to further updates (to the service unit not just the path) a git pull does it all, writing a new file into place is far more complex as you need to ascertain and apply the right path.

a script only needs to be written once (with the pull method), not every time the service unit is updated, a git pull is always sufficient!

Also don’t forget that whilst we are indeed accommodating the ability to install to other paths, 99% of the time the paths will all be the same, why edit at all for that 99% of the time? hardcode the service unit as /usr/local/bin/emonhub.py (or whatever it ends up as) and only “edit” if a different path is needed, that is much, much simpler than editing and writing the same exact file with the same paths on the fly a majority of the time. It doesn’t make sense to complicate the install or updating for the huge majority just to accommodate the very few that need a different path, when there is a way to only edit the minority!

For years we have struggled against the /home/pi and emonpi specific paths, now we are opening up to using more generic paths “instead” not “as well as” (backward compatibility aside), we do not need to overly complicate things by making it mandatory to edit/specify every path. That seems to be over-correcting the issue somewhat. /usr/local/bin will be present on far more installs and distros than /home/pi and that is what we are addressing here, if someone wants to then install to /some/other/less/mainstream/place then there is a mechanism to do that via the drop-in, we do not need to build the whole install and every update around the few users that might do that!

Yes, because they do not use cloning as an update method, we need to embrace the fact we do it that way, or do it another way, not half one way and half the other, that’s how we get in these positions.

If or when we move any software to a release package (debian, pip or just a download zip & install) then yes, the finalised file would be written or copied into place and not symlinked as there would obviously be either a better mechanism or no need to update frequently. I’m all for that, but we’re not there yet, whilst we are frequently cloning to make changes, symlinks are the better tool for the job.

YMMV here then as I have definitely tried it and it complained that the root folder did not exist.

I can only guess you were not running the edit command as the root user

pi@emonpi:~ $ ls -la /etc/systemd/system/emoncms_mqtt.service.d
ls: cannot access '/etc/systemd/system/emoncms_mqtt.service.d': No such file or directory
pi@emonpi:~ $ systemctl edit emoncms_mqtt
Failed to create directories for "/etc/systemd/system/emoncms_mqtt.service.d/override.conf": Permission denied
pi@emonpi:~ $ sudo systemctl edit emoncms_mqtt
# edited and saved the edit here.
pi@emonpi:~ $ ls -la /etc/systemd/system/emoncms_mqtt.service.d
total 12
drwxr-xr-x  2 root root 4096 May 21 19:44 .
drwxr-xr-x 19 root root 4096 May 21 19:44 ..
-rw-r--r--  1 root root   10 May 21 19:44 override.conf
pi@emonpi:~ $ sudo systemctl revert emoncms_mqtt
Removed /etc/systemd/system/emoncms_mqtt.service.d/override.conf.
Removed /etc/systemd/system/emoncms_mqtt.service.d.
pi@emonpi:~ $ ls -la /etc/systemd/system/emoncms_mqtt.service.d
ls: cannot access '/etc/systemd/system/emoncms_mqtt.service.d': No such file or directory

[edit] TBH it’s not relevant either way, I wasn’t suggesting anyone manually edits the service unit, it would be done via a script not via systemctl edit.

I’ve tested the install scripts with the new EmonScripts repo and it appears to have built fine with no apparent issues yet. The update process will still trigger the emonpi repo location, I need to look into that.

Yeah, it’s not much fun not being allowed to run the update script of your choice is it? :grin: