emonSD issues created using new scripts + emonhub environment file

No, systemctl creates the folder, it’s only because I was manually writing the override file that I needed to create the folder first.

Not at all.

rather than edit on the fly at install with sed you would just create a folder and write to a blank file (if it needs editing at all) and when it comes to further updates (to the service unit not just the path) a git pull does it all, writing a new file into place is far more complex as you need to ascertain and apply the right path.

a script only needs to be written once (with the pull method), not every time the service unit is updated, a git pull is always sufficient!

Also don’t forget that whilst we are indeed accommodating the ability to install to other paths, 99% of the time the paths will all be the same, why edit at all for that 99% of the time? hardcode the service unit as /usr/local/bin/emonhub.py (or whatever it ends up as) and only “edit” if a different path is needed, that is much, much simpler than editing and writing the same exact file with the same paths on the fly a majority of the time. It doesn’t make sense to complicate the install or updating for the huge majority just to accommodate the very few that need a different path, when there is a way to only edit the minority!

For years we have struggled against the /home/pi and emonpi specific paths, now we are opening up to using more generic paths “instead” not “as well as” (backward compatibility aside), we do not need to overly complicate things by making it mandatory to edit/specify every path. That seems to be over-correcting the issue somewhat. /usr/local/bin will be present on far more installs and distros than /home/pi and that is what we are addressing here, if someone wants to then install to /some/other/less/mainstream/place then there is a mechanism to do that via the drop-in, we do not need to build the whole install and every update around the few users that might do that!

Yes, because they do not use cloning as an update method, we need to embrace the fact we do it that way, or do it another way, not half one way and half the other, that’s how we get in these positions.

If or when we move any software to a release package (debian, pip or just a download zip & install) then yes, the finalised file would be written or copied into place and not symlinked as there would obviously be either a better mechanism or no need to update frequently. I’m all for that, but we’re not there yet, whilst we are frequently cloning to make changes, symlinks are the better tool for the job.

YMMV here then as I have definitely tried it and it complained that the root folder did not exist.

I can only guess you were not running the edit command as the root user

pi@emonpi:~ $ ls -la /etc/systemd/system/emoncms_mqtt.service.d
ls: cannot access '/etc/systemd/system/emoncms_mqtt.service.d': No such file or directory
pi@emonpi:~ $ systemctl edit emoncms_mqtt
Failed to create directories for "/etc/systemd/system/emoncms_mqtt.service.d/override.conf": Permission denied
pi@emonpi:~ $ sudo systemctl edit emoncms_mqtt
# edited and saved the edit here.
pi@emonpi:~ $ ls -la /etc/systemd/system/emoncms_mqtt.service.d
total 12
drwxr-xr-x  2 root root 4096 May 21 19:44 .
drwxr-xr-x 19 root root 4096 May 21 19:44 ..
-rw-r--r--  1 root root   10 May 21 19:44 override.conf
pi@emonpi:~ $ sudo systemctl revert emoncms_mqtt
Removed /etc/systemd/system/emoncms_mqtt.service.d/override.conf.
Removed /etc/systemd/system/emoncms_mqtt.service.d.
pi@emonpi:~ $ ls -la /etc/systemd/system/emoncms_mqtt.service.d
ls: cannot access '/etc/systemd/system/emoncms_mqtt.service.d': No such file or directory

[edit] TBH it’s not relevant either way, I wasn’t suggesting anyone manually edits the service unit, it would be done via a script not via systemctl edit.

I’ve tested the install scripts with the new EmonScripts repo and it appears to have built fine with no apparent issues yet. The update process will still trigger the emonpi repo location, I need to look into that.

Yeah, it’s not much fun not being allowed to run the update script of your choice is it? :grin:

Thinking more about this, I think I will go with your recommendations @pb66 re emonhub installation procedure:

  • Using /usr/share/emonhub for the emonhub.py symlink
  • Using /etc/emonhub/emonhub.conf
  • We’ve already agreed on emonhub.log
  • Creating the logfile and folder using the service file rather than in the install scripts

I’ve got this all working on my test SD card build and Im happy with it./

In order to use the same emonhub.service on the current image and new build process I need to place emonhub.conf in /etc/emonhub on the current image. The original reason for using /home/pi/data was due to the read only partition. That said it could have easily been symlinked to /etc/emonhub/emonhub.conf.

In order to provide backwards compatibility for older images with a read only partition, I think a good solution is to 1. update emonhub.service to our desired configuration as above for the new build, 2. symlink the existing /home/pi/data/emonhub.conf to /etc/emonhub/emonhub.conf on current and older images. Would you be happy with this?

Indeed I would, I’ve had to read it multiple times to be sure I wasn’t reading it wrong, it appears we agree on every point! Are you sure you wrote what you meant to say? :smile:

I think the same can be said for any displaced files or folders during the transitional period. Adding the missing symlinks now isn’t a bad thing. I would have prefered the mysql folder to have been symlinked and the my.cnf file changes reverted to use the original paths to resolve the ProtectHome issue for example, by moving the existing emonSD closer to “normal” using symlinks closes the gap between old and new images minimising the amount of extra “backwards compatibility” needed in the new images.

1 Like

Great, glad I’ve understood your intention, I will go forward with that change then :slight_smile:

I have also done a bit more work on the update process in the EmonScripts repository, mainly cleaning it up / removing legacy update code and updating it to use your suggested emonhub install approach:
https://github.com/openenergymonitor/EmonScripts/commits/master

Actually! my bad! Didn’t read it closely enough!

The path to emonhub.py is NOT as I suggested. to use /usr/share/emonhub isn’t right, that is for Debian packaged executables, that path was set when emonhub was first written as Dave was going to package it (as he had done with emoncms at the time).

The path I suggested was /usr/local/bin/. That’s what L2R, lwrfmqtt and your wifi-check already use, and no sub-folder req’d either ie /usr/local/bin/emonhub.py.

Ok great, il change it to /usr/local/bin/emonhub.py then

1 Like

There are also a couple of things installed to /usr/local/sbin too and emonPiLCD seems to be installed to /usr/share as well. So we should try and decide on a common location for them all really (eventually)

pi@emonpi:~ $ ls -la /usr/local/bin
total 164
drwxrwsr-x  2 root staff  4096 Apr 23 11:57 .
drwxrwsr-x 10 root staff  4096 Jun 27  2018 ..
-rwxr-xr-x  1 root staff  2060 Apr 23 11:57 log2ram
-rwxr-xr-x  1 root staff 13796 Oct  4  2018 lwrfmqtt
-rwxr-xr-x  1 root staff 31988 Oct  4  2018 paho_c_pub
-rwxr-xr-x  1 root staff 27012 Oct  4  2018 paho_cs_pub
-rwxr-xr-x  1 root staff 27088 Oct  4  2018 paho_cs_sub
-rwxr-xr-x  1 root staff 27788 Oct  4  2018 paho_c_sub
-rwxr-xr-x  1 root staff 13396 Oct  4  2018 paho_c_version
-rw-r--r--  1 root staff   675 Apr 23 11:57 uninstall-log2ram.sh
lrwxrwxrwx  1 root staff    26 Oct 22  2018 wifi-check -> /home/pi/emonpi/wifi-check
pi@emonpi:~ $ ls -la /usr/local/sbin
total 8
drwxrwsr-x  2 root staff 4096 Oct 22  2018 .
drwxrwsr-x 10 root staff 4096 Jun 27  2018 ..
lrwxrwxrwx  1 root staff   46 Oct 22  2018 emonSDexpand -> /home/pi/usefulscripts/sdpart/sdpart_imagefile
lrwxrwxrwx  1 root staff   32 Jul 31  2018 wifiAP -> /home/pi/emonpi/wifiAP/wifiAP.sh
pi@emonpi:~ $ ls -la /usr/share/emon*
lrwxrwxrwx 1 root root 20 Aug  2  2018 /usr/share/emonhub -> /home/pi/emonhub/src
lrwxrwxrwx 1 root root 20 Aug 23  2018 /usr/share/emonPiLCD -> /home/pi/emonpi/lcd/

Seems that bin is for any user and sbin for superuser command line - Differences between /bin, /sbin, /usr/bin, /usr/sbin, /usr/local/bin, /usr/local/sbin - Ask Ubuntu, given that emonhub runs with user emonhub it sounds like /usr/local/bin is the correct location?

Sounds good to me!

The thing that concerns me most rather than the particular choice of directories for everything is the ease of changing them.

I believe the best-practice approach is to only use variables containing the path names everywhere. The actual path names can all be collected together and mapped to the variable names in a single place. Traditionally this would have been a list of environment variables, and that is still a viable approach. Another approach might be to store the mappings in a JSON file, or similar portable data format. The various pieces of code that need to know the paths can then extract them from the storage by whatever facilities are provided by the language they are written in. Most uses of the paths can be resolved at runtime like this. Some special cases might need to be resolved at build time by rewriting the code unit.

Failing such an approach, I suggest that best alternative is to maintain the list of all paths in human-readable form in a single place, together with a mapping to variable names or a key string. A list of all the places the paths are used is also very helpful, but it should also be possible to find all occurrences in the code by searching. There are many ways this technique can go wrong.

Yes, althought not quite there yet, we are moving in the direction of greater path configurability in the rest of the install scripts, although this emonhub.service file discussion is at the same time fixing a set of paths… Its hard to break out the paths into variables if you cant edit the service file. I dont have a strong opinion either way on this, but Im happy to adopt @pb66’s recomendations on this one.

@pb66 currently on the emonpi /usr/share/emonhub is a symlinked copy of the whole emonhub/src folder, its possible however to just symlink emonhub.py to /usr/local/bin, which works fine, any view on which way is best? I assume just the emonhub.py symlink?

Well I don’t think you need to edit them, although I believe that is possible. It’s simpler to just read the mappings using an EnvironmentFile directive in the service file.

Ah yes that makes sense, emonhub is not a single file executable (like log2ram etc) it’s a collection of files and IMO they should all (appear to) be together, I do not think we should be symlinking just the emonhub.py. I was thrown by this bit of your commit when I looked

that’s why I quickly commented that the path should NOT be /usr/share/emonhub and that the sub-folder was unnecessary, as it looked like you were just creating a folder to put the symlink in (well in fact you were!).

In actual fact whilst the sub-folder doesn’t need to be created, the path with the sub-folder level is correct. What we should be doing is symlinking the emonhub/src folder to /usr/local/bin so that the path to emonhub is /usr/local/bin/emonhub/emonhub.py and the output from

ls -la /usr/local/bin/emonhub

is a symlink to emonhub/src and the output of

ls -la /usr/local/bin/emonhub/

is a full list of all the emonhub source files. (not the repo root, but src root).

I sort of panicked a bit and made a hasty call because these changes are going into the live repo’s, I’m really not comfortable with that. It means we must get it right at every commit because we don’t know when anyone might update. The emonhub logfile is all that needed to be sorted on existing images, the path works fine on the emonSD, once we make a firm choice, implement and test the new paths for the new installer and updater scripts we can then update the old images in a controlled transition, we can’t really alter the emonSD symlink paths now as the repo is in the wrong place ie /home/pi/emonhub not /opt/emonhub (or /opt/emon/emonhub?) unless you plan on chaining symlinks which may bring it’s own issues.

Can we please try and separate “fixing immediate issues with the current image” and “developing the new installer and updater” and get it right, before rolling it out?

I dont mind too much, given that it works as a single file it seems unneeded to symlink the whole folder? Is a bit arbitrary in the end, although it does seem a bit messy… I dont mind

Well hopefully if you aren’t too bothered about it you will be ok with keeping it all together and symlinking the /src folder then. I really don’t find it even remotely arbitrary. the content of the /src folder is a single python package, we simply can not have a fraction of that package found via a path whilst the remainder isn’t.

The fact that emonhub.py is found at /usr/local/bin/emonhub/emonhub.py we cannot then say that emonhub_interfacer.py is at /opt/emonhub/emonhub/src/emonhub_interfacer.py tell me that’s not confusing! So we end up forced to interchangeable use 2 paths for emonhub.py on a single install depending on the discussion at hand. That’s the exact opposite of what we’re trying to achieve.

The whole intention of the “clone&symlink” install method is that it closely mimic’s a normal “download-zip&install” type install (bar the symlinks to a cloned repo to allow easy updates via cloning) all files in service should be accessible via a non-/opt path, this distances the cloned repo’s from the filesystem.

If emonhub was downloaded and installed conventionally (like L2R for example) the whole source would be in a single emonhub folder in /usr/local/bin not split up, that’s how it must appear even when cloned&symlinked. Especially since it’s Python, someone new to python trying to understand emonhub’s inheritance and imports will have an unnecessarily hard time trying to get a handle on what’s going on. And that should apply to anything that uses relative paths really, for example if you were to decide not to install emoncms direct to /var/www, you would symlink the whole folder, not just index.php (no idea if that would even work or not, just using it as an illustration). We should aim to make things clear and easy to understand where ever possible, I know it’s not always possible but where we can we should.

1 Like

To try and explain my reasoning, not as an argument, just thinking aloud. Personally I think I would go to /opt/emon/emonhub if I wanted to look at the source, the /usr/local/bin location from my perspective is just a link to simplify the process of running emonhub from the service file. Having the source appear in two places feels more messy than just having it in /opt/emon/emonhub. I think I understand your wider rationale about symlinking everything into debian recommended package locations but then we can also just use /opt as the recommended location for packages that are not installed via apt-get. Would it maybe be a better solution to stick with /opt and use an EnvironmentFile as @djh suggested?