emonSD issues created using new scripts + emonhub environment file

Thinking more about this, I think I will go with your recommendations @pb66 re emonhub installation procedure:

  • Using /usr/share/emonhub for the emonhub.py symlink
  • Using /etc/emonhub/emonhub.conf
  • We’ve already agreed on emonhub.log
  • Creating the logfile and folder using the service file rather than in the install scripts

I’ve got this all working on my test SD card build and Im happy with it./

In order to use the same emonhub.service on the current image and new build process I need to place emonhub.conf in /etc/emonhub on the current image. The original reason for using /home/pi/data was due to the read only partition. That said it could have easily been symlinked to /etc/emonhub/emonhub.conf.

In order to provide backwards compatibility for older images with a read only partition, I think a good solution is to 1. update emonhub.service to our desired configuration as above for the new build, 2. symlink the existing /home/pi/data/emonhub.conf to /etc/emonhub/emonhub.conf on current and older images. Would you be happy with this?

Indeed I would, I’ve had to read it multiple times to be sure I wasn’t reading it wrong, it appears we agree on every point! Are you sure you wrote what you meant to say? :smile:

I think the same can be said for any displaced files or folders during the transitional period. Adding the missing symlinks now isn’t a bad thing. I would have prefered the mysql folder to have been symlinked and the my.cnf file changes reverted to use the original paths to resolve the ProtectHome issue for example, by moving the existing emonSD closer to “normal” using symlinks closes the gap between old and new images minimising the amount of extra “backwards compatibility” needed in the new images.

1 Like

Great, glad I’ve understood your intention, I will go forward with that change then :slight_smile:

I have also done a bit more work on the update process in the EmonScripts repository, mainly cleaning it up / removing legacy update code and updating it to use your suggested emonhub install approach:
https://github.com/openenergymonitor/EmonScripts/commits/master

Actually! my bad! Didn’t read it closely enough!

The path to emonhub.py is NOT as I suggested. to use /usr/share/emonhub isn’t right, that is for Debian packaged executables, that path was set when emonhub was first written as Dave was going to package it (as he had done with emoncms at the time).

The path I suggested was /usr/local/bin/. That’s what L2R, lwrfmqtt and your wifi-check already use, and no sub-folder req’d either ie /usr/local/bin/emonhub.py.

Ok great, il change it to /usr/local/bin/emonhub.py then

1 Like

There are also a couple of things installed to /usr/local/sbin too and emonPiLCD seems to be installed to /usr/share as well. So we should try and decide on a common location for them all really (eventually)

pi@emonpi:~ $ ls -la /usr/local/bin
total 164
drwxrwsr-x  2 root staff  4096 Apr 23 11:57 .
drwxrwsr-x 10 root staff  4096 Jun 27  2018 ..
-rwxr-xr-x  1 root staff  2060 Apr 23 11:57 log2ram
-rwxr-xr-x  1 root staff 13796 Oct  4  2018 lwrfmqtt
-rwxr-xr-x  1 root staff 31988 Oct  4  2018 paho_c_pub
-rwxr-xr-x  1 root staff 27012 Oct  4  2018 paho_cs_pub
-rwxr-xr-x  1 root staff 27088 Oct  4  2018 paho_cs_sub
-rwxr-xr-x  1 root staff 27788 Oct  4  2018 paho_c_sub
-rwxr-xr-x  1 root staff 13396 Oct  4  2018 paho_c_version
-rw-r--r--  1 root staff   675 Apr 23 11:57 uninstall-log2ram.sh
lrwxrwxrwx  1 root staff    26 Oct 22  2018 wifi-check -> /home/pi/emonpi/wifi-check
pi@emonpi:~ $ ls -la /usr/local/sbin
total 8
drwxrwsr-x  2 root staff 4096 Oct 22  2018 .
drwxrwsr-x 10 root staff 4096 Jun 27  2018 ..
lrwxrwxrwx  1 root staff   46 Oct 22  2018 emonSDexpand -> /home/pi/usefulscripts/sdpart/sdpart_imagefile
lrwxrwxrwx  1 root staff   32 Jul 31  2018 wifiAP -> /home/pi/emonpi/wifiAP/wifiAP.sh
pi@emonpi:~ $ ls -la /usr/share/emon*
lrwxrwxrwx 1 root root 20 Aug  2  2018 /usr/share/emonhub -> /home/pi/emonhub/src
lrwxrwxrwx 1 root root 20 Aug 23  2018 /usr/share/emonPiLCD -> /home/pi/emonpi/lcd/

Seems that bin is for any user and sbin for superuser command line - Differences between /bin, /sbin, /usr/bin, /usr/sbin, /usr/local/bin, /usr/local/sbin - Ask Ubuntu, given that emonhub runs with user emonhub it sounds like /usr/local/bin is the correct location?

Sounds good to me!

The thing that concerns me most rather than the particular choice of directories for everything is the ease of changing them.

I believe the best-practice approach is to only use variables containing the path names everywhere. The actual path names can all be collected together and mapped to the variable names in a single place. Traditionally this would have been a list of environment variables, and that is still a viable approach. Another approach might be to store the mappings in a JSON file, or similar portable data format. The various pieces of code that need to know the paths can then extract them from the storage by whatever facilities are provided by the language they are written in. Most uses of the paths can be resolved at runtime like this. Some special cases might need to be resolved at build time by rewriting the code unit.

Failing such an approach, I suggest that best alternative is to maintain the list of all paths in human-readable form in a single place, together with a mapping to variable names or a key string. A list of all the places the paths are used is also very helpful, but it should also be possible to find all occurrences in the code by searching. There are many ways this technique can go wrong.

Yes, althought not quite there yet, we are moving in the direction of greater path configurability in the rest of the install scripts, although this emonhub.service file discussion is at the same time fixing a set of paths… Its hard to break out the paths into variables if you cant edit the service file. I dont have a strong opinion either way on this, but Im happy to adopt @pb66’s recomendations on this one.

@pb66 currently on the emonpi /usr/share/emonhub is a symlinked copy of the whole emonhub/src folder, its possible however to just symlink emonhub.py to /usr/local/bin, which works fine, any view on which way is best? I assume just the emonhub.py symlink?

Well I don’t think you need to edit them, although I believe that is possible. It’s simpler to just read the mappings using an EnvironmentFile directive in the service file.

Ah yes that makes sense, emonhub is not a single file executable (like log2ram etc) it’s a collection of files and IMO they should all (appear to) be together, I do not think we should be symlinking just the emonhub.py. I was thrown by this bit of your commit when I looked

that’s why I quickly commented that the path should NOT be /usr/share/emonhub and that the sub-folder was unnecessary, as it looked like you were just creating a folder to put the symlink in (well in fact you were!).

In actual fact whilst the sub-folder doesn’t need to be created, the path with the sub-folder level is correct. What we should be doing is symlinking the emonhub/src folder to /usr/local/bin so that the path to emonhub is /usr/local/bin/emonhub/emonhub.py and the output from

ls -la /usr/local/bin/emonhub

is a symlink to emonhub/src and the output of

ls -la /usr/local/bin/emonhub/

is a full list of all the emonhub source files. (not the repo root, but src root).

I sort of panicked a bit and made a hasty call because these changes are going into the live repo’s, I’m really not comfortable with that. It means we must get it right at every commit because we don’t know when anyone might update. The emonhub logfile is all that needed to be sorted on existing images, the path works fine on the emonSD, once we make a firm choice, implement and test the new paths for the new installer and updater scripts we can then update the old images in a controlled transition, we can’t really alter the emonSD symlink paths now as the repo is in the wrong place ie /home/pi/emonhub not /opt/emonhub (or /opt/emon/emonhub?) unless you plan on chaining symlinks which may bring it’s own issues.

Can we please try and separate “fixing immediate issues with the current image” and “developing the new installer and updater” and get it right, before rolling it out?

I dont mind too much, given that it works as a single file it seems unneeded to symlink the whole folder? Is a bit arbitrary in the end, although it does seem a bit messy… I dont mind

Well hopefully if you aren’t too bothered about it you will be ok with keeping it all together and symlinking the /src folder then. I really don’t find it even remotely arbitrary. the content of the /src folder is a single python package, we simply can not have a fraction of that package found via a path whilst the remainder isn’t.

The fact that emonhub.py is found at /usr/local/bin/emonhub/emonhub.py we cannot then say that emonhub_interfacer.py is at /opt/emonhub/emonhub/src/emonhub_interfacer.py tell me that’s not confusing! So we end up forced to interchangeable use 2 paths for emonhub.py on a single install depending on the discussion at hand. That’s the exact opposite of what we’re trying to achieve.

The whole intention of the “clone&symlink” install method is that it closely mimic’s a normal “download-zip&install” type install (bar the symlinks to a cloned repo to allow easy updates via cloning) all files in service should be accessible via a non-/opt path, this distances the cloned repo’s from the filesystem.

If emonhub was downloaded and installed conventionally (like L2R for example) the whole source would be in a single emonhub folder in /usr/local/bin not split up, that’s how it must appear even when cloned&symlinked. Especially since it’s Python, someone new to python trying to understand emonhub’s inheritance and imports will have an unnecessarily hard time trying to get a handle on what’s going on. And that should apply to anything that uses relative paths really, for example if you were to decide not to install emoncms direct to /var/www, you would symlink the whole folder, not just index.php (no idea if that would even work or not, just using it as an illustration). We should aim to make things clear and easy to understand where ever possible, I know it’s not always possible but where we can we should.

1 Like

To try and explain my reasoning, not as an argument, just thinking aloud. Personally I think I would go to /opt/emon/emonhub if I wanted to look at the source, the /usr/local/bin location from my perspective is just a link to simplify the process of running emonhub from the service file. Having the source appear in two places feels more messy than just having it in /opt/emon/emonhub. I think I understand your wider rationale about symlinking everything into debian recommended package locations but then we can also just use /opt as the recommended location for packages that are not installed via apt-get. Would it maybe be a better solution to stick with /opt and use an EnvironmentFile as @djh suggested?

I’m not sure how adding another location and a symlink “simplifies” the service file, surely just call the emonhub.py direct from the service unit if that is your sole aim, essentially all you will be doing is moving the content of /home/pi to /opt. My reasons for doing the “clone&symlink” is that conf files are found where conf files live and the executables or binaries are found where the executables/binaries live etc etc tec.

I see no good reason to symlink just the executable that a service unit needs to find, the service unit path is dictated by us it can literally be anywhere, the reason for putting the executables in /usr/local/bin is so that users can find them where they should be. The fact we clone them to a repo in /opt is just a easy backdoor way to keeping things updated, “our dirty little secret” if you like, yes, you and I would look to /opt for the source code as we are developers and the “clone&symlink” method is primarily aimed as providing an easy way to develop, the fact we still use that version control system as a package manager after deployment isn’t something the average user will be aware of or even need to know if well implemented. And yet any budding developer/programmer will recognise exactly what is going on once they see the symlinks.

These are 2 separate things IMO, no we shouldn’t necessarily “stick to using /opt”, regardless of whether we use EnvironmentFile, in fact if you stick to using /opt, there is no need to use EnvironmentFile as the path is never likely to change, it’s no different to using /home/pi where every setup does have a user pi’s homedir to install to.

Using EnvironmentFile is not entirely a bad idea IMO, but it is a very “init.d” way of doing things, it’s how the emonhub init.d service has worked for years with the default paths in /etc/default/emonhub. The “systemd” way of doing things is to use edit files. But which ever way you get the path into the service unit, it is only necessary if you want to be able to vary the path, if all installs are going to be in /opt why edit the unit at all?

I would rather see the whole src folder symlinked to /usr/local/bin and that path used in the service unit as that will serve 99% or more users and be a FHS compliant’ish use of cloned repo’s. Only on more exotic installs that don’t use FHS will that path need to be overridden by a drop-in file, it seems pointless to add the confusion of mandatory edits, environment files or non-fhs paths just to serve those very few users when there is a perfectly good other way. Remember /usr/local is going to be present on far more installs than /home/pi is, the issues with installing to /home/pi are not being carried forward to /usr/local.

If we can’t do it that way, then yes I guess the next best thing is just use /opt directly and forget about /usr/local symlinks, this is marginally better than “all in /home/pi”, but it is pretty much the same thing different location. (But it is better than splitting the src folder, that’s just a no no!)

I can’t see why symlinking the src folder has caused such a reaction. Does it matter if all the other files from the src folder can be found at the same place (as expected)? You were happy to create a folder just to sit the emonhub.py symlink inside, but symlinking the folder rather than using mkdir, so that all the src content is kept together is apparently a problem, the path to emonhub.py would be exactly the same!

It’s clear this is not a decision that should be made off the cuff so again I suggest we hold off putting any changes into live repo’s.

Strangely enough a user has hit the problem of hardcoded paths here How to set MQTT server / what are expected mqtt data? - #9 by notme

. . . and if that path isn’t generic enough, perhaps we should look closer at that. The less generic, the more likely the need to edit across different distro’s. We are trying to find a common path for the majority of distro’s so that (for example) the emonhub.py file IS found in the same place regardless of distro where possible, just because we can make it more flexible and user configurable, we don’t want to encourage installing it in a wide variety of places “just because it’s possible”, that’s at the opposite reach of wrong to installing absolutely everything in the same folder.

Yeah I saw that and decided not to wade in as you seemed to have it covered, but yes the /html vs not-/html folder is on my radar too, the symlinking there is just to make it accessible in both locations, when in fact it should be in neither! So perhaps when the new installer comes out there will be 3 (or more?) incestuously linked folders?

This dates back to before apache decided that source files should not be installed directly into /var/www/ for security reasons and added a html folder to put their demo index.html page in. Somehow emoncms ended up moving and/or being linked into the html folder because the 000-default vhost is used rather than writing a proper emoncms vhost, thus it couldn’t find /var/www/emoncms post-html folder mod. The emoncms source is not directly in the /var/www folder, it’s in a emoncms folder, so it doesn’t need to be in the html folder (plus it’s not html, it’s php!), but now of course we have to support both /var/www/html/emoncms and /var/www/emoncms as both have been used interchangeably for some time now.

Actually, as @borpin as just pointed out that discussion about emoncms location. It has reminded me that emoncms is actually already installed to one place and served via a symlink from another place.

In theory you should be able to move /var/www/emoncms to /opt/emon/emoncms and edit the symlink in /var/www/html and everything work as it does now! Unless both the paths are being used interchangeably, but even then another link from /var/www/ would fix that (2 links pointing to /opt/emon/emoncms, in parallel, not daisy-chained)

However! how it works now (ie from /var/www/html/emoncms) is wrong, it should be served from /var/www/emoncms by a dedicated emoncms.conf vhost, not via the default 000-default.conf vhost and some edits to apache.conf. The edits made to apache.conf are actually specifically warned against somewhere as it is considered a security concern to make those changes system wide.

this bit from https://github.com/emoncms/emoncms/tree/master/docs/RaspberryPi#installation

For <Directory /> and <Directory /var/www/> change AllowOverride None to AllowOverride All . This should be on, or very close to lines 161 and 172 of /etc/apache2/apache2.conf

sudo nano /etc/apache2/apache2.conf

Save & exit, then restart Apache:

[edit]

The method used here https://github.com/emoncms/emoncms/blob/master/docs/LinuxInstall.md#configure-apache is better, although a file in the repo to be symlinked would be better than the heredoc and the ServerName can be set in the vhost rather than edit apache.conf. Plus the 000-default site should/could be disabled.