emonSD issues created using new scripts + emonhub environment file

Ah yes that makes sense, emonhub is not a single file executable (like log2ram etc) it’s a collection of files and IMO they should all (appear to) be together, I do not think we should be symlinking just the emonhub.py. I was thrown by this bit of your commit when I looked

that’s why I quickly commented that the path should NOT be /usr/share/emonhub and that the sub-folder was unnecessary, as it looked like you were just creating a folder to put the symlink in (well in fact you were!).

In actual fact whilst the sub-folder doesn’t need to be created, the path with the sub-folder level is correct. What we should be doing is symlinking the emonhub/src folder to /usr/local/bin so that the path to emonhub is /usr/local/bin/emonhub/emonhub.py and the output from

ls -la /usr/local/bin/emonhub

is a symlink to emonhub/src and the output of

ls -la /usr/local/bin/emonhub/

is a full list of all the emonhub source files. (not the repo root, but src root).

I sort of panicked a bit and made a hasty call because these changes are going into the live repo’s, I’m really not comfortable with that. It means we must get it right at every commit because we don’t know when anyone might update. The emonhub logfile is all that needed to be sorted on existing images, the path works fine on the emonSD, once we make a firm choice, implement and test the new paths for the new installer and updater scripts we can then update the old images in a controlled transition, we can’t really alter the emonSD symlink paths now as the repo is in the wrong place ie /home/pi/emonhub not /opt/emonhub (or /opt/emon/emonhub?) unless you plan on chaining symlinks which may bring it’s own issues.

Can we please try and separate “fixing immediate issues with the current image” and “developing the new installer and updater” and get it right, before rolling it out?

I dont mind too much, given that it works as a single file it seems unneeded to symlink the whole folder? Is a bit arbitrary in the end, although it does seem a bit messy… I dont mind

Well hopefully if you aren’t too bothered about it you will be ok with keeping it all together and symlinking the /src folder then. I really don’t find it even remotely arbitrary. the content of the /src folder is a single python package, we simply can not have a fraction of that package found via a path whilst the remainder isn’t.

The fact that emonhub.py is found at /usr/local/bin/emonhub/emonhub.py we cannot then say that emonhub_interfacer.py is at /opt/emonhub/emonhub/src/emonhub_interfacer.py tell me that’s not confusing! So we end up forced to interchangeable use 2 paths for emonhub.py on a single install depending on the discussion at hand. That’s the exact opposite of what we’re trying to achieve.

The whole intention of the “clone&symlink” install method is that it closely mimic’s a normal “download-zip&install” type install (bar the symlinks to a cloned repo to allow easy updates via cloning) all files in service should be accessible via a non-/opt path, this distances the cloned repo’s from the filesystem.

If emonhub was downloaded and installed conventionally (like L2R for example) the whole source would be in a single emonhub folder in /usr/local/bin not split up, that’s how it must appear even when cloned&symlinked. Especially since it’s Python, someone new to python trying to understand emonhub’s inheritance and imports will have an unnecessarily hard time trying to get a handle on what’s going on. And that should apply to anything that uses relative paths really, for example if you were to decide not to install emoncms direct to /var/www, you would symlink the whole folder, not just index.php (no idea if that would even work or not, just using it as an illustration). We should aim to make things clear and easy to understand where ever possible, I know it’s not always possible but where we can we should.

1 Like

To try and explain my reasoning, not as an argument, just thinking aloud. Personally I think I would go to /opt/emon/emonhub if I wanted to look at the source, the /usr/local/bin location from my perspective is just a link to simplify the process of running emonhub from the service file. Having the source appear in two places feels more messy than just having it in /opt/emon/emonhub. I think I understand your wider rationale about symlinking everything into debian recommended package locations but then we can also just use /opt as the recommended location for packages that are not installed via apt-get. Would it maybe be a better solution to stick with /opt and use an EnvironmentFile as @djh suggested?

I’m not sure how adding another location and a symlink “simplifies” the service file, surely just call the emonhub.py direct from the service unit if that is your sole aim, essentially all you will be doing is moving the content of /home/pi to /opt. My reasons for doing the “clone&symlink” is that conf files are found where conf files live and the executables or binaries are found where the executables/binaries live etc etc tec.

I see no good reason to symlink just the executable that a service unit needs to find, the service unit path is dictated by us it can literally be anywhere, the reason for putting the executables in /usr/local/bin is so that users can find them where they should be. The fact we clone them to a repo in /opt is just a easy backdoor way to keeping things updated, “our dirty little secret” if you like, yes, you and I would look to /opt for the source code as we are developers and the “clone&symlink” method is primarily aimed as providing an easy way to develop, the fact we still use that version control system as a package manager after deployment isn’t something the average user will be aware of or even need to know if well implemented. And yet any budding developer/programmer will recognise exactly what is going on once they see the symlinks.

These are 2 separate things IMO, no we shouldn’t necessarily “stick to using /opt”, regardless of whether we use EnvironmentFile, in fact if you stick to using /opt, there is no need to use EnvironmentFile as the path is never likely to change, it’s no different to using /home/pi where every setup does have a user pi’s homedir to install to.

Using EnvironmentFile is not entirely a bad idea IMO, but it is a very “init.d” way of doing things, it’s how the emonhub init.d service has worked for years with the default paths in /etc/default/emonhub. The “systemd” way of doing things is to use edit files. But which ever way you get the path into the service unit, it is only necessary if you want to be able to vary the path, if all installs are going to be in /opt why edit the unit at all?

I would rather see the whole src folder symlinked to /usr/local/bin and that path used in the service unit as that will serve 99% or more users and be a FHS compliant’ish use of cloned repo’s. Only on more exotic installs that don’t use FHS will that path need to be overridden by a drop-in file, it seems pointless to add the confusion of mandatory edits, environment files or non-fhs paths just to serve those very few users when there is a perfectly good other way. Remember /usr/local is going to be present on far more installs than /home/pi is, the issues with installing to /home/pi are not being carried forward to /usr/local.

If we can’t do it that way, then yes I guess the next best thing is just use /opt directly and forget about /usr/local symlinks, this is marginally better than “all in /home/pi”, but it is pretty much the same thing different location. (But it is better than splitting the src folder, that’s just a no no!)

I can’t see why symlinking the src folder has caused such a reaction. Does it matter if all the other files from the src folder can be found at the same place (as expected)? You were happy to create a folder just to sit the emonhub.py symlink inside, but symlinking the folder rather than using mkdir, so that all the src content is kept together is apparently a problem, the path to emonhub.py would be exactly the same!

It’s clear this is not a decision that should be made off the cuff so again I suggest we hold off putting any changes into live repo’s.

Strangely enough a user has hit the problem of hardcoded paths here How to set MQTT server / what are expected mqtt data? - #9 by notme

. . . and if that path isn’t generic enough, perhaps we should look closer at that. The less generic, the more likely the need to edit across different distro’s. We are trying to find a common path for the majority of distro’s so that (for example) the emonhub.py file IS found in the same place regardless of distro where possible, just because we can make it more flexible and user configurable, we don’t want to encourage installing it in a wide variety of places “just because it’s possible”, that’s at the opposite reach of wrong to installing absolutely everything in the same folder.

Yeah I saw that and decided not to wade in as you seemed to have it covered, but yes the /html vs not-/html folder is on my radar too, the symlinking there is just to make it accessible in both locations, when in fact it should be in neither! So perhaps when the new installer comes out there will be 3 (or more?) incestuously linked folders?

This dates back to before apache decided that source files should not be installed directly into /var/www/ for security reasons and added a html folder to put their demo index.html page in. Somehow emoncms ended up moving and/or being linked into the html folder because the 000-default vhost is used rather than writing a proper emoncms vhost, thus it couldn’t find /var/www/emoncms post-html folder mod. The emoncms source is not directly in the /var/www folder, it’s in a emoncms folder, so it doesn’t need to be in the html folder (plus it’s not html, it’s php!), but now of course we have to support both /var/www/html/emoncms and /var/www/emoncms as both have been used interchangeably for some time now.

Actually, as @borpin as just pointed out that discussion about emoncms location. It has reminded me that emoncms is actually already installed to one place and served via a symlink from another place.

In theory you should be able to move /var/www/emoncms to /opt/emon/emoncms and edit the symlink in /var/www/html and everything work as it does now! Unless both the paths are being used interchangeably, but even then another link from /var/www/ would fix that (2 links pointing to /opt/emon/emoncms, in parallel, not daisy-chained)

However! how it works now (ie from /var/www/html/emoncms) is wrong, it should be served from /var/www/emoncms by a dedicated emoncms.conf vhost, not via the default 000-default.conf vhost and some edits to apache.conf. The edits made to apache.conf are actually specifically warned against somewhere as it is considered a security concern to make those changes system wide.

this bit from https://github.com/emoncms/emoncms/tree/master/docs/RaspberryPi#installation

For <Directory /> and <Directory /var/www/> change AllowOverride None to AllowOverride All . This should be on, or very close to lines 161 and 172 of /etc/apache2/apache2.conf

sudo nano /etc/apache2/apache2.conf

Save & exit, then restart Apache:

[edit]

The method used here https://github.com/emoncms/emoncms/blob/master/docs/LinuxInstall.md#configure-apache is better, although a file in the repo to be symlinked would be better than the heredoc and the ServerName can be set in the vhost rather than edit apache.conf. Plus the 000-default site should/could be disabled.

It seems to me that this is a separate but extremely important issue. Surely it would make more sense to have the classical three repos: development, testing and production?

So all this stuff would be going on in the development repo, which people who wish could use on test installation, but nobody would use for their actual energy monitoring. Then when there was an apparently stable release candidate, it could be cloned to testing and more people would use it, including me if we resolve this paths issue, and a few brave souls might actually use it for live monitoring. Finally, when its known to work, it’s cloned to a production repo and downloaded by the great unwashed and used for producing factory product etc.

+99 - see previous post

I part company with you here. /opt is a sensible place for binaries and /etc/opt is a sensible place for their config files etc. So there’s no need to symlink them to other places.

But again, the first step is an environment file. Then the actual paths are moot. I can install it on my openSUSE machine. I could submit an OBS ( https://build.opensuse.org/ ) job to build packaged versions for openSUSE and Debian etc.

Note that /var/www is not where FHS says to put web data - that’s /srv/www - and that’s where openSUSE puts it.

There is a bit of history in play and how it has ‘developed’ over time. For some of the repos, there is a ‘stable’ branch which is effectively a release branch. However, some repos do not have a ‘stable’ branch.

I have some heavily customised wiki software on my PC that I use for my diary and making notes etc. It’s installed in /opt and runs happily from there. It isn’t symlinked below /srv/www; I just tell apache where to find it with an apache config file - I need that anyway because there are specials like rewrite rules that the wiki software needs.

Because this isn’t just about the path to the binary, it’s also about the apache path. It’s essential for ALL paths to be in the Environment File (and that is a systemd concept just as much as a sysV one). There are very few runtime environments that can’t read environment variables, and that’s for a very good reason.

As I pointed out elsewhere, Debian and Raspbian don’t follow FHS, so you’re arguing against yourself. One reason there aren’t more installs on other distros is precisely because paths are hardcoded to Debian and other distros use other paths.

IMHO, one Environment File is a whole lot simpler and cleaner than a bunch of symlinks.

My immediate reaction to that was to file a bug for it. But then I thought that you know a whole lot more than me about git and the best ways to use it, so perhaps you’d like to write a bug that proposes a sensible setup? Pretty please :slight_smile:

I definitely would not go that far! It isn’t that simple. The releases for the main repo now use tags so that is an improvement but for now we need to live with how it is and just recognize the limitations.

I think Im edging towards the Environment file solution for setting the service paths and using /opt/emon/emonhub directly.

We discussed the use of /opt over /usr here:
https://community.openenergymonitor.org/t/emonsd-next-steps/10426/106 @djh said:

Given that in this case the distro is not packaging the ‘product’ is it not therefore correct to use /opt directly?

The precise location aside, one of the goals of this installation script is to provide flexible installation paths rather than nailing everything down. I know we are still a number of steps away from that (e.g /var/www discussion), but perhaps using the environment file to allow emonhub to either be installed by our installer or perhaps if someone wants to do so be packaged independently and installed by the distro into /usr/local or other relevant location?

I haven’t tried using an environment file with systemd yet so I’m assuming this is possible and not bad practice

Lets do this, seems like a sensible step.

Both ideas sound sensible to me, FWIW @TrystanLea

Note that the environment file should include the /var/www/emoncms path and all other paths. You can read the file into the environment in a systemd unit with the EnvironmentFile directive. It looks like the best way to read them into a PHP environment is with the dotenv package as described at How to Use Environment Variables ― Scotch.io . There’s also a python-dotenv package that does something similar.

I think apache requires special handling, since it only accepts environment variables from the process that starts it, and the startup mechanism for apache varies between distros and is changing over time. It may be simplest to make the apache vhost conf file use Define statements within it for the paths. Then in order to install it in a non-default place it will be necessary to edit that file as well as the environment file before copying/moving it to wherever it is needed on that system. A bit more hassle but I don’t see a simpler way and it shouldn’t be a problem as long as it is documented. If anybody can think of a better way, I’m all ears!

I suggest using an EMON_ prefix for all the variables.