Move data & settings file to a common location?

Emoncms
1

I’ve just submitted a git pull request to open a discussion about moving the emoncms settings file & it’s data files to a common location, and have suggested - /home/pi/.emoncms to align with node-red - /home/pi/.node-red for Raspberry Pi’s, but the PR can also easily accommodate other operating systems such as DietPi etc.

I’ve also put together a quick video to show how this would work in practice.

The changes are non-breaking so shouldn’t affect already established systems, but of course any existing system could easily be changed to use the common directory if they wished.

Paul

3 Likes
2

Not sure if I should be commenting here or there, so I’ll start here.

This ties in with what i have been doing regards a emoncms install script and the updater script.

I share your reasoning and have some suggestions of my own regarding simplifying the standard locations.

However! IMO the pi user folder is NOT the place to put this stuff and with the needs of the emonSD filestructure and non-pi installs this is unlikely to become THE standard. Once you factor out the emonSD/Pi users, RPi users that use a different username and non-pi installs this actually caters for quite a small group of users.

My efforts have been to move away from any rasperrypi/raspbian or pi user configuration, in fact I avoid using any user directories altogether. Even if we try to expand inclusion to RPi users that choose to change their RPi username by editing all guides to use /home/pi instead of /home/<username> or something similar, every command they encounter needs editing, they cannot just copy and paste. Likewise instructions, guides and forum comment like “usually found in the users home directory” or " usually found at /home/pi or what ever your username you use" are not as good as “you WILL find it at x/y.z”.

The use of a hidden folder or file in the users home folder is usually done for user specific settings and files that are positioned as such to over rule the defaults, if you delete a user, the lost settings and files should not affect any softwares for other users.

Do you also intend to move the mysql tables? If so do you intend to edit the mysql default location for all users/databases or specific to the emoncms table(s) or use symlinks?

The emonPi already uses a different location for the data files, settings.php and mysql tables etc but not the same as you suggest here, it would be good to try and unify rather than add another structure.

Since the RO filesystem of the emonSD/Pi dictates the datafiles settings and tables must go to a writable location, which must be a separate partition to allow it to be mounted RW and the FS RO, it might make sense to follow this directive by way of a new root folder /data and to hold the data, settings and tables there, but with symlinks to the “correct” locations, ie the tables and data files remain under /var/lib (although the phpfina and phptimeseries data should be sub-foldered /var/lib/emoncms as they are emoncms specific files, this would match other services like /var/lib/mysql).

IMO we should try to resolve to a fixed location for the settings.php file that is outside, outside the emoncms folder rather than a list of possible locations that are biased to the pi username. Not only do the checks add to the processing load every time the settings file is accessed, it is also very open to issues arising from multiple settings.php files eg a user may want to experiment and to do so makes a copy of their settings.php file in their home/user folder, which if they are using the default pi username, could mean emoncms will find that copy before the one they are editing in the emoncms folder so no changes will take effect.

Maybe this is an opportunity to change the settings.php file name too, it’s name doesn’t tie it in with emoncms at all, so from that prospective the emoncms folder is the best and possibly the only place it should be located. even calling it emoncms_settings.php would be a big step forward, however I would like to see some form of instancing so if you have multiple emoncms installs it is clear which settings file belongs to which instance, again, the obvious place would be the emoncms folder it relates to, but I agree it would be better placed elsewhere, but in a sub-folder or with a name that ties it back to it’s instance.

Whilst writing the installer and updater I have also decided the non-www modules need to be located in a single folder, for operational/security reasons this must not be in the /var/www path, but nor should it be in the users home directory. I have been installing this folder to /srv which is also the preferred location for emonhub and any other non-packaged softwares.

In short. I think the default Linux installation should follow Linux filesystem guideline as close as possible, when these agreed standard locations are used throughout, then backup routines and any other interaction with the emoncms filesystem will be easily performed, documented and supported. To that we can add a further optional layer of “move and symlink” of the essential variable parts (tables, settings and datafiles etc) to a root folder eg /data, this would allow users such as yourself to find all the files in one location and since this /data folder can just as easily be a mounted partition, it would be the default for an emonSD/Pi type install.

All installs would then have the files in the “correct” locations and meet the demands of the RO image and those that want a single location for back up purposes or even just for HDD or USB stick (for data only) solution.

a couple of links re linux FS structure
http://www.pathname.com/fhs/pub/fhs-2.3.html
https://wiki.debian.org/FilesystemHierarchyStandard

3

I don’t know what the right answer is, but a similar discussion took place relating to DietPi The DietPi installation does not follow the *nix Filesystem Hierarchy Standard · Issue #1297 · MichaIng/DietPi · GitHub.

Might be relevant as DietPi does a similar thing for logs (using RAM log). :smile:

4

Makes sense, PR already updated.

This was my initial thought, as it simplifies things, but…

  • There are various backup tools currently in use, and linking by symlinks would break those, as they would archive the actual symlink and not the data. Although it’s an easy fix - add a ‘h’ switch to the tar statement, but it would mean that they would need updating too.
  • It would double the work during installation, as 2 sets of folders would need creating & ownership changing etc, once for the actual data files, and secondly the folders holding the symlinks.

Easily done, just a few minor changes in process_settings.php file, although updating the readme.md’s, and all of the translations would be a pain. Also backwards compatibility could be problematic.
We couldn’t however have a file named settings.php in a /home/data directory, as it gives no clue what it belongs to… maybe use a sub-directory to identify it’s owner, such as /home/data/emondata/settings.php or even just /home/emondata/settings.php??
The change of path for emonpi’s could easily be made as part of the emonpi update process, without breaking existing systems.

Ideally, and maybe emonhub.conf?? (eventually)

Paul

5

looking at the PR I think you maybe mis-understood. you have pointed emoncms at a possible psuedo location that could definitely exist if running emoncms on an SD card, might be there if users choose to group their emoncms stuff for backing up, but is unlikely to be there if users stick to a Linux FS structure.

This is where we are looking at it differently. I am saying the files and folders should be moved and symlinked from their “correct” linux FS positions not to those positions. If done in the right order you only need to set the ownership and permissions once.

For example. lets say during the initial set up steps we create a settings file in a sub-folder /etc/emoncms/settings.php (file name to be discussed else where, I’m using the existing/familiar name here for ease) and set the correct ownership and permissions, then for all Linux users the settings file is always found at the same place and it is in line with the Linux FS structure etc.

This next install step is then compulsory for SDcard based installs and entirely optional for everyone else, based on whether they choose to group their emoncms stuff for back-up or want to locate it on a USB stick or HDD etc.

sudo mv /etc/emoncms /data && sudo ln -s /data/emoncms /etc

That one line will move the /etc/emoncms folder keeping the ownership and permissions intact for both the folder and the files therein, plus create a symlink from the original Linux FHS location.

emoncms will still find (a writable) settings file at '/etc/emoncms/settings.php`, no need to make any changes for alternative installs from within emoncms, it’s always there (or at least it appears to be and works as if it was there). All documentation can be the same, all support can say “check your settings at /etc/emoncms/settings.php” there will be no variation. But in reality, IF this second stage has been implemented, the file is actually on a separate partition, in the same place as all the other emoncms stuff.

This is done for 3 locations, the settings sub-folder, the data sub-folder (containing the phpfina and phptimeseries folders) and the mysql tables. That’s just 3 additional and optional (mandatory for low-write SDcard) lines to make all linux based installs the same, with no “check here then check there” type code in emoncms.

… ok so we could have one fall back so that if no settings file is found at /etc/emoncms/settings.php use ./settings.php.

True but once it’s done it’s done, the unification and simplification of the installation and the easier support will hopefully save a significant amount of effort on-going.

6

So any Linux install would look like this (FHS compliant)

├── etc
│   └── emoncms
│       └── emoncms.settings.php
└── var
    └── lib
        ├── emondata
        │   └── emoncms
        │       ├── phpfina
        │       └── phptimeseries
        └── mysql
            └── emoncms

And optionally for most users, but mandatory for SDcards it would look like this after the 3x “move and symlink” folder commands.

├── data
│   ├── emoncms
│   │   └── emoncms.settings.php
│   ├── emondata
│   │   └── emoncms
│   │       ├── phpfina
│   │       └── phptimeseries
│   └── mysql
│       └── emoncms
├── etc
│   └── emoncms -> /data/emoncms
└── var
    └── lib
        ├── emondata -> /data/emondata
        └── mysql -> /data/mysql

then if you want more emoncms instances

├── data
│   ├── emoncms
│   │   ├── emoncms.settings.php
│   │   └── emoncms2.settings.php
│   ├── emondata
│   │   ├── emoncms
│   │   │    ├── phpfina
│   │   │    └── phptimeseries
│   │   └── emoncms2
│   │       ├── phpfina
│   │       └── phptimeseries
│   └── mysql
│       ├── emoncms
│       └── emoncms2
├── etc
│   └── emoncms -> /data/emoncms
└── var
    └── lib
        ├── emondata -> /data/emondata
        └── mysql -> /data/mysql

Most of the naming can then be done dynamically using the dirname(__FILE__) so by just creating a folder “emoncms2” and cloning emoncms into that, the paths can all be derived from the folder name.

We need to think about file naming, I have deliberately called the emoncms data folder emondata as we cannot “copy and symlink” a “emoncms” data folder, “emoncms” settings folder and a “emoncms” mysql folder all to the same target, so I have used “emondata” as the top-level emoncms data folder to avoid that conflict and also as the first emoncms instance is usually called “emoncms” it also avoids nesting repetition ie /var/lib/emoncms/emoncms/phpfina.

7

There is another option, It is possible to use different names in the symlinks, it would allow all the “emoncms” folders to reside in one place as they would be renamed “emoncms_settings” “emoncms_data” and “emoncms_mysql” or similar. BUT I think that should be avoided if at all possible so that both ends of the symlinks are the same naming to avoid confusion, ie the same folder being called 2 different names depending on where it’s viewed in the FS.

8

Ah that’s made it clearer.

As /data will contain emoncms data in at least 3 sub-folders, wouldn’t it make things clearer to call the top level folder something that makes it clear that it is specifically for emoncms use.
ie. instead of /data use something specific like /emon_userdata (I’m sure we could think of something better!!)
That would deter people from using that folder for other ‘data’ storing purposes.

Paul

9

I think there will be other data in there too like emonhub.conf or the node-red stuff. The /data folder I have used is either just another folder for non sdcard users or it is another partition/disk mounted as \data. IMO there’s only so many times you can sub-folder, and does the emoncms-node-red stuff and emonhub.conf class as emoncms or not? Strictly speaking not, but for this purpose I would say yes.

I’m proposing this as an alternative to /home/pi/data on emonPi’s, for the transitional period we would use another symlink so the /data folder also appears as /home/pi/data but if you delete the pi user, the /data folder and all the data remains intact.

I chose /data as it is more in tune with existing top-level folders, they are all 3 or 4 letters long, plus it fits with the emonPi/SD and if you are using a separate HDD or staging files for backing-up, the chances are you will want other “data” there too that isn’t emoncms.

10

Ok, I’ll take a look over the coming nights and try and pull this together in a feature branch, so we can fine tune it, and discuss further.
Thanks for the input.

Paul

1 Like
11

It would be good to see some other feedback and opinions before we/you sink too much time into it.

12

TBH where files were and why, always used to be a mystery to me and, until recently, I’d never even heard of the FHS!

In order to mount the RW partition, does the folder have to be a root folder or is this just a convenient way of linking it together under a single folder?

Slightly OT but related, what are the advantages of having the settings in a PHP file other than ‘that is how it has always been’. Thinking that if it was a structured data format, if an update required a new setting this could (I suspect) be added more easily.

Other than that this all looks sensible.

13

No, the current 3rd “data” partition of the emonSD is mounted as /home/pi/data. Using FHS the data folders should be mounted under /var/lib but there is other stuff on the 3rd partition too, a separate drive should really be mounted under /mnt but we are discussing a common location that may be a seperate partition or maybe just a single folder for collating this stuff, so /mnt won’t work. Since we have a variety of stuff going into a single folder that maybe a mounted device/disc, I think a root folder would take it outside of FHS reach, we can do what we like in /data there is no convention in FHS. That way the components that reside in /data will be symlinked according to their own standing in terms of FHS.

I believe it has just evolved that way, I would like to see it change too but I think changing it’s format/structure may be a step too far for some, I’m not sure we will even get to change the name TBH.

There are some other related things I would like to address too, like why do we need to specify multiple locations for the different feed folders in the settings when they are always in the same folder? Why can’t we just spec the root location as (for example) a default of /var/lib/emondata/dirname(__FILE__) and both the phpfina and phptimeseries folders (and any others that come and go such as phpfiwa and timestore) can all be found there rather than typing the same path multiple times in the settings.php?

The topic of FHS and the emoncms structure was discussed at some length back in 2013/14 when Dave (Schism) packaged emoncms 8.4 and a couple of modules as debian packages, Even the emonhub install locations were originally chosen for imminent debian packaging, but since emonhub has never been packaged for debian, the emonhub installer installs some of the stuff to wrong locations (by FHS) on the emonSD emonhub is also located in the pi user folder, that location was fine during development but for deployment it should have been located in /srv/emonhub as it’s not packaged.

1 Like
14

where do you propose the main emoncms folder is installed. i.e where would you run git clone http://github.com/emoncms/emoncms.git?

I have been thinking that I would like to move the emoncms/scripts folder out of /var/www/ and docs for that matter

15

The only way I can see to do that is to have a public_html folder within the emoncms repository which is then symlinked to /var/www/emoncms. But that’s not particularly backwards compatible if you just run git pull in the /var/www/emoncms folder.

16

The answer to that is to have some form of script that does the update and move away from simply doing a git pull. So use curl to get the script and execute it. First run, the script looks for the various existing folders and moves things around as appropriate, fresh installs would set up correctly in the first place. Script a new settings file structure while at it :slight_smile: GRD

17

One of the really nice things about using git is that you can easily see any changes to files. If the files are not moved but symlinked I can see that working, is this the intention? i.e the installation script would symlink files to the correct locations?

18

Paul would be in the best place to answer that but from the bits above I think not. Also, for at least 95% of those using Emoncms, they never make any use of git for this and it is simply a delivery method to get the updates. It could be regarded as a logical progression from the EmonSD/EmonPi update button.

19

I hadn’t imagined that to be an area where change would be concidered, but for the record I personally do not usually install to /var/www anyway. I prefer to install to /srv/emoncms and symlink to one or more “sites” in the /var/www path, this is the reason I would prefer to see the settings outside the emoncms folder.

Sub-foldering the web-content would make is easier to install to another location eg /srv and just symlink the web-content folder, much like the emailreport and postprocess modules are already, perhaps we can move to a standard name for the web-content folder.

I had already suggested installing the emoncms modules that need to be outside the /var/www path, to a single folder eg “emoncms-modules” so that they were easier to manage and to update. By simply installing all emoncms repos to /srv/emoncms (eg /srv/emoncms/emoncms, /srv/emoncms/app /srv/emoncms/dasboard etc) this would make the auto updater I’m working on even simpler, as it can just loop all folders in /srv/emoncms.

Please see Break out the version to a separate file. by pb66 · Pull Request #745 · emoncms/emoncms · GitHub for more on this.

If you are referring to the source files, yes, or at least initially, once we have a reliable installation and update method, and established locations, we could perhaps review that, but initially nirroring the github repos gives us a small level of consistency to build from.

The data and the settings are not source code and they should not reside in the /srv location. as discussed above, they should reside in the FHS locations unless symlinked to /data for RO sdcards or backing up etc.

1 Like
20

Possible the easiest and safest way to approach this is for me to commit the first (non-breaking) stage of the ‘proposed’ changes to a feature branch, where the proposal will be clearer, and we can then discuss & develop it further from that point.

Paul