I’ve merged the hierarchical settings development originally proposed and developed by @anna_carboncoop and @borpin into the emoncms master branch. This work has been ongoing for the last year as its quite a significant change to the way settings are handled within emoncms as well as providing a new ini based format.
The idea behind hierarchical settings is to have a default settings file as a starting point that includes all available settings, and then a user settings file that inherits the default settings and overwrites the defaults for only the settings that the user has included in their settings file. This makes updating and adding new settings easier and allows for a cleaner more compact user settings file.
; -----------------------------------------------------
; Example emoncms settings.ini file
;
; default-settings.ini contains the default settings
; enter only the settings you wish to customise here
; the default settings are inherited
;
; The following is a barebones example, copy across
; other settings from default-settings.ini as required
;
; -----------------------------------------------------
; MYSQL Database settings
[sql]
server = "localhost"
database = "emoncms"
username = "emoncms"
password = "password"
; Redis Database (used as a cache for improved performance)
[redis]
enabled = true
; MQTT Used with emoncms_mqtt service to send and receive data over MQTT
[mqtt]
enabled = false
user = 'username'
password = 'password'
; Feed engine settings
[feed]
; 5:phpfina and 2:phptimeseries are the recommended emoncms feed engines
; its possible to hide unused engines e.g: 0:mysql, 6:phpfiwa & 10:cassandra
engines_hidden = [0,6,10]
; Buffer data to be written to
redisbuffer[enabled] = false
phpfina[datadir] = '/var/opt/emoncms/phpfina/'
phptimeseries[datadir] = '/var/opt/emoncms/phptimeseries/'
; Enable the graph module if you have it installed
[interface]
feedviewpath = "graph/"
This development is backwards compatible, the old settings file and format still works, it is parsed into the new format if detected.
As this is a big change, it would be great to have help testing this while it is in the master branch before we merge it into stable. I’ve tested it quite a bit here myself and it all looks fine, but I will usually miss something.
To test
If you have an emonSD system running the emoncms master branch, run emonpi/emonbase update to pull in the latest changes. Emoncms will parse the old settings file at this point - and should work fine with no apparent change.
The only part I have not tested is the environment variables side of things used by some for docker installations. There is an example env settings file here: emoncms/settings.env.ini at master · emoncms/emoncms · GitHub, rename to settings.ini in order to use this approach.
default-settings.ini and default-settings.php are not to be edited.
settings.ini is edited by the user to suit local installation - only values to be changed from default need to be included. If settings.ini is not found emoncms will look for settings.php
Its possible to use either the new object based format for settings.php or the old settings format for backwards compatibility.
It is recommended to use the new .ini format. The process of switching from the old php format to the new .ini format will be automated for emonSD installations assuming no user modifications to the settings file.
The associated changes to support the new settings object for the sync module are available in the sync module settings_object branch here https://github.com/emoncms/sync/tree/settings_object, this will merge to master when the main emoncms branch gets merged to stable
Actually I have got an old installation running 9.9.6-beta version on the basis of the emonSD-26Oct17 image…it is a RPI with the old ro/rw option activated…
The emoncms main dir was used for dev and experimentations and was dirty but I have to specify I always use the whole default settings…
I could not update, had to rename the /var/www/emoncms to /var/www/emoncms_old and installed a fresh copy of emoncms with a git clone
then created a settings.ini as you mentionned
I only had to reinstall the emoncms_mqtt service.
I managed to update via the admin page
After a reboot the situation is as follow :
In the admin page it says :
The log file has no write permissions or does not exists. To fix, log-on on shell and do:
touch /var/log/emoncms.log/emoncms.log
chmod 666 /var/log/emoncms.log/emoncms.log
/var/log/emoncms.log/emoncms.log does not exist
on the input page when I open the process list setup on an input, it says Feedid "+feedid+" does not exists or was deleted but the associated feed is still being regularly updated…
Hmm, sometimes the cache refresh doesnt work, the errors still look like js cache issues, I’ve added js file version’s to the process interface, to force a cache refresh. Could pull in the latest and try again?
I’ve pulled your latest changes
situation is still the same
when I follow the route /emoncms/feed/list.json, I’ve got the following output :
**Notice** : Undefined variable: data_sampling in **/var/www/emoncms/Modules/process/process_processlist.php** on line **1498**
**Notice** : Undefined variable: data_sampling in **/var/www/emoncms/Modules/process/process_processlist.php** on line **1498**
**Warning** : Cannot modify header information - headers already sent by (output started at /var/www/emoncms/Modules/process/process_processlist.php:1498) in **/var/www/emoncms/index.php** on line **300**
[{"id":"1","userid":"1","name":"t1","datatype":"1","tag":"Node emonpi","public":"0","size":"12340432","engine":"5","processList":"","unit":"\u00b0C","value":25.8,"time":1569403066,"start_time":1529417690,"interval":10},
blabla......
please note it is the same message when I follow the route /emoncms/vis/list
<br />
<b>Notice</b>: Undefined variable: data_sampling in <b>/var/www/emoncms/Modules/process/process_processlist.php</b> on line <b>1498</b><br />
<br />
<b>Notice</b>: Undefined variable: data_sampling in <b>/var/www/emoncms/Modules/process/process_processlist.php</b> on line <b>1498</b><br />
for the admin page, I had the following parameters at the end of default-settings.php
// Log file configuration
"log"=>array(
"enabled" => true,
// On windows or shared hosting you will likely need to specify a
// different logfile directory
"location" => "/var/log/emoncms.log",
// Log Level: 1=INFO, 2=WARN, 3=ERROR
"level" => 2
)
I’ve changed the “location” line to :
"location" => "/var/log/emoncms",
and now the admin page is very OK
dont know why it was /var/log/emoncms.log…after the pull, it should have been /var/log/emoncms, as default-settings.php is not in the .gitignore
but in the default-settings.ini of the repo, you still have :
Did you edit the settings.ini to match any settings in the pre-existing settings.php file? Things like credentials and file paths (as these have changed over time).
@TrystanLea Nice it is working now…I had to commit the change I had done on default-settings.php and to launch the pull by hand but it is OK now for :
input page
feed page
sync module
graph module
vis module
postprocess module
for backup module and to remove the warning, I only had to replace all # by // in the config.cfg files
Anyway backup is not running and still make its search for timeseries files in the /var/lib dir, whereas the database_path is correct in config.cfg (/home/pi/data
2019-09-25 12:28:58.135|WARN|PHPFina.php|get_meta() meta file does not exist '/var/lib/phpfina/74.meta'
At last, I still have :
the following line in the emoncms.log
2019-09-25 12:17:13.850|ERROR|run.php|Can't connect to database, please verify credentials/configuration in settings.php
run.php is the demandshaper module. Theres a dedicated branch for the time being called settings_object. The sync and postprocess modules also have similarly named branches.
@borpin : run.php is not so bad it is just the script name for something which looks like a background worker
@TrystanLea: Ok I’ve installed the branch settings_object of the demandshaper
still the same error in the log
ERROR|run.php|Can’t connect to database, please verify credentials/configuration in settings.php
but i dont know how to use demandshaper so i am not really a good tester on that point
Except it is in a separate Repository/Module. If each of those had a run.php we’d be in a right mucky fuddle. It is a bit like the multitude of config.cfg files that float about. You have to know where the file is to know what it is doing.
.
it is just the script name for something which looks like a background worker