New emoncms settings format - help testing needed

I’ve merged the hierarchical settings development originally proposed and developed by @anna_carboncoop and @borpin into the emoncms master branch. This work has been ongoing for the last year as its quite a significant change to the way settings are handled within emoncms as well as providing a new ini based format.

The idea behind hierarchical settings is to have a default settings file as a starting point that includes all available settings, and then a user settings file that inherits the default settings and overwrites the defaults for only the settings that the user has included in their settings file. This makes updating and adding new settings easier and allows for a cleaner more compact user settings file.

The full list of default settings is available here:
emoncms/default-settings.ini at master · emoncms/emoncms · GitHub

and an example of a cut down settings file here:
emoncms/example.settings.ini at master · emoncms/emoncms · GitHub

; -----------------------------------------------------
; Example emoncms settings.ini file
; default-settings.ini contains the default settings
; enter only the settings you wish to customise here
; the default settings are inherited
; The following is a barebones example, copy across
; other settings from default-settings.ini as required
; -----------------------------------------------------

; MYSQL Database settings
server = "localhost"
database = "emoncms"
username = "emoncms"
password = "password"

; Redis Database (used as a cache for improved performance)
enabled = true

; MQTT Used with emoncms_mqtt service to send and receive data over MQTT
enabled = false
user = 'username'
password = 'password'

; Feed engine settings
; 5:phpfina and 2:phptimeseries are the recommended emoncms feed engines
; its possible to hide unused engines e.g: 0:mysql, 6:phpfiwa & 10:cassandra
engines_hidden = [0,6,10]
; Buffer data to be written to
redisbuffer[enabled] = false
phpfina[datadir] = '/var/opt/emoncms/phpfina/'
phptimeseries[datadir] = '/var/opt/emoncms/phptimeseries/'

; Enable the graph module if you have it installed
feedviewpath = "graph/"

This development is backwards compatible, the old settings file and format still works, it is parsed into the new format if detected.

As this is a big change, it would be great to have help testing this while it is in the master branch before we merge it into stable. I’ve tested it quite a bit here myself and it all looks fine, but I will usually miss something.

To test

  1. If you have an emonSD system running the emoncms master branch, run emonpi/emonbase update to pull in the latest changes. Emoncms will parse the old settings file at this point - and should work fine with no apparent change.

  2. Copy the default emonSD settings.ini example from here: emonpi/emonpi.settings.ini at master · openenergymonitor/emonpi · GitHub place in /var/www/emoncms and rename to settings.ini - check to ensure everything keeps working as expected.

  3. Try restarting the emoncms_mqtt and feedwriter services e.g:

    sudo systemctl restart emoncms_mqtt
    sudo systemctl restart feedwriter

The only part I have not tested is the environment variables side of things used by some for docker installations. There is an example env settings file here: emoncms/settings.env.ini at master · emoncms/emoncms · GitHub, rename to settings.ini in order to use this approach.

@borpin asked me to clarify a couple of points:

  • default-settings.ini and default-settings.php are not to be edited.

  • settings.ini is edited by the user to suit local installation - only values to be changed from default need to be included. If settings.ini is not found emoncms will look for settings.php

  • Its possible to use either the new object based format for settings.php or the old settings format for backwards compatibility.

  • It is recommended to use the new .ini format. The process of switching from the old php format to the new .ini format will be automated for emonSD installations assuming no user modifications to the settings file.

1 Like

The associated changes to support the new settings object for the sync module are available in the sync module settings_object branch here, this will merge to master when the main emoncms branch gets merged to stable

Actually I have got an old installation running 9.9.6-beta version on the basis of the emonSD-26Oct17 image…it is a RPI with the old ro/rw option activated…

The emoncms main dir was used for dev and experimentations and was dirty but I have to specify I always use the whole default settings…
I could not update, had to rename the /var/www/emoncms to /var/www/emoncms_old and installed a fresh copy of emoncms with a git clone
then created a settings.ini as you mentionned
I only had to reinstall the emoncms_mqtt service.
I managed to update via the admin page

After a reboot the situation is as follow :

In the admin page it says :

The log file has no write permissions or does not exists. To fix, log-on on shell and do:
touch /var/log/emoncms.log/emoncms.log
chmod 666 /var/log/emoncms.log/emoncms.log
/var/log/emoncms.log/emoncms.log does not exist

the input page is working nearly fine

the feed page cannot load

backup is unhappy

sync wants redis, which is running

postprocess doe not see the feeds

graph and visu are OK

Thanks @alexandrecuer the input page and feed page maybe javascript cache issues, can you try a full page refresh on those pages?

The sync module needs the settings_object branch as mentioned above.

thanks but situation is the same after a full refresh…
but when I log on other users accounts via the admin page, the feed page loads correctly

On the input page I had the following message at a time :

EmonCMS Error
Message: ReferenceError: table is not defined
Route: Modules/process/Views/process_ui.js
Line: 26
Column: 3

EmonCMS Error
Message: TypeError: processlist_ui is undefined
Route: input/view
Line: 727
Column: 3

EmonCMS Error
Message: TypeError: processlist_ui is undefined
Route: Modules/input/Views/input_view.js?v=4
Line: 1236
Column: 1 

vis and graph are not completely OK
vis can see the feeds :

graph cannot and can only see the existing graphs

on the input page when I open the process list setup on an input, it says Feedid "+feedid+" does not exists or was deleted but the associated feed is still being regularly updated…

Hmm, sometimes the cache refresh doesnt work, the errors still look like js cache issues, I’ve added js file version’s to the process interface, to force a cache refresh. Could pull in the latest and try again?

I’ve pulled your latest changes
situation is still the same
when I follow the route /emoncms/feed/list.json, I’ve got the following output :

**Notice** : Undefined variable: data_sampling in **/var/www/emoncms/Modules/process/process_processlist.php** on line **1498**

**Notice** : Undefined variable: data_sampling in **/var/www/emoncms/Modules/process/process_processlist.php** on line **1498**

**Warning** : Cannot modify header information - headers already sent by (output started at /var/www/emoncms/Modules/process/process_processlist.php:1498) in **/var/www/emoncms/index.php** on line **300**
[{"id":"1","userid":"1","name":"t1","datatype":"1","tag":"Node emonpi","public":"0","size":"12340432","engine":"5","processList":"","unit":"\u00b0C","value":25.8,"time":1569403066,"start_time":1529417690,"interval":10},

please note it is the same message when I follow the route /emoncms/vis/list

<br />
<b>Notice</b>:  Undefined variable: data_sampling in <b>/var/www/emoncms/Modules/process/process_processlist.php</b> on line <b>1498</b><br />
<br />
<b>Notice</b>:  Undefined variable: data_sampling in <b>/var/www/emoncms/Modules/process/process_processlist.php</b> on line <b>1498</b><br />

for the admin page, I had the following parameters at the end of default-settings.php

// Log file configuration
    "enabled" => true,
    // On windows or shared hosting you will likely need to specify a
    // different logfile directory
    "location" => "/var/log/emoncms.log",
    // Log Level: 1=INFO, 2=WARN, 3=ERROR
    "level" => 2

I’ve changed the “location” line to :

"location" => "/var/log/emoncms",

and now the admin page is very OK
dont know why it was /var/log/emoncms.log…after the pull, it should have been /var/log/emoncms, as default-settings.php is not in the .gitignore
but in the default-settings.ini of the repo, you still have :

location = '/var/log/emoncms.log' 

I’ve got the following emoncms log :

2019-09-25 10:16:24.011|ERROR|run.php|Can't connect to database, please verify credentials/configuration in settings.php

is there a run.php file ? and there is no more settings.php file now am I wrong ?

Did you edit the settings.ini to match any settings in the pre-existing settings.php file? Things like credentials and file paths (as these have changed over time).

Thanks @alexandrecuer I’ve ammended both the log setting and fixed the data_sampling error. Could you pull in the latest and try again?

@TrystanLea Nice it is working now…I had to commit the change I had done on default-settings.php and to launch the pull by hand but it is OK now for :

  • input page
  • feed page
  • sync module
  • graph module
  • vis module
  • postprocess module

for backup module and to remove the warning, I only had to replace all # by // in the config.cfg files
Anyway backup is not running and still make its search for timeseries files in the /var/lib dir, whereas the database_path is correct in config.cfg (/home/pi/data

2019-09-25 12:28:58.135|WARN|PHPFina.php|get_meta() meta file does not exist '/var/lib/phpfina/74.meta'

At last, I still have :

  • the following line in the emoncms.log
2019-09-25 12:17:13.850|ERROR|run.php|Can't connect to database, please verify credentials/configuration in settings.php

Good to hear @alexandrecuer

run.php is the demandshaper module. Theres a dedicated branch for the time being called settings_object. The sync and postprocess modules also have similarly named branches.

I will look at the backup module next

Perhaps a more intuitive name could be user :grinning:.

@TrystanLea could you add a warning to your top post that testers need these branches?

@borpin : run.php is not so bad :slight_smile: it is just the script name for something which looks like a background worker

@TrystanLea: Ok I’ve installed the branch settings_object of the demandshaper
still the same error in the log
ERROR|run.php|Can’t connect to database, please verify credentials/configuration in settings.php
but i dont know how to use demandshaper so i am not really a good tester on that point

Except it is in a separate Repository/Module. If each of those had a run.php we’d be in a right mucky fuddle. It is a bit like the multitude of config.cfg files that float about. You have to know where the file is to know what it is doing.

1 Like

Thanks @alexandrecuer did you restart the demandshaper service, sorry forgot to mention that.

sudo systemctl restart demandshaper

Yes agree that run.php is not particularly useful, I will change that

quote=“TrystanLea, post:18, topic:12002”]
sudo systemctl restart demandshaper

Yes i did it and i also reboot the pi but still the same error