Community
OpenEnergyMonitor

Development: Devices, Inputs and Feeds in emoncms

device
development
emoncms
Tags: #<Tag:0x00007fc66ddf8258> #<Tag:0x00007fc66ddf80f0> #<Tag:0x00007fc660577f50>

(Trystan Lea) #1

Over the last couple of months I’ve started to use @nchaveiro’s device module introduced originally in emoncms v9 and now available as an optional module. I’ve made a couple of modifications to make it possible to create and initialise a device from a device template by sending a ‘describe’ property alongside other node input data, and in the last week I’ve made a start on integrating device configuration into the emoncms input user interface itself.

I’ve been wondering for sometime if the grouping of inputs and feeds and the relation between them can be improved with a better definition of nodes or devices. I like the idea that when an emontx or emonth posts to emoncms it appears more like the detection of a new device rather than a loosely grouped set of inputs or feeds - at the same time the flexibility of the current implementation is I think a good thing and so any changes need to be implemented carefully.

Backing up a bit. Inputs, input processing and Feeds are key parts of emoncms allowing a degree of pre-processing and providing the option to record only what you wish to record.

At present inputs can be grouped by nodeid which may be a number such (e.g 10) or an alphanumeric name (e.g emontx).

Feeds can be grouped by ‘tag’ which may or may not be associated with the input node grouping.

@nchaveiro introduced the device module for defining ‘devices’ in emoncms v9. Devices are associated with an input ‘nodeid’. The device module can be used to automatically configure emoncms inputs and create emoncms feeds according to a device template definition. The device module also provides device or input node level authentication keys.

Devices:

  • Linked to input ‘nodeid’
  • Has seperate device name
  • Configures inputs and creates feeds from a template definition

Inputs:

  • Grouped by ‘nodeid’
  • nodeid/inputname must be unique

Feeds:

  • Grouped by ‘tag’
  • feedname only must be unique
  • Feeds are created from Inputs.

Does emoncms have too many different grouping types?, feed ‘tag’, input ‘nodeid’, device ‘name’, device ‘nodeid’. We could have all of these refer to the same thing. Feeds could belong to a device identified by ‘devicename’, inputs to a device identified by ‘devicename’, a device would then just be a device rather than having a name and then a seperate association with a nodeid.


A start on integrating the device module in the input interface

I’ve attached some initial screenshots of the new inputs UI I’ve been working on that integrates the device module more closely, with the ability to select and initialise a device template from the input interface.

Input list with new look that emphasises ‘node’ or ‘device’ grouping + multiselect for deleting multiple inputs at once and perhaps other multi-input operations:

Clicking on the spanner icon for the node/device brings up the device configure dialog, where the Location and device template can be set:

Clicking on initialise, initialises the device, building the input processing and creating feeds according to the selected device template:

My other thought is that you could almost call the inputs interface ‘My Devices’ and the interface could even provide the ability to toggle between device inputs and device feeds within the same device list interface… That would be a big step, Im not 100% sure if its the right way forward.

If anyone has any thoughts on it Id be glad to hear them.

The code for the above so far is in the device-integration branch of emoncms here:
https://github.com/emoncms/emoncms/tree/device-integration

This development topic builds on the ‘emoncms UI improvement’ item outlined in the emoncms development outline - 14th of April 2017:


OpenEnergyMonitor goals 2018/2019
New: emonPi / emonBase Network Setup Wizard
MQTT Auth transfer for EmonESP WIFI devices + Device module
CydYnni, EnergyLocal Community hydro smart grid - blog post
Copy users on private server, including dashboard, inputs, feeds, graphs, etc?
Important Emoncms security update for multi-user emoncms installations V9.8.11
Using the new emoncms device module integration
(Paul Reed) #2

Looks like a welcome major step forward, but what about backward compatibility?
Will this break existing systems?

Paul


(Trystan Lea) #3

Thanks @Paul, all of my modifications so far are fully backward compatible. Changing the naming could start to have backward compatibility issues.

Changing feed ‘tag’ to feed ‘nodeid’ or feed ‘devicename’ or changing input ‘nodeid’ to input ‘devicename’ would be difficult at the database level. It would require migration scripts etc. Alternatively Feed Tag could easily be renamed to ‘nodeid’ or ‘devicename’ in the user interface at the final point.

Keeping with the ‘nodeid’ field, property, variable name is probably easiest. But is the device name being the nodeid confusing? or nodeid actually being a node name…

Cleaning up the naming at the database and code level is not crucial to improving the user interface but something to think about.


(Simon) #4

Trystan,

NodeId, Field Nr etc are all internal system concepts so to speak. A change to Device name, Sensor name etc. would make things a lot more understandable. Obviously you’d have to have the ability to assign a NodeId to a Device name and likewise a NodeId + Field nr to a Sensor as you seem to be doing.

The concept of Inputs and Fields in the current interface is confusing - witness the number of questions about this from new users, so any change to make it more ‘real world’ and less ‘system’ oriented can only be good. And I’m sure we could argue for a long time about whether an emonTX should be called a Node or a Device, etc.

As a final thought on compatibility, you should maybe introduce a toggle on the management screen that allows users to switch back to ‘Classic’ view. Also I think if you kept the database as is then you should be able to have a simple file to translate between the different views. This file would house all the additional information needed by the new interface to ‘use’ the database format as is.

Obviously this will need some analysis but if you can come up with a solution the next step might be a graphical interface that starts off as Nodes with Sensors (automatically detected) where the user can name the Nodes and the attached sensors and also name and define the various feeds derived from the sensor data. Just a thought to keep you busy while you’re enjoying yourself coding :wink:

Simon


(Trystan Lea) #5

Thanks @Bramco.

Also I think if you kept the database as is then you should be able to have a simple file to translate between the different views. This file would house all the additional information needed by the new interface to ‘use’ the database format as is.

That’s true if we keep the fields in the database the same and just change the user interface and potentially some of the variable naming that would keep backwards compatibility without a need for migration.

maybe introduce a toggle on the management screen that allows users to switch back to ‘Classic’ view

Yes I think this could be possible, we could certainly start off like that, perhaps providing the option to try new interfaces in a beta mode, similar to how say gmail provides the option to try a new interface.


(Johan Kanflo) #6

I think this is a good addition. The concept of node Id and node name both are needed. A user will typically not know about node IDs but will give each node a descriptive name. Will there be an API call for the “Device configuration” step above? If so, the device could be automatically configured whenever it appears for the first time.

Cheers,
Johan


(Simon) #7

One issue with automatic configuration, or inclusion in the set of nodes is that with the RFM frequencies you can easily get random nodes appearing which aren’t part of your network. All sorts of things use these frequencies, baby alarms, remotes for lighting etc. Every once in a while I go into my inputs list and remove any of these random nodes.

So I guess what you’d want to have is some way of switching between the set of ‘good’ nodes and the full list so that you could include any new nodes in the set of ‘good’ ones.

Simon


(Andreas Messerli) #8

Thats nice! I’m thinking of using this for 1wire devices like the DS Temp sensors as they have some standard things like Alarm settings etc. which are always there to use but hardly everyone knows etc. But what if your “device” sends a lot of data which you actually don’t need to log?
One thing which came to my mind all the time is: Why do we need to log everything?
I have quite some use cases where no feeds are necessary. Inputs for example which are just used to show a LED status value. There all you need is to show the actual value of the input and not the feed. I my case I have a lot of Status LEDs which show process information. I do not need to log them as it is more used as a real-time dashboard, in my case a water treatment system of the town or in my house values if a window/door is closed etc.

I think this is also change which should be considered to allow to use an input for the widgets etc. I think this would fit nicely as you can then create device templates which do not need a feed creation or just where necessary.
What do others think about that?


(Trystan Lea) #9

Thanks @kanflo

Will there be an API call for the “Device configuration” step above?

Yes, there are a number of APIs in the device module and I’ve been adding a few more as part of this development which will be documented once this stabilises.

@Bramco good point, I think we can at least aim to stop spurious nodes initialising device templates as this step requires either the sending of a specific device describe property or user input to initialise the device.

@Andreas_Messerli yes thats one of the strengths of the inputs vs feeds seperation in that its possible to choose not to log an input as you describe, yet I agree it would be a nice feature to make these inputs available in dashboards for status widgets. Happy to accept pull requests to that effect.


(Trystan Lea) #10

I’ve been making some more progress on the device module integration development, so far I have added:

For anyone interested in trying and testing you can switch to the device-integration branch with the following steps:

cd /var/www/emoncms
git pull
got checkout device-integration

You will need to perform a database update if you dont already have the device module installed.

This development is still some way from being complete so I can only recommend trying it if you want to see an early preview.


(Dave) #11

@TrystanLea
I’ll tell you what would be handy is to have the rules on javascript so you can drag and drop them as on my supply CT I’ve got 100 processing rules.

Regards
Dave


(Trystan Lea) #12

Wow 100 processing rules!! the rules are all in javascript but drag and drop functionality is a bit more complicated, Id be happy to accept a code contribution if anyone fancies the challenge of implementing that. Im going to focus on getting the basics up and running first :slight_smile:

I’ve been doing some further work on this, primarily on the responsive design of the interface. But also on an extension to the device module that allows for the definition of a control device where the emoncms inputs are control parameters to which a device such as a smart plug can subscribe (this is for a project that we are working on where smart plugs are to be switched in response to excess available hydro power - blog to come soon). Latest commits are on the control_prototype branch:

git checkout control_prototype

(Dave) #13

I would love to input my 2p worth but I don’t have the necessary skills required.
I may have nearly 100 processing rules for my supply CT but I would say that’s needed to get all the useful information that I want to see.
For anybody that’s intrigued…

1	Log to feed	Node emonTx_Node9: Supply	(feed last value:336.00)		
	2	- input	Node emonTx_Diverter:Grid_Watts Sub	(input last value:158.00)		
	3	- input	Node emonTx_Node8:Commando_Skt Commando Socket	(input last value:-2.00)		
	4	Log to feed	Node emonTx_Node9: Corrected Supply	(feed last value:180.00)		
	5	Allow positive				
	6	Log to feed	Node emonTx_Node9: Supply Import	(feed last value:180.00)		
	7	Power to kWh	Node emonTx_Node9: Supply Import KWh	(feed last value:5580.80)		
	8	Power to kWh/d	Node emonTx_Node9: Supply Import KWh Day	(feed last value:9.64)		
	9	x	0.09375			
	10	Log to feed	Node emonTx_Node9: Daily Import Cost Transfer	(feed last value:16.88)		
	11	Power to kWh	Node emonTx_Node9: Total Import Cost	(feed last value:523.20)		
	12	Power to kWh/d	Node emonTx_Node9: Daily Import Cost	(feed last value:0.90)		
	13	x	0.001			
	14	Log to feed	Node emonTx_Node9: Supply CPH	(feed last value:0.02)		
	15	Reset to Original				
	16	- input	Node emonTx_Diverter:Grid_Watts Sub	(input last value:158.00)		
	17	- input	Node emonTx_Node8:Commando_Skt Commando Socket	(input last value:-2.00)		
	18	If <=, skip next	0			
	19	Reset to ZERO				
	20	+ feed	Node emonpi: Total PV Generation	(feed last value:842.00)		
	21	Power to kWh	Node emonTx_Node9: Solar Power Utilised KWh	(feed last value:2234.37)		
	22	Power to kWh/d	Node emonTx_Node9: Solar Power Utilised KWh Day	(feed last value:12.44)		
	23	x	0.09375			
	24	Power to kWh	Node emonTx_Node9: Solar Power Utilised Total Savings	(feed last value:209.47)		
	25	Power to kWh/d	Node emonTx_Node9: Solar Power Utilised Daily Savings	(feed last value:1.17)		
	26	x	0.001			
	27	Log to feed	Node emonTx_Node9: Solar Utilised CPH	(feed last value:0.08)		
	28	x	1000			
	29	+ feed	Node emonpi: Total FIT Transfer	(feed last value:119.71)		
	30	Power to kWh	Node emonTx_Node9: Total Savings Plus FIT	(feed last value:915.01)		
	31	Power to kWh/d	Node emonTx_Node9: Daily Savings Plus FIT	(feed last value:4.92)		
	32	- feed	Node emonTx_Node9: Daily Import Cost Transfer	(feed last value:16.88)		
	33	Power to kWh	Node emonTx_Node9: Total Revenue Minus Import Costs	(feed last value:391.88)		
	34	Power to kWh/d	Node emonTx_Node9: Daily Revenue Minus Import Costs	(feed last value:4.02)		
	35	- feed	Node emonTx_Node9: Gas Cost Transfer	(feed last value:215.50)		
	36	Power to kWh	Node emonTx_Node9: Total Revenue Minus Import and Gas Costs	(feed last value:-461.74)		
	37	Power to kWh/d	Node emonTx_Node9: Daily Revenue Minus Import and Gas Costs	(feed last value:0.53)		
	38	Reset to Original				
	39	- input	Node emonTx_Diverter:Grid_Watts Sub	(input last value:158.00)		
	40	- input	Node emonTx_Node8:Commando_Skt Commando Socket	(input last value:-2.00)		
	41	Allow negative				
	42	x	-1			
	43	Log to feed	Node emonTx_Node9: Supply Export	(feed last value:0.00)		
	44	Power to kWh	Node emonTx_Node9: Supply Export KWh	(feed last value:2705.08)		
	45	Power to kWh/d	Node emonTx_Node9: Supply Export KWh Day	(feed last value:13.64)		
	46	+ feed	Node emonTx_Diverter: Immersion	(feed last value:17.00)		
	47	If >, skip next	0			
	48	Reset to ZERO				
	49	Log to feed	Node emonTx_Node9: Avaliable Power Before Diversion	(feed last value:17.00)		
	50	Reset to Original				
	51	- input	Node emonTx_Diverter:Grid_Watts Sub	(input last value:158.00)		
	52	- input	Node emonTx_Node8:Commando_Skt Commando Socket	(input last value:-2.00)		
	53	+ feed	Node emonpi: Total PV Generation	(feed last value:842.00)		
	54	Log to feed	Node emonTx_Node9: Local Demand	(feed last value:1022.00)		
	55	Power to kWh	Node emonTx_Node9: Local Demand KWh	(feed last value:7814.17)		
	56	Power to kWh/d	Node emonTx_Node9: Local Demand KWh Day	(feed last value:22.08)		
	57	/ input	Node emonTx_Node9:vrms Voltage	(input last value:242.87)		
	58	Log to feed	Node emonTx_Node9: Local Demand in Amps	(feed last value:4.21)		
	59	Reset to Original				
	60	- input	Node emonTx_Diverter:Grid_Watts Sub	(input last value:158.00)		
	61	- input	Node emonTx_Node8:Commando_Skt Commando Socket	(input last value:-2.00)		
	62	/ input	Node emonTx_Node9:vrms Voltage	(input last value:242.87)		
	63	If >=, skip next	0			
	64	x	-1			
	65	Log to feed	Node emonTx_Node9: Supply Current	(feed last value:0.74)		
	66	Reset to Original				
	67	Allow positive				
	68	Power to kWh	Node emonTx_Node9: True Import KWh	(feed last value:7452.89)		
	69	Power to kWh/d	Node emonTx_Node9: True Import KWh Day	(feed last value:11.07)		
	70	Reset to Original				
	71	Allow negative				
	72	x	-1			
	73	Log to feed	Node emonTx_Node9: True Export KWh	(feed last value:0.00)		
	74	Power to kWh/d	Node emonTx_Node9: True Export KWh Day	(feed last value:12.56)		
	75	Reset to ZERO				
	76	+ feed	Node emonpi: Total PV Generation Daily KWh	(feed last value:26.11)		
	77	/ feed	Node emonTx_Node9: Local Demand KWh Day	(feed last value:22.08)		
	78	x	100			
	79	Log to feed	Node emonTx_Node9: Efficiency Percentage	(feed last value:118.24)		
	80	Reset to ZERO				
	81	+ feed	Node emonTx_Node9: Solar Power Utilised KWh Day	(feed last value:12.44)		
	82	/ feed	Node emonpi: Total PV Generation Daily KWh	(feed last value:26.11)		
	83	x	100			
	84	Log to feed	Node emonTx_Node9: Solar Utilisation Percentage	(feed last value:47.64)		
	85	Reset to Original				
	86	/ input	Node emonTx_Node9:vrms Voltage	(input last value:242.87)		
	87	If >=, skip next	0			
	88	x	-1			
	89	Log to feed	Node emonTx_Node9: True Supply Current	(feed last value:1.38)		
	90	Reset to Original				
	91	Allow positive				
	92	Power to kWh/d	Node emonTx_Node9: True Supply KWh Day	(feed last value:11.07)		
	93	Power to kWh/w	Node emonTx_Node9: True Supply KWh Week	(feed last value:90.86)		
	94	Power to kWh/m	Node emonTx_Node9: True Supply KWh Month	(feed last value:319.82)

Regards
Dave


(stuart) #14

I’d like the ability to prevent non-authorised nodes from being created automatically - as @Bramco mentions above you do get random node ids being created automatically which need clearing out now and again.

I still feel that were missing a trick here, surely the device should describe to emonCMS what it is and what outputs/inputs its going to provide/supports rather than the need for a manually applied template ?


(Adrian) #15

Hi guys,

first, as a small offtopic: I wanted to ask, if it is intentionally, that with the new UI (which I love btw) does not provide a notice box anymore with a hint towards the API link for empty tables. Neither for inputs, feeds, nor devices.

Further and mainly, I wanted to ask about the control-integration plan :wink:
We (here at ISC Konstanz) are currently revamping the results of an old smart grid project with several controllable devices like smart plugs, heat pumps or ev charge control systems. I had the idea for a while now, to provide an overall improved UI for all of this in the form of an emoncms app to switch those manually, explore their induvidual consumption (if possible) and provide a generic API for other algorithms or input processes to control things automatically.

Hence, it would be awesome to maybe coordinate some things… for us to contribute our work and make it compatible with your thoughts.
Like some sort of dynamic [module]_control.php class or to include a set_state($id, $state) function [module]_template.php class


(Trystan Lea) #16

@Adminde UI notice boxes are back in again now, along with all the other features that I had not added back into the interface e.g: virtual feeds, feed export. I will get back to you on the control implementation plan :slight_smile:

First I’ve been trying to work out how to bring in the changes developed as part of this effort including yours @Adminde into the main emoncms master branch.

I’ve gone full circle back to the idea of keeping the device module a separate module and then only updating the emoncms master branch with the required device module integration support which can be disabled, if the device module is not present.

I think I’m there now with both a pull request to the master branch:
https://github.com/emoncms/emoncms/pull/724

and a pull request to the emoncms device module (including your developments @Adminde)
https://github.com/emoncms/device/pull/7

If anyone already running the device module would like to test the device-integration branch of the device module and the device-support branch of emoncms, that would be much appreciated.

The steps are:

cd /var/www/emoncms
git pull
git checkout device-support
cd /var/www/emoncms/Modules/device
git pull
git checkout device-integration

To test the new input and feed interfaces, enable them by adding the line:

$ui_version_2 = true;

to settings.php.


(Trystan Lea) #17

Here’s a screenshot of the new device dialog developed by @Adminde for configuring devices:


Emoncms Graph module developments
(Trystan Lea) #18

Screenshot of new inputs interface (not sure about the rename to My Devices…):

Screenshot of new feeds interface:


(Chaveiro) #19

I just noticed this topic now. Looking good Trystan!
It’s all backward compatible right?
Will give it a try later.


(Glyn Hudson) #20

Nice, looking good! How about just ‘Devices’? I think this would be more intuitive than ‘Inputs’.