Development: Devices, Inputs and Feeds in emoncms

Wow 100 processing rules!! the rules are all in javascript but drag and drop functionality is a bit more complicated, Id be happy to accept a code contribution if anyone fancies the challenge of implementing that. Im going to focus on getting the basics up and running first :slight_smile:

I’ve been doing some further work on this, primarily on the responsive design of the interface. But also on an extension to the device module that allows for the definition of a control device where the emoncms inputs are control parameters to which a device such as a smart plug can subscribe (this is for a project that we are working on where smart plugs are to be switched in response to excess available hydro power - blog to come soon). Latest commits are on the control_prototype branch:

git checkout control_prototype

I would love to input my 2p worth but I don’t have the necessary skills required.
I may have nearly 100 processing rules for my supply CT but I would say that’s needed to get all the useful information that I want to see.
For anybody that’s intrigued…

1	Log to feed	Node emonTx_Node9: Supply	(feed last value:336.00)		
	2	- input	Node emonTx_Diverter:Grid_Watts Sub	(input last value:158.00)		
	3	- input	Node emonTx_Node8:Commando_Skt Commando Socket	(input last value:-2.00)		
	4	Log to feed	Node emonTx_Node9: Corrected Supply	(feed last value:180.00)		
	5	Allow positive				
	6	Log to feed	Node emonTx_Node9: Supply Import	(feed last value:180.00)		
	7	Power to kWh	Node emonTx_Node9: Supply Import KWh	(feed last value:5580.80)		
	8	Power to kWh/d	Node emonTx_Node9: Supply Import KWh Day	(feed last value:9.64)		
	9	x	0.09375			
	10	Log to feed	Node emonTx_Node9: Daily Import Cost Transfer	(feed last value:16.88)		
	11	Power to kWh	Node emonTx_Node9: Total Import Cost	(feed last value:523.20)		
	12	Power to kWh/d	Node emonTx_Node9: Daily Import Cost	(feed last value:0.90)		
	13	x	0.001			
	14	Log to feed	Node emonTx_Node9: Supply CPH	(feed last value:0.02)		
	15	Reset to Original				
	16	- input	Node emonTx_Diverter:Grid_Watts Sub	(input last value:158.00)		
	17	- input	Node emonTx_Node8:Commando_Skt Commando Socket	(input last value:-2.00)		
	18	If <=, skip next	0			
	19	Reset to ZERO				
	20	+ feed	Node emonpi: Total PV Generation	(feed last value:842.00)		
	21	Power to kWh	Node emonTx_Node9: Solar Power Utilised KWh	(feed last value:2234.37)		
	22	Power to kWh/d	Node emonTx_Node9: Solar Power Utilised KWh Day	(feed last value:12.44)		
	23	x	0.09375			
	24	Power to kWh	Node emonTx_Node9: Solar Power Utilised Total Savings	(feed last value:209.47)		
	25	Power to kWh/d	Node emonTx_Node9: Solar Power Utilised Daily Savings	(feed last value:1.17)		
	26	x	0.001			
	27	Log to feed	Node emonTx_Node9: Solar Utilised CPH	(feed last value:0.08)		
	28	x	1000			
	29	+ feed	Node emonpi: Total FIT Transfer	(feed last value:119.71)		
	30	Power to kWh	Node emonTx_Node9: Total Savings Plus FIT	(feed last value:915.01)		
	31	Power to kWh/d	Node emonTx_Node9: Daily Savings Plus FIT	(feed last value:4.92)		
	32	- feed	Node emonTx_Node9: Daily Import Cost Transfer	(feed last value:16.88)		
	33	Power to kWh	Node emonTx_Node9: Total Revenue Minus Import Costs	(feed last value:391.88)		
	34	Power to kWh/d	Node emonTx_Node9: Daily Revenue Minus Import Costs	(feed last value:4.02)		
	35	- feed	Node emonTx_Node9: Gas Cost Transfer	(feed last value:215.50)		
	36	Power to kWh	Node emonTx_Node9: Total Revenue Minus Import and Gas Costs	(feed last value:-461.74)		
	37	Power to kWh/d	Node emonTx_Node9: Daily Revenue Minus Import and Gas Costs	(feed last value:0.53)		
	38	Reset to Original				
	39	- input	Node emonTx_Diverter:Grid_Watts Sub	(input last value:158.00)		
	40	- input	Node emonTx_Node8:Commando_Skt Commando Socket	(input last value:-2.00)		
	41	Allow negative				
	42	x	-1			
	43	Log to feed	Node emonTx_Node9: Supply Export	(feed last value:0.00)		
	44	Power to kWh	Node emonTx_Node9: Supply Export KWh	(feed last value:2705.08)		
	45	Power to kWh/d	Node emonTx_Node9: Supply Export KWh Day	(feed last value:13.64)		
	46	+ feed	Node emonTx_Diverter: Immersion	(feed last value:17.00)		
	47	If >, skip next	0			
	48	Reset to ZERO				
	49	Log to feed	Node emonTx_Node9: Avaliable Power Before Diversion	(feed last value:17.00)		
	50	Reset to Original				
	51	- input	Node emonTx_Diverter:Grid_Watts Sub	(input last value:158.00)		
	52	- input	Node emonTx_Node8:Commando_Skt Commando Socket	(input last value:-2.00)		
	53	+ feed	Node emonpi: Total PV Generation	(feed last value:842.00)		
	54	Log to feed	Node emonTx_Node9: Local Demand	(feed last value:1022.00)		
	55	Power to kWh	Node emonTx_Node9: Local Demand KWh	(feed last value:7814.17)		
	56	Power to kWh/d	Node emonTx_Node9: Local Demand KWh Day	(feed last value:22.08)		
	57	/ input	Node emonTx_Node9:vrms Voltage	(input last value:242.87)		
	58	Log to feed	Node emonTx_Node9: Local Demand in Amps	(feed last value:4.21)		
	59	Reset to Original				
	60	- input	Node emonTx_Diverter:Grid_Watts Sub	(input last value:158.00)		
	61	- input	Node emonTx_Node8:Commando_Skt Commando Socket	(input last value:-2.00)		
	62	/ input	Node emonTx_Node9:vrms Voltage	(input last value:242.87)		
	63	If >=, skip next	0			
	64	x	-1			
	65	Log to feed	Node emonTx_Node9: Supply Current	(feed last value:0.74)		
	66	Reset to Original				
	67	Allow positive				
	68	Power to kWh	Node emonTx_Node9: True Import KWh	(feed last value:7452.89)		
	69	Power to kWh/d	Node emonTx_Node9: True Import KWh Day	(feed last value:11.07)		
	70	Reset to Original				
	71	Allow negative				
	72	x	-1			
	73	Log to feed	Node emonTx_Node9: True Export KWh	(feed last value:0.00)		
	74	Power to kWh/d	Node emonTx_Node9: True Export KWh Day	(feed last value:12.56)		
	75	Reset to ZERO				
	76	+ feed	Node emonpi: Total PV Generation Daily KWh	(feed last value:26.11)		
	77	/ feed	Node emonTx_Node9: Local Demand KWh Day	(feed last value:22.08)		
	78	x	100			
	79	Log to feed	Node emonTx_Node9: Efficiency Percentage	(feed last value:118.24)		
	80	Reset to ZERO				
	81	+ feed	Node emonTx_Node9: Solar Power Utilised KWh Day	(feed last value:12.44)		
	82	/ feed	Node emonpi: Total PV Generation Daily KWh	(feed last value:26.11)		
	83	x	100			
	84	Log to feed	Node emonTx_Node9: Solar Utilisation Percentage	(feed last value:47.64)		
	85	Reset to Original				
	86	/ input	Node emonTx_Node9:vrms Voltage	(input last value:242.87)		
	87	If >=, skip next	0			
	88	x	-1			
	89	Log to feed	Node emonTx_Node9: True Supply Current	(feed last value:1.38)		
	90	Reset to Original				
	91	Allow positive				
	92	Power to kWh/d	Node emonTx_Node9: True Supply KWh Day	(feed last value:11.07)		
	93	Power to kWh/w	Node emonTx_Node9: True Supply KWh Week	(feed last value:90.86)		
	94	Power to kWh/m	Node emonTx_Node9: True Supply KWh Month	(feed last value:319.82)

Regards
Dave

I’d like the ability to prevent non-authorised nodes from being created automatically - as @Bramco mentions above you do get random node ids being created automatically which need clearing out now and again.

I still feel that were missing a trick here, surely the device should describe to emonCMS what it is and what outputs/inputs its going to provide/supports rather than the need for a manually applied template ?

Hi guys,

first, as a small offtopic: I wanted to ask, if it is intentionally, that with the new UI (which I love btw) does not provide a notice box anymore with a hint towards the API link for empty tables. Neither for inputs, feeds, nor devices.

Further and mainly, I wanted to ask about the control-integration plan :wink:
We (here at ISC Konstanz) are currently revamping the results of an old smart grid project with several controllable devices like smart plugs, heat pumps or ev charge control systems. I had the idea for a while now, to provide an overall improved UI for all of this in the form of an emoncms app to switch those manually, explore their induvidual consumption (if possible) and provide a generic API for other algorithms or input processes to control things automatically.

Hence, it would be awesome to maybe coordinate some things… for us to contribute our work and make it compatible with your thoughts.
Like some sort of dynamic [module]_control.php class or to include a set_state($id, $state) function [module]_template.php class

@Adminde UI notice boxes are back in again now, along with all the other features that I had not added back into the interface e.g: virtual feeds, feed export. I will get back to you on the control implementation plan :slight_smile:

First I’ve been trying to work out how to bring in the changes developed as part of this effort including yours @Adminde into the main emoncms master branch.

I’ve gone full circle back to the idea of keeping the device module a separate module and then only updating the emoncms master branch with the required device module integration support which can be disabled, if the device module is not present.

I think I’m there now with both a pull request to the master branch:
https://github.com/emoncms/emoncms/pull/724

and a pull request to the emoncms device module (including your developments @Adminde)
https://github.com/emoncms/device/pull/7

If anyone already running the device module would like to test the device-integration branch of the device module and the device-support branch of emoncms, that would be much appreciated.

The steps are:

cd /var/www/emoncms
git pull
git checkout device-support
cd /var/www/emoncms/Modules/device
git pull
git checkout device-integration

To test the new input and feed interfaces, enable them by adding the line:

$ui_version_2 = true;

to settings.php.

Here’s a screenshot of the new device dialog developed by @Adminde for configuring devices:

Screenshot of new inputs interface (not sure about the rename to My Devices…):

Screenshot of new feeds interface:

1 Like

I just noticed this topic now. Looking good Trystan!
It’s all backward compatible right?
Will give it a try later.

1 Like

Nice, looking good! How about just ‘Devices’? I think this would be more intuitive than ‘Inputs’.

@nchaveiro Im pretty sure its backwards compatible, in that a device list created prior to an update is maintained as well as device keys. Existing configured Inputs and Feeds created via the device module are unaffected as these are separate apart from initialisation.

While I have tested backwards compatibility briefly without obvious issues, I have only really started using the device module in earnest since making these new changes, so it would be great to get some feedback from anyone using the device module for some time.

I use the original device module so I’m interested in how this compares, I’m short on time at the moment but I will try to free up some time to set up another emoncms instance so I can try this out. sorry I can’t say when that might be yet.

1 Like

Thanks @pb66

Hello All, I have added a readme with installation and basic usage documentation to the emoncms device module repository: https://github.com/emoncms/device/.

I think we are good to go now with merging the device-integration branch of the device module into the master branch of the device module. @nchaveiro, @pb66, @Adminde if you can think of anything else before I go forward with this please let me know. I will merge and write a forum post launching the new version on wednesday this coming week.

This is separate from merging the changes to emoncms core itself, these are just the device module changes as opposed to the input and feed list interface changes.

I still haven’t had a chance to play with this in any depth, but I have just setup a test server for another reason and installed the device module “device-integration” branch.

Am I right in thinking the “new device dialog by @Adminde” is the main change when just looking at the device module via emoncms/master (ie not the emoncms/device-support branch) ?

Are there plans to drop the old input page style edit button from the device page? seems we have 2 buttons for the same job, either to edit in-line or to open the new modal screen which might be a little confusing?

Why is the device key not in the modal screen?

Since we have this big modal box, could we please have a box that displays the selected device template so we can review which one we want? This means we can expand the comments in the templates and this becomes less of a dark art and more self explanatory.

Sub-dividing the left panel is a nice touch, however the actual files are all lumped together in “device/data”. Are you planning to add every individual file to a .gitignore or do users need to be doing that? Would it not be better to have folders in device/data so that the templates are grouped and users can keep their own device templates in a repo of thier own. In fact this might encourage other projects/developers to contribute drop in folders for their own product/devices. (Please see Custom and pre-defined template sets · Issue #5 · emoncms/device · GitHub)

Somewhere I think I recall reading that triggering the initiation multiple times would add missing inputs or feeds, is that correct? If so is the decision for this based on a single field eg feedname or a combination eg tag name and feedname combined so that multiple nodes with similar feeds can be created?

eg if a single emontx is created and then the template is changed and re run, only the additional (and changed?) feeds will be created, however if we setup a second emontx with a different node id all the feeds need to be created, even if there are already feeds of the same name.

Good point, happy to remove the in-line editing.

Sure can add it in.

Im not sure that I follow you here?, the selected device template is displayed when selected.

Sure happy for the templates to be in subfolders.

It is based on the feedname and feed tag combined. If a user changes a feed tag after initialisation and presses initialise again, a new feed with feedname and tag corresponding to the template will be created.

That’s really good…

That’s not so quite so good…

Since the tag can also be defined in the device template.json, it might also be nice, further down the line to add the use of variables to the .json templates so that we could get a little creative with the naming and grouping.

Oh. That’s not what I’m seeing. Here I have selected the emonTH template and aside from it’s entry in the list being a different colour there is no other indication,

What I’d like to see is a (collapsible?) text box area like the emonhub config or the emoncms/emonhub.log viewers that displays the actual template file like so

{
	"name": "EmonTH",
	"category": "OpenEnergyMonitor",
	"group": "Temperature & Humidity",
    "description": "Automatic inputs and feeds creation for emonTH device.",
    "inputs": [
        {
            "name": "temperature",
            "description": "Temperature C",
            "processList": [
            {
                "process": "1",
                "arguments": {"type": "ProcessArg::FEEDID", "value": "emonth_temperature" }
            }
            ]
        },
        {
            "name": "humidity",
            "description": "Humidity Rh%",
            "processList": [
            {
                "process": "1",
                "arguments": {"type": "ProcessArg::FEEDID", "value": "emonth_humidity" }
            }
            ]
        }
    ],

    "feeds": [
        {
            "name": "emonth_temperature",
            "type": "DataType::REALTIME",
            "engine": "Engine::PHPFINA",
            "interval": "60"
        },
        {
            "name": "emonth_humidity",
            "type": "DataType::REALTIME",
            "engine": "Engine::PHPFINA",
            "interval": "60"
        }
    ]
}

In fact perhaps even (in-time) a “dry-run” could result is a “you are about to create these feeds and inputs - click ok to continue or exit to abort” type list for confirmation, this list would also highlight any clashes (ie feeds it isn’t going to create) and make it obvious to users that feeds are being created again because the tag has been altered.

On that note of a changed tag. if a user creates a device and then changes the tag on one feed, when the initialisation is re-run and a new feed is created, does the original input processing get changed to post to the new feed or is it the new feed just redundant (the latter I guess).

Thanks Paul.

The selected configuration is stated just below the title Configuration, in your case as you selected EmonTH the text says: Automatic inputs and feeds creation for emonTH device.

Adminde sent me a pull request yesterday with a couple of changes, see : Device integration by adminde · Pull Request #8 · emoncms/device · GitHub. I’ve merged these in if you wish to run git pull again to bring these changes in.

@Adminde I think you are more familiar that me with the device initialisation code, how hard might it be to show the user a dry-run as Paul is suggesting?

The user can be creative with the tag grouping by changing setting the ‘node’ field appropriately. E.g creating a device with the node set to emonth5 will create two feeds (emonth_temperature and emonth_humidity) under the tag emonth5

So it does, I had missed that, but that isn’t what I was asking about, I can already see I have selected that template by the left hand panel. What I would like to see is the template itself.

I have been using the device module for a considerable time and I have some quite complex templates. That’s fine on first use for a new account and stock device. But It can nerve racking when blindly adding devices to an existing set up, I tend to have a bash console open to view the template whilst using this to double check I’m getting what I was expecting.

We may have different opinions on what “creative” is, being able to edit the one label that gets applied to every input and feed isn’t quite what I would call “creative”. But as I said “down the line” so this isn’t a stumbling block at all, the editable nodename/tag is definitely good enough for now as it allows multiple feeds of the same name.

I’ve just had a quick look at the PR you linked and I have some concerns about the fact that processlists are over written by default, that is where and personalisation is likely to occur. users may start off with a default emonTx and add hours of work getting the processing right and a simple click of the initialise button and it’s all gone. Previously it would no create/edit an existing input.

I think it is a great addition to add the ability to update inputs and feeds via the device module, but this particular item (overwriting processlist) is a bit of a concern, perhaps if we did have a dry-run type confirmation screen, it would at least alert the user(if the output isn’t too busy to read). Perhaps something like adding a allow/disallow check box to each proposed change (all checked by default) so a user could deseclect the changes to a cherished processlist before accepting the dry-run?

Think of a simple case of a CT being orientated the wrong way and corrected with a x-1, even being aware the processlist has been over written might prompt a user to re-edit the processlist before the feed data is ruined.

I will pull in the changes, but it looks like they are bug fixes rather than noticeable changes.

First of all: yes, the last pull request contained only some few bugfixes :slight_smile:
One of the bugfixes results in a quite noticable change though, as a device will be initialized immediately at device creation now. I did not see any reason to wait with the initialization to be honest… but I get the feeling that the device module may be used differently than I anticipated^^ Which is the reason why the current version rewrites all processlists to the template default.
In the use-case I had mainly in mind, a device will be created and initialized just once - and a re-initialization would then only be necessary if I experimented too much with my processes or feeds and something went terribly wrong^^

With the latest pull-request I added at least a small warning-line regarding the processList reset at the init-dialog.

Regarding the dry-run: nothing is impossible :wink: and most of the necessary code can be copied but currently there is only one single api route ./device/template/init.json?id=1 with the device id as sole parameter to initialize the device… as the device type etc. can be fetched backend-wise.
A collapseable text-box would be much easier at first, maybe even parsed or “nicely” displayed JSON like chrome or firefox does it.

To get a little more creative with the input and feed names in terms of e.g. variables in the template, I’d be strongly in favor^^
What I currently do is quite patchy/not very creative as (in my device-control branch) I just added a “prefix”: <name|node> option in the templates, which will be taken into account for all inputs and feeds if the keyvalue is present. For example the feeds with the names “use” and “use_kwh” in the template would be created as “foo_use” and “foo_use_kwh” if the device name was chosen “foo” with “prefix”:“name”.
If we can agree on something nice and sustainable for future developments, I’d gladly implement it :wink:

Hi Adrian

It sounds like we have been using the device module in similar ways. Prior to now, I thought of it as a very powerful tool but not overly user friendly, the sort of thing I would recommend a dev/admin using for rapid replication of complex processing, but not really ready for the casual user, mainly because the learning curve to only use it once or twice was a bit steep.

I have a load of templates and quite often they get edited for specific jobs as it’s easier to edit the template before each run than manually add all the inputs, processing and feeds the original way or even to edit after device creation.

I also have a bash script with some default.json files that I can just edit a couple of variables and it uses sed routines to change various keywords I have placed throughout my base templates, it then spits out a new single use json file for use in the device module. It sounds odd I know to produce single use templates but it really cuts my workload down no end, alas I know much of what I use is bespoke so I do not expect the device module to accommodate all my needs.

Personally, I would prefer the devices were not initialized as they were created. But it’s a minor point and may need rethinking anyway if there is to be a dry-run type confirmation.

Regards the API, there are many options, First is that users/scripts using the API directly will probably have greater confidence than the casual GUI user so there could be a simple assumed “force yes” arg to skip the dry-run. Or a second API can be used so the emoncms webpage can call “dryrun” first and “init” if confirmed, I imagined these to be the same process, where the expected result is output to screen rather than actually completing as I assumed the device module should check it can do the full initialization before starting it and fail part way through, although I’m guessing the new method should reduce the chance of it failing, previously it could create half the feeds before finding a duplicate and aborting, often ( I speak from experience) this could be caused by something as simple as a typo, the feed “power1” was created then the device initialization failed when it couldn’t find feed “pwer1” when incorrectly defined in the input processing, IMO any attempt to create this device (via API or GUI) should complain/fail before creating any feeds so IMO it should not create the first feed until it has done a “dry-run” regardless of whether user confirmation is requested, so pausing at the end of the first pass for the user to confirm would be a small addition in this instance.

The thing that causes me the biggest headache is cross device totaling, I’m guessing that with this new version, because of the “just add what doesn’t already exist” approach. I’m hoping that I can add to that a facility to define feeds by tag and feedname in the input processing so I can create “summary” nodes, in which I can define 3x “Power1” feeds (tagged as L1, L2 & L3) and define a process list of +feed “L1: Power1”, +feed “L2: Power1”. +feed “L3: Power1”, logtofeed “Summary:PowerTotal” (not sure that would be acceptable syntax, but you get the gist), Then (hopefully) when the L1, L2 and L3 nodes are created they should use those 3 existing Power1 feeds. Currently I expect the tag for any feeds named in a processlist to be assumed to be equal to the nodename of that device, so defining +feed Power1 three times isn’t going to have the desired effect, but that is pretty much what I do now, I add 3x +feed “dummy_feed” and then go in afterwards and edit the target feed by selecting from the dropdown on the inputprocessing editing page.

I’m happy to help with any of this if I can as I do use it quite a lot, my php isn’t great but I can muddle though.

PS this isn’t a list of demands, just throwing idea’s out there for consideration, if my php was better and my diary slimmer, I would happily give it a go on my own.