Development: Devices, Inputs and Feeds in emoncms

[TrystanLea]

@nchaveiro Im pretty sure its backwards compatible, in that a device list created prior to an update is maintained as well as device keys. Existing configured Inputs and Feeds created via the device module are unaffected as these are separate apart from initialisation.

While I have tested backwards compatibility briefly without obvious issues, I have only really started using the device module in earnest since making these new changes, so it would be great to get some feedback from anyone using the device module for some time.

[pb66 (Paul)]

I use the original device module so I’m interested in how this compares, I’m short on time at the moment but I will try to free up some time to set up another emoncms instance so I can try this out. sorry I can’t say when that might be yet.

[TrystanLea]

Thanks @pb66

[TrystanLea]

Hello All, I have added a readme with installation and basic usage documentation to the emoncms device module repository: https://github.com/emoncms/device/.

I think we are good to go now with merging the device-integration branch of the device module into the master branch of the device module. @nchaveiro, @pb66, @Adminde if you can think of anything else before I go forward with this please let me know. I will merge and write a forum post launching the new version on wednesday this coming week.

This is separate from merging the changes to emoncms core itself, these are just the device module changes as opposed to the input and feed list interface changes.

[pb66 (Paul)]

I still haven’t had a chance to play with this in any depth, but I have just setup a test server for another reason and installed the device module “device-integration” branch.

Am I right in thinking the “new device dialog by @Adminde” is the main change when just looking at the device module via emoncms/master (ie not the emoncms/device-support branch) ?

Are there plans to drop the old input page style edit button from the device page? seems we have 2 buttons for the same job, either to edit in-line or to open the new modal screen which might be a little confusing?

Why is the device key not in the modal screen?

Since we have this big modal box, could we please have a box that displays the selected device template so we can review which one we want? This means we can expand the comments in the templates and this becomes less of a dark art and more self explanatory.

Sub-dividing the left panel is a nice touch, however the actual files are all lumped together in “device/data”. Are you planning to add every individual file to a .gitignore or do users need to be doing that? Would it not be better to have folders in device/data so that the templates are grouped and users can keep their own device templates in a repo of thier own. In fact this might encourage other projects/developers to contribute drop in folders for their own product/devices. (Please see Custom and pre-defined template sets · Issue #5 · emoncms/device · GitHub)

Somewhere I think I recall reading that triggering the initiation multiple times would add missing inputs or feeds, is that correct? If so is the decision for this based on a single field eg feedname or a combination eg tag name and feedname combined so that multiple nodes with similar feeds can be created?

eg if a single emontx is created and then the template is changed and re run, only the additional (and changed?) feeds will be created, however if we setup a second emontx with a different node id all the feeds need to be created, even if there are already feeds of the same name.

[TrystanLea]

@pb66

Are there plans to drop the old input page style edit button from the device page? seems we have 2 buttons for the same job, either to edit in-line or to open the new modal screen which might be a little confusing?

Good point, happy to remove the in-line editing.

@pb66

Why is the device key not in the modal screen?

Sure can add it in.

@pb66

Since we have this big modal box, could we please have a box that displays the selected device template so we can review which one we want? This means we can expand the comments in the templates and this becomes less of a dark art and more self explanatory.

Im not sure that I follow you here?, the selected device template is displayed when selected.

@pb66

Would it not be better to have folders in device/data so that the templates are grouped and users can keep their own device templates in a repo of thier own.

Sure happy for the templates to be in subfolders.

@pb66

is the decision for this based on a single field eg feedname or a combination eg tag name and feedname combined

It is based on the feedname and feed tag combined. If a user changes a feed tag after initialisation and presses initialise again, a new feed with feedname and tag corresponding to the template will be created.

[pb66 (Paul)]

@TrystanLea

It is based on the feedname and feed tag combined.

That’s really good…

@TrystanLea

If a user changes a feed tag after initialisation and presses initialise again, a new feed with feedname and tag corresponding to the template will be created.

That’s not so quite so good…

Since the tag can also be defined in the device template.json, it might also be nice, further down the line to add the use of variables to the .json templates so that we could get a little creative with the naming and grouping.

@TrystanLea

Im not sure that I follow you here?, the selected device template is displayed when selected.

Oh. That’s not what I’m seeing. Here I have selected the emonTH template and aside from it’s entry in the list being a different colour there is no other indication,

What I’d like to see is a (collapsible?) text box area like the emonhub config or the emoncms/emonhub.log viewers that displays the actual template file like so

{
	"name": "EmonTH",
	"category": "OpenEnergyMonitor",
	"group": "Temperature & Humidity",
    "description": "Automatic inputs and feeds creation for emonTH device.",
    "inputs": [
        {
            "name": "temperature",
            "description": "Temperature C",
            "processList": [
            {
                "process": "1",
                "arguments": {"type": "ProcessArg::FEEDID", "value": "emonth_temperature" }
            }
            ]
        },
        {
            "name": "humidity",
            "description": "Humidity Rh%",
            "processList": [
            {
                "process": "1",
                "arguments": {"type": "ProcessArg::FEEDID", "value": "emonth_humidity" }
            }
            ]
        }
    ],

    "feeds": [
        {
            "name": "emonth_temperature",
            "type": "DataType::REALTIME",
            "engine": "Engine::PHPFINA",
            "interval": "60"
        },
        {
            "name": "emonth_humidity",
            "type": "DataType::REALTIME",
            "engine": "Engine::PHPFINA",
            "interval": "60"
        }
    ]
}

In fact perhaps even (in-time) a “dry-run” could result is a “you are about to create these feeds and inputs - click ok to continue or exit to abort” type list for confirmation, this list would also highlight any clashes (ie feeds it isn’t going to create) and make it obvious to users that feeds are being created again because the tag has been altered.

On that note of a changed tag. if a user creates a device and then changes the tag on one feed, when the initialisation is re-run and a new feed is created, does the original input processing get changed to post to the new feed or is it the new feed just redundant (the latter I guess).

[TrystanLea]

Thanks Paul.

The selected configuration is stated just below the title Configuration, in your case as you selected EmonTH the text says: Automatic inputs and feeds creation for emonTH device.

Adminde sent me a pull request yesterday with a couple of changes, see : Device integration by adminde · Pull Request #8 · emoncms/device · GitHub. I’ve merged these in if you wish to run git pull again to bring these changes in.

@Adminde I think you are more familiar that me with the device initialisation code, how hard might it be to show the user a dry-run as Paul is suggesting?

@pb66

Since the tag can also be defined in the device template.json, it might also be nice, further down the line to add the use of variables to the .json templates so that we could get a little creative with the naming and grouping.

The user can be creative with the tag grouping by changing setting the ‘node’ field appropriately. E.g creating a device with the node set to emonth5 will create two feeds (emonth_temperature and emonth_humidity) under the tag emonth5

[pb66 (Paul)]

@TrystanLea

The selected configuration is stated just below the title Configuration, in your case as you selected EmonTH the text says: Automatic inputs and feeds creation for emonTH device.

So it does, I had missed that, but that isn’t what I was asking about, I can already see I have selected that template by the left hand panel. What I would like to see is the template itself.

I have been using the device module for a considerable time and I have some quite complex templates. That’s fine on first use for a new account and stock device. But It can nerve racking when blindly adding devices to an existing set up, I tend to have a bash console open to view the template whilst using this to double check I’m getting what I was expecting.

@TrystanLea

The user can be creative with the tag grouping by changing setting the ‘node’ field appropriately. E.g creating a device with the node set to emonth5 will create two feeds (emonth_temperature and emonth_humidity) under the tag emonth5

We may have different opinions on what “creative” is, being able to edit the one label that gets applied to every input and feed isn’t quite what I would call “creative”. But as I said “down the line” so this isn’t a stumbling block at all, the editable nodename/tag is definitely good enough for now as it allows multiple feeds of the same name.

I’ve just had a quick look at the PR you linked and I have some concerns about the fact that processlists are over written by default, that is where and personalisation is likely to occur. users may start off with a default emonTx and add hours of work getting the processing right and a simple click of the initialise button and it’s all gone. Previously it would no create/edit an existing input.

I think it is a great addition to add the ability to update inputs and feeds via the device module, but this particular item (overwriting processlist) is a bit of a concern, perhaps if we did have a dry-run type confirmation screen, it would at least alert the user(if the output isn’t too busy to read). Perhaps something like adding a allow/disallow check box to each proposed change (all checked by default) so a user could deseclect the changes to a cherished processlist before accepting the dry-run?

Think of a simple case of a CT being orientated the wrong way and corrected with a x-1, even being aware the processlist has been over written might prompt a user to re-edit the processlist before the feed data is ruined.

I will pull in the changes, but it looks like they are bug fixes rather than noticeable changes.

[Adminde (Adrian)]

First of all: yes, the last pull request contained only some few bugfixes
One of the bugfixes results in a quite noticable change though, as a device will be initialized immediately at device creation now. I did not see any reason to wait with the initialization to be honest… but I get the feeling that the device module may be used differently than I anticipated^^ Which is the reason why the current version rewrites all processlists to the template default.
In the use-case I had mainly in mind, a device will be created and initialized just once - and a re-initialization would then only be necessary if I experimented too much with my processes or feeds and something went terribly wrong^^

With the latest pull-request I added at least a small warning-line regarding the processList reset at the init-dialog.

Regarding the dry-run: nothing is impossible and most of the necessary code can be copied but currently there is only one single api route ./device/template/init.json?id=1 with the device id as sole parameter to initialize the device… as the device type etc. can be fetched backend-wise.
A collapseable text-box would be much easier at first, maybe even parsed or “nicely” displayed JSON like chrome or firefox does it.

To get a little more creative with the input and feed names in terms of e.g. variables in the template, I’d be strongly in favor^^
What I currently do is quite patchy/not very creative as (in my device-control branch) I just added a “prefix”: <name|node> option in the templates, which will be taken into account for all inputs and feeds if the keyvalue is present. For example the feeds with the names “use” and “use_kwh” in the template would be created as “foo_use” and “foo_use_kwh” if the device name was chosen “foo” with “prefix”:“name”.
If we can agree on something nice and sustainable for future developments, I’d gladly implement it

[pb66 (Paul)]

Hi Adrian

It sounds like we have been using the device module in similar ways. Prior to now, I thought of it as a very powerful tool but not overly user friendly, the sort of thing I would recommend a dev/admin using for rapid replication of complex processing, but not really ready for the casual user, mainly because the learning curve to only use it once or twice was a bit steep.

I have a load of templates and quite often they get edited for specific jobs as it’s easier to edit the template before each run than manually add all the inputs, processing and feeds the original way or even to edit after device creation.

I also have a bash script with some default.json files that I can just edit a couple of variables and it uses sed routines to change various keywords I have placed throughout my base templates, it then spits out a new single use json file for use in the device module. It sounds odd I know to produce single use templates but it really cuts my workload down no end, alas I know much of what I use is bespoke so I do not expect the device module to accommodate all my needs.

Personally, I would prefer the devices were not initialized as they were created. But it’s a minor point and may need rethinking anyway if there is to be a dry-run type confirmation.

Regards the API, there are many options, First is that users/scripts using the API directly will probably have greater confidence than the casual GUI user so there could be a simple assumed “force yes” arg to skip the dry-run. Or a second API can be used so the emoncms webpage can call “dryrun” first and “init” if confirmed, I imagined these to be the same process, where the expected result is output to screen rather than actually completing as I assumed the device module should check it can do the full initialization before starting it and fail part way through, although I’m guessing the new method should reduce the chance of it failing, previously it could create half the feeds before finding a duplicate and aborting, often ( I speak from experience) this could be caused by something as simple as a typo, the feed “power1” was created then the device initialization failed when it couldn’t find feed “pwer1” when incorrectly defined in the input processing, IMO any attempt to create this device (via API or GUI) should complain/fail before creating any feeds so IMO it should not create the first feed until it has done a “dry-run” regardless of whether user confirmation is requested, so pausing at the end of the first pass for the user to confirm would be a small addition in this instance.

The thing that causes me the biggest headache is cross device totaling, I’m guessing that with this new version, because of the “just add what doesn’t already exist” approach. I’m hoping that I can add to that a facility to define feeds by tag and feedname in the input processing so I can create “summary” nodes, in which I can define 3x “Power1” feeds (tagged as L1, L2 & L3) and define a process list of +feed “L1: Power1”, +feed “L2: Power1”. +feed “L3: Power1”, logtofeed “Summary:PowerTotal” (not sure that would be acceptable syntax, but you get the gist), Then (hopefully) when the L1, L2 and L3 nodes are created they should use those 3 existing Power1 feeds. Currently I expect the tag for any feeds named in a processlist to be assumed to be equal to the nodename of that device, so defining +feed Power1 three times isn’t going to have the desired effect, but that is pretty much what I do now, I add 3x +feed “dummy_feed” and then go in afterwards and edit the target feed by selecting from the dropdown on the inputprocessing editing page.

I’m happy to help with any of this if I can as I do use it quite a lot, my php isn’t great but I can muddle though.

PS this isn’t a list of demands, just throwing idea’s out there for consideration, if my php was better and my diary slimmer, I would happily give it a go on my own.

[pb66 (Paul)]

So this morning I need to set up another user account, 4 devices on my live server which was fully updated last night so it is emoncms master 9.8.11 but there is no versioning in the device module yet, but it is fully updated from master branch as of now (10:25 16/11/2017) .

When I used a template I used before the screens etc were all familiar (as not running device-integration branch on live server) and all appeared to go well, except… No processlists were created! Feeds and inputs all seem to be present, but no processlists at all.

Since I have a test server i’ve just set up, I thought I could try that too. That is also fully updated but is running the device-integration branch of the device module and emoncms itself is the master branch. I was able to create a device, all looked ok until I tried to initialise it, I got this error message

as you can see, not much to go on and a quick check found no inputs or feeds created, looking at the device page again I could see there was no “type”

returning into the modal device setup page again, I can select the template and edit the fields, but when saved it does not retain the type. I have confirmed this in phpmyadmiin.

Since this was using the “device-integration” branch of the device module, I thought I would try switching the branch and try the master branch with a default template (in case there was something wrong with my template) on a totally stock, fully updated, master branch only install.

This time, familiar actions and responses all as expected and the emonTH was created, but without any processlists.

Now I’m a bit stuck, I have a deadline to meet and cannot currently use the device module, any clues how to fix? device module isn’t tagged so I cannot easily roll back. Right now I do not now which is the faster option, manually creating all the processing by hand, trying to locate/fix the issue or to try and roll back the repos to a known good point by trying to extract files from an image backup.

Current situation for the account/user I’m working on is there are no processlists and even if I find a fix, I cannot re-use the “master” device module without deleting all the created inputs and feeds. I also currently cannot use the “device-integration” branch as that has no “type” for initiation or re-initiation.

I’m gonna move on to another part of that same project for now, in hope I get a reply with some assistance here, otherwise I think I will have to start manually adding the horrendously long processlists.

Any suggestions please @TrystanLea or @Adminde

[pb66 (Paul)]

Just tried another quick test, I have rolled back my test server to emoncms 9.8.8 and switched back to the device-integration branch of the device module. Re-running the initialisation again for the emonTH did not add the processlists.

The roll back to 9.8.8 suggests this isn’t anything to do with the security fixes by @TrystanLea yesterday.

Since the device now has a type from the first run being completed by the master branch device module, re-running the initialization with the "device-integration branch device module should have added the missing processlists but it hasn’t, this suggests the failure to save the “type” is unrelated to this issue.

Clicking the new edit button from the device page brings up the modal page with the emonTH already highlighted to the new branch is definitely able to see the type saved by the old branch.

[Adminde (Adrian)]

Hi @pb66

sorry for that

I tested all my stuff with the emoncms/device-support branch and yesterday @TrystanLea merged quite some changes, including the group-support or at least something related to that.

This changed the $input/$feed->set_processlist function, which broke the whole device module. I created a pullrequest - maybe this fixes your problem?

[TrystanLea]

Thanks @Adminde, I updated the set_processlist function yesterday to sort the issue here: Important Emoncms security update for multi-user emoncms installations V9.8.11 as the processlists where not being validated for input and feed ownership permissions.

@pb66 I will make a patch for the master branch of the device module for you, @Adminde has fixed it in the device-integration branch

[pb66 (Paul)]

Thanks guys but I have already tried backdateing to 9.8.8 emoncms and the problem still exists, so are you saying there were changes made after 9.8.8 that effectively got undone yesterday?

I’m guessing I will have to use the “device-integration” branch if I’m to recover this situation as the original “master” device module does not do re runs unless I delete all the feeds already created (and there are a lot).

As it stands if there’s only a fix for creating the processlists I will still need to create devices with the master branch and only update devices with the “device-integration” branch.

[TrystanLea]

@pb66 can you not update to latest emoncms master and then I will patch the device master branch to fix the set_processlist function?

[TrystanLea]

I just tried backdateing emoncms master to commit 778bd22cd949bd80f6238bdde1a5c466149f3f51 - your patch from a couple of days ago @pb66 with the device module master branch and device initialisation worked fine?

[pb66 (Paul)]

@TrystanLea

@pb66 can you not update to latest emoncms master and then I will patch the device master branch to fix the set_processlist function?

That’s not the point, if the problem exists in 9.8.8 with the updated device module (either branch) it isn’t something you did last night that’s causing the issue.

My live server IS on the latest emoncms but I’m not doing any more experimenting on there, if anything I will be rolling everything back. I only updated because of your security fix, prior to that everything was fine with v9.8.8 and an out of date device module, everything worked.

I will first try it on the test server, That was originally on 9.8.10 until I rolled it back to 9.8.8, again both of these were pre-security fix updates to 9.8.11.

@Adminde

I tested all my stuff with the emoncms/device-support branch

Can we please be sure the device module “device-integration” branch works with both the master and stable branches of emoncms before meging to the master device module branch? And can we start using a version number?

[TrystanLea]

I did add this change in April to ensure that inputs where not created if there was an existing input with the same nodeid and name: https://github.com/emoncms/device/commit/69ea64343d7e94e53dd9f7e1100f402a587c7d4c - I dont know if your version of the device module was prior to that?