Development: Devices, Inputs and Feeds in emoncms

A key advantage of redis over mysql is that it reduces your disk IO significantly, emoncms will spent much less time waiting on the disk.

Alongside introducing redis, using PHPFina, redis based input queues and switching to SSD’s where the other major performance improvements we made on emoncms.org.

If you can check that redis is running ok, it should be possible to just set $redis_enabled = true; and redis will load anything it needs from the disk. That said its not something I’ve tried on a live server since we first introduced redis quite some time back.

Shall I merge the device-integration branch into the master branch @pb66, @Adminde? I know @glyn.hudson is keen to have the latest device module developments for simplifying the OpenEVSE setup procedure as well as the SolarPV type1 & type2 configurations. Testing it here looks like its working well and thankyou @Adminde for all of your hard work on the latest pull request.

Which repo are you referring to? Both the emoncms and the device module have “device-integration” and “master” branches. Or do you mean both?

Can you please recap on what this entails? This thread started out with some quite significant changes to the way inputs and feeds currently work, I 'm no longer sure what the current status or direction is.

I will try to find time to try the “device-integration” branches later this week, I have not previously tried them as there are still some bugs with the latest (non-integrated) device module. I started to debug at the turn of the year but ran out of time. I’m still using a much earlier version on my live server, so these bugs have not effected my work, but that also means I don’t have recent experience of the bugs is the later version, I will need to run some tests to refresh if you are going to pick up device module dev again.

@pb66 I’d be happy to help with the debugging or features if I can.
I personally use a different branch of our fork with some more changes but I tried to merge all stability-related changes in my latest PR.

@TrystanLea I’d love to see the overhauled device module and its support in the emoncms repo to go live as I use it productivly for quite a while now and absolutely love it. We did not experience any bugs in a while but I’m sure some will emerge… I’d gladly continue to help looking into it, as we still have some additional features in testing related to our OpenMUC protocol driver integration into emoncms.
Just as a small outlook, we added an OpenHAB thing/item/channel-like functionality for devices a while ago to be able to have an entry-point to control e.g. Smartplugs or any communication over ModBus, M-Bus, DLMS/COSEM, IEC 61850 (generally smartmeters or EVs) or whatever with only emoncms as a central GUI.

This can be achieved with modules providing a *_thing.php file and an additional “items” tag in templates:

  "items": [
      {
           "id": "enable",
           "type": "Switch",
           "label": "Enable",
           "mapping": {
               "ON": { "channel": "foo_enable", "value": true },
               "OFF": { "channel": "foo_enable", "value": false }
           },
           "input": "foo_state"
       }
   ]

At the moment the necessary function public function set_item($itemid, $mapping) in device_thing.php is only a placeholder that directly writes a value into an input but I thought this could be filled e.g. with a more general MQTT command usable for EVSE or Smartplugs. I’m not very experienced with MQTT though^^
I will get an intern at around August to be able to better test and especially document some stuff here :slight_smile:
Hence anything in the direction of going stable for the basis of this would be awesome.

1 Like

I’m referring to specifically the device-integration branch in the device module at this stage: https://github.com/emoncms/device/tree/device-integration

The emoncms device-support branch is a second part to come back to later.

Its a while back now and its stretching my memory, but I think the main changes are:

  • new device templates, using PHPFina for standard OEM configurations, e.g: Solar PV and HomeEnergyMonitor setups
  • option to place devices in subfolders
  • device template selection dialog thanks to @Adminde
  • dry run device initialisation, again thanks @Adminde

Wow that looks really nice! Looking forward to hearing more about it, looks like it could fit in well with the scheduler I built last year for the energy local project that uses the device module :slight_smile: CydYnni, EnergyLocal Community hydro smart grid - blog post - #3 by TrystanLea

I actually was strongly hoping so @TrystanLea, as I followed those blog posts closely :wink: we are trying to push our local City/DSO here to do something extremly similar with solar power for quite a while now… but german bureaucracy you know.
I’d gladly help anywhere I can here, as especially the template database should benefit the more the module will be used^^

Further, I just started another PR, trying to improve overall stability of the module by overhauling the redis caching… the present delete-everything-and-reload-approach for cached devices originally by @nchaveiro seemed to be a little error-prone in asynchronous environments.
I was only able to test it in a small context but have high hopes as it actually reduces complexity instead of increasing it^^
@pb66 if you maybe find some time to look into the branch, it would be strongly appreciated if you could take a look and see if your known bugs still exist there as well.

Thanks @Adminde, I’ve merged your pull request , thanks again for all the hard work you’ve put into this!

Thanks, that is great. The bigger hurdle right now is finding the time to try and recreate the issues, especially as they are no longer fresh in my mind. I previously had to abandon several times as time was thin and I wasn’t able to put my finger on how to reliably recreate the issue to post here.

From what I can recall, I think there were some “unique-ness” issues around templates of the same name in a different subset, eg if I duplicated the OEM:emonTH template as pb66:emonTH to add my own processes and feeds. Also with the initialize/create device, sometimes it would report the feed as existing as it found a similar feed with a different tag. But I could be miles out, those descriptions are purely from fading memories.

I need to to some testing to confirm. Especially as there have been changes since I last tried it, it may of been fixed already. My test emoncms has not been updated in the last couple of weeks, so I think I need to find the issues before updating otherwise I will not know if if the issue’s fixed or I’ve just not found it.

Ah! Good, all these features are familiar and I’ve just checked my test server, it is running the device-integration branch, so it was just where I didn’t link the branch name to the feature set, we are on the same page I think.

I will gladly look at all the new stuff once I am able to confirm/reproduce/describe the issues known to me on the older setup. Previously I never progressed any further than finding these issues so I cannot say if the rest of it was bug-free or not, but it would obviously make more sense now to continue any testing after I do update to the latest version.

@pb66 just thought Id say, dont feel you have to jump in with testing now if you have a lot on, we could merge this into the master branch and let others use and test it, and then when its a convenient time for you, you could always come back to it then?

I’ve installed the device module on emoncms.org. I needed to make a couple of changes for compatibility as emoncms.org has a different implementation of the process model GitHub - emoncms/device at emoncmsorg

The device-integration branch is now merged into master.

I’ve created a release for the device module that marks the state of the device module prior to merging the device-integration branch.
https://github.com/emoncms/device/releases/tag/1.0.0

and a second release for the launch of the device-integration branch version:
https://github.com/emoncms/device/releases/tag/1.1.0

See full release list:
https://github.com/emoncms/device/releases

To switch back to the old device module v1.0.0:

cd /var/www/emoncms/Modules/device
git checkout 1.0.0
1 Like

Thanks to a lot of hard work by @emrys, The device-support branch of emoncms is now merged into master with the option to try out the new interface via a toggle-able switch on the emoncms My Account page (after a database update).

We will be writing updated documentation and a post to describe all the changes over the next week :slight_smile:

Hi again,

with the new input (device_)view I was wondering if there is still a real purpose for devices to have both a node and a name?

Both node and name need to be unique at device creation eitherway and inputs are grouped by their node in the current version at emoncms master…
I liked the prior way of grouping inputs by their corresponding device name, but naturally understand the reasoning of reverting this. I currently tend to chose both node and name of a device to be identical, to recreate the original effect thogh… as the My Devices view feels a little wrong right now with the “device” blocks not being identified by their name but their node?

Just asking for opinions :wink:

Maybe “repurposing” the device node to be used as the device name and drop the original name in the database on the long run could reduce confusion without being such a horrible backwards incompatible thing to do? If I’m not totally wrong and keeping them both serves an important role that I am unaware of.

Hello @Adminde, yes I vote for removing the device name and having just the node, I find the duplication confusing, but Im sure we discussed before why its needed? One alternative could be to disable the node name field when used in the inputs interface.

My naming of the new device integration in the inputs page as ‘My Devices’ may also become a source of confusion do you think we should change this back to Inputs for now?

PS: Your latest changes to the device module @Adminde, the animated sidebar menu, are looking great!

I just thought of maybe “rebranding” or repurposing the database fields like you did with the input name/description to key/name would be sufficient as well to reduce confusion?
Rebranding node/name/description(location) to key/description/location or even key/name/description with some explainatory info hover icons to explain their purpose could eliminate any confusion?
I guess it should be clarified at some point that the node/key should be a short no-special-char string, as it is used to post input values? And the name then could potentially allow spaces and everything to e.g. “dishwasher/G 4203 Active/Kitchen”

Further, I was asking myself if the “updated” time is intuitive to understand right now. Everyone who knows the old table view knows what a green or yellow time element means but without any sort of header new users could be confused as well?

And thanks! I was wondering in general to be honest, why the bootstrap collapse and accordions(panels for bs3) are not beeing used, as they provide an easy way to have a more fluid and responsive UI.

I used those for our emonmuc channels module and was wondering if you would want to stop me, if I planned on taking a look into the new input and feed views, when I find some time over the next weeks/months? :wink:
Some of the small things I’d try to change/add if you want would be:

  • Animated collapses for group blocks
  • Indeterminate selections both for “select all” as well as for group blocks
  • Clicking a row anywher except for the buttons selects the row
  • Updates of the “updated” time span once per second
  • Hover row colour changes

Those would be somewhat copy-paste from our channels module (for which I btw have an intern at the moment to hopefully write some guides for):

And yes, I’d kinda support renaming the input view to Inputs again. I guess since the menu element is called “Inputs” as well, this would be generally more consistent. I mean, the view still shows inputs… they are grouped by their device but the main elements are still the inputs^^
The connection between inputs and devices could be mentioned in the alert-header though?

But maybe a New device button at the bottom of the view could somewhat further “merge” the two views?

Hi @TrystanLea and @emrys!

I just noticed all your very nice changes to the input and feed views regarding collapses and wanted to ask if I may use my experience with this design approach through our pretty rigorously tested channels module… to contribute to this without offending anyone :wink:
I very much like the approach to put most of the css into the theme, as this opens up a whole lot of custumization possibilities, e.g. to merge this with the new app colors like in MyElectric2 and use the same css classes for those as well - or move some of their css to the themes as well.
The “.node” naming scheme could seem a little off for this though… maybe this could be renamed to “.group” or something like that?

Further I wanted to ask if the currently plain style of collapses was intended as a reduction of the UI visuals or if this was the easier thing to do?^^ I personally liked the in-block rows of the prior design a lot

Thanks @adminde, Sorry for the slow reply on your above posts. We’ve just been on a bit of a focus stint trying to bring this release together over the last couple of weeks as well as sorting divergent code between emoncms core and emoncms.org :slight_smile:

As well as css changes the other large change is in the way the column sizing and positioning and hiding of column fields on smaller screen sizes uses javascript (although much of that has been in the view for some time, @emrys has recently significantly refactored and abstracted this code, see: emoncms/Lib/responsive-linked-tables.js at master · emoncms/emoncms · GitHub).

@emrys suggested the plainer style rather than the in-block rows, it simplifies the sizing code a bit as columns in the header row match the widths and positions of those in the non-heading rows, we are persuadable either way on that one.

I renamed the new inputs view to be called Inputs again rather than My Devices to avoid confusion.

Perhaps node/description/location? with description appended with (optional)? I would still vote for removing description as I think location is sufficient

Interesting thought and approach regarding last updated time… It would be interesting to see if we get any queries on that… perhaps appending ‘ago’ would be sufficient e.g 15s ago?

Aha, @emrys has added this in now :slight_smile: We did try adding in “Indeterminate selections both for “select all” as well as for group blocks” but disabled it for now as it conflicts with the expand/contract arrows

Hi @TrystanLea, as I take my time to reply as well I hope we’re even :wink:

Why though?^^ Shouldn’t @media(max-width: 979px) be sufficient? Having Javascript doing more than necessary can cause a lot of problems in the long run.
But eh :slight_smile: it looks great overall! I’d very much like to suggest some small desgin specific things and/or make a pull request regarding e.g. the chevrons. But I will probably wait with any of this until your milestone is reached and you have room for potentially bothersome discussions?^^

I love the new lean design as well actually. I just adjusted my design to it to keep it consistent.

I actually totally agree with this. Having 2 input fields should be sufficient. I’d personally prefer it to be just node/description though. The connection to inputs is logically and clear and the description could still be used as a location description if the user wants it to be. But it leaves room open for e.g. different model descriptors if you have e.g. two devices in the same room or whatever. Just add some small tooltips to the dialog to describe it. The original reasoning of “grouping” as done in the table.js is gone now eitherway with the groups view?
It just feels wrong to have an empty name lingering in the database. Shouln’t it be possible to drop the column in the longterm, if the user won’t be able to see it in the GUI eitherway?

My only problem with this would be translateability :confused:
In German this would roughly translate to 15s her which sounds just…meh^^
Maybe having it connected with the value to be 1.23 kW since 15s (1,23 kW seit 15s) could be more generic for other languages? I’m just not sure if a device or tag with 20 inputs and 20 “since” wouldn’t be bothersome to look at as well.


Even though our replys get longer and longer I wanted to ask if you planned on creating a Lib/groupjs/groups.js script to generically create and fill those groups as table.js does for tables? I just started doing so to consolidate my modules (and because it felt right to do so) and I wanted to ask if I should stop that because you are doing so as well or if this could even be appreciated? I can’t do that on working hours but would continue with it over the next few weeks otherwise^^