Hi,
New to this forum but a long time user of emoncms. Sorry in advance for the longs post.
I am currently running my emoncms (9.7.5) instance under a Docker container on a unRAID host.
I have been looking at importing data from other applications such as Home Assistant (HA). There has been some work done by others for HA to provide integration. They ran into an issue that I also encountered in the past in that the input api for bulk data input has limitations, such as:
- json will not accept spaces, requiring manipulation before input.
- Not being able to use keynames for bulk data input. Also raised in the old emoncms forum, Emoncms Link.
I have implemented work around solutions for this in the past myself in other projects.
I would like to start a discussion on a json input format that would cover all/most of the existing data input solutions. This would be an additional bulk input format that is run along side existing formats to keep the legacy support.
You might be asking “why do we need this”. I will use the Home Assistant (HA) example to illustrate why is see a need for it.
- HA sensors collect data at random time. Each sensor has a unique alphanumeric id and this is used as a keyname.
- Put all the data on the same emoncms node for simplicity.
- The current HA implementation captures the sensor data at a point in time and sends all of the sensors using the current json input format.
- To track each change (assuming they dont happen at the same point in time) would require an individual api call for each change to be able to use keynames. Using this proposed json format a dictionary (Python language) could be created over the period of a minute collecting the information, converted to a json string and then sent using one api call with all the information including keynames and time of change.
- Increase the scenario to 100 sensors with individual change times and this becomes a lot of api calls.
Another example closer to what I have run into in the past:
- Arduino battery device collects more than one sensor information locally.
- Arduino to save power only transmits every 60min.
- This can be done using existing methods but not with keynames in one api call.
I have put a sample json format below that I think might cover most needs.
Disclaimer. I am not a programmer or json expert… This is just a hobby.
{"3" : { "keyname": {"value" : 103}},
"4" : { "keyname1": {"value" : 104}, "keyname2": {"value" : 204}, "keyname3": {"value" : 304}},
"100" : { "keyname" : [
{"value" : 502, "timeoffset" : -10},
{"value" : 503, "timeoffset" : -20},
{"value" : 504, "timeoffset" : -30}
] },
"101" : { "keyname" : [
{"value" : 502, "timesentat" : 443},
{"value" : 503, "timesentat" : 543},
{"value" : 504, "timesentat" : 643}
] },
"102" : { "keyname" : [
{"value" : 502, "time" : 1477263129},
{"value" : 503, "time" : 1477263229},
{"value" : 504, "time" : 1477263329}
] }
}
In this json example:
- Node 3: input the current value into the defined keyname.
- Node 4: input the current value of multiple keynames.
- Node 100: input multiple values into the defined keyname with using the defined timeoffset method.
- Node 101: input multiple values into the defined keyname with using the defined timesentat method.
- Node 102: input multiple values into the defined keyname with using the defined absolute time method.
Ideally the input api would do the replacing of any spaces after the input has been received. This would allow for simpler sending direct from applications, by converting an object to json formatted string and sending it as is.
I am no php nor json expert but might have a go at writing the code to accept this kind of json input through the api and submitting a pull request on GitHub if i get time.
I thought some discussion about this before I commit any significant time to this would be good place to start.
Queue the feedback…