Emoncms.org says OK then drops the ball

Emoncms Emoncms.org
1

Uploading from IoTaWatt to emoncms using the bulk protocol, if an entry in the array is empty, Emoncms responds ok (in the case of unsecure GET) or with the proper SHA256 validation (in the case of secure POST), but doesn’t post the data.

Here are normal secure transactions with the SHA256 response:
time=1512339910&data=[[0,"IotaWatt",118.3,1393.7,307.3,152.9,0,0,0,0,0,0,0,0,0,0,0]]
GI0eIXIrPtI4EoVe3BYV8zi2mo_JRLY-WZjkq5s7bTQ=
time=1512339920&data=[[0,"IotaWatt",117.9,1382.4,305.2,152.0,0,0,0,0,0,0,0,0,0,0,0]]
IMGKZNQnrTpDcRpgHoMz7JVgPWjN-oaQe8yrAvvPOgo=
time=1512339930&data=[[0,"IotaWatt",118.0,1388.8,306.9,152.7,0,0,0,0,0,0,0,0,0,0,0]]
cDIXMOen5PtsPT2Z3c_5aEEkCDUH1T9ynCr5p32JStM=

During this, the Emoncms.org input display shows the updates happening.
Now one field is removed and the input string has no value for one entry:

time=1512339960&data=[[0,"IotaWatt",117.9,1400.1,309.5,154.0,0,0,0,0,0,0,0,,0,0,0]]
-_kt6f3P27qP0xYEI6dSRJPnc1kul8Ewkx1cvuuYPuA=
time=1512339970&data=[[0,"IotaWatt",117.9,1430.5,316.4,157.5,0,0,0,0,0,0,0,,0,0,0]]
VwsTaFpkIEu47V3PUhor0P2Kdq75Vbyf9bk2JsgQmDY=

As you can see, a proper SHA256 acknowledgement is returned, yet the input screen doesn’t change, and the data is not posted.
Switching over to unsecured GET:

time=1512340020&data=[[0,"IotaWatt",118.7,1409.5,311.4,154.8,0,0,0,0,0,0,0,,0,0,0]]
ok
time=1512340030&data=[[0,"IotaWatt",118.7,1413.7,314.3,156.5,0,0,0,0,0,0,0,,0,0,0]]
ok

The ok response is received, but again, no updates occur.
Adding that input back into the array:

time=1512340090&data=[[0,"IotaWatt",118.4,1407.8,311.6,155.1,0,0,0,0,0,0,0,0,0,0,0]]
ok
time=1512340100&data=[[0,"IotaWatt",118.1,1396.8,308.9,154.0,0,0,0,0,0,0,0,0,0,0,0]]
ok
time=1512340110&data=[[0,"IotaWatt",118.1,1496.5,331.8,164.6,0,0,0,0,0,0,0,0,0,0,0]]
ok

Now still getting OK, but the data is actually being posted.

This is disconcerting. Not because the null array entry causes the problem, but that there is no indication of any problem.

The bulk protocol is positional, so in order to post higher entries, this position must be included. Previously, I believe this worked OK, although the input was created and treated as zero. I can put a zero in there, but it would be better if I could leave it with no value (or “null”) and not have an input created or updated.

But most importantly, the acknowledgement should be failure if the inputs don’t post.

2

Just keeping this alive. It’s a serious problem and the cause and full extent isn’t well understood.

3

I can’t help with the false ok’s.

Regarding the positional data. emoncms will work with null values when used via the bulk api.

https://myserver/input/bulk.json?data=[[0,31,11,null,null,null,55]]

is valid and only creates/updates inputs 1 and 5.

Using zero will most likely have unexpected consequences, try null (no quote marks as that would be a string “null”)

 time=1512340020&data=[[0,"IotaWatt",118.7,1409.5,311.4,154.8,0,0,0,0,0,0,0,null,0,0,0]]

should work fine (when using the standard “bulk” api, I have only ever tested via url strings and I have not tried the ssl alternative api at all if that makes any difference or not)

4

Yeah, that’s really my issue. I can send zeroes, or nulls, but this used to work with comma-comma and now it doesn’t. I could even see if everything after the commas were gone, but this sure looks like I’m being sent an OK before the original request has been successfully parsed.

Maybe I’m doing something wrong. Like to have someone take a look under the hood and see what’s going on. When you run into something like this that affects released firmware, you have to wonder what else might be going on.

5

That may have been a quirk, you can send partial packets eg

https://myserver/input/bulk.json?data=[[0,31,11,22,33,44,55]]
https://myserver/input/bulk.json?data=[[0,31,11,22]]
https://myserver/input/bulk.json?data=[[0,31,11,22,33,44,55]]

will work and just not update the last 3 values on the second update, but empty positions mid packet have never (intentionally?) worked to the best of my knowledge. It certainly wasn’t the case when the “null input” functionality was added back in Dec '14 and here are 2 posts from Apr '15 and Nov '15 discussing passing null values to emoncms.

6

Therein lies the problem. I don’t have a major problem with there being a “quirk” in a protocol that is only documented by example rather than definition, that’s pretty common nowadays, It’s when the quirk is new (pretty sure it used to work as I described) and there is an explicit indication that it worked when it didn’t.

I have to confess that I didn’t research the documentation in the forum archives from 2015. My feeling is that should be reflected in the current documentation, such as it is. But there is an interesting issue there that I don’t believe was picked up and is somewhat related:

ok, this doesn´t work:

http://domain.com/emoncms/input/bulk.json?data=[[-7,4,34,null,null,null,null,null,null,null,null,null],[-4,4,null,null,,null,null,null,null,null,null,null],[-2,4,null,null,244,null,null,null,null,null,null,null]]&time=1429614609&apikey=bd10a9b18fa094f0xxxxxxxxxxxxxxxx

but this:

http://domain.com/emoncms/input/bulk.json?data=[[-7,4,34,null,null,null,null,null,null,null,null,null]]&time=1429614609&apikey=bd10a9b18fa094f0xxxxxxxxxxxxxxxx

My eyesight isn’t what it used to be, but it looks to me that the difference between the two is that the Json array in the first example contains three Json Arrays as elements, while the second contains only one. Syntactically, this should not be a problem.

7

I’m not entirely sure what you are referring to there. I don’t specifically recognize those examples so I have no context without re-reading those threads, but at first glance the issue appears to be the 2nd of those 3 arrays in the 1st example has an empty “comma comma” (3rd csv value is missing not null) where as the latter example is complete.

8

Likle I said, my eyes aren’t that good anymore. You picked that up on “first glance”. Tough to get old. Anyway, the guy apparently received a response that there was a problem.

Thanks for your interest Paul, but we’re getting off point, as this forum has a tendency to do.

9

The way i see this should work is not to send the string “null”, but an empty array:
Eg:
https://myserver/input/bulk.json?data=[[0,31,11,,,,55]]
instead of
https://myserver/input/bulk.json?data=[[0,31,11,null,null,null,55]]

10

I give up.

11

I don’t disagree with you Nuno, I was just bringing Bob up to speed with what was and what is possible. Personally I would much prefer to send

https://myserver/input/bulk.json?data=[[0,31,11,,,,55]]

than

https://myserver/input/bulk.json?data=[[0,31,11,null,null,null,55]]

but as demonstrated in the last couple of posts, it is easy to miss an absent value, not so much with null values and having the functionality in any format is better than not having it at all. Besides IMO it would be useful if emoncms could handle null and NaN as well as absent values rather than objecting.

@overeasy - have you got a self-hosted emoncms instance? I think what you are seeing is peculiar to emoncms.org and therefore only @TrystanLea can help you.

https://emoncms.org/input/bulk.json?data=[[0,30,11,22,33,44,55]]
https://emoncms.org/input/bulk.json?data=[[0,30,11,22,33,null,55]]
https://emoncms.org/input/bulk.json?data=[[0,30,11,22,33,,55]]

all return “ok” on emoncms.org, but the latter one doesn’t update anything.
On a local instance

https://myserver.com/input/bulk.json?data=[[0,30,11,22,33,44,55]]
https://myserver.com/input/bulk.json?data=[[0,30,11,22,33,null,55]]

both return “ok” but

https://myserver.com/input/bulk.json?data=[[0,30,11,22,33,,55]]

correctly returns (correct in that it wasn’t successful not that it’s the preferred method)

{"success": false, "message": "Format error, json string supplied is not valid"}
12

I suspect so.

Exactly

13

Ops, i just re-read the post and notice this is about the bulk.json, so the “null” should be required and accepted.
If it was the CSV input API then the empty string would make sense eg: a comma followed by a comma.

14

Please try and keep discussion polite, on topic and productive. The above post is non of the above. We’re all here for the same reason to progress the development of open source energy monitoring.

I agree, this is an issue worth flagging. Lets work together to discuss a solution. @nchaveiro has been a significant contributor to Emoncms for many years.

I would recomend using your emonPi to test running latest Emoncms master branch. Emoncms.org differs (often lagging) developments on the Emoncms master branch. If the issue only effects emoncms.org please contact @TrystanLea directly.

1 Like