Community
OpenEnergyMonitor

Community

Emoncms feed API


(franck102) #1

Hi all,

I am trying to use the feed API to update a single (bogus) feed data point:

1546778700, 0.9
1546779600, 0.6
1546780500, 1.0

I have tried both GET and POST using curl, both calls return a 200 OK however the data point is not updated in the Graph view (show CSV output)… am I missing something?
I have double checked the feed ID; it is a Wh accumulator feed if that makes a difference.

Thanks!

curl -v -X POST "http://emonpi/emoncms/feed/insert.json?id=13&time=1546779600&value=0.9&apikey=<my write API key>"
*   Trying 2a01:cb15:81c2:e500:b10d:4809:3e2d:c9bf...
* TCP_NODELAY set
* Connected to emonpi (2a01:cb15:81c2:e500:b10d:4809:3e2d:c9bf) port 80 (#0)
> POST /emoncms/feed/insert.json?id=13&time=1546779600&value=0.9&apikey=... HTTP/1.1
> Host: emonpi
> User-Agent: curl/7.63.0
> Accept: */*
>
< HTTP/1.1 200 OK
< Date: Sun, 06 Jan 2019 16:58:59 GMT
< Server: Apache/2.4.25 (Raspbian)
< Content-Length: 3
< Content-Type: application/json
<
0.9* Connection #0 to host emonpi left intact

C:\Users\Franck>curl -v "http://emonpi/emoncms/feed/insert.json?id=13&time=1546779600&value=0.9&apikey=<key>"
*   Trying 2a01:cb15:81c2:e500:b10d:4809:3e2d:c9bf...
* TCP_NODELAY set
* Connected to emonpi (2a01:cb15:81c2:e500:b10d:4809:3e2d:c9bf) port 80 (#0)
> GET /emoncms/feed/insert.json?id=13&time=1546779600&value=0.9&apikey=<key> HTTP/1.1
> Host: emonpi
> User-Agent: curl/7.63.0
> Accept: */*
>
< HTTP/1.1 200 OK
< Date: Sun, 06 Jan 2019 17:00:32 GMT
< Server: Apache/2.4.25 (Raspbian)
< Content-Length: 3
< Content-Type: application/json
<
0.9* Connection #0 to host emonpi left intact

(franck102) #2

Oops, if I replace insert.json with update.json (I was using the wrong url) something does happen… but not quite what I expected, a new value is appended to the feed with a bogus unix time:

curl -v "http://emonpi/emoncms/feed/update.json?id=13&time=1546779600&value=0.9&apikey=<key>

1546794000, 0.9

The value at 1546779600 is left unchanged??

Franck


(franck102) #3

Answering my own question… I was naively taking the unix time values from the “show CSV output” on the Graph tab.
If I export the feed instead and use the unix times found there to update the bogus values the API works like a charm.

Franck

Edit: another reason is the lack of acknowledgements in the JeeLib library. Today the power went down, my custom node and the emonPi came back up at the same time, the emonPi missed the “0” that the custom node sends when starting up to signify the Wh accumulator processor that it was reset, and my feed is now all messed up.
I have now added a delay to the node for that use case, but the only solid fix would be waiting for an acknowledgement from the emonPi…


(Paul) #4

That doesn’t sound right, the WhAccumulator doesn’t need to see a zero. It just recognises that a value is less than the last value and assumes a rollover/reset, it will only use that lesser value to calculate the next delta and no value is added for the curent post.

posting the following series of values

10,20,30,40,20,30,40

will increase the whaccumulator like so

0,10,20,30,30,40,50

ie the delta would be

0,10,10,10,0,10,10

so even if your custom node took ages to fire up or for it’s data to be accepted by emonpi only the increase in energy logged before the first packet reaches emoncms will be lost as that first value is used as “the zero point” from which emoncms continues counting.


(franck102) #5

Ok, unfortunately I ran a post-process “Remove resets” on the feed and I replaced the original with the corrected feed - I’ll check what’s happening if the problems comes up again.

The outage lasted about 3h, maybe that is the problem…
Franck


(Paul) #6

Quite possibly.

There still hasn’t been any build guide published for the current emonSD so I cannot comment on the current position, but definately prior to the latest image, the values held in redis are not persisted accross reboots (it is/was a low-write “optimisation”) so the first data in is/was always unused for increasing the total, it created the “reset point” only.

For example if you had a battery operated emonTH pulse counter and your emonpi was off line for whatever reason, whilst your emonpi is off line the pulsecount on the emonTH continues to grow, when your emonpi comes online the first value received is assumed to be the new “zero point” (as redis is flushed on the reboot there is no “last value”) then the second value received will lead to a calculation based on the 1st value, now held in redis.
So the emonPi’s “low-write” emoncms is not able to “catch up” with missed energy/pulse values after a reboot, it is as if the world stopped for the duration of the downtime. This is not the case for regular emoncms as the last redis value is regularly persisted to disc by redis and read back into memory at start up, so the 1st value recieved after boot up, is compared to the last value received before going offline, in the case of the battery emonTH pulsecounter, no pulses would be lost.

There is a awy around this (is it is still an issue in the latest image) by using the rather catchlyl named “Total pulse count to pulse increment” process instead. (see Wh accumulator - strange behavior) don’t be put off processes that don’t have generic names, the only difference between a pulse and a Wh is possibly scaling in many instances (but more often not even that)


emonCMS Configuration For Recording Pulse Counting Gas Consumption
(franck102) #7

Great thread, thanks! I am using the whAccumulator to track water meter pulses, so I gave up worrying about the names already :slight_smile: The pulse process you suggest seems to be much robust for the job, I will definitely give it a try.

Franck


(franck102) #8

Regarding the Reddis / low write problem with the whAccumulator, why not prime Reddis with the last persisted feed value upon startup (or have the process read from disk if it can’t find a value in Reddis)?
Naïve question I guess, I haven’t been into the code so far…

Franck


(Paul) #9

Because it’s not persisted on a low write install (with RO filesystem at least)

Because the feed value on disk is the calculated running total, the last value is only kept in redis, this is why the pulse process works better, because it does persist the last values to a feed and then you must save the calculated running total to another feed, so the price is it uses twice the space on disc.


(franck102) #10

Ahh, got it, thanks!


(franck102) #11

The pulse increment to pulse count feed works great. With the wh accumulator on the other hand I see a problem that has been reported in other threads I believe, i.e. periodically the feed contains one or two values that are below the running total:

21/01/2019 13:37:00 2650
21/01/2019 13:38:00 2650
21/01/2019 13:39:00 2648.5
21/01/2019 13:40:00 2650
21/01/2019 13:41:00 2650

According to the emonhub log there is nothing wrong with the input (1h shift due to timezone), the input is the third value, and it is 7 in all packets:

2019-01-21 12:37:59,405 DEBUG    RFM2Pi     1011 NEW FRAME : OK 18 73 1 0 0 0 0 0 0 7 0 0 0 (-56)
2019-01-21 12:37:59,407 DEBUG    RFM2Pi     1011 Timestamp : 1548074279.41
2019-01-21 12:37:59,408 DEBUG    RFM2Pi     1011 From Node : 18
2019-01-21 12:37:59,409 DEBUG    RFM2Pi     1011    Values : [329, 0, 7]
2019-01-21 12:37:59,409 DEBUG    RFM2Pi     1011      RSSI : -56
2019-01-21 12:37:59,410 DEBUG    RFM2Pi     1011 Sent to channel(start)' : ToEmonCMS
2019-01-21 12:37:59,411 DEBUG    RFM2Pi     1011 Sent to channel(end)' : ToEmonCMS
2019-01-21 12:37:59,613 DEBUG    MQTT       Publishing: emon/emonmeter/fuel 329
2019-01-21 12:37:59,615 DEBUG    MQTT       Publishing: emon/emonmeter/plaque 0
2019-01-21 12:37:59,617 DEBUG    MQTT       Publishing: emon/emonmeter/water 7
2019-01-21 12:37:59,618 DEBUG    MQTT       Publishing: emon/emonmeter/rssi -56
2019-01-21 12:37:59,620 INFO     MQTT       Publishing: emonhub/rx/18/values 329,0,7,-56
...
2019-01-21 12:38:59,535 DEBUG    RFM2Pi     1018 NEW FRAME : OK 18 73 1 0 0 0 0 0 0 7 0 0 0 (-56)
2019-01-21 12:38:59,538 DEBUG    RFM2Pi     1018 Timestamp : 1548074339.54
...
2019-01-21 12:39:59,672 DEBUG    RFM2Pi     1025 NEW FRAME : OK 18 73 1 0 0 0 0 0 0 7 0 0 0 (-55)
2019-01-21 12:39:59,673 DEBUG    RFM2Pi     1025 Timestamp : 1548074399.67
...
2019-01-21 12:40:59,764 DEBUG    RFM2Pi     1031 NEW FRAME : OK 18 73 1 0 0 0 0 0 0 7 0 0 0 (-55)
2019-01-21 12:40:59,766 DEBUG    RFM2Pi     1031 Timestamp : 1548074459.76

There is nothing special in the logs between the entries above, though I don’t know if Redis logs anything (i.e. flushes).

Franck


(Paul) #12

No, although I do recall that issue, I don’t recall if we ever got to the bottom of it. Is the error actually in the feed data source file or just in the graphs and/or csv produced from the feed?

I find the whaccumulator way too quirky which is why I use the pulse increment to pulse count process (minor point but the feed behind the process is just another feed, it’s the process that is “pulse increment to pulse count”), the other advantage of the pulse increment to pulse count process over the whaccumulator is that even when it does misbehave, you can see exactly what data was presented and used to determine the increment value, the whaccumulator overwrites the redis entry each time.

Although I haven’t used it yet, there is a new “trim” feature for the feeds by @emrys, so every few months you could “trim” the feed created by the pulse increment to pulse count process and just keep the last month or so if space is short.


(franck102) #13

The error is in the csv export from the feed (not the graph).
I haven’t looked at the file, but the graph shows the dip as well, the feed REST API returns it, and things are the same after a reboot.
I looked at the code and I can’t imagine how the value handed out to feed->insert_data would not be strictly greater than last value returned by feed->get_timevalue($feedid) - so this could be a lower level problem. In particular there is a paddingMode=‘join’ argument that might trigger some low-level interpolation?

Are the pulse processes still “low-write” in my low-write install? I see that the function calls feed->insert_data which posts to the Redis buffer it seems, so I should be good? I only have 3 inputs using pulse processes, with a 60s period, so it shouldn’t be too bad but I’d rather not kill my SD card too quickly…

Thanks for the help!
Franck


(franck102) #14

The padding option is described here it seems:
https://openenergymonitor.org/forum-archive/node/10373.html

… and a quick look at the phpfina code shows floating point calculations for interpolation, that is where I’d start looking but I am not set up to add logging statements to my install :slight_smile:

Franck


(Paul) #15

I think you could be right, that is where I would expect it to creep in.

I don’t know for absolute certainty, but ASAIK all the data from any processes is written to redis and held in memory, then once every min, the feedwriter will write all the data to disc, so it shouldn’t matter what processes or how many feeds you have, the writes are still limited.


(franck102) #16

A small improvement for the pulse increment to pulse count process maybe:if the process receives float values (or if the value is scaled down before coming into the process), adding a small hysteresis to the comparison to account for floating point (un-)precision would avoid unwanted jumps:

Pulses:

1548229920,7.70
1548229980,7.70
1548230040,7.70
1548230100,7.70

Process output:

1548229920,15.70
1548229980,15.70
1548230040,23.40
1548230100,23.40

Franck