Unearthed a quirk with editrealtimedata timestamps

[pb66 (Paul)]

Today I spotted a freak value in some temperature data and decided to try and correct the one bad datapoint using vis/editrealdata. This way of editing values has always been a bit temperamental and I may have stumbled across some useful info whilst making my correction.

For this particular project I am recording csv locally and also sending to a self-hosted emoncms. It was in emoncms that I noticed that one of 34 temp sensors momentarily read -2043.125°C, thinking that was a tad chilly for this time of year I checked the local csv and found the spike was prior to sending and not an emoncms issue.

This is a snippet of the csv file, at the end of the 3rd line you can see the offending value.

1535463900.0,"w1_bus_23",21.75,21.562,21.312,20.937,21.625,21.687,21.625,21.812,20.687,20.312,20.687,21.75,20.937
1535463930.0,"w1_bus_23",21.75,21.562,21.312,20.937,21.625,21.687,21.562,21.75,20.687,20.25,20.687,21.75,20.937
1535463960.0,"w1_bus_23",21.687,21.562,21.312,21.0,21.625,21.687,21.625,21.75,20.687,20.312,20.687,21.812,-2043.125
1535463990.0,"w1_bus_23",21.687,21.562,21.312,21.0,21.625,21.687,21.562,21.75,20.687,20.312,20.687,21.812,20.875
1535464020.0,"w1_bus_23",21.687,21.562,21.312,21.0,21.625,21.687,21.562,21.75,20.687,20.312,20.687,21.812,20.875

This is what it looked like isolated in emoncms/graph and below is the csv created by the graphs page.

the timestamp and the value are recorded and reported exactly as posted from the csv above.

1535463720, 21.000
1535463750, 21.000
1535463780, 20.937
1535463810, 20.937
1535463840, 20.937
1535463870, 20.937
1535463900, 20.937
1535463930, 20.937
1535463960, -2043.125
1535463990, 20.875
1535464020, 20.875
1535464050, 20.937
1535464080, 20.937
1535464110, 20.937
1535464140, 20.937
1535464170, 20.937
1535464200, 20.875

When I went to vis/editrealtimedata and searched for that timestamp I was surprised it wasn’t found, but I picked out the obvious spike from the graph and it apparently had a timestamp that was 3 seconds later at 1535463963 not 1535463960 as expected.

once I had the spike selected it corrected without a problem

here’s the graph reloaded (still open in another tab, nothing changed just clicked “reload”)

and the csv also confirms the spike has gone.

1535463720, 21.000
1535463750, 21.000
1535463780, 20.937
1535463810, 20.937
1535463840, 20.937
1535463870, 20.937
1535463900, 20.937
1535463930, 20.937
1535463960, 20.875
1535463990, 20.875
1535464020, 20.875
1535464050, 20.937
1535464080, 20.937
1535464110, 20.937
1535464140, 20.937
1535464170, 20.937
1535464200, 20.875

This only really stood out because I know all my posted timestamps are exact multiples of 30 and I use the “time=0” arg with the bulk upload and all the feed start times (30s phpfina) are an exact multiple of 30 too (therefore every recorded data point is too), so I know the 1535463963 datapoint doesn’t exist.

Where did those 3 seconds come from???

Is this possibly why when ever I try to correct a particular data point it fails (because the datapoint with the selected timestamp doesn’t actually exist to edit)???

Right now I’m more interested in finding what caused that freak temp reading, but I thought I would post all this here as it may help debug or improve the editrealtimedata functionality.

[EDIT] Issue raised at An anomoly with editrealtimedata timestamps · Issue #1006 · emoncms/emoncms · GitHub

[pb66 (Paul)]

A bit more info I found playing with it whilst waiting for the kettle boil.

3 points from the same zoomed in section, they are all out by 4 secs, they should be 1535463240, 1535463270, 1535463300.

At a different level of zoom these three are all consistently 16secs out (should be 1535466930, 1535466960, 1535466990)

if I zoom in slightly on those same last three points again (or what i think is the same points)

These appear to be just 3 secs out, but could it actually be 33 or 27 seconds out? The values are too similar to tell, I will have to do more tests with some varying data.

Needless to say, there is some serious cross talk between the zoom level or size of data and the timestamps that should not be affected by zoom or anything else for that matter.

[pb66 (Paul)]

Further to the above, I have since had to edit a few more of these errors and discovered that when the timestamp error/offset is negative, the edits do not work.

1535712180, 18.062
1535712210, 18.062
1535712240, -2045.937
1535712270, 18.125
1535712300, 18.125

I tried multiple times to edit this datapoint but the screen never changed, no error, no refresh (as it does when successful) it silently failed.

eventually by zooming in and out I eventually got the edit to stick, the difference was that the apparent timestamp was now greater than the true timestamp (as per all the examples above), with the prior failed attempts at this correction I had noticed the apparent timestamp was actually 3s earlier than the actually timestamp.

the above screenshot is immediately after the edit succeeded as I did not take screenshots of each failed attempt.

[pb66 (Paul)]

I have briefly looked at the code and see no obvious errors, but I vaguely suspect that what is happening is that the vis is requesting feed data by the usual means, start time, end time and interval and the data returned is presented with the requested datapoint timestamps rather than the found datapoint timestamps.

Here is a example request

https://myserver.co.uk/feed/data.json?id=103396&start=1535709532613.64&end=1535712793245.9226&interval=4&skipmissing=1&limitinterval=1&apikey=abc123

and that returns

[
   [
      1535709532000,
      16.812000274658
   ],
   [
      1535709562000,
      16.875
   ],
   [
      1535709592000,
      16.875
   ],
   [
      1535709622000,
      16.875
   ],
   [
      1535709652000,
      16.875
   ],
   [
      1535709682000,
      16.937000274658
   ],
   [
      1535709712000,
      16.937000274658
   ],
   [
      1535709742000,
      16.937000274658
   ],
   [
      1535709772000,
      16.937000274658
   ],
   [
      1535709802000,
      16.937000274658
   ],
   [
      1535709832000,
      17
   ],
   [
      1535709862000,
      17
   ],
   [
      1535709892000,
      17
   ],
   [
      1535709922000,
      17
   ],
   [
      1535709952000,
      17.062000274658
   ],
   [
      1535709982000,
      17.062000274658
   ],
   [
      1535710012000,
      17.062000274658
   ],
   [
      1535710042000,
      17.062000274658
   ],
   [
      1535710072000,
      17.062000274658
   ],
   [
      1535710102000,
      17.125
   ],
   [
      1535710132000,
      17.125
   ],
   [
      1535710162000,
      17.125
   ],
   [
      1535710192000,
      17.125
   ],
   [
      1535710222000,
      17.187000274658
   ],
   [
      1535710252000,
      17.187000274658
   ],
   [
      1535710282000,
      17.187000274658
   ],
   [
      1535710312000,
      17.187000274658
   ],
   [
      1535710342000,
      17.25
   ],
   [
      1535710372000,
      17.25
   ],
   [
      1535710402000,
      17.25
   ],
   [
      1535710432000,
      17.25
   ],
   [
      1535710462000,
      17.25
   ],
   [
      1535710492000,
      17.312000274658
   ],
   [
      1535710522000,
      17.312000274658
   ],
   [
      1535710552000,
      17.312000274658
   ],
   [
      1535710582000,
      17.312000274658
   ],
   [
      1535710612000,
      17.312000274658
   ],
   [
      1535710642000,
      17.312000274658
   ],
   [
      1535710672000,
      17.375
   ],
   [
      1535710702000,
      17.375
   ],
   [
      1535710732000,
      17.375
   ],
   [
      1535710762000,
      17.375
   ],
   [
      1535710792000,
      17.375
   ],
   [
      1535710822000,
      17.437000274658
   ],
   [
      1535710852000,
      17.437000274658
   ],
   [
      1535710882000,
      17.437000274658
   ],
   [
      1535710912000,
      17.437000274658
   ],
   [
      1535710942000,
      17.5
   ],
   [
      1535710972000,
      17.5
   ],
   [
      1535711002000,
      17.5
   ],
   [
      1535711032000,
      17.5
   ],
   [
      1535711062000,
      17.5
   ],
   [
      1535711092000,
      17.562000274658
   ],
   [
      1535711122000,
      17.562000274658
   ],
   [
      1535711152000,
      17.562000274658
   ],
   [
      1535711182000,
      17.562000274658
   ],
   [
      1535711212000,
      17.625
   ],
   [
      1535711242000,
      17.625
   ],
   [
      1535711272000,
      17.625
   ],
   [
      1535711302000,
      17.625
   ],
   [
      1535711332000,
      17.687000274658
   ],
   [
      1535711362000,
      17.687000274658
   ],
   [
      1535711392000,
      17.687000274658
   ],
   [
      1535711422000,
      17.687000274658
   ],
   [
      1535711452000,
      17.687000274658
   ],
   [
      1535711482000,
      17.75
   ],
   [
      1535711512000,
      17.75
   ],
   [
      1535711542000,
      17.75
   ],
   [
      1535711572000,
      17.812000274658
   ],
   [
      1535711602000,
      17.812000274658
   ],
   [
      1535711632000,
      17.812000274658
   ],
   [
      1535711662000,
      17.812000274658
   ],
   [
      1535711692000,
      17.812000274658
   ],
   [
      1535711722000,
      17.875
   ],
   [
      1535711752000,
      17.875
   ],
   [
      1535711782000,
      17.875
   ],
   [
      1535711812000,
      17.875
   ],
   [
      1535711842000,
      17.875
   ],
   [
      1535711872000,
      17.937000274658
   ],
   [
      1535711902000,
      17.937000274658
   ],
   [
      1535711932000,
      17.937000274658
   ],
   [
      1535711962000,
      17.937000274658
   ],
   [
      1535711992000,
      18
   ],
   [
      1535712022000,
      18
   ],
   [
      1535712052000,
      18
   ],
   [
      1535712082000,
      18
   ],
   [
      1535712112000,
      18.062000274658
   ],
   [
      1535712142000,
      18.062000274658
   ],
   [
      1535712172000,
      18.062000274658
   ],
   [
      1535712202000,
      18.062000274658
   ],
   [
      1535712232000,
      18.062000274658
   ],
   [
      1535712262000,
      18.125
   ],
   [
      1535712292000,
      18.125
   ],
   [
      1535712322000,
      18.125
   ],
   [
      1535712352000,
      18.125
   ],
   [
      1535712382000,
      18.187000274658
   ],
   [
      1535712412000,
      18.187000274658
   ],
   [
      1535712442000,
      18.187000274658
   ],
   [
      1535712472000,
      18.25
   ],
   [
      1535712502000,
      18.25
   ],
   [
      1535712532000,
      18.25
   ],
   [
      1535712562000,
      18.25
   ],
   [
      1535712592000,
      18.25
   ],
   [
      1535712622000,
      18.25
   ],
   [
      1535712652000,
      18.312000274658
   ],
   [
      1535712682000,
      18.312000274658
   ],
   [
      1535712712000,
      18.312000274658
   ],
   [
      1535712742000,
      18.375
   ],
   [
      1535712772000,
      18.375
   ],
   [
      1535712802000,
      18.375
   ]
]

Ignoring the end 3 zeros for each data stamp we can see that they are not multiples of 30, eg the first datapoint

[
      1535709532000,
      16.812000274658
   ],

is 1535709532 which sits between the real datapoint timestamps of 1535709510 and 1535709540

1535709450, 16.812
1535709480, 16.812
1535709510, 16.812
1535709540, 16.812
1535709570, 16.875
1535709600, 16.875

This I suspect is why the edits appear to fail when they are negative errors and succeed when they are positive errors. It’s because when the datapoint is saved, the positive error datapoint’s edit will land in the correct datapoint, but the negative error datapoint’s edit will actually edit the previous datapoint.

for example, when targeting 1535709540 for editing if the returned datapoint is timestamped 1535709543 it will be saved to 1535709540, but it the returned datapoint is timestamped 1535709532, when the edited value is saved it will be saved to datapoint 1535709510 not 1535709540 due to the way writing feed values works.

This latest quirk is just side-effect of using the wrong datapoint timestamps, the core issue is that the api is effectively returning misleading info, 1535709532 doesn’t exist, it should return

[
      1535709540000,
      16.812000274658
   ],

I understand that other applications might not be so sensitive, but I think there needs to be an option (arg?) to optionally return “true data” that the editrealtime vis could use (and many of us might actually prefer that approach for other things too.

Perhaps the read and write functions should possibly work the same way, ie when searching for a datapoint, always use the next “previous” datapoint rather than being bi-directional and grabbing the closest “previous” or “next” datapoint, just a thought.

[pb66 (Paul)]

Opened another issue for a feature request to offer options to return “true” timestamps and also to seek datapoints in one direction only to offer the inverse of the write functionality.

option to return "true" data in feed api's

[borpin (Brian Orpin)]

Firstly this is a rerally useful feature that is hidden away really well!

Just tried to edit some data (having come across this thread) and can concur with @pb66 finding.

I think it would be useful to provide the text output of the data as you can, in the graphs view.

It may be that this is just how the graphing works. By providing the text output, you should be able to identify the correct timestamp to edit easily.