Best way to guarantee times when using BULK load with PHPTIMESERIES?

Hi, New to forum… posting to emoncms.org for 2 months.

I am posting pulse data from a gas meter - so slow pulse rate.

I am working out how to present infrequent pulse data with emoncms.org (any slow pulse data!)

I am using a PHPTIMESERIES feed. I noticed missing data so changed to posting using the json BULK format, to dictate times.

How can I ensure the times I post are actually used by emoncms.org?

As an example I sent…

http…&data=[…[1480115210,21,21337],[1480115260,21,21337],[1480115270,21,20958],[1480115310,21,20958],[1480115320,21,21146],[1480115360,21,21146]]&offset=1480115380&time=1480115380

The 2 times per value are generated at a pulse, power set for 10s after the last pulse and for when the pulse occurred - helps make nice graphs!

For the above data I expected in emoncms.org (in csv output)…

1480115210, 21337.0
1480115260, 21337.0
1480115270, 20958.0
1480115310, 20958.0
1480115320, 21146.0
1480115360, 21146.0

What I see is…

1480115217, 21337.0 (time is out by 7)
1480115267, 21337.0 (time is out by 7)
1480115277, 20958.0 (time is out by 7)
1480115317, 20958.0 (time is out by 7)
1480115321, 20958.0 (extra!!!)
1480115329, 21146.0 (time is out by 9)
1480115367, 21146.0 (time is out by 7)

(times taken from very zoomed in editrealtime and also graph view with 1 sec interval)

In this case I see an extra entry. In others posts entries go missing. The '‘out’ amount has also been 8.

The lack of a guaranteed posting time means that emoncms.org is not reliably presenting the posts.

Can anyone explain this and how I can post to guarantee emoncms.org stores/uses/presents the posted times?

Do forgive me if this post is not quite to community standard!

Thanks - in anticipation.

Unfortunately there is no “absolute timestamp” bulk api, I started looking into this a while back as I was seeing some unusual timestamps logged,

Firstly as you are using “offset=” the “time=” doesn’t get a look in as the offset option is before the time option in the bulk input api’s if else block.

As far as I could make out the only way you might be able to coerce emoncms to use the supplied timestamp is to use time=0, that way the calculated $time_ref is zero and there should be no “adjustment” made when each individual frames timestamp is processed further along the code.

All the other options “sentat=”, “offset=” or even legacy mode (by omitting an overall timestamp mode) use the timestamp of the server to establish a $time_ref, so time difference between the machines and even time in transit could skew the timestamps.

So try (reusing your example)

http…&data=[…[1480115210,21,21337],[1480115260,21,21337],[1480115270,21,20958],[1480115310,21,20958],[1480115320,21,21146],[1480115360,21,21146]]&time=0

Also bear in mind that because the input api uses (int)$itemtime the timestamp gets truncated to the whole second, not rounded so 1480115210.999 would become 1480115210 not 1480115211.

and please let us know how you get on.

1 Like

Thanks Paul,

Especially useful to point me to the code. I can see the use of ‘offset’ and ‘time’ etc. are mutually exclusive.

The example in the API description for ‘&time’ of

"Absolute time format (-6 is 1387730121 seconds since 1970-01-01 00:00:00 UTC)

https://emoncms.org/input/bulk.json?data=[[-10,16,1137],[-8,17,1437,3164],[-6,19,1412,3077]]&time=1480193370"

looks to be wrong numbers to me (I think). A typo?

The use proposed here is desired timestamps in the data and ‘&time=0’.

After testing, as it is easy to make an input unusable if you get it wrong, I have adjusted my live BASH code to use ‘&time=0’…

Thanks for the warning about ‘(int)’. Happily my code uses integer timestamp values rounded to 10 secs.

I am seeing timestamps on emoncms.org that match the posts at this point. I am hopeful of stable storage of data and that we have a solution!

At this point I am BULK uploading to a PHPTIMESERIES feed using the wanted timestamps and &time=0 - I will monitor this solution and update this thread with a conclusion.

Thanks again for the pointers, Paul.

Quite possibly, I do not 100% understand the use examples and the code for the “sentat=” and “offset=” looks absolutely identical to me, so I assume one is incorrect, I assume it to be the “sentat=” because is it essentially applying an “offset” and ignoring the time it was “sent at” for any other purpose than calculating an offset for each item that can be applied to the current time, effectively adding travel-time to the actual point in time it was sampled.

It seems all 4 methods do exactly the same in various ways and are all relative to the time received at emoncms unless you use “time=0”, I think this needs further investigation and if it is correct, emonhub would benefit from updating to use “time=0” too.

Thanks for trying and ;please keep us updated.

Ah yes, the input_controller.php code does seem to be identical for “sentat” and “offset” - strange.

BULK load to emoncms.org with PHPTIMESERIES feed using wanted timestamps and ‘&time=0’ does appear to store the wanted timestamps in emoncms.org.

So for example…

http://emoncms.org/input/bulk.json?apikey=…&data=[…[1480115210,21,21337],[1480115260,21,21337],[1480115270,21,20958],[1480115310,21,20958],[1480115320,21,21146],[1480115360,21,21146]]&time=0

will store in emoncms the timestamps used in the BULK call for a PHPTIMESERIES feed. (as tested, unless you know different!).

This method is used by me with an uploading process that robustly handles timestamps to load pulse data.

For my simple setup I see data sent over serial from a single EmonTx at 10s intervals… this is picked up by a script. The script gets the real time once, rounds it to 10s, and then for each emontx data set processed increments this ‘virtual’ time by 10s. This gives 10s timestamp intervals.

When a pulse is found in the data the script now uses BULK upload with ‘&time=0’ to a PHPTIMESERIES emoncms.org feed. This stores using the time of that 10s interval - great. (non pulse data is sent every 10s).

The virtual time will drift - time will tell by how much. If this is a problem I can code a solution. No need so far.

This is my attempt to guarantee good quality of data in emoncms.org. I perhaps need this absolute approach for pulse data, where I prefer to dictate the stored data. It appears to handle the known issues of time differences between machines or time in transit, you kindly detailed.

The BULK upload, ‘&time=0’, rounded timestamp approach may help with PHPFINA as well. Whilst the absolute time may not be stored for these feeds, it might eliminate the issues caused by time differences between machines etc.

Thanks, Paul, for the support.

A work in progress.