# Best way to guarantee times when using BULK load with PHPTIMESERIES? **URL:** \ **Category:** Emoncms\ **Tags:** pulsecount, emoncms\ **Created:** [26 November 2016 13:21 UTC](https://community.openenergymonitor.org/t/best-way-to-guarantee-times-when-using-bulk-load-with-phptimeseries/2320 "2016-11-26T13:21:05Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![iov](https://community.openenergymonitor.org/letter_avatar_proxy/v4/letter/i/f17d59/32.png) [@iov](https://community.openenergymonitor.org/u/iov)\ **Post date:** [26 November 2016 13:21 UTC](https://community.openenergymonitor.org/t/best-way-to-guarantee-times-when-using-bulk-load-with-phptimeseries/2320/1 "2016-11-26T13:21:05Z")
Hi, New to forum… posting to [emoncms.org](http://emoncms.org) for 2 months. I am posting pulse data from a gas meter - so slow pulse rate. I am working out how to present infrequent pulse data with [emoncms.org](http://emoncms.org) (any slow pulse data!) I am using a PHPTIMESERIES feed. I noticed missing data so changed to posting using the json BULK format, to dictate times. How can I ensure the times I post are actually used by [emoncms.org](http://emoncms.org)? As an example I sent… http…&data=[…[1480115210,21,21337],[1480115260,21,21337],[1480115270,21,20958],[1480115310,21,20958],[1480115320,21,21146],[1480115360,21,21146]]&offset=1480115380&time=1480115380 The 2 times per value are generated at a pulse, power set for 10s after the last pulse and for when the pulse occurred - helps make nice graphs! For the above data I expected in [emoncms.org](http://emoncms.org) (in csv output)… 1480115210, 21337.0 1480115260, 21337.0 1480115270, 20958.0 1480115310, 20958.0 1480115320, 21146.0 1480115360, 21146.0 What I see is… 1480115217, 21337.0 (time is out by 7) 1480115267, 21337.0 (time is out by 7) 1480115277, 20958.0 (time is out by 7) 1480115317, 20958.0 (time is out by 7) 1480115321, 20958.0 (extra!!!) 1480115329, 21146.0 (time is out by 9) 1480115367, 21146.0 (time is out by 7) (times taken from very zoomed in editrealtime and also graph view with 1 sec interval) In this case I see an extra entry. In others posts entries go missing. The '‘out’ amount has also been 8. The lack of a guaranteed posting time means that [emoncms.org](http://emoncms.org) is not reliably presenting the posts. Can anyone explain this and how I can post to guarantee [emoncms.org](http://emoncms.org) stores/uses/presents the posted times? Do forgive me if this post is not quite to community standard! Thanks - in anticipation. ---
**Author:** ![pb66](https://community.openenergymonitor.org/user_avatar/community.openenergymonitor.org/pb66/32/27_2.png) [@pb66](https://community.openenergymonitor.org/u/pb66)\ **Post date:** [26 November 2016 16:12 UTC](https://community.openenergymonitor.org/t/best-way-to-guarantee-times-when-using-bulk-load-with-phptimeseries/2320/2 "2016-11-26T16:12:04Z")
Unfortunately there is no “absolute timestamp” bulk api, I started looking into this a while back as I was seeing some unusual timestamps logged, Firstly as you are using “offset=” the “time=” doesn’t get a look in as the offset option is before the time option in the bulk input [api’s `if else` block](https://github.com/emoncms/emoncms/blob/master/Modules/input/input_controller.php#L93-L114). As far as I could make out the only way you might be able to coerce emoncms to use the supplied timestamp is to use `time=0`, that way the calculated [`$time_ref`](https://github.com/emoncms/emoncms/blob/master/Modules/input/input_controller.php#L107) is zero and there should be no “adjustment” made when each [individual frames timestamp is processed](https://github.com/emoncms/emoncms/blob/master/Modules/input/input_controller.php#L126) further along the code. All the other options “[sentat=](https://github.com/emoncms/emoncms/blob/master/Modules/input/input_controller.php#L95)”, “[offset=](https://github.com/emoncms/emoncms/blob/master/Modules/input/input_controller.php#L101)” or even [legacy mode](https://github.com/emoncms/emoncms/blob/master/Modules/input/input_controller.php#L113) (by omitting an overall timestamp mode) use the timestamp of the server to establish a `$time_ref`, so time difference between the machines and even time in transit could skew the timestamps. So try (reusing your example) http…&data=[…[1480115210,21,21337],[1480115260,21,21337],[1480115270,21,20958],[1480115310,21,20958],[1480115320,21,21146],[1480115360,21,21146]] **&time=0** Also bear in mind that because the input api uses `(int)$itemtime` the timestamp gets truncated to the whole second, not rounded so `1480115210.999` would become 1480115210 not 1480115211. and please let us know how you get on. ---
**Author:** ![iov](https://community.openenergymonitor.org/letter_avatar_proxy/v4/letter/i/f17d59/32.png) [@iov](https://community.openenergymonitor.org/u/iov)\ **Post date:** [26 November 2016 21:14 UTC](https://community.openenergymonitor.org/t/best-way-to-guarantee-times-when-using-bulk-load-with-phptimeseries/2320/3 "2016-11-26T21:14:29Z")
Thanks Paul, Especially useful to point me to the code. I can see the use of ‘offset’ and ‘time’ etc. are mutually exclusive. The example in the API description for ‘&time’ of "Absolute time format (-6 is 1387730121 seconds since 1970-01-01 00:00:00 UTC) [https://emoncms.org/input/bulk.json?data=[[-10,16,1137],[-8,17,1437,3164](https://emoncms.org/input/bulk.json?data=%5B%5B-10,16,1137%5D,%5B-8,17,1437,3164)],[-6,19,1412,3077]]&time=1480193370" looks to be wrong numbers to me (I think). A typo? The use proposed here is desired timestamps in the data and ‘&time=0’. After testing, as it is easy to make an input unusable if you get it wrong, I have adjusted my live BASH code to use ‘&time=0’… Thanks for the warning about ‘(int)’. Happily my code uses integer timestamp values rounded to 10 secs. I am seeing timestamps on [emoncms.org](http://emoncms.org) that match the posts at this point. I am hopeful of stable storage of data and that we have a solution! At this point I am BULK uploading to a PHPTIMESERIES feed using the wanted timestamps and &time=0 - I will monitor this solution and update this thread with a conclusion. Thanks again for the pointers, Paul. ---
**Author:** ![pb66](https://community.openenergymonitor.org/user_avatar/community.openenergymonitor.org/pb66/32/27_2.png) [@pb66](https://community.openenergymonitor.org/u/pb66)\ **Post date:** [27 November 2016 13:10 UTC](https://community.openenergymonitor.org/t/best-way-to-guarantee-times-when-using-bulk-load-with-phptimeseries/2320/4 "2016-11-27T13:10:21Z")
> [@iov](#): > > looks to be wrong numbers to me (I think). A typo? Quite possibly, I do not 100% understand the use examples and the code for the “sentat=” and “offset=” looks absolutely identical to me, so I assume one is incorrect, I assume it to be the “sentat=” because is it essentially applying an “offset” and ignoring the time it was “sent at” for any other purpose than calculating an offset for each item that can be applied to the current time, effectively adding travel-time to the actual point in time it was sampled. It seems all 4 methods do exactly the same in various ways and are all relative to the time received at emoncms unless you use “time=0”, I think this needs further investigation and if it is correct, emonhub would benefit from updating to use “time=0” too. Thanks for trying and ;please keep us updated. ---
**Author:** ![iov](https://community.openenergymonitor.org/letter_avatar_proxy/v4/letter/i/f17d59/32.png) [@iov](https://community.openenergymonitor.org/u/iov)\ **Post date:** [28 November 2016 19:30 UTC](https://community.openenergymonitor.org/t/best-way-to-guarantee-times-when-using-bulk-load-with-phptimeseries/2320/5 "2016-11-28T19:30:32Z")
Ah yes, the input\_controller.php code does seem to be identical for “sentat” and “offset” - strange. BULK load to [emoncms.org](http://emoncms.org) with PHPTIMESERIES feed using wanted timestamps and ‘&time=0’ does appear to store the wanted timestamps in [emoncms.org](http://emoncms.org). So for example… [http://emoncms.org/input/bulk.json?apikey=](http://emoncms.org/input/bulk.json?apikey=)…&data=[…[1480115210,21,21337],[1480115260,21,21337],[1480115270,21,20958],[1480115310,21,20958],[1480115320,21,21146],[1480115360,21,21146]]&time=0 will store in emoncms the timestamps used in the BULK call for a PHPTIMESERIES feed. (as tested, unless you know different!). This method is used by me with an uploading process that robustly handles timestamps to load pulse data. For my simple setup I see data sent over serial from a single EmonTx at 10s intervals… this is picked up by a script. The script gets the real time once, rounds it to 10s, and then for each emontx data set processed increments this ‘virtual’ time by 10s. This gives 10s timestamp intervals. When a pulse is found in the data the script now uses BULK upload with ‘&time=0’ to a PHPTIMESERIES [emoncms.org](http://emoncms.org) feed. This stores using the time of that 10s interval - great. (non pulse data is sent every 10s). The virtual time will drift - time will tell by how much. If this is a problem I can code a solution. No need so far. This is my attempt to guarantee good quality of data in [emoncms.org](http://emoncms.org). I perhaps need this absolute approach for pulse data, where I prefer to dictate the stored data. It appears to handle the known issues of time differences between machines or time in transit, you kindly detailed. The BULK upload, ‘&time=0’, rounded timestamp approach may help with PHPFINA as well. Whilst the absolute time may not be stored for these feeds, it might eliminate the issues caused by time differences between machines etc. Thanks, Paul, for the support. A work in progress.