Been scratching my head on this for a while now and discovered it must be at the emoncms.org server as when I send duplicate data to another server it has the correct times.
Below shows nodes 29 and 30 being sent to 2 emoncms instances with timestamps. node 29 at emoncms.org has stopped updating or is running very slow.
One for @TrystanLea I suspect?
Interesting! The queues all look fine over the last couple of days. If one node is ahead and the other on time within a single account that suggests an issue outside of emoncms.org as the emoncms queues are divided in blocks of users, so any slow down should affect all inputs in a given account. Has it all caught up now?
Yes, thanks, it all caught up later that evening. Node 29 was something I was testing and not running 24/7 so I can’t say whether it would have remained that way or not. But as odd as it might seem, the node 29 and node 30 frames were delivered to emoncms.org via the same single bulk upload with timestamps within a couple of seconds of each other. That’s how I came to conclude it was at the server end, I was watching the emonhub.logs and could see the frames arrive, get processed with similar timestamps and dispatched as 2 identical bulk upload requests to 2 emoncms targets and got “ok” replies from each.
The emoncms.org target showing this delay whilst the other showing correctly.
If not the queuing, the only other thing I could speculate towards might be IF during my tests I might of accidentally sent a frame with an incorrect timestamp in the future, perhaps the different servers handled the subsequent inputs differently??? I’ve no evidence or suspicion of using incorrect timestamps, but I did change from untimestamped to timestamped data at some point in writing the script. I had also checked the clocks on all the devices and found them ok during debugging this.
The other server is running older v9.??? emoncms without any queue management.
It isn’t a problem for me, but the delayed input times displayed at emoncms.org were definitely incorrect for node 29 whilst correct for 30. node 29 was being updated alongside 30 so the update times should have been the same (within a couple of seconds).
In fact, the fact it slowly caught up would suggest a queuing issue too, if the timestamps were to blame it would correct instantly, either immediately the correction was made or after passing the future timestamp if the inputs do now function like a low-write phpfina feeds in some way.
Oh well, not a problem for me, so don’t spend any time on it.
Thanks @pb66 interesting I can see how youd think it points to emoncms.org, perhaps there’s something Im missing, I will keep an eye out.
Hi @TrystanLea this has happened again
Identical data posted via http bulk api from the same emonhub source to 2 emoncms instances and emoncms.org is showing as last updated 2 hrs ago.
This time I can confirm that the clock was out on one of the Pi’s and there most likely was a period were the timestamp supplied was wrong. However to be 2 hrs out after correction, I would have expected to see the inputs page showing “last updated” minus 2 hours ago prior to correction? And that certainly didn’t happen.
It would seem there is something funky going on when submitted timestamps change. In this instance “sentat” would have been 2 hours behind so (old) emoncms would have added 2hrs to all the timestamps, putting them in the future. But that doesn’t explain why now that all timestamps are correct, emoncms shows last updated 2 hrs ago unless last updated is based on the difference between supplied timestamps.
The data values are static at the values presented with “2hr slow” timestamps and I suspect it will remain that way for 2 hour until the new data timestamps pass the stuck timestamp.
This is a very old emonhub setup that still uses “sentat=” in the bulk api, I now always use “time=0” so that’s probably why I haven’t seen this before. But this error doesn’t occur on emoncms version 10.1.6-beta (without input queuing), it’s just a redundant test server that happens to still be live, it’s quite by chance that we have this to compare against.
This bug really doesn’t bother me, but I thought you should be aware and I can perhaps recreate to test if needed.
@TrystanLea I just spotted that the OEM emonhub repo still uses “sentat” not “time=0”.
I thought this had already been discussed and changed.
If a users clock is out (say) 2 hours slow, when the data is submitted to emoncms.org via emonhub and the bulk input api, sentat="now_minus_2_hours" emoncms will try to correct that error and add 2hrs to all the timestamps, If the user is using timestamped data from another source with a correct clock, the correct timestamps get increased by 2hrs ie 2hrs into the future and then once the emonhub hosting Pi’s clock gets corrected, there will be a 2 hour period where inputs do not update until the new timestamped data + 0hrs correction (because sentat = now) is beyond the incorrectly posted data.
All this is avoided by using a fixed time=0 rather than sentat=. There is a discussion somewhere here on the forum about achieving absolute timestamps via the various emoncms options
This is what caught me out as the data was coming from a very old emonhub, all my more recent installs use time=0 I no longer use sentat and it seems that emoncms.org handles this in a less helpful way. My old test server running emoncms 10.1.6-beta behaves very differently, the inputs page still shows as updating when the new data overlaps the old data rather than just staying static until the post dated data becomes historic.
