I am seeing delays of 14 minutes on input processing on emoncms.org. it was 10 minutes at 9:00. So appears to be getting worse. Hopefully @TrystanLea will confirm emoncms.org health?
Delay 34 minutes at 15:50
Delay 48 minutes at 17:30
На сервере часы перевели на час … ???
via a translate module this gives : On the server, the clock was translated for an hour … ???
Forgot to translate
At 19:50 49 minutes. My data includes a heartbeat from which I know the send time and I also dictate the time emoncms stores. This means my last updated delay is how long it takes emoncms to process that heartbeat stamped post after receipt. It appears emoncms.org is running a queue which has grown. Just reporting for investigation.
The delay is now normal.
The only issue for me, as I dictate stored times to emoncms.org, is my graphs etc are not up to date when emoncms.org gets large delay/queue.
Perhaps (?) for those posting with no times, so seeing emoncms.org add time when it processes it from it’s queue, this will cause distortion of data. If a post is sent now but fully processed 48 minutes later it will be stored with a time 48 minutes late? As the queue changes size it will stretch out or (worse?) compress data.
I think that because I force the times emoncms.org uses that my delay times reflect large queues.
Those who understand the code may know if this is an issue or not.
Kind regards
Sorry for the slow processing time on emoncms.org over the last day. It cleared over night but the queues are now growing again today and the usual load balancing solution has not solved the issue this morning. I’m looking into it.
@iov Timestamps are assigned as the data arrives at emoncms.org, before going into the queues so datapoint timing is preserved.
@TrystanLea Thanks for the update. Sounds good.
I am guessing others were also missing the ‘near live’ monitoring.
Interestingly, I dictate timestamps using ‘bulk’ api posts (explicit time and &time=0) with these timestamps exactly 10s apart. emoncms.org uses these as-is and reliably puts data into PHPFINA 10s periods. As I post gas pulses, I do not want to miss a single post!
With timestamp on arrival by emoncms.org they were affected by my local 10s loop variance, time difference with emoncms.org and variable Internet transmission times - this often saw 2 posts going into the same PHPFINA 10s period and leaving a PHPFINA period with no posts, when bad it would see clusters of missing data. Since changing to ‘dictated’ timestamps emoncms.org has done a great job at presenting data.
Thanks for looking at queue handling. Your work to keep emoncms.org as ‘near live’ as possible is really appreciated.