5 hours of no updating of Feeds

David_Eager · 2018-01-09T18:13:06.821Z

Interestingly I’ve looked at two other (and older) installations and they are still working fine

pb66 · 2018-01-09T19:00:59.573Z

David_Eager:

Interestingly I’ve looked at two other (and older) installations and they are still working fine

The OEM hosted emoncms is operating across multiple linked servers to handle the traffic, so it’s totally possible for some users to be effected but not others, When I checked my account earlier it was ok too, I didn’t post that as it wasn’t helpful to you guys.

albertras · 2018-01-09T19:04:47.919Z

I experienced same problem today. My feeds were also 5 hours delayed but it is now 4 hours late. The graphs are correct and the same as the local graphs but just delayed. On the emonhub the data send to https://emoncms.org is getting an OK. Anything that I can check or do?

Vicbitter57 · 2018-01-09T20:46:37.647Z

Hi, I,m the same, says last update 4 hours ago, Emonhub also says data sent OK.

Vicbitter57 · 2018-01-10T03:47:30.218Z

Down to 2 hours now, waiting in anticipation

BThomas · 2018-01-10T09:57:41.776Z

Now its below 1 hour.

CelsoHenriques · 2018-01-10T10:19:25.606Z

Hi there.
I am still having problems too, with 53 min last updated.
Yesterday MQTT wasn’t able to connect.
Local connection was working always fine.
I did a full reset to my emonpy (didn’t mind about the data) but results were the same.

TrystanLea · 2018-01-10T11:24:03.526Z

Apologies for the long queue, the issue seemed to have been caused by an IotaWatt uploading a couple of months of history yesterday - it should really be able to handle it so I will need to look into that once the queue clears.

The queue should be fully clear in just over 12 hours from now.

CelsoHenriques · 2018-01-10T11:43:45.303Z

The only problem was the issue was making me think my emonpi was having some problem until i check this thread. Otherwise, everything is fine since now i know the problem.
I wonder, maybe emoncms.org need some solution to prevent a single upload to cause this big slowdown. Imagine more of this happens.
Best regards

overeasy · 2018-01-10T12:57:59.392Z

TrystanLea:

the issue seemed to have been caused by an IotaWatt uploading a couple of months of history yesterday -

I was aware of one IoTaWatt user that was affected by the backlog, but that was less than a day backup and seemed to be more of a result than a cause.

Nevertheless, it’s to be expected that one might upload several months, or even a years worth of data. I’d like to know more about what it is that triggers the problem. Whether it’s simply volume or has to do with specific input processes employed by this user.

Right now, there isn’t any feedback system to indicate any problem. In fact, I had recently posted an issue where EmonCMS apparently responded OK to a POST message, then couldn’t parse the data and discard it.

So we should investigate some method of validating successful posting, both to verify that it is done, and possibly as a mechanism to throttle large upload streams.

rearden · 2018-01-11T13:31:24.890Z

I still do not see changes I made to the Emonbase Inputs and Feeds show up in EmonCMS Inputs and Feeds. I am new at this so I may have done something wrong, but it approaching 2 days since I made the changes and the Updated column is reporting seconds rather than hours like it was.

Has the database worked through its backlog or have I done something incorrect in my setup?

The other thread is
here

thanks
rearden

blaal02 · 2018-01-11T14:55:16.360Z

Hi Rearden,
So, you have an EmonBase, that’s configured to send the data to 2 Emoncms instances - one local (either within the Pi that’s running the data acquisition or separate), and one on emoncms.org. And the emonHuib piece seems to be working; the data values are indeed being sent to both the local & .org instances - correct?
I think the issue might be expectations! These 2 Emoncms instances are independent - they get fed the same data, but they need to be configured separately.
Thanks,
Sandy

pb66 · 2018-01-11T16:19:30.424Z

rearden:

but it approaching 2 days since I made the changes and the Updated column is reporting seconds rather than hours like it was.

Has the database worked through its backlog or have I done something incorrect in my setup?

It would be useful if someone could confirm the backlog is now clear and normal service has resumed, but in the absence of that confirmation it sounds like it has from the fact the update column values you are seeing are now green and just seconds, previously you were seeing orange values of 4hrs. In which case, yes further debugging should return to your original thread, I had only pointed you here as it was pointless trying to debug your input processing using input data that was 4hrs old.

rcojoe · 2018-01-11T16:40:38.030Z

All my feeds are caught up. They finished catching up around 6pm EST (New York timezone) yesterday. I was part of the group that was back logged. When I look now, everything is within 10 seconds of being updated last.

barche · 2018-01-11T18:54:29.804Z

My feeds are 45 hours behind now. I am uploading data using Node Red, but it doesn’t seem to report any errors. Anything I can check?

CelsoHenriques · 2018-01-11T19:49:26.152Z

Well i am not even getting anything. I deleted the inputs, waiting for them to refresh, rebooted the emonpi, update it and still no inputs available. Emonpi is connected to emoncms.org but emoncms dont receive inputs info or refreshing it.
Any sugestions?

TrystanLea · 2018-01-11T20:20:41.012Z

Thankyou All. To confirm the queue is full clear as of 24h ago.

@barche, @CelsoHenriques I cant see anything that might be causing the issue on emoncms.org, so Im not sure whats going wrong there, @barche do you see any ‘ok’ replies?

barche · 2018-01-11T21:25:21.190Z

TrystanLea:

@barche do you see any ‘ok’ replies?

Sorry, it was my mistake. When initially seeing no updates, I rebooted the emonPi, and it seems I misconfigured the static network settings so DNS was not set at boot. Working fine again now, thanks!

CelsoHenriques · 2018-01-12T02:30:05.918Z

Sorry, i made a newbie mistake too. Since i reset my emonpi, i forgot to add the emoncms api key to emonhub, so, its normal that it didnt receive the information. Sorry TrystanLea. I edited the api key and its working fine now.
Thanks!

TrystanLea · 2018-01-12T10:07:31.085Z

Great, good to hear!