eMonCMS.org losing input data?

Emoncms
1

This morning, my wifi connection was down for about an hour while the vendor did some work on my cable hookup. During that time my power monitoring device (IotaWatt) continued to record power on the local SDcard but was unable to upload to eMonCMS.org. When service was restored, the device sent all of the backed up data at a rate of about one per second. (Each one represents about 10 seconds of monitoring). In examining the log, they were apparently all accepted (“ok” response). This was a test bench unit that was reporting voltage and a couple of small power channels. The voltage channel (first one), is configured to post to a feed, and that is where I see the data missing. This doesn’t seem to occur at “normal” posting rates.

When I look at the csv data of a plot of data that includes that time period, a few of the data points (identified by the unixtime stamp) are there, but most are not. All of the transmissions use the GET input/post?json&time=14…&csv=n,n,n,&apikey= format.

iotalog.pdf (12.0 KB)

So I’m wondering if you have an idea why this might have happened. I was soon going to change the way the device reports to use the input/bulk protocol and report up to ten different consecutive 10 second observations per transaction. Is that handled differently such that this problem might not occur? Any other thoughts?

2

Sorry about that, found the issue and fixed it. Post was implementing a fixed post limit based on the time as received on the server while Bulk was applying the limiter based on the timestamps sent in the request.

Both will now work with the time set in the request.

3

Thanks for the fast response. About the bulk mode…
I went ahead and recoded the eMonCMS posting service to use bulk mode to catch up. It now sends up to 1K urls that typically contain about 20 data intervals. It shovels them at a rate of about 2 per second. That may be too fast, and I can throttle it if need be.

So I had it start resending from where that lapsed period began to the present. Just tried it again to verify and the feed is not being updated. The protocol seems to work OK when sending new data, but doesn’t seem to do history very well. I can imagine that could be a problem for cumulative feeds like kWh, but shouldn’t be a problem for a basic power feed. If it’s not possible to update history, that’s fine, just so I know. Most of the time I would be just catching up the stream of new data.

But on a related topic, is it possible to choke the server? After about a minute of shoveling this historical data out, I start getting timeouts (no reply). Resending gets the same response. This could be something on my end, because after several retries, the ESP gets an exception while in the WiFiclient send code.

The server seems to make short work of just about everything and responds very fast to queries (especially since it has to cross the ocean), but still, there must be quite a bit of I/O on your end and maybe a queue that fills up?

As I said, this ought to work pretty well as is for basic catchup from down WiFi, and the choke point seems to about 5 hours worth of back data.

Thanks.

4

Trystan,

That was my problem. The only issue is that can’t update history.

Thanks

5

Sorry for the slow response, only just had a chance to look at this.

You’re right that both post and bulk upload have limiters on emoncms.org that are configured for catching up and normal posting rather than updating historic readings or filling in blanks. I had not fully appreciated that that was the case as I have only ever used the API for either normal operation or catching up (that was also my primary development goal) and Im aware that the PHPFina engine as it is configured for multi-server scaling on emoncms.org does not currently support updating of historic data anyway.

I have added a note to the API overview to clarify this (Emoncms - site api)

“Note: Both the input/post and input/bulk API’s can be used for real-time posting of data and for catching up (such as when an internet connection goes down). Emoncms.org does not however support updating or filling in of historic data after newer data has been written.”

On your other point, it should not be possible to choke the server if your script is sending out requests sequentially. If you send a large number of requests in parallel that may well cause problems. There are queue’s on the server but these are not bounded and no items are dropped so this should not be causing your issue.

Is there any chance that your bulk upload uploads a current time data point which then blocks historic data uploads perhaps?

6

Thanks for the reply,

Not a problem. The only reason I discovered it was that the previous problem that you fixed had created holes. That shouldn’t happen anymore. Now I’ve got a real time clock and keep track of what has been sent, so there should be no need to revisit history.

All of my transactions are now input/bulk and are synchronous and sequential, so no problem.

btw/ there’s someone by the handle of thundersun that reports bulk update problems that I suspect are the same issue.

Thanks.

7

I say I’m not an expert, I’m using self-built emonesp but for some time at a certain time data is lost on emoncms! why ?
I use the string:
…

 cmd = "GET /api/post?apikey=" + String(apikey) + "&node=12" ;
  cmd += "&json={light:" +  String(light) + ",wind:" + String(f) + "}/r/n";

  esp.print("AT+CIPSEND=");
  esp.println(cmd.length() + 10);
  
  if (esp.find(">")) {
    esp.println(cmd);

   
    Serial.println("cmd > OK");
  } else {
    errori++;
    Serial.print("errori :");
    Serial.println(errori);
    if (errori >10 ) {
      errori = 0;
      errore();  // reset arduino
    }

Any advice?