Adding Missing Data into a Feed

Emoncms
1

We had an issue with a network outage, so some data did not get uploaded from my emonPi2 to the emonCMS cloud. The data does exist on the emonPi2 so I downloaded the missing data and attempted to import it to the emonCMS cloud. Because the data predates the data on the emonCMS cloud, it does not get loaded into the emonCMS cloud feed.

How do I add this missing data?

thank you,

Gordon

2

Hello @grod55

How long is the outage and how many feeds do you wish to fix?

I wonder if it might be worth using the sync module as standard for your applications, turning off the emonhub http method and having it upload the data every hour via the sync module instead…?

3

Hi Trystan,

As this happens frequently (we are using cellular connections) my preference is to use the sync module then I don’t have to worry about it. What is the most frequent allowable interval to run sync?

Is there a document describing how to set up sync? I tried to set it up quite a while ago and was not successful.

Thank you,

Gordon

image001.png

4

Hi Trystan,

In doing some more research in the forums, it seems that the sync module has a downside that does not address my situation “One key downside with the current implementation is that it only uploads or downloads data that is new i.e. after last end time of the feed that is behind local or remote. It does not upload edited datapoints if they are made before the end time.”

With cellular, it is possible to have an outage that only lasts a few minutes and then restores itself. So, my latest data timestamp is beyond the missing period of data. Missing a few 10 second data intervals is fine here and there, but large gaps are what I want to be able to address. I find the gaps when I look for “null”. We run a simple check of each sensor should generate 8,640 data points in a 24-hour period (24606 for 10 second data).

If I used sync instead of the emonhub http method, would I never have gaps? Also, does sync use http or https? (I can’t use https as it consumes too much bandwidth).

Thanks,

Gordon

5

Hello @grod55

A couple of parallel threads here:

  1. Technically emonHub should buffer data during a connection outage and then upload this data once the connection resumes. It seems that this has not been working as it should and Im not quite sure why. I’ve made a minor change to emonhub to increase the maximum upload size from 100 frames to 1000 frames this morning and tested uploading 21,600 frames (~11 days at 20s interval). This seems to have worked fine even if I turn off my laptop internet connection mid-way… I’ve uploaded this minor change to version 11.2.13 of emonhub. Could you try updating emonhub to see if this makes any difference for you?

  2. Sync module: Sync module should never have gaps. You can configure the sync module to use http instead of https by setting the host name to http here:

image

  1. Bandwidth for the sync module when used for regular uploads is not optimised, I want to do more work on this. I would say the sync module is great for hourly uploads or slower, but not really recommended for less than that. This is something I want to fix/improve.
6

Hi Trystan,

Thank you for looking into this. I will update and see what happens.

I’m curious how long you disconnected your laptop for? In a previous instance we had a site lose connectivity for almost a week (it turned out to be a bad power cord to the cellular router) would emonHub buffer the data for that long of a period? Based on what you say regarding emonCMS sync, I am going to hold off trying it for now. With sync I am assuming it is a choice of either use emonHub or sync, but not both – correct?

Lastly, on a somewhat related topic, is it possible to merge two feeds? Specifically, we finalized our feed naming convention for consistency and have a couple of older systems where we want to bring in historical feed data into the newly named feed.

Best regards,

Gordon

7

I created a test where I filled the emonHub buffer with 11 days worth of data , I had the internet disconnected at first to make sure that the upload attempts did not clear the data. As soon as I connected the internet again the data started uploading correctly.

One thing to watch out for is if you have cron task that restarts emonHub that will also clear the buffer. We have such a script on the emonHP image due to a data lockup issue on some MBUS meters and it’s something I need to look at again - I did add some code to the MBUS interfacer recently to reset the serial connection internally. The standard emonSD image does not have this cron reset.

There is a merge feed post process in the post process module, that should do what you are after.

8

Hi Trystan,

I think you may have identified the cause of my problem. After correcting the connectivity issue, I restarted emonHub just to give things a fresh start, not realizing this would have cleared the buffer of data.

I will have a look at the feed merge.

Thank you,

Gordon

1 Like