I have a number of feeds with slowly changing status information (a few changes per day max). I am using PHPTimeSeries for this. Sometimes I would like to be able to load historic data which I understand is not supported. I think there might be two options:
If the old mysql engine existed I guess I could use this, but in current emoncms this seems to be limited to an in-memory database, whereas I would like persistence.
I know I can clear a feed data through the web interface; if there was a public API call for this I could extract the existing data, clear the feed and load a merged dataset.
Are either of these possible, or is there a better way?
If I was doing this, I would import historic data into a new feed (via button at bottom of Feeds page), then add a log step to the input process to append to that new feed. Once everything looks fine, remove old feed from the input, and delete.
Sorry, I don’t follow. I can create a new feed and add a log process to an input to append to it, but as soon as I do so I can’t add any more historic data (as I understand). My problem is that the historic data arrives with some non-trivial delay, so what do I do with the input while I’m waiting for it? I almost need a sort of buffer to wait while the historic data catches up, but it would be a lot easier to add entries out of sequence…
You need to import* the historic data before adding a log process, as timeseries cannot add values before the start of the feed.
Any new values that come in while importing historic data to a new feed will be recorded on the original feed, so you’ll be able to export/import the last few values when all the historic data is imported. As the data is infrequent, this shouldn’t be much.
*actually just need to import the 1st historic value to set the start of the feed, then you can append new values while importing old ones. I can’t guarantee this will work though.
OK, I get the need to do the import to get the feed start time correct. I think I understand what you are suggesting is that
start with input-X logging to feed-X
import historic data to feed-X-historic (in time order)
keep doing this until feed-X-historic has caught up with the start of feed-X
export feed-X and import it into feed-X-historic and update input-X logging to feed-X-historic
And I make sure I do the last stage reasonably fast so as to not miss any values. Is that right? It’s a bit manual but it would be possible as a one-off I suppose.
Note that I’m pretty certain that you can’t import old values while appending new ones to the same feed. I’ve just tried it with a test feed, and while the import dialogs say success, actually the data doesn’t get in. I think it would work for PHPFina, but not for PHPTimeSeries.
I think this won’t work. PHPtimeseries is now called Variable Interval Time Series & stores the time and the value for each record.
I think records must be in strict order of increasing time.
Therefore, all the historic data must be transferred first before any incoming data is logged.
Fixed Interval Time Series stores the start time and Interval and computes the record position from the time (and vice versa).
The help says Feed engine:
Emoncms Fixed Interval TimeSeries (PHPFina) is the recommended feed engine, it is a fixed interval timeseries engine.
Emoncms Variable Interval TimeSeries (PHPTimeseries) is for data posted at a non regular interval.
Robert - I think Tim’s solution would work for Variable Interval Time Series because the idea is to run two feeds in parallel, both in strict order of time. As soon as they overlap, you paste any new data from the live feed onto the end of the data in the historic feed and then use that as the new live feed.
In any case, I’m now using a slightly different approach, that when I want to load data pre-dating the existing feed contents I
download all the existing data (feed/data.json)
clear the feed entirely (feed/clear.json)
merge it with my historic data
upload the new data (feed/insert.json)
It seems to work OK. The only wrinkle is that the download step is oddly slow, but I don’t have to do it often.