I have emon CMS running locally on on emonPI (not emoncms.org) - writing data to a feed from an external machine using the API, when graphing the data I see gaps depending on the time range I plot over:
The data is written approximately 1/minute, I have a local log on the machine (PI Zero W) making the API call, this is receiving the {“success”: true} message from emon CMS. for each call.
See attached three graphs - over 12 hours there are gaps in the graph (and the CSV export) between 01:00 and 03:00. If I then zoom in over 2 Hours I start to see the data both in the graph and in the CSV but with gaps again, if I zoom to 30 mins see all the data at 1 minute resolution.
I just tried extracting the data from the emon using the API, using some python uilities see below, data is is all present between 01:00 and 03:00- so it looks like something related to the rendering in the graph module
One thing is to check the box at the top ‘Show missing data’, but if you then extrcat via API and it is all there, something is off - one for @TrystanLea
What is the Feed Interval set to? Hover over the Feed name on the Feeds page, or look at the setting in the processing steps on the Inputs page, and how often does data arrive? Is it exactly on the minute?
If the interval in which data arrives is longer than your feed interval, you will get null values in the database. When graphed, the points are selected according to the timebase and if it happens to land on a null value, it doesn’t look for a near adjacent one, nor does it average over a range.
Your feed interval is based on Internet time (as known to your PiZero), you need to adjust the rate at which data is sent so that it is slightly faster than the Feed Interval you set. This means that occasionally, depending on the ratio of the two times, a second value will arrive before the time slot expires and the previous value is overwritten and lost, but you will avoid empty (‘null’) slots.
So for example, if data arrives every 1’ 10", you need a feed interval of 2 mins to guarantee every ‘slot’ has valid data, but you’ll lose some of the values sent.
The source PI runs a service (systemd) that is basically a loop, that reads a couple of DS18B20 1 wire sensors (assumed a 2 second timeout) and then waits - the wait is set to 55 seconds, so the data does indeed get send slightly faster than 1/minute. It’s a linux stack, not a microcontroler so there will a bit of variability in this, depending on what else is going on but it should still be < 60 seconds
This is news to me! my prior experience in in industrial historians where you just get the time-value pairs, is confusing me. PHPFINA has no timestamp.
The feed interval is set to 10seconds (no idea how perhaps was the default way back in time). So there are indeed a lot of null’s
If the graph point hits a null it’s ignored, this is different behavior to the feed page where it displays the last (non null) value e.g. below. there have been 5 null values after the 24.31 but it still shows 24.31
My API python library pulls these Null as np.nan and is set to dump them so you don’t see them in the final screenshot.
NEXT QUESTION…
On the input page I can’t seem to set the feed interval for an existing feed, is there a way to do this? or do I have to create a new feed?
If I create a new feed is there a way to write the old feed data back to it? - I can always extract it via the API drop the null and write it back myself, but wondered if there is a way in emon?
This is the difference between Emoncms Fixed Interval Timeseries and Emoncms Variable Interval Timeseries. The second does indeed work the way you describe, storing time-value pairs. The first stores the creation time and the interval as metadata, and computes the time for each data point from this, which explains:
so yes, you must create a new Feed.
(Incidentally, emonCMS has always been this way - the recent change is the names, how the two databases work is unchanged. There used to be detailed explanation in ‘Learn’, but it’s disappeared and not made it into ‘Docs’.)
If you create a new Feed with a 60 second interval, I think you should be OK.
(And yes, 10 s is the default.)
No it isn’t really - this is showing you the last value going into the Feed, not a historic value from the database.
I think there will be, but I’m not an emonCMS expert, so I’m ignorant on that subject. Beware, once you’ve put data into a Feed, you can never send earlier data to it.
I think where you’ve got yourself has happened before, so a search might yield a result. Failing that, you need Trystan. (An @mention normally works, given a day or two.)
It’s possible to downsample feeds using the emoncms postprocess module. It’s also possible to remove null values, by interpolating across these missing values, again using the post process module.
You can also use the Log to feed (join) input process to do this automatically as data comes in, though be aware that it will interpolate with a straight line across large data gaps if these occur…
@carl_afgte45, one other solution is to send the data as a JSON and include a time element. This is then used by the system and avoids drift and nulls (if the time is exactly 10s increments ).
Thanks I will probably just resample myself and drop the Null/nan values outwith the emonCMS in pandas and use the API to put back to a new feed at 1 minute frequency, I feel more confident doing this than using the front end.
Few remaining questions, sorry if these are in the docs but I cant see them. PHPFINA uses a start date/time, sample frequency and a count to work out the timestamp of any given sample,
What happens if I make a call using the API and it falls between two timestamps e.g.
Start at midnight 1 minute resolutions, the samples would be
00:00
00:01
00:02 etc
What happens if I make an API call with time of 00:01:30? (given I am processing the existing 10s data data this is feasible) is it mapped to a timestamp of 00:01 or 00:02 or is it ignored?
I understand I can’t write data before the start of the feed, but can I write data in the middle of an existing feed, or have all my API calls got to be increasing in time?
When a new feed is created what is used as the start time, is it the actual time e.g. my timestamp samples could be
00:01:37
00:02:37
00:03:37 etc or are the started from 00:01 ?
I guess doing PHPFINA is less storage intensive than storing a datetime-value tuple? but what if I corrupt the data - if I am missing samples in middle I can’t calculate the timestamps?
I have progressed a bit with this - I have all my old data extracted via the API, at 60 second resolution and now want to write it back to the emon.
I have created a new feed id=77, and am following the instructions on the emon API help: http://localhost:8080/feed/api to insert multiple data points where I seem to have to write a list of lists of the format
[timestamp,value]
To be explicit the emonpi is on my LAN at home, I am not at home so I have a SSH tunnel running from my laptop this is why my url’s begin http://localhost:8080/ doing this I can easily pull data via the API so I dont think it’s relevant to the problem, but thoughts welcome…
OK so I have progressed, first problem was my fault, the “406” error comes from using the read API key when trying to write, previously the library only ever did reads the single point write worked (with no APIKEY) as the browser was already authenticated by username and password.
Trying now with 1 days worth of data I get an error of.
414 Request-URI Too LongThe requested URL's length exceeds the capacity\nlimit for this serverApache/2.4.54 (Raspbian) Server at localhost Port 80
Checking the string the length of the request is 23,881 bytes, I don’t know the actual limit on the emon apache server, but a bit of googling seems to indicate a default of 8192
trying with 6 hours of data I have a length of 6078, trying this I get ‘{“success”:true}’
SO I wrote a script to generate 6 hours chunks and managed to writeback just short of a years worth of tidied up data to a 1 minute frequency new feed. script took about 5 minutes to run, I then changed the input to write to the new feed, job done…
I have a few more feeds like his that took the default 10seconds I will fix in coming days, when python code is tidied up I will do final post with it attached
).