Data Viewer stats by API call

borpin · 17 October 2018 22:06

Continuing the discussion from Data Viewer questions:

I asked this question a while ago and the answer got rather side tracked!

I also asked previously about a rate of change calculation, actually, if these stats were available (for this use case, Diff), and I knew the time period over which I had requested the data, I could easily calculate a Rate of Change.

My use case is to determine whether heating is required and that is to some degree dependent on rate of change of external temperature.

I could store the data in Node-red and do it, but that seems daft!

As a point of note, the data viewer exports the data in milliseconds. Is this deliberate or is this just because Javascript always uses date/time in milliseconds? Internally I am pretty sure seconds are used (I could be wrong).

pb66 · 18 October 2018 18:43

I did answer on that thread (although it did get buried by significant side tracking)

Looking closer at the code, there are api endpoints for average but not for min and max etc

github.com

emoncms/emoncms/blob/9.9.0/Modules/feed/feed_controller.php#L90-L95


      
          else if ($route->action == 'average') {
              if (isset($_GET['interval'])) {
                  $result = $feed->get_average($feedid,get('start'),get('end'),get('interval'));
              } else if (isset($_GET['mode'])) {
                  $result = $feed->get_average_DMY($feedid,get('start'),get('end'),get('mode'));
              }

This tallies with the phpfina engine only having functions for average but no mention of min and max functions.

github.com

emoncms/emoncms/blob/9.9.0/Modules/feed/engine/PHPFina.php#L744-L1091


      
          Averaging (mean)
          
          The standard data request method returns the value of the data point at the specified timestamp 
          this works well for most data requests and is the required method for extracting accumulating kWh data
          
          However its useful for many applicaitons to be able to extract the average value for a given period
          An example would be requesting hourly temperature over number of days and wanting to see the average temperature for each hour
          rather than the temperature at the start of each hour.
          
          Emoncms initially did this with a dedicated feed engine called PHPFiwa the implementation of that engine calculated the average 
          values on the fly as the data came in which is not a particularly write efficient method as each average layer is opened for every
          update and only one 4 byte float is appended.
          
          A better approach is to post process the average layers when the data request is made and to cache the calculated averages at this 
          point. This significantly reduces the write load and converts the averaged layers into a recompilable cached property rather than
          and integral part of the core engine.
          
          */
              
          public function get_average($id,$start,$end,$interval)

This file has been truncated. show original

However, there doesn’t appear to be any mention of average in the phptimeseries engine so use of the “average” feed api must be limited to phpfina only.

The graphs calculate these things “on page” as it were, I too think it would be very useful to have api endpoints (and the functions in the feed engines if needed) so that we could query that data directly, as then could the graphs page and apps etc thus reducing the need for any duplication of the calculations, it would offer consistent results from one place (per engine?).

I believe use of millis is indeed due to the use of JS, but the lowest resolution emoncms can work to is 1 second so the end 3 zeros in any unixmillis timestamp are just padding.

pb66 · 18 October 2018 19:40

Just tested that theory and the results are…

for a phpfina feed (id 77), and…

for a phptimeseries feed (id 71)

Seems there is no mention of the feed/average.json api endpoint in the main repo feed api help page, only the emoncms.org help page shows it, but I have just confirmed it’s working on my own server.

borpin · 19 October 2018 16:20

Yes I did see it but significant is right.

I dug through the code a bit more and realised that those ‘stats’ are calculated using a javascript library I think. I’ll look for a different solution.

TrystanLea · 19 October 2018 16:35

Yes that’s correct, a specific feature of the graph module Im afraid. Again If anyone would like to help with adding min/max API end points for date ranges as you suggest I would be happy for these to be added, Ideally building on the existing feed/data API rather than a full search of the underlying data as that will have scaling issues on multi user installations.