Could not do basic feed API query

jrbenito · 16 October 2017 19:19

Hi,

I am playing with my feeds in Emoncms.org but I could not make a feed API query to work. As per feed API help page, I can reach List.json for a list of my feeds. This works OK! However, same help page says I can reach

https://emoncms.org/feed/data.json?id=xxxxxx&start=1507647600&end=1508179408&interval=10

ID xxxxx → my KWh feed,
Start and End UNIX timestamp (oct-10-2017 and oct-16-2017)

The answer is:

[]

But the feed claims have data since Aug-18-2017, why I can´t ask for data in the period above?

Remarks:
→ In fact I only need two datapoints for this feed, the one in the start and one for the end timestamp. Is there a better way to get it?
→ if I put wrong timestamps the Emoncms.org complains as expected.

Sorry for this dumb question but I could not spot what I am doing wrong.

Thanks and best regards,
Benito

pb66 · 16 October 2017 21:12

Hi Benito

You are not the first, nor will you be the last to get caught out by this. The API expects the unix time stamps to be in milliseconds, so add 3 zeros to the start and end time stamps and it should work as originally expected, the interval, however is still expected in seconds.

jrbenito · 17 October 2017 12:14

thanks a lot, it works now. Shame on me !

GeorgeB · 17 October 2017 12:45

@pb66 any idea why the API provides an extra data point at 1507647610000 with an end time of 1507647600000 at 10s intervals?

pb66 · 17 October 2017 15:32

What feed type are you using George? I suspect phpfina, I do not have much experience with that feed type

Taking a quick peek at the code I would expect it is due to the “less than or equal to” test at L319 of the phpfina engine (assuming that’s the engine you are using)

github.com

emoncms/emoncms/blob/master/Modules/feed/engine/PHPFina.php#L319

    
      
              return false;
          }
          if($start<$meta->start_time) $start=$meta->start_time;
          $end_time=$this->lastvalue($id)['time'];
          if($end>$end_time) $end=$end_time;
          
          
//calculates address in dat file and number of values to write
          $pos_start=4*floor(($start-$meta->start_time)/$meta->interval);
          $pos_end=4*floor(($end-$meta->start_time)/$meta->interval);
          $nbwrites=($pos_end-$pos_start)/4;
          //echo("<br>$nbwrites");
          
          
//open the dat file
          $fh = fopen ($this->dir.$id.".dat","c+");
          if (!$fh){
              $this->log->warn("scale_range() : unable to open data file with id=$id");
              return false;
          }
          $this->log->warn("scale_range() : going to write $nbwrites values from address $pos_start to $pos_end");
          
          
//create a buffer with the processed values

since the $time variable is incremented at the beginning of each loop rather than at the end, each loop is processing the next incremented $time so the “or equal to” results in an additional iteration of $end + $interval.

You could try editing that line to read

        while($time<$end)

however I do not know what (if any) possible ramifications that might have elsewhere if other parts rely on that error.

pb66 · 17 October 2017 15:36

Not at all, we have all done it. It is not obvious or intuitive to use unixtime in millis, especially as the additional resolution has no effect, the default/maximum resolution in emoncms is whole seconds, the extra zero’s just get removed.

jrbenito · 17 October 2017 15:40

Yes, it is PHPFINA. I can see the same behavior, one extra data point after $end period.

pb66 · 17 October 2017 15:55

Looking at the commit history it looks like there were changes made in Mar '15 that might have knocked this code out of wack. Prior to that the $time var was incremented after reading the data point value from the data file,

github.com

emoncms/emoncms/blob/cd73f08504a718da14abc92d6951d96b7653d9aa/Modules/feed/engine/PHPFina.php#L218-L223

    
      
          // read from the file
          fseek($fh,$pos*4);
          $val = unpack("f",fread($fh,4));
          
          
// calculate the datapoint time
          $time = $meta->start_time + $pos * $meta->interval;

GeorgeB · 17 October 2017 15:55

Yes that’s the one.

I guess @TrystanLea will need to fix the bug on the EmonCMS.org server.

TrystanLea · 17 October 2017 16:26

Thanks for spotting @pb66 @GeorgeB , Il have to spend some time double checking this to ensure there are no knock on effects but I take note, thanks.

pb66 · 17 October 2017 16:38

Thanks @Trystan, It’s the same on both local and emoncms.org so I have opened an issue on both repos as a reminder

CDuffy · 12 January 2018 17:27

All,

I submitted the following pull request last night as the PHPFina get_data_dmy() was not returning the last full data point and also noticed the additional data points from get_data() detailed above.

github.com/emoncms/emoncms

Change get_data_DMY() to return all requested values

emoncms:master ← CDuffers:phpfina_tweak

opened 10:18PM - 11 Jan 18 UTC

CDuffers

+9 -10

Changed the logic in PHPFina.php for get_data_DMY() to: 1. When storing the val…ue in the array, remove the "$time<$end" as this was preventing the return of the last value and is also not required as the 'break' will prevent going past the end. This was also causing disparity with get_data() for the same date ranges 2. 'streamlined' the loop by making $increment a value which is defined once outside of the loop

Using an example of a monthly interval, I also found that where the data started to be collected within that interval that this interval is overlooked. e.g. month data starting 1st July, but the data was first recorded 15th July, is a null data point. An equivalent happens if requested end is after the last captured data point.

I propose that the get_data*() and get_average*() functions are expanded to include a ‘partial’ option, meaning that when requested it will return these first/ last entries and if appropriate could even place them against the datetime for that the start/ end, 15th July with the data point for the example above.

I’m willing to write something up, but wanted confirmation and thoughts before starting. We could also resolve Paul’s request as the same time.

Clive.

pb66 · 25 July 2018 10:46

Is this possibly related? Email report complains has not enough data, while it has