Could not do basic feed API query

jrbenito · 2017-10-16T19:19:52.675Z

Hi,

I am playing with my feeds in Emoncms.org but I could not make a feed API query to work. As per feed API help page, I can reach List.json for a list of my feeds. This works OK! However, same help page says I can reach

https://emoncms.org/feed/data.json?id=xxxxxx&start=1507647600&end=1508179408&interval=10

ID xxxxx → my KWh feed,
Start and End UNIX timestamp (oct-10-2017 and oct-16-2017)

The answer is:

[]

But the feed claims have data since Aug-18-2017, why I can´t ask for data in the period above?

Remarks:
→ In fact I only need two datapoints for this feed, the one in the start and one for the end timestamp. Is there a better way to get it?
→ if I put wrong timestamps the Emoncms.org complains as expected.

Sorry for this dumb question but I could not spot what I am doing wrong.

Thanks and best regards,
Benito

pb66 · 2017-10-16T21:12:38.128Z

Hi Benito

You are not the first, nor will you be the last to get caught out by this. The API expects the unix time stamps to be in milliseconds, so add 3 zeros to the start and end time stamps and it should work as originally expected, the interval, however is still expected in seconds.

jrbenito · 2017-10-17T12:14:26.136Z

pb66:

You are not the first, nor will you be the last to get caught out by this.

thanks a lot, it works now. Shame on me !

GeorgeB · 2017-10-17T12:45:34.271Z

@pb66 any idea why the API provides an extra data point at 1507647610000 with an end time of 1507647600000 at 10s intervals?

pb66 · 2017-10-17T15:32:00.784Z

What feed type are you using George? I suspect phpfina, I do not have much experience with that feed type

Taking a quick peek at the code I would expect it is due to the “less than or equal to” test at L319 of the phpfina engine (assuming that’s the engine you are using)

github.com

emoncms/emoncms/blob/master/Modules/feed/engine/PHPFina.php#L319

    
      
              return false;
          }
          if($start<$meta->start_time) $start=$meta->start_time;
          $end_time=$this->lastvalue($id)['time'];
          if($end>$end_time) $end=$end_time;
          
          
//calculates address in dat file and number of values to write
          $pos_start=4*floor(($start-$meta->start_time)/$meta->interval);
          $pos_end=4*floor(($end-$meta->start_time)/$meta->interval);
          $nbwrites=($pos_end-$pos_start)/4;
          //echo("<br>$nbwrites");
          
          
//open the dat file
          $fh = fopen ($this->dir.$id.".dat","c+");
          if (!$fh){
              $this->log->warn("scale_range() : unable to open data file with id=$id");
              return false;
          }
          $this->log->warn("scale_range() : going to write $nbwrites values from address $pos_start to $pos_end");
          
          
//create a buffer with the processed values

since the $time variable is incremented at the beginning of each loop rather than at the end, each loop is processing the next incremented $time so the “or equal to” results in an additional iteration of $end + $interval.

You could try editing that line to read

        while($time<$end)

however I do not know what (if any) possible ramifications that might have elsewhere if other parts rely on that error.

pb66 · 2017-10-17T15:36:28.132Z

jrbenito:

Shame on me !

Not at all, we have all done it. It is not obvious or intuitive to use unixtime in millis, especially as the additional resolution has no effect, the default/maximum resolution in emoncms is whole seconds, the extra zero’s just get removed.

jrbenito · 2017-10-17T15:40:48.152Z

Yes, it is PHPFINA. I can see the same behavior, one extra data point after $end period.

pb66 · 2017-10-17T15:55:12.511Z

Looking at the commit history it looks like there were changes made in Mar '15 that might have knocked this code out of wack. Prior to that the $time var was incremented after reading the data point value from the data file,

github.com

emoncms/emoncms/blob/cd73f08504a718da14abc92d6951d96b7653d9aa/Modules/feed/engine/PHPFina.php#L218-L223

    
      
          // read from the file
          fseek($fh,$pos*4);
          $val = unpack("f",fread($fh,4));
          
          
// calculate the datapoint time
          $time = $meta->start_time + $pos * $meta->interval;

GeorgeB · 2017-10-17T15:55:22.185Z

pb66:

I suspect phpfina

Yes that’s the one.

I guess @TrystanLea will need to fix the bug on the EmonCMS.org server.

TrystanLea · 2017-10-17T16:26:07.058Z

Thanks for spotting @pb66 @GeorgeB , Il have to spend some time double checking this to ensure there are no knock on effects but I take note, thanks.

pb66 · 2017-10-17T16:38:18.808Z

Thanks @Trystan, It’s the same on both local and emoncms.org so I have opened an issue on both repos as a reminder

github.com/emoncms/emoncms

phpfina feed/data.json returns extra data point

opened 04:35PM - 17 Oct 17 UTC

closed 02:21PM - 11 Jul 19 UTC

pb66

bug

See https://community.openenergymonitor.org/t/could-not-do-basic-feed-api-query/…5375?u=pb66

github.com/emoncms/emoncmsorg

phpfina feed/data.json returns extra data point

opened 04:35PM - 17 Oct 17 UTC

pb66

bug

See https://community.openenergymonitor.org/t/could-not-do-basic-feed-api-query/…5375?u=pb66

CDuffy · 2018-01-12T17:27:22.656Z

All,

I submitted the following pull request last night as the PHPFina get_data_dmy() was not returning the last full data point and also noticed the additional data points from get_data() detailed above.

github.com/emoncms/emoncms

Change get_data_DMY() to return all requested values

emoncms:master ← CDuffers:phpfina_tweak

opened 10:18PM - 11 Jan 18 UTC

CDuffers

+9 -10

Changed the logic in PHPFina.php for get_data_DMY() to: 1. When storing the val…ue in the array, remove the "$time<$end" as this was preventing the return of the last value and is also not required as the 'break' will prevent going past the end. This was also causing disparity with get_data() for the same date ranges 2. 'streamlined' the loop by making $increment a value which is defined once outside of the loop

Using an example of a monthly interval, I also found that where the data started to be collected within that interval that this interval is overlooked. e.g. month data starting 1st July, but the data was first recorded 15th July, is a null data point. An equivalent happens if requested end is after the last captured data point.

I propose that the get_data*() and get_average*() functions are expanded to include a ‘partial’ option, meaning that when requested it will return these first/ last entries and if appropriate could even place them against the datetime for that the start/ end, 15th July with the data point for the example above.

I’m willing to write something up, but wanted confirmation and thoughts before starting. We could also resolve Paul’s request as the same time.

Clive.

pb66 · 2018-07-25T10:46:44.653Z

Is this possibly related? Email report complains has not enough data, while it has