Request datapoint limit until 8928 points

RDCH106 · 2016-06-03T08:18:12.525Z

PHPFina and PHPTimseries have implemented a hardcoded datapoint limit until 8928 points.

What is the reason to this specific limit? And what is the reason this amount of hardcoded points is present in the two class.

Could be the limit moved to settings file?

Why “PHPFIWA” knows no bounds in the “get_data” function?

amgican · 2016-08-25T21:47:16.592Z

I have the same question. How can I change this limit to return any number of datapoints for export? What’s the procedure? I’m having a hell of a time getting the data I need from my emonPi because of this limitation.

amgican · 2016-08-30T03:46:42.215Z

Can someone respond to this please? It is seriously limiting for exporting data without knowing how to up the limit. It has really put a clog in the toilet of progress for my energy auditing work

This error occurs when I’m trying to “repost” from the data viewer (http://emonpi.local/emoncms/graph/) after setting the interval to 60 (feed input was sampled every 10 seconds).
Request error {"success":false,"message":"Request datapoint limit reached (8928), increase request interval or time range, requested datapoints = 12510"}

Where and how can I make the change to this limit?

pb66 · 2016-08-30T09:18:33.038Z

amgican:

It has really put a clog in the toilet of progress for my energy auditing work

A quick search of the old forum turned up this metaphorical plunger!

CSV export problems - Request datapoint limit reached (8928)

amgican · 2016-08-30T13:14:47.921Z

amgican:

Where and how can I make the change to this limit?

I did see that earlier post on the old forum, but it does not answer my question. I searched for the paths described in that post and could not find them.

Jon · 2016-08-30T14:33:41.515Z

I cannot answer the questions about the limits or PHPFIWA. But I have a small PHP script that grabs all of the data directly from PHPTimeseries and drops it into a .csv file. It will grab all data points. It was written for the emonPi. Would that help?

amgican · 2016-08-30T14:59:01.943Z

That sounds great! But unfortunately, all of my feeds are fixed interval PHPFINA (I still haven’t yet determined which one is better).
I’m very strong with SQL, so potentially I could simply grab the data that way. But I still haven’t figured out how to run queries on my Mac to the MySQL DB on the emonPi. And if I’m not mistaken, this data is not actually stored in MySQL, right?
I would be interested in your script though, as I might be able to modify it to access the PHPFINA data instead.

Jon · 2016-08-30T15:10:14.037Z

Got one for phpfina also! The code is from something Trystan wrote and then I modified slightly for my needs:

#!/usr/bin/php
<?php
$time_start = microtime(true);

// Directory of phpfina feeds, see: settings.php
chdir ('/home/pi/data/myApps');
$dir = "/home/pi/data/phpfina/";
    
// Feed id to read: 
$feedid = 2;
 
//----------------------------------------------------

    // read meta data
    $meta = new stdClass();
    $metafile = fopen($dir.$feedid.".meta", 'rb');
    fseek($metafile,8);
    $tmp = unpack("I",fread($metafile,4)); 
    $meta->interval = $tmp[1];
    $tmp = unpack("I",fread($metafile,4)); 
    $meta->start_time = $tmp[1];
    fclose($metafile);
    
    $fh = fopen($dir."$feedid.dat", 'rb');
    $filesize = filesize($dir."$feedid.dat");
    
    $npoints = floor($filesize / 4.0);
    
    $csv_file = "feed_" .$feedid. ".meta.csv";
    $fp = fopen($csv_file, 'w');

    
    for ($i=0; $i<$npoints; $i++)
    {
        $val = unpack("f",fread($fh,4));
        $time = $meta->start_time + $i * $meta->interval;
        $value = $val[1];
        $array = array($time, $value); 
        if (!is_nan($value))
        {
            fputcsv($fp, $array);
            //print $time."\t".$value."\n";
        }
        
        //print $time."\t".$value."\n";
    }
    fclose($fp);

print "feedDB: " .$dir.$feedid.".meta\n";
print "csv_file: " .$csv_file."\n";
print "filesize: " .$filesize."\n";
print "npoints: " .$npoints."\n\n";

$time_end = microtime(true);
$time = $time_end - $time_start;
echo "Ran in $time seconds\n";

?>

All of the apps/scripts I add are placed in a “myApps” directory. Make sure you change these three lines for your needs:

chdir ('/home/pi/data/myApps');
$dir = "/home/pi/data/phpfina/";

$feedid = 2;

I know there is some data in mySQL (maybe some settings?!?) but I don’t believe it is the data you are looking for.

pb66 · 2016-08-30T19:23:36.478Z

emoncms/Modules/feed/engine/PHPFina.php

emoncms/Modules/feed/engine/PHPTimeSeries.php

These paths are relative to the location of your emoncms directory, if that is, for example /var/www/html/emoncms then the path you need for PHPFina.php is /var/www/html/emoncms/Modules/feed/engine/PHPFina.php. You will need to check the path of your install, if it’s an emonSD it think it maybe just /var/www/ then the relevant path.

Here are the exact lines with the hardcoded limit direct from the repo.

github.com

emoncms/emoncms/blob/master/Modules/feed/engine/PHPFina.php#L301


      
          
          /**
           * scale a portion of a feed
           * added by Alexandre CUER - january 2019
           *
           * @param integer $id The id of the feed
           * @param integer $start unix time stamp in ms of the start of the data range
           * @param integer $end unix time stamp in ms of the end of the data rage
           * @param float $scale : numeric value for the scaling
          */
          public function scalerange($id,$start,$end,$scale){
          
              $id = (int) $id;
              $start = (int) $start;
              $end = (int) $end;
          
              //echo("test on $scale not started");
              //case1: NAN
              if(preg_match("/^NAN$/i",$scale)){
                  $this->log->warn("scale_range() : going to erase data range with NAN");
                  $scale = NAN;

github.com

emoncms/emoncms/blob/master/Modules/feed/engine/PHPTimeSeries.php#L236


      
                      }
                  } else {
                      $buffer .= pack("CIf",249,$time,$value);
                  }
              }
              // Otherwise append a new value
              if ($buffer!="") {
                  fseek($fh, $npoints*9);
                  fwrite($fh,$buffer);
              }
              fclose($fh);
              return true;
          }
          
          /**
           * Get array with last time and value from a feed
           *
           * @param integer $id The id of the feed
          */
          public function lastvalue($id)
          {

Does this answer your original question?

amgican:

How can I change this limit to return any number of datapoints for export? What’s the procedure?

or do you need further guidance on how to edit the file?

amgican · 2016-08-30T20:09:01.107Z

Well while you were writing your response, I had finally found the files needing editing and I’d saved the file PHPFina.php. Thought it would be a good idea to reboot the system, so I did. Then after reboot, under feeds, there were none listed, only a message “loading…” (or something like that). So I rebooted again, and now I’m stuck in a loop of reboot > loading raspberry pi > shutdown in 5… 4… 3… 2… 1… > reboot > ad nauseum. I am so fed up with this thing, I’m about ready to throw it across the room and smash it into pieces! GAH!
How can I get out of this reboot loop?

pb66 · 2016-08-30T20:24:38.607Z

I’m afraid I am not familiar enough with the emonSD to know the answer to that.

My best guess would be (assuming your edits to correct etc) perhaps on reboot your emonPi was updating and that’s why there were no feeds displayed yet, and by rebooting again, perhaps before it has finished you now have another issue.

I think you need @glyn.hudson expertise or perhaps @jon can confirm if the updating runs automatically at boot, when did you last reboot the Pi?

I know It’s a bit late now but you only needed to restart the apache server at most with sudo service apache2 restart but to be honest I think it probably would have picked up the changes without a restart.

What changes did you make? just up’d the limit or commented out the line altogether? or something else ?

EDIT - The more I think about it, I think the auto update might only occur on first boot of a new emonSD image, so that might not be relevent.

Jon · 2016-08-30T20:45:37.629Z

pb66:

confirm if the updating runs automatically at boot,

The only auto update I know of is the first boot update. But I would guess Rodney is past that one.

.

Rodney - is the emonPi going thru this sequence by itself? Or is there some manual intervention (like leaning on the switch for 5 seconds)??

amgican:

reboot > loading raspberry pi > shutdown in 5… 4… 3… 2… 1… > reboot > ad nauseum.

amgican · 2016-08-31T03:41:54.709Z

I’m recreating my steps with re-imaging all the way through where I had the problem and documenting along the way. Will post more after I get farther along in the process. Thanks for stickin’ with me.

Jon · 2016-08-31T05:06:56.177Z

glad to hear! based on the above it sounds like the right choice.

glyn.hudson · 2016-08-31T13:50:34.758Z

amgican:

now I’m stuck in a loop of reboot > loading raspberry pi > shutdown in 5… 4… 3… 2… 1… > reboot > ad nauseum.

It sounds like the emonPi shutdown switch is being help down or something is shorting on the GPIO pins on the RaspberryPi. The emonPiLCD.py script performs the shutdown when it receives a signal via GPIO.

amgican · 2016-08-31T15:28:44.670Z

Alrighty gentlemen, thanks much for sticking with me on this journey.
While I still plan to check out your script @Jon , I’ve answered my original question by doing the following (thanks for all the tips and hints). I’m writing this long and detailed description so that other folks like myself won’t have the same problem finding the solution to enabling more than 8928 data points.
< optional >
Re-imaged the SD card using the latest build (May, 2016) and booted the system to that newest version.
After first boot, updated the system from http://emonpi.local/emoncms/admin/view > Update emonPi
After reboot, restored my backup data from http://emonpi.local/emoncms/backup > Import
Checked that feeds existed, they did, although my inputs (and their unique names) were not there any longer. Bug? No biggy, I can configure those again later.
< /optional >
Accessed the system with SSH: $ SSH pi@EmonPi.local (pwd: emonpi2016)
Put the filesystem in read-write mode: $ rpi-rw
Change to the directory: $ cd /var/www/emoncms/Modules/feed/engine
Backup the file: $ cp PHPFina.php PHPFina.bkp
Open the file in a text editor: $ nano PHPFina.php
scroll down to this line:
if ($req_dp>8928) return array('success'=>false, 'message'=>"Request datapoint limit reached (8928), increase request interval or time range, requested datapoints = $req_dp");
change to:
if ($req_dp>100000) return array('success'=>false, 'message'=>"Request datapoint limit reached (100000), increase request interval or time range, requested datapoints = $req_dp");
Optionally, you could add a variable and edit that variable as needed:
$dp_max = 100000
if ($req_dp>$dp_max) return array('success'=>false, 'message'=>"Request datapoint limit reached ($dp_max), increase request interval or time range, requested datapoints = $req_dp");

Control + X, save the file, exit nano.
Put the filesystem into read only mode: $ rpi-ro
Restart apache: $ sudo service apache2 restart
Open a data viewer for a feed, load lots of datapoints. No issues.
Was able to view CSV output with 100s of thousands of datapoints AND no gaps in the data!

I have no idea what was causing the cycled reboots at this point. I could very well have been that I’d pinched the button when installing the end cap. If that is the case, then, D’oh!

And now for the kudos…
This new version 9.7.2 and it’s graphing and data output are the bomb!!! Excellent work all of you hard working, open-source-dedicated, making me have hope in humanity, programmers and geeks and experimenters involved in this project. Good on ya!
Let me just tell you how happy it makes me to see these options for data output:
Time format: Three choices! and Null values: Three choices! This is exactly what I needed. You guys rock!

Some observations… this might be a bug. If I export to CSV from http://emonpi.local/emoncms/graph/ using the same start/end time as I would using the download/export popup listed for the same feed, the quantity of records is not the same. Perhaps I should move this question to another thread?

Feature request: enable mouse hover over options to popup a description of the option. i.e., when I hover over the text “scale” in the data viewer, it would sure be nice to know what the heck that option means.

I learned a lot from this three day adventure. I’m really hoping to retain a bit of this new knowledge too
Thank you @pb66, @Jon, @TrystanLea, and @glyn.hudson for being responsive and creating such a useful tool for my business. I’ve developed a rather nice niche market here in the southwest US because of the openenergymonitor project. Cheers, all.

pb66 · 2016-08-31T20:00:32.700Z

Good to hear you’ve sussed it!