CSV Export Problem

Getting Started
1

@TrystanLea

I’m running ver 10.1.6 and trying to do a CSV Export …
Meter pulse data from Apr 2019 to Sep 2019 at 1 min intervals - so approx 216,000 data lines
I use phptimeseries with a 10 sec feed interval

But I get the following error message after approx 89,000 downloaded lines …

Fatal error: Maximum execution time of 30 seconds exceeded in /var/www/emoncms/Modules/feed/engine/PHPTimeSeries.php on line 588

I’ve found 3 instances of php.ini at …

/etc/php/7.0/cli/php.ini
/etc/php/7.0/cgi/php.ini
/etc/php/7.0/apache2/php.ini

… and in each changed maximum_execution_time to 240
But still the same 30 secs error report - both before and after a reboot.

If I use a 5 mins interval then no problem.

Any suggestions most welcome
Thx

2

Hello @johnbanks

not sure why changing the setting in the ini file is not working but I notice I have the equivalent setting in php in the mysqltimeseries engine here: emoncms/MysqlTimeSeries.php at master · emoncms/emoncms · GitHub

You could try placing that line at the start of the export function here:
emoncms/PHPTimeSeries.php at master · emoncms/emoncms · GitHub

if that works Id be happy to include in the code.

3

@TrystanLea
Thx - I tried your suggestion but it did not work unfortunately.
After a bit more googling and on a hunch …
I inserted set_time_limit(300); as the second line immediately after the <?php in /var/www/emoncms/Modules/feed/engine/PHPTimeSeries.php
Then to ‘stress test’ I tried exporting 10 sec data in Excel time format for a 5 months period …
I got the warning message about the file size - 34 MB
It took 5 mins before Excel opened the csv file - 597K lines - and a fatal error that the 300 sec execution time had been exceeded.

It seems that second line in the php script is the correct place for the set time limit.

Might this also be the case for the ~~~~MysqlTimeSeries.php script?

Incidently, my googling suggested that changing the php.ini file did not work because php.ini was not in the /home directory

Thx

4

@TrystanLea
Does the above demonstrate the case for changing the code?
Regards

5

Raise a PR in GitHub proposing the change?

6

@borpin
Would if I knew how
@TrystanLea
Is this one for you?

7

Everyday a learning day…

For simple changes, it is easy to do within GitHub itself.

Go to the file page and click on edit
image
Don’t worry about the message.

Make the change and then submit it with a comment as to why. It will ask you to create a ‘Pull Request’. Again it will ask for a comment as to why.

There is nothing as satisfying as seeing one of your suggestions (PRs) being merged i.e. included in the system.

8

Hi,

Is this issue has been fixed already?
I run on low-write 10.2.3, and I have the same problem:
When exporting in csv, the last 4 lines of the .csv file are:

1575899620,223.08
1575899630,220.22
<br />
<b>Fatal error</b>:  Maximum execution time of 30 seconds exceeded in <b>/var/www/emoncms/Modules/feed/engine/PHPFina.php</b> on line <b>710</b><br />

which is pretty annoying because I don’t know if I got all the requested data (but it seems so), and I need to get rid of the last 2 lines to use it in any timeserie software (pandas in python for instance).

In my case I have 18 feeds recording every 10sec. And I had to download 6 month per 6 month for each one of the 18 feeds. So to overpass the ‘25Mo limit’ I had to download by hand 32 times. But even when being below the red limit of 25Mo, I’ve got the ‘Fatal error’ above.

Is there any workaround?

Thx!

9

@NPL
There has been no official change as far as I’m aware.

See my post above …

After a bit more googling and on a hunch …
I inserted set_time_limit(300); as the second line immediately after the <?php in /var/www/emoncms/Modules/feed/engine/PHPTimeSeries.php

That certainly circumvented the 30 secs restriction in 10.1.6.

You would need make the change in the corresponding file for PHPFina.