Community
OpenEnergyMonitor

Community

Deleting outliers from PHPFINA time series

phpfina
editing
Tags: #<Tag:0x00007f1be55985f0> #<Tag:0x00007f1be5598460>

(A.G.) #1

Hi folks,

I have some outlier data from T/H sensors in m time series that I would lie o remove. Is there a way of editing PHPFINA time series data? I did a bit of a search in the archives and there is a mention of emoncms post editing tool but I don’t think it does what I need it to do. Basically, I would like to edit OUTLIERS to NULL in the time series database, point by point. Is this possible?

Thanks


(Paul) #2

Did you look at the “spike removal” post-processing? You would need to manually set the min/max levels, it would not automatically identify the values to be removed based on the data itself if that is what you seek.


(A.G.) #3

Hi @pb66,

I’v installed this in the /home/user/local folder on my server and created to the link to /var/www/html/emoncms as directed here and changed the postprocess.settings.php

<?php

$basedir = "/home/user/local/postprocess/";
$dir = "/var/lib/phpfina/";

I was getting errors at start-up of the PHP script with lines in the postprocess_run.php so decided to make the relative path changes in the main file as is below (you’ll see that I have EmonCMS in /var/www/html).

?php

define('EMONCMS_EXEC', 1);
chdir("/var/www/html/emoncms");
require "process_settings.php";
if (!isset($homedir)) $homedir = "/home/user/local";
$basedir = "$homedir/postprocess/";
$dir = $feed_settings["phpfina"]["datadir"];
chdir($basedir);

Now, when I launch the main script I get:

[email protected]:~/local/postprocess$ sudo php /home/user/local/postprocess/postprocess_run.php 
PHP Warning:  chdir(): No such file or directory (errno 2) in /home/user/local/postprocess/postprocess_run.php on line 9

Warning: chdir(): No such file or directory (errno 2) in /home/user/local/postprocess/postprocess_run.php on line 9
PHP Warning:  scandir(/home/pi/postprocess/processes): failed to open dir: No such file or directory in /home/user/local/postprocess/postprocess_run.php on line 19

Warning: scandir(/home/pi/postprocess/processes): failed to open dir: No such file or directory in /home/user/local/postprocess/postprocess_run.php on line 19
PHP Warning:  scandir(): (errno 2): No such file or directory in /home/user/local/postprocess/postprocess_run.php on line 19

Warning: scandir(): (errno 2): No such file or directory in /home/user/local/postprocess/postprocess_run.php on line 19
ERROR: Redis is not enabled

So the script is still picking up /home/pi when it does the scandir but I’m not quite sure why. The other thing is that its saying that REDIS is not running but that is not quite true as:

user  14372  0.0  0.0   4580  2264 pts/0    S+   21:11   0:00 grep redis
redis    25238  0.0  0.0  29664  2288 ?        Ssl  nov15  39:55 /usr/bin/redis-server 127.0.0.1:6379 

Any clues?

Cheers


(Paul) #4

What are these 2 settings set to in emoncms/settings.php?

$redis_enabled = true;

$homedir = "/home/username";

[Edit - As the use of /home/pi is so ingrained in the emoncms projects, you may even find it easier to just create a symlink at /home/pi to /home/user/local (or even /home/user/local/emoncms_modules to keep the modules together?) so that it appears the modules are in /home/pi.]


(A.G.) #5

I’ve now set redis to Trus and change /home/username to /home/user. Reloaded Apache and now get:

[email protected]:~/local/postprocess$ php postprocess_run.php 
PHP Warning:  chdir(): No such file or directory (errno 2) in /home/user/local/postprocess/postprocess_run.php on line 9

Warning: chdir(): No such file or directory (errno 2) in /home/user/local/postprocess/postprocess_run.php on line 9
PHP Warning:  fopen(/tmp/postprocess-runlock): failed to open stream: Permission denied in /home/user/local/postprocess/postprocess_run.php on line 11

Warning: fopen(/tmp/postprocess-runlock): failed to open stream: Permission denied in /home/user/local/postprocess/postprocess_run.php on line 11
PHP Warning:  flock() expects parameter 1 to be resource, boolean given in /home/user/local/postprocess/postprocess_run.php on line 12

Warning: flock() expects parameter 1 to be resource, boolean given in /home/user/local/postprocess/postprocess_run.php on line 12
Already running

It seems to me that there is some permission problem somewhere?

Regarding your suggestion, are you suggesting creating a user pi on my server? and creating symlinks in there? I’d have to get all the permissions right … seems quite a lot of work.

Wouldn’t it be better to just install the modules in /var/www/htm/emoncms? Although the installation instructions recommend not doing this but just creating a symlink. Because of security?


(Paul) #6

No, I suggested a single symlink where the user pig’s homedir would reside IF there was a user pi. Since you are using it specifically for emoncms modules only, that single symlink would point to the folder you are actually installing the modules to.

The symlink is just for locating so permissions should not be an issue if your user has the right permissions and the folders are owned by that user (use “pi” as just a path, not a user). It was just a suggestion to avoid any hiccups borne from not using the location that is (wrongly) assumed to be where everyone is using.

Of course this isn’t “the right way to do it” so by all means try and get the paths right via settings etc and edit quoted commands etc to your user if you choose, my suggestion was just something else to try.

No, for security reasons. These modules should not be in the www path and have been designed to be installed, symlinked and used that way intentionally.

I also think you need to use sudo with the command according to the last paragraph of the readme

 sudo php postprocess_run.php

(A.G.) #7

Really strange that it fails on the chdir() command of line 9. The directory is there. Could it be that by running this as root (using sudo) it won’t access the /home/user folder?

I’ve changed all the “/home/pi” to “/home/user” in the following files:

/var/www/html/emoncms/process_settings.php
/var/www/html/emoncms/default.emonpi.settings.php
/var/www/html/emoncms/default.settings.php

but its made no difference :frowning: