Deleting outliers from PHPFINA time series

MegaTux · 2018-12-28T00:30:45.484Z

Hi folks,

I have some outlier data from T/H sensors in m time series that I would lie o remove. Is there a way of editing PHPFINA time series data? I did a bit of a search in the archives and there is a mention of emoncms post editing tool but I don’t think it does what I need it to do. Basically, I would like to edit OUTLIERS to NULL in the time series database, point by point. Is this possible?

Thanks

pb66 · 2018-12-28T16:38:15.613Z

Did you look at the “spike removal” post-processing? You would need to manually set the min/max levels, it would not automatically identify the values to be removed based on the data itself if that is what you seek.

MegaTux · 2018-12-30T21:12:44.852Z

pb66:

post-processing

Hi @pb66,

I’v installed this in the /home/user/local folder on my server and created to the link to /var/www/html/emoncms as directed here and changed the postprocess.settings.php

<?php

$basedir = "/home/user/local/postprocess/";
$dir = "/var/lib/phpfina/";

I was getting errors at start-up of the PHP script with lines in the postprocess_run.php so decided to make the relative path changes in the main file as is below (you’ll see that I have EmonCMS in /var/www/html).

?php

define('EMONCMS_EXEC', 1);
chdir("/var/www/html/emoncms");
require "process_settings.php";
if (!isset($homedir)) $homedir = "/home/user/local";
$basedir = "$homedir/postprocess/";
$dir = $feed_settings["phpfina"]["datadir"];
chdir($basedir);

Now, when I launch the main script I get:

user@server:~/local/postprocess$ sudo php /home/user/local/postprocess/postprocess_run.php 
PHP Warning:  chdir(): No such file or directory (errno 2) in /home/user/local/postprocess/postprocess_run.php on line 9

Warning: chdir(): No such file or directory (errno 2) in /home/user/local/postprocess/postprocess_run.php on line 9
PHP Warning:  scandir(/home/pi/postprocess/processes): failed to open dir: No such file or directory in /home/user/local/postprocess/postprocess_run.php on line 19

Warning: scandir(/home/pi/postprocess/processes): failed to open dir: No such file or directory in /home/user/local/postprocess/postprocess_run.php on line 19
PHP Warning:  scandir(): (errno 2): No such file or directory in /home/user/local/postprocess/postprocess_run.php on line 19

Warning: scandir(): (errno 2): No such file or directory in /home/user/local/postprocess/postprocess_run.php on line 19
ERROR: Redis is not enabled

So the script is still picking up /home/pi when it does the scandir but I’m not quite sure why. The other thing is that its saying that REDIS is not running but that is not quite true as:

user  14372  0.0  0.0   4580  2264 pts/0    S+   21:11   0:00 grep redis
redis    25238  0.0  0.0  29664  2288 ?        Ssl  nov15  39:55 /usr/bin/redis-server 127.0.0.1:6379

Any clues?

Cheers

pb66 · 2018-12-30T21:29:17.264Z

What are these 2 settings set to in emoncms/settings.php?

$redis_enabled = true;

$homedir = "/home/username";

[Edit - As the use of /home/pi is so ingrained in the emoncms projects, you may even find it easier to just create a symlink at /home/pi to /home/user/local (or even /home/user/local/emoncms_modules to keep the modules together?) so that it appears the modules are in /home/pi.]

MegaTux · 2018-12-30T22:10:35.483Z

I’ve now set redis to Trus and change /home/username to /home/user. Reloaded Apache and now get:

user@server:~/local/postprocess$ php postprocess_run.php 
PHP Warning:  chdir(): No such file or directory (errno 2) in /home/user/local/postprocess/postprocess_run.php on line 9

Warning: chdir(): No such file or directory (errno 2) in /home/user/local/postprocess/postprocess_run.php on line 9
PHP Warning:  fopen(/tmp/postprocess-runlock): failed to open stream: Permission denied in /home/user/local/postprocess/postprocess_run.php on line 11

Warning: fopen(/tmp/postprocess-runlock): failed to open stream: Permission denied in /home/user/local/postprocess/postprocess_run.php on line 11
PHP Warning:  flock() expects parameter 1 to be resource, boolean given in /home/user/local/postprocess/postprocess_run.php on line 12

Warning: flock() expects parameter 1 to be resource, boolean given in /home/user/local/postprocess/postprocess_run.php on line 12
Already running

It seems to me that there is some permission problem somewhere?

Regarding your suggestion, are you suggesting creating a user pi on my server? and creating symlinks in there? I’d have to get all the permissions right … seems quite a lot of work.

Wouldn’t it be better to just install the modules in /var/www/htm/emoncms? Although the installation instructions recommend not doing this but just creating a symlink. Because of security?

pb66 · 2018-12-30T22:30:21.503Z

MegaTux:

Regarding your suggestion, are you suggesting creating a user pi on my server? and creating symlinks in there?

No, I suggested a single symlink where the user pig’s homedir would reside IF there was a user pi. Since you are using it specifically for emoncms modules only, that single symlink would point to the folder you are actually installing the modules to.

The symlink is just for locating so permissions should not be an issue if your user has the right permissions and the folders are owned by that user (use “pi” as just a path, not a user). It was just a suggestion to avoid any hiccups borne from not using the location that is (wrongly) assumed to be where everyone is using.

Of course this isn’t “the right way to do it” so by all means try and get the paths right via settings etc and edit quoted commands etc to your user if you choose, my suggestion was just something else to try.

MegaTux:

Wouldn’t it be better to just install the modules in /var/www/htm/emoncms? Although the installation instructions recommend not doing this but just creating a symlink. Because of security?

No, for security reasons. These modules should not be in the www path and have been designed to be installed, symlinked and used that way intentionally.

I also think you need to use sudo with the command according to the last paragraph of the readme

 sudo php postprocess_run.php

MegaTux · 2018-12-31T00:09:14.569Z

Really strange that it fails on the chdir() command of line 9. The directory is there. Could it be that by running this as root (using sudo) it won’t access the /home/user folder?

I’ve changed all the “/home/pi” to “/home/user” in the following files:

/var/www/html/emoncms/process_settings.php
/var/www/html/emoncms/default.emonpi.settings.php
/var/www/html/emoncms/default.settings.php

but its made no difference

Raffaele_Morelli · 2019-12-05T08:43:20.640Z

Any progress on this?
I have this DHT22 sensor which this evening reported a couple of bad values. Can’t see anything useful in postprocess scripts.

Robert.Wall · 2019-12-05T09:13:30.674Z

Search. I answered a similar question only a few days ago. If you can’t find it, I’ll try to look later.

borpin · 2019-12-05T19:37:46.102Z

On the Visualisation, there is an ‘editrealtime’ vis.

You can edit data points. Beware though, the code as it stands does not use exact data points.

If you work out what the exact data point is by it’s UNIX timestamp you can edit it (just enter the timestamp).

You can identify the exact time stamp from the graphing module, use the show CSV button at the bottom.

Beware (again) the graph does not easily show the actual data points. I have found the way to do this is to zoom right in, select

and possibly reduce the fixed interval time down to 1 and then click reload. Even then you may need to shift left/right to get an actual data point!

A pain I know .

Alternatively look at the Feed API help - there is an edit API call. You need to know the timestamp.

To prevent it, you can add a check in the input processing to skip if the value is greater than X.

Raffaele_Morelli · 2019-12-06T05:42:03.009Z

Yep, I finallly got rid of those two bad guys using editrealtime and added a min/max to inputs.
The data is every 2 minutes so easy to identify.
Thank you