Hi folks,
I have some outlier data from T/H sensors in m time series that I would lie o remove. Is there a way of editing PHPFINA time series data? I did a bit of a search in the archives and there is a mention of emoncms post editing tool but I don’t think it does what I need it to do. Basically, I would like to edit OUTLIERS to NULL in the time series database, point by point. Is this possible?
Thanks
Did you look at the “spike removal” post-processing? You would need to manually set the min/max levels, it would not automatically identify the values to be removed based on the data itself if that is what you seek.
Hi @pb66,
I’v installed this in the /home/user/local folder on my server and created to the link to /var/www/html/emoncms as directed here and changed the postprocess.settings.php
<?php
$basedir = "/home/user/local/postprocess/";
$dir = "/var/lib/phpfina/";
I was getting errors at start-up of the PHP script with lines in the postprocess_run.php so decided to make the relative path changes in the main file as is below (you’ll see that I have EmonCMS in /var/www/html).
?php
define('EMONCMS_EXEC', 1);
chdir("/var/www/html/emoncms");
require "process_settings.php";
if (!isset($homedir)) $homedir = "/home/user/local";
$basedir = "$homedir/postprocess/";
$dir = $feed_settings["phpfina"]["datadir"];
chdir($basedir);
Now, when I launch the main script I get:
user@server:~/local/postprocess$ sudo php /home/user/local/postprocess/postprocess_run.php
PHP Warning: chdir(): No such file or directory (errno 2) in /home/user/local/postprocess/postprocess_run.php on line 9
Warning: chdir(): No such file or directory (errno 2) in /home/user/local/postprocess/postprocess_run.php on line 9
PHP Warning: scandir(/home/pi/postprocess/processes): failed to open dir: No such file or directory in /home/user/local/postprocess/postprocess_run.php on line 19
Warning: scandir(/home/pi/postprocess/processes): failed to open dir: No such file or directory in /home/user/local/postprocess/postprocess_run.php on line 19
PHP Warning: scandir(): (errno 2): No such file or directory in /home/user/local/postprocess/postprocess_run.php on line 19
Warning: scandir(): (errno 2): No such file or directory in /home/user/local/postprocess/postprocess_run.php on line 19
ERROR: Redis is not enabled
So the script is still picking up /home/pi when it does the scandir but I’m not quite sure why. The other thing is that its saying that REDIS is not running but that is not quite true as:
user 14372 0.0 0.0 4580 2264 pts/0 S+ 21:11 0:00 grep redis
redis 25238 0.0 0.0 29664 2288 ? Ssl nov15 39:55 /usr/bin/redis-server 127.0.0.1:6379
Any clues?
Cheers
What are these 2 settings set to in emoncms/settings.php?
$redis_enabled = true;
$homedir = "/home/username";
[Edit - As the use of /home/pi is so ingrained in the emoncms projects, you may even find it easier to just create a symlink at /home/pi to /home/user/local (or even /home/user/local/emoncms_modules to keep the modules together?) so that it appears the modules are in /home/pi.]
I’ve now set redis to Trus and change /home/username to /home/user. Reloaded Apache and now get:
user@server:~/local/postprocess$ php postprocess_run.php
PHP Warning: chdir(): No such file or directory (errno 2) in /home/user/local/postprocess/postprocess_run.php on line 9
Warning: chdir(): No such file or directory (errno 2) in /home/user/local/postprocess/postprocess_run.php on line 9
PHP Warning: fopen(/tmp/postprocess-runlock): failed to open stream: Permission denied in /home/user/local/postprocess/postprocess_run.php on line 11
Warning: fopen(/tmp/postprocess-runlock): failed to open stream: Permission denied in /home/user/local/postprocess/postprocess_run.php on line 11
PHP Warning: flock() expects parameter 1 to be resource, boolean given in /home/user/local/postprocess/postprocess_run.php on line 12
Warning: flock() expects parameter 1 to be resource, boolean given in /home/user/local/postprocess/postprocess_run.php on line 12
Already running
It seems to me that there is some permission problem somewhere?
Regarding your suggestion, are you suggesting creating a user pi on my server? and creating symlinks in there? I’d have to get all the permissions right … seems quite a lot of work.
Wouldn’t it be better to just install the modules in /var/www/htm/emoncms? Although the installation instructions recommend not doing this but just creating a symlink. Because of security?
No, I suggested a single symlink where the user pig’s homedir would reside IF there was a user pi. Since you are using it specifically for emoncms modules only, that single symlink would point to the folder you are actually installing the modules to.
The symlink is just for locating so permissions should not be an issue if your user has the right permissions and the folders are owned by that user (use “pi” as just a path, not a user). It was just a suggestion to avoid any hiccups borne from not using the location that is (wrongly) assumed to be where everyone is using.
Of course this isn’t “the right way to do it” so by all means try and get the paths right via settings etc and edit quoted commands etc to your user if you choose, my suggestion was just something else to try.
No, for security reasons. These modules should not be in the www path and have been designed to be installed, symlinked and used that way intentionally.
I also think you need to use sudo with the command according to the last paragraph of the readme
sudo php postprocess_run.php
Really strange that it fails on the chdir() command of line 9. The directory is there. Could it be that by running this as root (using sudo) it won’t access the /home/user folder?
I’ve changed all the “/home/pi” to “/home/user” in the following files:
/var/www/html/emoncms/process_settings.php
/var/www/html/emoncms/default.emonpi.settings.php
/var/www/html/emoncms/default.settings.php
but its made no difference 
Any progress on this?
I have this DHT22 sensor which this evening reported a couple of bad values. Can’t see anything useful in postprocess scripts.
Search. I answered a similar question only a few days ago. If you can’t find it, I’ll try to look later.
On the Visualisation, there is an ‘editrealtime’ vis.
You can edit data points. Beware though, the code as it stands does not use exact data points.
If you work out what the exact data point is by it’s UNIX timestamp you can edit it (just enter the timestamp).
You can identify the exact time stamp from the graphing module, use the show CSV button at the bottom.
Beware (again) the graph does not easily show the actual data points. I have found the way to do this is to zoom right in, select
and possibly reduce the fixed interval time down to 1 and then click reload. Even then you may need to shift left/right to get an actual data point!
A pain I know 
.
Alternatively look at the Feed API help - there is an edit API call. You need to know the timestamp.
To prevent it, you can add a check in the input processing to skip if the value is greater than X.
Yep, I finallly got rid of those two bad guys using editrealtime and added a min/max to inputs.
The data is every 2 minutes so easy to identify.
Thank you