Postprocess - new process to facilitate calculations

Emoncms
1

Just to share some infos about the postprocess module

I’ve developped a new process allowing to make some basic calculations between feeds : multiplication, division, soustraction and addition…
In a single operation, you can make for example : 2*f1+f2/3 -f3/f4. Brackets are not implemented but it is less complicated than using the Polish notation :slight_smile: The validity of the formula is tested and analysed through different regular expressions…
Actually you have to know the feed numbers but it can be easy to add an helper with a select menu…

postprocess
postprocess1896×1359 189 KB

@TrystanLea @glyn.hudson : I’ve submitted a PR for this on the official postprocess repo
I’ve added a log visualiser also…I did not have time to go through the automation of processrun as it was reported in one issue…When unprocessed values come into feeds, you still have to click on the “Run Process” button…anyway the formula process can make things easier…

Alex

1 Like
2

Thanks a lot, this looks very interesting. I can think of many use case.

I will link to your PR here:

Do you have any thoughs on this @TrystanLea?

3

This looks great @alexandrecuer, thanks a lot for sharing, Testing here I cant seem to create a formula process, I’ve added a comment to the pull request A process to manipulate feeds through arithmetic operation by alexandrecuer · Pull Request #8 · emoncms/postprocess · GitHub

4

@TrystanLea
what is your formula ?
maybe ouput feed name already exists ?
both boxes (formula & output) are going green ?
it happens to me once and when I renew the cache, it worked…
it is strange…I only added an extra test in the validate function of the view.js. if the type is “formula”, it takes the formula and test it with 2 simple regular expressions

var regex1 = /[^-\+\*\/\df]/;
var regex2 = /f/;
if (formula.match(regex1) || !formula.match(regex2)){
    $(".process_option[option="+z+"]").css("background-color","#ffeeee");
    valid = false;
} else {
    $(".process_option[option="+z+"]").css("background-color","#eeffee");
}

for feeds involved in the formula , the existence check is done later in the controller (create)

5

I cannot manage to reproduce the bug…I took a new raspberry, checkout to my fork

git remote set-url origin https://github.com/alexandrecuer/postprocess.git
sudo git fetch
sudo git pull

and the create button appears if both fields are green :

postpro
postpro1920×1080 142 KB

6

Thanks @alexandrecuer I will dig into it a bit more and see if I can find out what’s going wrong, I’m also seeing the points behind as negative numbers so there must be a difference somehow in our setups

7

It seems to me that processes should only perform operations on the feeds on their common portion, that is to say between :

  • the maximum of the starting timestamps of all the feeds involved in the process
  • and the minimum of the ‘ending’ timestamps of all the feeds involved in the process

That’s why I propose to add a compute_meta function in common.php, in order not to repeat those basic calculations when defining the start_time, the writing end time and the interval…

This is different from your original choice I think, as I could remark some of your postprocesses (addfeeds for exemple) could realize operations out of what I call the common portion…
so If you see negative values, it is probably due to this : if you’ve added two feeds with different start_time with the addfeeds process, you’ve got more points in the resulting feed than in the common portion, and so the number of points behind will be negative…

actually, I tried to improve the algorithm in view.js in order to calculate the number of points behind but I am still not happy with it…

the question is : is it needed to realize operations on feeds out of the common portion ?

From what I expect from the post-processing algorithm, I would tend to say no but maybe I am wrong…

8

My mistake for the first, I missed placing f for feed before the feedid!! :slight_smile: It works fine now.

9

I agree it makes sense for the processes to run only for the common portion of the source feeds. Im testing the points behind with the powertokwh process which has once source feed and one output feed, do you see negative points behind if you try with the powertokwh process?

Digging through the code so far suggests that the npoints returned by the destination feed is larger than that from the source feed. Both feeds are actually the same length and have the same start time. If I uncomment:

//base_npoints = processlist[z][key].npoints;

and comment

base_npoints=Math.round((Math.min(...ftime)-Math.max(...fstart_time))/Math.max(...finterval));

it works ok, but I guess that’s just returning to the previous implementation

10

yes it is like taking over your initial implementation

anyway, I do not have negative points with single feeds processes

I’ve just corrected a little point in the powertokwh process, which was causing some minor troubles when reading a non existing value…there is no link with the number of points…

your code :

if ($om->npoints>0) fseek($of,($om->npoints-1)*4);

$tmp = unpack("f",fread($of,4));
$wh = $tmp[1]*1000.0;

my proposition :

if ($om->npoints>0) {
    fseek($of,($om->npoints-1)*4);
    $tmp = unpack("f",fread($of,4));
    $wh = $tmp[1]*1000.0;
}

My mistake should be somewhere else…can you send me a feed which could cause negative values when processed by the Kwh accumulator ?

11

Thanks @alexandrecuer, I noticed that issue with the powertokwh processor as well and have some local changes to upload here once I merge your pull request. The processor also needs a max power limit to avoid going to infinity if there’s an erroneous value in the source power feed…

I get negative points behind using the formula process as well, this output feed is created from two phpfina feeds with the same start times, intervals and npoints. It should show 0 points behind

image
image1192×47 5.71 KB

12

Adding a few console logs into view.js it seems that the issue is the last time in the feed as reported by the feed model…

ftime: [1544659200]         incorrect
fstart_time: [1475276400]   correct
finterval: [1800]           correct
base_npoints: 38546         incorrect
out_npoints: 41906          correct
points_behind: -3360        incorrect

feed size as reported in feed list 164kb (41906 dp = 163.7kb)

Result of call to last timevalue api:
http://localhost/emoncms/feed/timevalue.json?id=165161

{"time":1550707200,"value":2.9000015258789}

the api returns the correct value and is calling the same model method as the postprocess_controller, hmm… I will keep digging.

13

My fault, my redis cached timevalues for the source feeds where wrong, I had synchronised my feeds across to my computers emoncms installation and the timevalue cache had not updated correctly!
Clearing redis sorted it! sorry about that

14

I’ve merged in your pull request (A process to manipulate feeds through arithmetic operation by alexandrecuer · Pull Request #8 · emoncms/postprocess · GitHub), thanks a lot @alexandrecuer

I’ve merged in a couple of minor changes to the layout and added in the more general powertokwh modifications at the same time: Merge branch 'alexandrecuer-emonpi-mod' into emonpi · emoncms/postprocess@6583d87 · GitHub the list2 api was a mistake now removed in subsequent commit.

15

@TrystanLea : thanks especially to you for the Emoncms ecosystem :slight_smile: I am very happy with the PHPFINA timeserie which is so fast and simple.

16

That’s really great to hear, I’ve also been really happy with PHPFina as well, inspired originally by Mike Stirlings timestore Timestore – a Time Series Database | Mike's Lab Notes it has scaled really well on emoncms.org and provided a lot of versatility in terms of structuring the data requests, data synchronisation, post processing as needed.

17

I’ve read what you wrote about timestore in the learn section. I like C but I really think you made the right choice by switching to PHP for the data engine. It’s very flexible like that.

1 Like