What type of input processing should I use for an always increasing value that may reset to zero

Emoncms
1

I’ve just moved my Internet modem throughput logging from an RRD to EmonCMS for easier graphing/dashboard capabilities but I’m not sure what Input Processing would be most appropriate.

I’m collecting 4 different values, but two of them could be problematic: UpBytesTotal and DownBytesTotal.
They represent the total bytes uploaded or downloaded since the device rebooted (or maybe when the VDSL2 connection was last reset).

At the moment, I have a “Log to Feed” followed by a “Rate of Change” which seem to be giving me exactly what I need, but if the Internet modem resets, or the connection drops, I imagine my “Log to Feed” feed is going to have a cliff where it drops back to a near zero number before once again starting to accumulate, causing a huge negative value in my “Rate of Change” feed for that interval.

I tried switching the “Log to Feed” over to the “Wh Accumulator” or the “kWh Accumulator”, but it seems to be falling foul of the “spike limit” as neither were updating values on Input changes.

Any suggestions on how best to handle this data in EmonCMS?

I didn’t have to do anything special with my RRD solution because it is designed specifically with this use case in mind.

I’ve just chosen to use EmonCMS because I have it already setup and it was simple enough to feed my data into it, but if it just isn’t the right tool for this job, I’m happy to be told that :slight_smile:

2

I’ve just discovered the “If rate <” process, so I’ve added that step before the “Rate of Change” step.

I assume that will prevent any negative “Rate of Change” values from being logged, which, when I think about it, is actually what I want to achieve. I don’t really care that much about the total values.

I would still like to hear anyone else’s thoughts on how to approach this though.

3

My guess is you’re referring to a Round Robin Database.

1 Like
4

I think the accumulator process should do this, though I have occasionally seen these reset. They do require Redis to be running IIRC.

5

Yes these are the input processors that are designed for use with this at least when the data is energy data. If it’s a local system, one option would be to change the maximum power limit in the code itself emoncms/Modules/process/process_processlist.php at master · emoncms/emoncms · GitHub makes updating harder later on but it’s one approach…

6

Could this be a setting?

2 Likes
7

Yes that would be nice, the easiest way would be a setting in settings.ini. The ideal way is a setting via the input processing UI but that would require a fundamental redesign of the input processing implementation…

1 Like
8

It would make sense :slight_smile:

9

So it seems either there’s a bug in the ?rate< process, or my logic doesn’t work the way I expected.

All values are in kb (or kb/s for the rates) - the actual raw numbers are divided by 1024 before being fed to EmonCMS.

Yellow line is raw number (which seems to have rolled over to 0 at 2^32), scale on the left
Blue line is rate, scale on the right.

image
image1131×566 30.9 KB

Input Processing is:

image
image1006×161 7.66 KB

What I was expecting to happen was the “downloadRate” data point at 23:35 would have been missing.

Any thoughts? Is this a bug or expected behaviour?

10

@TrystanLea, any thoughts on whether this could be a bug in the “?rate<” input process?

11

Here’s the code for that processs, it seems it’s just diff rather than a rate? and the abs of that diff… does that explain it?

public function ifRateLtSkip($arg, $time, $value, $options)
{
    global $redis;
    if ($redis) {
        $redispath = "process:ifRateLtSkip:".$options['sourcetype'].":".$options['sourceid']."_".$this->parentProcessModel->proc_goto;
        //$this->log->info("ifRateLtSkip() time=$time value=$value redispath=$redispath");
        if ($redis->exists($redispath)) {
            $lastvalue = $redis->hmget($redispath,array('time','value'));
            $change = abs(floatval($value) - floatval($lastvalue['value']));
            if ($change < $arg)
                $this->parentProcessModel->proc_skip_next = true;
        }
        $redis->hMset($redispath, array('time' => $time, 'value' => $value));
    }
    return $value;
}
12

Ah-ha! very much that explains it, so its nothing to do with the rate at all!

Is it worth updating the help text for that Input process to be clearer about that? Particularly about the fact that its an absolute value check - so testing for a “rate < 0” is completely pointless.
[EDIT] I just re-read the help text, and maybe I just didn’t read it properly the first time!

What I think I need to do is ignore a difference of a larger number than my maximum rate… I’ll give that a shot now.

1 Like
13

Ok, that didn’t work either…

I took a punt that a “skip next” type input process would fail if there was no process to skip to, so I added a “reset to original” at the end as a “do nothing” step if my counter has reset.

After most of today, finally I have success! I can see a missing data point for the “rate” feed at the time when the total feed resets to zero.

To re-iterate, a “skip next” input process will not work if there isn’t another input process to skip to.

1 Like