Trying to update older datapoints in a feed, RedisBuffer issue?

Emoncms
1

I need to be able to update older datapoints in a pulse-based feed of gas usage, but cannot reliably go back further than a handful of intervals. I am beginning to suspect that RedisBuffer may be impacting on my attempts, giving me inconsistent results. Can anyone tell me if this is an issue with RdisBuffer? Or suggest a better way to acheive the updates? I am currently pursuing two strategies, the first is a modified log to feed to update the datapoints in the actual feed, and the other is to do so in the web app. I have modified a copy of mysolarpv that replaces the solar data with my gas data. I will probably end up renaming this to MyDualFuel when it is properly polished. But that’s a separate issue.

I am recording the durations between pulses from my gas meter, each pulse representing one cubic foot of gas consumed. Dividing the Joules of that cuFt (1146052.8J) by the duration in seconds gives me a flat rate of gas consumption in Watts. This is logged to a feed at 10s intervals.

The resulting value is valid not just at the time of the pulse, but for every second of the duration back to the previous pulse, so I need to write the Watts value into all of the relevant datapoints in that duration, though I limit that to 10 minutes if the duration was longer than that. However, it doesn’t matter how I limit the update range, if it is longer than about 3 intervals ago, it just runs away and overwrites anything from four to many tens of intervals, even when the loop is explicitly limited to four cycles writing a single datapoint per cycle. I am pulling my hair out as after weeks of trying to diagnose the problem I cannot seem to get a handle on it. Any ideas?

This is why I have started the second strategy of intervening in the in-browser Timeseries, and appying the retrowrites in the copied data there, as a self-contained process that doesn’t need to write anything to the actual feed. I have high hopes that this will work if it doesn’t take too long during every reload.

Mike Sims

2

For further information, I am running an emonTxShield v2.4 on an Arduino Uno with the Continuous Sampling firmware from the 20Feb2016 image for emonpi. The pulse is derived from a magnetically triggered reedswitch in my gas meter, and this is debounced by a hardware debouncing chip, a Max6816 which works flawlessly.

I have modified the pulse ISR to track the duration in seconds between pulses as well as the normal pulsecount. The duration is limited to 7200 seconds, then added to the serial output to emoncms on the Raspberry Pi as an Input called pulsegap. The duration is sent twice during the 5 second serial cycle, then the pulsegap is reset to zero until the next pulse is detected.

All of this works without issue. Input processing as mentioned in the previous post takes the duration and a constant Joules value and calculates a Watts value, which is logged to a feed using phpFINA, on a 10 second interval.

The basic problem is due to the nature of pulse-based data. The Input processlist cannot calculate a Watts value from a duration until the pulse that terminates the duration from a previous pulse is detected, so in the meantime, zeros are written instead.

So currently we are representing the entire duration between pulses with a single Watts value at the timestamp of the pulse, with all the other datapoints covering that duration sitting at zero. This is not a sensible policy. The zeros were written before the real figures became available, but once a pulse has arrived and that data is now available, it is just plain wrong not to place all that new data into their correct positions in the timestream of the feed, if it is possible to do so, and it certainly should be.

It just requires a working update process, and I am hoping that someone can help me with my efforts to create one.