Correcting cumulative kWh reset with post process module

John_Fitton · 2017-07-21T11:50:55.445Z

During the storms the other day I lost power to the house, and this appears to have done something to the use_kwh log. The log has data but the running total seems to have been reset at power lost time, right before power loss it had a value of 2178 when the system came backline it started at 0 and is now showing 70.

The knock on effect is that the reports now show a - 2000 kwh for that day which is really skewing the reports.
As the data seems to be there and is working before and after correctly, is there anyway to process thats days data agian?
I can export the log but doesnt appear to be a way to import.

Many thanks
John

TrystanLea · 2017-07-21T11:59:39.166Z

Hello John, Yes there is a way to fix this, do you see a module called post process in your EmonPi setup menu?

There is an issue with the module having disappeared in a recent update which I need to fix

John_Fitton · 2017-07-21T12:05:14.703Z

Hi, No I do not have “post process” so am affected by it disappearing. How would I do it when it returns?

TrystanLea · 2017-07-21T13:09:25.862Z

Ok, thanks to @glyn.hudson the module disappearing issue has been fixed. If you can run EmonPi update again it should correct the link.

One you have it, you can use the ‘powertokwh’ post processor on the original power feed to rebuild the cumulative kwh use_kwh feed. Let me know if you get stuck.

I’m aware I have not yet written a guide for this module or announced its introduction on the emonpi, but I am working towards adding these resources to the openenergymonitor guide soon.

John_Fitton · 2017-07-21T13:54:55.782Z

Hi, post process is back, although I am unsure which input to use.

Create new - powertokwh
Select input “use” or “use_kwh” ??
output feed emonpi\use_kwh

Is that correct, how do I monitor progress as I tried the above and the create button appears but it doesnt make it to the “My Processes” list.

Sorry for such basic questions.
John

TrystanLea · 2017-07-21T14:31:25.907Z

Sorry, It sounds like you’re running the emoncms master branch. There was a mistake in a recent commit which I have now corrected, could you update again?

Then try powertokwh, select input “use”, name the output feed “use_kwh2” (creates a new feed which you can check before deleting the old).

Depending on how large your source feed is, it can take minutes to process and at the moment, there is no feedback on progress unfortunately.

John_Fitton · 2017-07-21T14:55:24.577Z

Awesome, data regenerated. Many thanks for the quick responses.

TrystanLea · 2017-07-21T15:35:31.400Z

great to hear! I have changed the thread title so that its hopefully easier to find for others

wabiloo · 2017-12-05T17:48:54.981Z

Hi Trystan,

Can this mechanism described above be used to fix an existing use_kwh feed, instead of creating a new one?
My issue is that I’m using the “Power to kWh” process on the input to log to that feed (which allows me to track instant use on the Apps). If I create a new feed in the way you describe, I’d have a static out-of-date feed. Can I then log to it as I would have done, after fixing it?

Also, how do you set the output feed name to include a tag / group name?

nicsergio · 2018-07-25T19:31:12.010Z

Continuing the discussion from Correcting cumulative kWh reset with post process module:

Hello all

I also have the same request as wabiloo: I recently installed emoncms locally on my Raspberry Pi with Raspbian Jessie, following the detailed guide that you provide on github.
I activated the low-write mode and everything works perfectly.
Congratulations to all for the wonderful work done and put at the disposal of everyone.

The only problem I had is a reset of the feeds related to the Power to kWh process after a restart of the Raspberry (at the time it happened only once, but the relative feeds are compromised).

I searched a bit in the forum and I saw that several users complain about the same thing:

https://community.openenergymonitor.org/t/kwh-feeds-reset-to-zero-after-power-failure/4162
https://community.openenergymonitor.org/t/phpfina-value-drop-to-zero/3142/8
https://community.openenergymonitor.org/t/help-kwh-feed-resets-to-zero-randomly/5994

Is there a way to restore these feeds? I saw that in the usefulscripts/process section there’s a php script power_to_kwh.php that also allows you to overwrite an existing feed giving input power data, can this be a way, or with post-process module?

Thanks all

nicsergio · 2018-07-30T07:39:05.930Z

Hi everybody,

not receiving answers, I tried to do tests independently.

I tried to post-process one of the feeds that had gone to zero, creating a new feed: the script really does a good job and the data are fixed
I tried to replace the reset feed with the newly created feed:
– I stopped apache service
– I did redis FLUSHALL command
– I replaced the damaged phpfina/feed id.dat file with the new one
– I restarted apache

I verify the data before new registrations: the modified feed is correct, either by displaying as a graphic or as a CSV output, but in the feed list, as a last value, NULL appears.

Upon arrival of new input data, the feed in question returns to zero.
Turning the Raspberry off and on again, magically, the feed returns all fixed by itself.

I’m a programmer, but I’m not expert in PHP, even less in SQL/redis, I tried to take a look at the code, in the script emoncms/Modules/process/process_processlist.php - function power_to_kwh():

if it is not possible to retrieve the last data point of the feed, it assumes the last value is zero.

// Get last value
$last = $this->feed->get_timevalue($feedid);
if (!isset($last['value'])) $last['value'] = 0;

this value, if more than two hours have elapsed or the time interval is not valid (flush redis), is written in the feed, causing the reset:

//in the event that redis is flushed the last time will
//likely be> 7200s ago and so kwh inc is not calculated
//rather than enter 0 we enter the last value
$new_kwh = $last_kwh;

In the script emoncms/Modules/feed/feed_model.php - function get_timevalue():

the code to read the last feed data point, in case redis is used:

  if ($this->redis)
  {
      if ($this->redis->hExists("feed: $id",'time')) {
          $lastvalue = $this->redis->hmget("feed: $id", array('time', 'value'));
          if (!isset ($lastvalue ['time']) || !is_numeric ($lastvalue['time']) || is_nan($lastvalue ['time'])) {
              $lastvalue['time'] = null;
          } else {
              $lastvalue['time'] = (int) $lastvalue['time'];
          }
          if (!isset ($lastvalue['value']) || !is_numeric($lastvalue['value']) || is_nan ($lastvalue['value'])) {
              $lastvalue['value'] = null;
          } else {
              $lastvalue['value'] = (float) $lastvalue['value'];
          }
          //CHAVEIRO comment: Can return NULL as a valid number or else the processlist logic will be broken
      } else {
          //if it does not, load it in to redis from the actual feed data
          $lastvalue = $this->EngineClass($engine)->lastvalue($id);
          $this->redis->hMset("feed: $id", array('time' => $lastvalue['time'], 'value' => $lastvalue['value']));
      }

Here I imagine that redis->hmget() returns an invalid value and then $lastvalue[‘value’] = null, my question as ignorant is: instead of returning a null value, it would not be possible to try to make some attempts to read a point even slightly earlier, this could save the feed from reset, which, in the case of the power to kwh process, is a deleterious effect

sorry for my bad english
thank you all

Sergio

nicsergio · 2018-08-04T13:55:39.114Z

no one helps me…
for the moment I’m trying a mod to the function get_timevalue():

            // CHAVEIRO comment: Can return NULL as a valid number or else processlist logic will be broken

            // Test for avoid to reset PowerToKWh process feed
            if (is_null($lastvalue['time']) || is_null($lastvalue['value']))
                $lastvalue = $this->EngineClass($engine)->lastvalue($id);

let’s see if it improves…
for the moment I restart Raspeberry (emoncms server) twice and no feed was reset.

I have attached the mod:
PhpMod.zip (16,9 KB)

Robert.Wall · 2018-08-04T15:16:48.893Z

I think @TrystanLea might be the best person to help you, or perhaps @nchaveiro as his name appears in the comment where you are looking.

TrystanLea · 2019-05-06T17:37:24.569Z

2 posts were split to a new topic: Missing postprocess and sync menu entries emoncms 9.9.9

TrystanLea · 2019-05-06T17:42:03.907Z

Hello @nicsergio sorry to miss your posts, good to hear you found a solution, you should also be able to create the new feed using the post process module, remove the old power_to_kwh process and create a new one linking the original input to the new feed.

nicsergio · 2019-05-06T18:52:24.491Z

Hi @TrystanLea, thanks

After about eight months I could say that my mod works well, the raspby has restarted several times and I have never lost data.

OlavKristiansen · 2020-09-25T06:43:37.411Z

@TrystanLea could I kindly ask you to take a look at my (quite similar) problem described in How is post processing supposed to work??

And could you explain the steps involved in

TrystanLea:

you should also be able to create the new feed using the post process module, remove the old power_to_kwh process and create a new one linking the original input to the new feed

?

borpin · 2023-01-20T17:16:45.385Z

bump @TrystanLea

github.com/emoncms/emoncms

Possible solution to Cumulative totals resetting

emoncms:master ← borpin:bpo-reset-cumulative

opened 06:08PM - 25 Sep 20 UTC

borpin

+4 -0

I have seen this happen to me; i.e. the cumulative process resetting to zero. … Community discussion https://community.openenergymonitor.org/t/correcting-cumulative-kwh-reset-with-post-process-module/4631/12 nicsergio suggested a fix to the get_timevalue() function. I have pulled that in here for discussion. As it relates to one of your comments @chaveiro perhaps you could look at the logic. I am **not** suggesting this is correct just the user reports that the issue stopped for him when using this code. Another user has noted this happening as well. https://community.openenergymonitor.org/t/how-is-post-processing-supposed-to-work/15478