Post Process Questions & Problems

[lhgray (Lawrence Gray)]

I should have known better than to try the shiny new “post process” features without reading the documentation which I just learned are not available yet. Hopefully soon. But I jumped right in and immediately got over my head.

@TrystanLea

I’m aware I have not yet written a guide for this module or announced its introduction on the emonpi, but I am working towards adding these resources to the openenergymonitor guide soon.

The “post process” feature appeared when I updated my emoncms from 9.8.6 to 9.8.18, while I was attempting to resolve issues with virtual feeds, which are now working.

I created 3 post processes that I would like to remove, but I’am not sure how. There is no explicit mechanism for doing so that I see.

The first process is simply summing the two power feeds from the emonPi into a new feed titled “Power Consumption”. On its own this process worked and updated when requested and would graph as expected. What’s unexpected is that the database file, 28.dat, is nearly 170% the size of either of the input database files, 2.dat or 3.dat.

I should have stopped there but I continued. I decided that it would be good to trim the feed start time for the two power feeds as they included data before I had the larger CT’s that would fit over the power feed lines in my residence. After doing this for Line1, I noticed that a new feed was not created. I expected a new feed, but the original feed, is not altered in size and when graphed is truncated at the new start time as expected, but does not update anymore in the graph. I suspect now that the data file is not modified by the “trimfeedstart” process, but the start time is altered in the corresponding metafile.

I see from the feeds page that the original feeds are updating and the corresponding database file, 2.dat, increases in file size periodically as does 3.dat. And the first post process feed “PowerConsumption” does not update graphically- even after pressing “Run Process”, but the file size of the data file does increase when I run the process. Does this have anything to do with the fact that the two feeds being summed do not now have the same start time?

How do I recover from this? Can I simply use a new “trimfeed start” process to reset the start time to an earlier time, which I can determine from the other intact feeds? This sounds logical to me, but I would like other, more knowledgeable opinions on the matter.

And finally, how do I remove the post processes from the page?

[TrystanLea]

Hello @lhgray

Just a quick reply for now,

Not sure about the feed size being 170% that one seemed to work ok for me. The resulting feed should match the shortest interval of the source feeds.

Trimming the feed actually deletes data out of the source feed up to the trim point, there is no way to roll back on that one Im afraid. But I see the issue testing here and I think its caused by the buffered writing of data thats active on the emonpi. You may notice new data being written in the future by the amount you trimmed?

I would keep the feed that you trimmed but disconnect its input processing and create a new feed for now so that you dont loose further data. I will then try and have a look at this next week.

[lhgray (Lawrence Gray)]

Thank you for the quick reply.

@TrystanLea

Not sure about the feed size being 170% that one seemed to work ok for me. The resulting feed should match the shortest interval of the source feeds.

Both feeds had the same interval, 5sec and the post process result as well. I’ll go out on a limb here and suggest that the increased size may be due to the fact the post process feed does not have benefit of scaling /rescaling to use integers to minimize storage. My scaling factors in emonhub.conf for Line 1 & Line 2 are 1.093. Are real numbers being stored here?

@TrystanLea

Trimming the feed actually deletes data out of the source feed up to the trim point, there is no way to roll back on that one Im afraid.

Not a problem, it was throwaway, setup data, but did constitute nearly 70% of what was recorded.

@TrystanLea

I would keep the feed that you trimmed but disconnect its input processing and create a new feed

I’m not sure how you do that. On the 'feeds’ page, the 'inputs’ page or post process page? You surely don’t mean remove the log process on the process list? That will delete the database data file

[pb66 (Paul)]

@lhgray

You surely don’t mean remove the log process on the process list? That will delete the database data file

No, removing the process from the processlist via the inputs page will only break the link between the input and the feed, if you go to the feeds page after removing the process you will see the feed is still there and the time since update will increase in real time as it is no longer being updated.

The easier way rather than delete or add a new process is to “edit” the process, just select “create new” and it will create a new feed in the same spot on the processlist and the old one will remain intact but disconnected.

[lhgray (Lawrence Gray)]

I followed your suggestion, I think, and uncovered new unrelated issues. I opened the process list for the Line 1 input and edited the first item Log to Feed. I selected CreateNew and entered a new feed name Line 1a. I wasn’t sure that the same feed name, ie., Line1, could have been used.

The new feed is logging data and can be viewed along with the old feed. I assume that there must be some way to merge the two feeds into one feed that can be used to log new data. But, I’m not sure that any of the PostProcess processes will do that. There is a mergedata process but the result needs to be updated from the PostProcess page to be current. Can we log new data into a merged data feed once created?

Onto my new issue. Now that I have created my new feed id#30 I needed to update the process list for each of my virtual feeds. In trying to change the initial source feed from Line 1 to Line1a and save the edit I would get an error:

ERROR: Could not save processlist. Feed processlist was not updated

Turns out this only occured only for process lists that contain a schedule process, so if I removed the scheduling process, saved it and reedited the source feed in the process list it would save. This is evident for feed id#16. But I could not readd the schedule process, it would not save. I resisted the temptation to edit the others so I created a Virtual test that I confimed the problem numerous times. My only thought about this is that the schedules I use and the virtual feeds were all setup using emoncms v9.8.6 as delivered on purchase. I have recently updated to emoncms v9.8.18 to get virtual feeds to graph, and to gain access to the PostProcess feature.

Update: I had a brief power interruption here last night, so the emonPi had rebooted. Now, I cannot edit any of the virtual feeds to alter the source feed to Line1a. And the three “Post processes” that I had previously questioned about how to remove, are gone. No action on my part.

Any one have any ideas on what’s going on?