Everything has been running well with my feeds and My Solar app collecting data (still need to do some power factor calibration to improve accuracy). However today I clicked on History view and was presented with a large anomaly which is skewing the view.
So my question is, how can I fix up this erroneous data in the history view to return the axis scales back to normal and report the correct history view? Worst case, can I delete that day from history?
You can use the graph module data viewer to explore the data around that time and most likely be able to find an anomaly like a huge step, a reset or minus value etc, once you know what the anomaly is you are better positioned to determine a fix.
Looks like a loss of data but no negative values or the like to push history view so skewed. Is there a data file with CMS history that i can play with?
emoncms.org view has no issues in history view, but still has the same gap in the data viewer graphs likely while i was playing with the pi/network/etc.
I think you need to be looking at the accumulating “energy” data rather than the instantaneous “power” data.
The history page of the mysolar app is based on data from the solar_kwh, use_kwh and import_kwh feeds (or their equivalent if you have a different naming scheme). Those 3 feeds should show as steadily increasing over time with no decreases or large spikes/steps etc.
From the info we have from the power feeds graph and the 2 mysolar apps (local vs emoncms.org) is suspect you are right that it may have been caused by some down time, either planned or otherwise, and I begin to suspect that there may have been a “hiccup” of some sort when the local emoncms resumed operation, most likely in the input processing.
I can see a few days before the day in question, there were 2 days with problems too and the errors are also presented differently on each emoncms instance. That could be due to the way it failed eg mqtt error on local server would allow the remote to continue collecting valid data whilst the local data stopped (I’m assuming an emonPi/SD here) or it could be due to a difference in the way the input processing is done at each emoncms.
There are several data files, one for each feed so you need to identify the feed(s) to fix first.
not easily, there are tools to repair feeds but the raw feeds are not easily viewed or edited manually.
Yes, emonPi/SD which has been backed up and re-imaged with the beta build and data imported (around this issue time, but not sure of specific date). This was due to me playing around with some external stuff and basically breaking things so I started fresh.
I definitely see a big drop off on the history date in question, resets to zero but still no negatives.
That reset is the cause of the issue. I cannot say why it happened but for some reason those feeds have been zeroed and that they are effectively massive negatives.
All 3 of these feeds should only ever increase, you cannot un-use what you have already used or un-generate what you have already generated etc. When you consider the delta of any given periods eg if you use 10kWh per day the feed will increase by 10kWh across each day, if the feed falls by 10k over the course of a day it indicates you have used minus 10kW’s which cannot happen, so by suddenly resetting to 0, that shows as a use of around minus 67kWh for that one moment and will dwarf any valid values for the rest of that day.
To correct this error you need to add the feed value immediately prior to the reset, to each value post reset. So that when you look at the graph above, the 3 feeds continue their upward trajectory rather than dropping to zero.
I believe post-process module is there for this purpose but I have never used it so I am not able to guide you further.
OK thanks Paul, appreciate your responses.
So the question becomes, who has done this processing before or can provide further guidance?
If I have to just remove previous history prior to this reset so we just start from there, then happy to do that too.
Every value since the reset needs to be increased by the value of the feed immediately prior to the reset. 19days at 10s intervals for 3 feeds is nearly half a million datapoints that need editing, that’s why you need a script/tool.
In the interests of damage limitation, you could create a duplicate set of new feeds and start collecting data asap, if you fix the old feeds you can delete the new ones, if you can’t fix the old ones you have the new ones with at least a little history if/when you decide to give up on the old ones. You can just name them “import_kwh2” etc for now and rename later. Position each new feed immediately after the old feeds in the input processing lists.
This sounds a bit drastic I know, but you previously said you would delete all data prior to the reset, so it’s only an additional 19 days data at risk.
Thinking about it, there is a new feature in the pipeline to “trim” phpfina feeds to clear data older than a certain point. That is in the master branch of emoncms I believe, I have not tried it and the emonPi/SD runs the stable branch not master by default, you might want to explore switching branches and trying that feature, but again, I’m not familiar with the feature and I do not know how confident you are to manually switch branches and more importantly, deal with any “hiccups” that may pop up, it may well be straightforward and go without any “hiccups” at all, but I’m not in a position to reassure you that would definitely be the case.
Confidence would be pretty low - this project is me learning from scratch!
So I either just start again for my history, which isn’t a big deal (it’s only just moving in to peak solar time in Melbourne), or attempt some trimming which is probably going to ruin my day!!
Do you have the postprocess module in your emoncms Setup menu?
You can create new kWh feeds from power data using this module.
Create new > powertokwh
Select a power feed e.g solar
Enter a name for the kwh feed e.g: solar_kwh2
Click on run process in the postprocess list
Give it a bit of time and then check the newly created solar_kwh2 feed. If your happy that it is correct, rename or delete the old feed and switch around the input processing to point to the new feed.
There was an update recently to switch the postprocess module to use the mysql database, you may need to run the emoncms database update from the Administration page.
Thanks @TrystanLea and @pb66 . I’ve started fresh and removed old data and all good. ecmoncms.org is tracking nicely, local instance is fresh. Took the chance to manually update power factor via hub config (7% use, 16% solar) and almost perfect accuracy against my smart meter figures. Happy camper.
