Getting spikes in emonCMS graph

djh · 2018-12-07T20:52:30.207Z

I’ve got some dipoles on order; I’ll let you all know how they work after they arrive.

In the meantime I’m curious about how the optical pulse sensor data is interpreted. There’s a glitch in the graph of my solar generation; perhaps the easiest way to show it is wih the CSV data:

2018-12-07 14:22:00, 12200.0
2018-12-07 14:24:00, 12213.0
2018-12-07 14:26:00, 12028.0
2018-12-07 14:28:00, 12235.0
2018-12-07 14:30:00, 12244.0

That feed uses a whacc process. I thought they were supposed to increase monotonically?

1 Wh Accumulator whacc Node emontx3: solar generation optical pulse sensor (10464.00)

(BTW, why is the current value shown wrong?)

Robert.Wall · 2018-12-07T21:29:23.145Z

Getting started with emonTx and emoncms

I thought they were supposed to increase monotonically?

That is my understanding too. You need someone who knows the inner workings of emonCMS to explain this.

Getting started with emonTx and emoncms

(BTW, why is the current value shown wrong?)

With just that to go on, out of context, I haven’t a clue.

pb66 · 2018-12-08T18:18:50.504Z

Getting started with emonTx and emoncms

There’s a glitch in the graph of my solar generation; perhaps the easiest way to show it is wih the CSV data:

2018-12-07 14:22:00, 12200.0
2018-12-07 14:24:00, 12213.0
2018-12-07 14:26:00, 12028.0
2018-12-07 14:28:00, 12235.0
2018-12-07 14:30:00, 12244.0

There isn’t anything obviously wrong with this data (to us at least) it is increasing only and the values seem believable (without any context to consider).

Getting started with emonTx and emoncms

1 Wh Accumulator whacc Node emontx3: solar generation optical pulse sensor (10464.00)

(BTW, why is the current value shown wrong?)

You have posted a link to a graph on your local emonpi (http://emonpi/emoncms/graph/3) which obviously we cannot view. I these instances we ask you post a screenshot rather than a (working) link because as time passes the link to live data would change and the context of the discussion is lost.

Can you specify what is wrong with the current value, ie what it is and what you think it should be and why?

Robert.Wall · 2018-12-08T19:17:42.854Z

I saw the middle value was 200 low, but thought it to be an error in copying the values.

pb66 · 2018-12-08T20:09:23.241Z

Getting started with emonTx and emoncms

I saw the middle value was 200 low, but thought it to be an error in copying the values.

Me too, I do not think a whaccumulator would fallback like that, it would usually drop to zero or spike when misbehaving and the fact it springs back to counter the missing 200 is too coincidental to believe without confirmation, I think the 12028 should have been 12228. Perhaps @djh can confirm.

djh · 2018-12-09T12:02:15.089Z

Getting started with emonTx and emoncms

There isn’t anything obviously wrong with this data (to us at least) it is increasing only and the values seem believable (without any context to consider).

I’m sorry but I think that 12028.0 is less than 12213.0?

And no, I didn’t copy the values wrong! It’s difficult to do that with copy-and-paste

Getting started with emonTx and emoncms

You have posted a link to a graph on your local emonpi (http://emonpi/emoncms/graph/3) which obviously we cannot view. I these instances we ask you post a screenshot rather than a (working) link because as time passes the link to live data would change and the context of the discussion is lost.

Can you specify what is wrong with the current value, ie what it is and what you think it should be and why?

I just posted what emoncms displays, to try to be sure I wasn’t mistaken when I said ‘whacc’. I am assuming that 10464.00 is the current value. If not, I’d be happy if you could explain what that number actually is?

djh · 2018-12-09T12:19:49.949Z

Little more info. Firstly the heading above the 10464.00 is ‘Latest’ and there’s a tooltip that says ‘Last recorded 2 value’. The number is still the same despite it being a day later.

Second I copied the data from /home/pi/data/phpfina/ and I wrote a perl program to read the .dat files. In 3.dat I can see the 12028.00 value sitting there:

12222.000000
12223.000000
12223.000000
12028.000000
12225.000000
12227.000000
12228.000000
12228.000000
12229.000000

So it’s definitely emon-whatever doing this, not my imagination!

Now the graph isn’t showing me the same data today, it’s obviously resampling or somesuch. Where is the explanation of how the graphs work? A link on the page would be useful.

djh · 2018-12-11T02:32:14.939Z

OK. Here’s a screenshot showing the bad data point in the optical sensor output:

Screenshot-2018-12-11%20Emoncms%20-%20graph%2031053×1661 89.5 KB

The energy feed is calculated from the CT sensor. My dump of the actual data files confirms that it’s a single bad value timed at 14:26:00 in the optical stream.

Is this a data collection error (bug) in the emonTx or a data recording error (bug) in the emonBase? Or is there some other possible explanation?

I gather from some googling that there is no user documentation for the graph app, so I’m still in the dark as to whether I’m using it right. Why are the colours shown as black in the ‘feeds in view’ for example?

I’ve got other oddities in the data that I’ll post later.

djh · 2018-12-11T12:56:35.595Z

Here’s another screenshot:

Screenshot-2018-12-11%20Emoncms%20-%20graph%2051053×1759 105 KB

This one concerns the electricity imported from the mains, measured by the other emonTx in my garage. There’s the optical sensor data, the raw power readings from the CT on the meter tail, and the integrated energy readings computed from the power. The readings have been scaled by different factors to make the graph show the problem.

The CT has a single missing data point at the critical time; the raw data in 4.dat confirms that:
2018-12-10 03:41:40 2160.000000
2018-12-10 03:41:50 2163.000000
2018-12-10 03:42:00 NaN
2018-12-10 03:42:10 2167.000000
2018-12-10 03:42:20 2184.000000

The computed energy has a single retrograde value at the same time and apparently catches up afterwards:
2018-12-10 03:41:40 113.359528
2018-12-10 03:41:50 113.365540
2018-12-10 03:42:00 112.734421
2018-12-10 03:42:10 113.377548
2018-12-10 03:42:20 113.389030
I don’t know what I expect the energy integration to do when there’s missing power data, but pretending that there was a sudden switch to energy outflow is definitely not it!

The optical sensor also has a retrograde measurement at the same time:
2018-12-10 03:41:40 115049.000000
2018-12-10 03:41:50 115055.000000
2018-12-10 03:42:00 114418.500000
2018-12-10 03:42:10 115067.000000
2018-12-10 03:42:20 115079.000000
This is very strange. How does a whacc feed produce retrograde data? And no, my meter doesn’t record exported power! And why does it occur at this precise time? What’s special about -636.5, approximately 100 times the magnitude of the actual import at that time?

djh · 2018-12-15T21:14:35.276Z

Thanks for splitting the thread, Bill.

I believe the problem occurs as a result of NaN data, but I haven’t got to the bottom of it yet. (and apologies but the lack of comments in the code or overview documentation is one of the main things in slowing me down diagnosing the problem, IMHO)

Bill.Thomson · 2018-12-17T00:52:26.815Z

djh:

Thanks for splitting the thread, Bill.

Not a problem, Dave.
Made sense and is easy to do.

Simsala · 2018-12-17T07:05:12.790Z

djh:

I believe the problem occurs as a result of NaN data,

I’ve noticed the same behavior and could associate it with NaN values as well:

Wh accumulator - strange behavior Emoncms

That’s definitely best practice and should also be in the official guide. My emonpi was accidentally without power for some days, which is not a big problem as the emonTh continues to increment the pulses. Once the Pi was back online, the wh_accumulator did a bad job: [16] For any reason (probably because the wh_accumulator needs also the elapsed time for the calculation), it did’t count the 1971 pulses and it just shows a horizontal line. However “Total pulse count to pulse increment” worke…

djh · 2018-12-17T21:08:58.797Z

Simsala:

I’ve noticed the same behavior and could associate it with NaN values as well

Thanks for noticing this thread and posting the cross-link. We do indeed seem to be experiencing the same problem. It’s clearly a bug that’s been around for a while but doesn’t seem to have been cleared. I raised a bug about it: incorrect data recorded in feeds · Issue #1137 · emoncms/emoncms · GitHub but it doesn’t seem to be attracting any attention.

I read the thread about your experiences and whilst a lot of it is still over my head I did notice some points that seem interesting:

(1) this problem seems to occur with the phpfina engine but not with the timeseries engine.

(2) I’ve also seen spikes in power_to_kwh data but they’re slightly different, in that they first go down when there’s null data underlying, then spike upwards by the same height above as the initial spike below, and then return to a sensible value. In each case, the underlying trend of the data is also accounted for.

(3) I haven’t got a clue what determines the size of the spikes, but I suspect it will prove to be significant. Here’s an interesting example with two spikes, one being three nulls wide:

Screenshot-2018-12-17%20Emoncms%20-%20graph%20211053×1707 127 KB

(4) In your thread a year ago there was a suggestion that there is a better technique for recording the data (optical pulse counters on revenue meters, in my case) instead of wh_accumulator, but I chose wh_accumulator precisely because it is the technique recommended in the current documentation. So what’s the situation here?

borpin · 2018-12-17T21:42:39.226Z

djh:

one being three nulls wide:

I had an issue with nulls, you can test in the process list and skip the log to feed if the value is null.

A null means that, for whatever reason, the input value was not sent correctly. Unfortunately, by default, Emoncms logs nulls to feed. Seems odd, but that is what it does.

borpin · 2018-12-17T21:50:24.553Z

djh:

this problem seems to occur with the phpfina engine but not with the timeseries engine.

@djh, have you tried to record it using both engines to demonstrate this? - just a thought.

djh · 2018-12-17T21:51:35.853Z

The nulls are not the problem. The whole point of phpfina, as I understand it is to minimise writes and increase efficiency. It does that by writing nulls, which are essential for it to keep the time stamps accurate. So nulls are to be be expected in the underlying data.

The problem is what wh_accumulator (in conjunction with the phpfina engine) does with those nulls.

I expect in my case the nulls are due to some co-channel interference from some device or other.

djh · 2018-12-17T21:52:21.396Z

borpin:

@djh, have you tried to record it using both engines to demonstrate this? - just a thought.

No need. Others did that over a year ago.

borpin · 2018-12-17T21:59:43.291Z

djh:

The problem is what wh_accumulator (in conjunction with the phpfina engine) does with those nulls.

Yes but if you skip the wh_accumulator process if the input is null, you may well eliminate the issue.

borpin · 2018-12-17T22:00:41.307Z

djh:

No need. Others did that over a year ago.

But you said ‘seems to’. By doing it yourself you can add to the weight of evidence.

djh · 2018-12-17T22:03:09.325Z

borpin:

Yes but if you skip the wh_accumulator process if the input is null, you may well eliminate the issue.

You mean I can fudge the results and sweep the bug under the carpet?

Shame on you! I’d rather try to see it fixed.

borpin:

But you said ‘seems to’. By doing it yourself you can add to the weight of evidence.

How to do that is part of the puzzle that goes over my head. Plus I’m doing more than enough fiddling with my system as it is and getting myself into a muddle.