Phase measurement and correction in IoTaWatt

overeasy · 28 November 2017 23:54

Once upon a time, I started to tackle the problems created by the phase shifting in the transformers that we use to sense AC voltage and current. I read up on this forum and looked into what others had done both academically and practically. Most of the solutions I saw were based on trial and error calibration.

I got into a few discussions on this forum. What I got was a tongue lashing for not having an Electrical Engineering Degree, and as far as how to solve the problem, not even the time-of-day.

So I dug in and solved the problem. Along the way, I exposed the tools that I use to measure these things in the device I was building so others could do the same without additional equipment or specialized knowledge.

The problem that I came to understand was that the signals that are coming out of these transformers are not simply the input signals shifted by some constant number of degrees. They shift more and less over the duration of a single cycle. In fact, what happens with a voltage transformer is that it shifts a lot more while the slope is high (zero crossing), and a lot less while the slope is low (peak voltage). But most of the time, the real power is being developed during peak voltage when there is less shift.

I have an instruction manual that I got with my oscilloscope. It tells me how to measure phase shift using an elliptical pattern on the screen. I did that (after being warned that I was a fool and would blow myself and my equipment up - leave it to the professionals). I got a number. I could reproduce it. Pretty good.

So I pump that into my device and try to reconcile the power factor by shifting the voltage and data vectors by that angle. Hmmm. Doesn’t seem to work. After awhile, I climbed back outside the box.

If I feed the AC directly into my IoTaWatt - that’s right, no transformer, just a couple of resistors voltage divider - I have a true reference to the line. During this I banish the IoTaWatt to an isolated place, power it with a USB battery box, and talk to it via WiFi. But I digress. I’ve got an IoTaWatt with a direct line voltage reference and I connect a CT to it that is clamped to a unity-load on the AC line. Are you following me? Now I take 640 samples of AC and CT and compute the power factor. The arccosine of that power factor is the phase shift of the CT.

I do the same with a VT. I get a phase shift for the VT vs AC line. So now the big test. If I measure the power factor of the CT relative to the VT. Interestingly, it all seems to work out, at least pretty close.

As an example, I used a crappy VT that I measure with 5.0 degrees of shift vs the AC line.
I use an SCT013-000 on the AC line at about 3 amps and get 2.9 vs the AC line.
Then when I measure the CT vs the VT, I get -2.6

Line                                 CT                                      VT
0....................................2.9....................................5.0...........
                                              Difference is -2.1.

So my method is .5 degrees off. That’s about as good as it gets in the IoTaWatt.

When I measure the VT phase shift with the traditional oscilloscope method, I get 7.5 degrees. That wouldn’t work in this application.

The API is there to read out the phase difference between any two inputs to an IoTaWatt. I’m working on a toolkit in the app that will make it as simple as pushing a button. These metrics work well and the result is reflected in the high accuracy of the IoTaWatt.

dBC · 29 November 2017 02:49

It’s probably better described as the net phase shift of the CT, your anti-aliasing filters, and any s/w induced shifts (unless you synchronise two AtoDs to start sampling at exactly the same time?). Although I’m not sure how far any of those additional sources of phase shift will go towards explaining the discrepancy you see between your two methods.

The other thing to keep in mind is that the arccosine (or arctan in my case) maths assumes a pure sinewave. The calibration instructions for all the energy ICs call for “ideal inputs”, i.e. you have to feed it two pure sinewaves as generated by an AC lab supply or an AC power calibrator. Unfortunately they’re out of the reach of most home hobbyists.

The AC signal most of us get from the grid can be pretty distorted with flattops in the sinewave, reportedly from all the switch mode power supplies coming on high in the cycle. I’m struggling to get my head around what that will mean for your calculations. Since you’re feeding the grid to a purely resistive load, I would expect the harmonics in V to also appear in I, so the power contained in those frequencies will show up in your Real power calculation, as well as your Apparent power calculation. In which case, they probably wouldn’t drive you towards a non-unity PF in themselves (like harmonics appearing solely in I do). The bit I’m struggling with is what does it mean to do an arccos on those values.

How many linecycles do your 640 V,I pairs represent? Out of interest, do you get a different (better?) result if you make that calculation over a much larger sample? Do you tune the sample count to make sure they represent a whole number of line cycles?

One thing I try to avoid during calibration is the effects of quantization errors, especially if you’re dealing with very small numbers (or numbers that are very close to each other, or PFs that are very close to 1). To that end, for the purpose of phase error calibration, I collect V,I pairs for 20 seconds at 8kHz and make sure they represent a whole number of line cycles. That’s helped by the fact that I know the calibrator is going to feed me the exact same sinewaves cycle after cycle at an extremely stable line frequency. If a particular point on the sinewave sometimes reads as 100 and sometimes as 101, then by collecting it 8000x20 times I’m effectively oversampling it and determining that it really represents 100.3 (say) and the maths ends up locking that value into the calibration instead of the 101 it might have seen had I just sampled it once.

overeasy · 29 November 2017 03:25

I mean that the power factor is the cosine of the effective phase-shift.

one (60Hz) More like 770 at 50Hz

No

dBC · 29 November 2017 03:38

Can you talk me through the maths on that please, when the signals aren’t pure sinewaves?

No as in not different, or No as in different but not better?

dBC · 29 November 2017 03:44

Do you calculate the PF the traditional way (Real/Apparent)? And if so, do you calculate Apparent the traditional way (Vrms * Irms). If you’re doing that over just once linecycle, I think you really need to make sure it’s exactly one linecycle. A shortcut is to do it over a very very large number of linecycles so that the “tail” doesn’t have much impact on the result. Or the gold standard is the approach I described above: a very very large number of linecycles with no tail (i.e. starting and ending on the zero-crossing of a pure sinewave).

overeasy · 29 November 2017 04:31

No as in not different.

I do it that way, and I’ve done it using a phasor diagram and the law of cosines, which if you play around with for awhile reduces to the former.

dBC · 29 November 2017 05:58

I think that’s only true when there are no harmonics present. Have a look at: Measuring reactive power to build NILM system - #7 by dBC

Your situation won’t be as bad as the one in that diagram because you’re using a purely resistive load, but you will have harmonics in your V (and I) due to the non-sinusoidal nature of Vgrid (flat-tops etc).

michaeljquinn · 29 November 2017 08:37

where is rob in this discussion, a chance to contribute

michaeljquinn · 29 November 2017 08:47

Very interesting discussion,
If i understand, its quite difficult to get a perfect measurement of power factor, and here in AUS (Victoria) we have “smart meters”, where the previous meters were basically mechanical devices (a sort of spinning wheel), replaced with 'digital" devices driven by software - Its a “commercial grade” meter - Landis & Gyr E350 - Type 1200.
Given all the issues you describe, how do these commercial dudes
1 - Run the software to do all you describe
2 - Do the calibration - and if its against some ‘reference source’, how does that map to reality
Or is the qualification just against the reference source anyway

Coz in reality (or at least my reality
all I want is the IotaWatt to be as close as possible to that being measured by my electricity provider - until I go pure “off grid” of course

dBC · 29 November 2017 09:31

“perfect” would be difficult ;-), but anything else is not that hard. The three basic ingredients you need are Vrms, Irms and RealPower, then PF is simply RealPower / (Vrms * Irms). That approach works regardless of the shape of the two signals (V and I).

It’s only if you start talking about angles that things get tricky. The textbook power triangle, and cos φ definition of PF only works when the two signals are pure sinewaves (no harmonics) and the only time that’s likely to happen is in the lab.

The commercial dudes calibrate their meters against a power calibrator. Something like this: https://www.timeelectronics.com/ate-bench-calibrators-multimeters/5077-power-calibrator/. Before they do that they’ll have to ensure the calibrator’s calibration certificates are current. If not they send the calibrator off to the local calibration lab, who’ll calibrate it against their equipment (even more accurate). That equipment will also need to have current calibration certificates. The whole chain is traceable back to god, via calibration certificates, or at least to your national standards lab who are in charge of deciding how big a volt is, and from there no doubt to some international standards body so that our volt doesn’t drift too far from a UK volt etc.

They also have the advantage of not using VTs. Some use CTs and some use shunts. So their starting point is from a relatively good place.

The generally accepted test around this forum is that so long as your energy monitor stays in broad agreement with your revenue meter over a suitably long period, that’s close enough. I’ve never been a huge fan of that test (unless maybe the period is extremely short). Passing that test simply means your errors are averaging out to zero, which is better than them not averaging out to zero but doesn’t really give you much confidence in answering questions like: how much is my sub-woofer using in standby… is it worth unplugging it? To me, that’s where energy monitors (especially per-breaker monitors) are far more useful than just checking your quarterly bill from Origin looks about right.

It’s a bit like Treasurers who promise to balance the budget over the economic cycle. It’s all dandy, but sometimes you want to know what the deficit is right now.

overeasy · 29 November 2017 13:57

Thanks to all who have helped make my point. If you’ve read this whole thread, and I commend you on your patience if you have, I’d ask you to go back and read my opening post.

My two main points:

Traditional methods of measuring phase-shift are not as useful in developing methods to correct for the effects of VTs and CTs.
This isn’t rocket science, but there are those who want you believe that it is.

dBC · 29 November 2017 14:31

Actually, I use a very similar technique to what you’re using. I use arctan (Reactive/Real) while you use arccos (Real/Apparent) but that’s neither here nor there. I have easy access to Reactive, you have easy access to Apparent.

I don’t think it’s rocket science. I’m just trying to help you improve on:

I’ve pointed out several possible causes for that error in my postings above. If I had to pick which I thought would be the most significant contributor I’d say it’s calculating Vrms (and Irms) based on a single cycles worth of data +/- some “tail” of the cycle. At least I think that’s what you said you do. Do you at least start near a zero-crossing so that the “tail” is relatively low amplitude?

overeasy · 29 November 2017 15:47

The variance changes with the quality of the wave. The more distorted, the more the variance. I think that’s because these are not pure sine waves. I’ve had it work to within .1 degree. When I did it while making that post, it was off more with that crappy VT.

The IoTaWatt is within 0% to .5% variation from several revenue grade meters in my home, and within .5% of the power reported by several solar edge inverters. I don’t think there’s a problem.

At 640 samples/cycle, one sample represents .5625 degrees. That would average out to about .28 degrees off over time.

I start collecting samples with the first one after zero crossing and I end with the sample after two more crossings. The “tail” is more like a stub. Why do you guys keep trying to fix something that aint broken?

dBC · 29 November 2017 19:41

Yes, that’s what I tried to explain above.

Great! But not entirely consistent with:

I don’t think anything is broken, we’re all striving for constant improvement and sharing techniques we’ve found that help in that regard. Well, ok, when you wrote:

I admit I thought it was somewhat broken, but your latest reply has put my mind at ease.

The power coming out of your Solaredge inverter is some of the easiest power around to measure. You could have significant phase errors in your monitor ( could - I’m not saying you do) and it would still match nicely. Now the power going into your Solaredge inverter as it puts itself to sleep at night is somewhat more interesting; mine goes very reactive for a short while on the way down. Some brands of micro-inverters go very reactive and stay that way all night. That’s when uncorrected phase errors in your monitor start to matter.

dBC · 29 November 2017 21:33

Here’s how my SolarEdge inverter looks when it’s putting itself to sleep. If you see a similar spike in current with no corresponding blip in Power then I’d say your phase correction is working well.

That small dip at about 18:27 from -4.4W to -2.5w happens every night. It seems about 20 minutes after putting itself to sleep (as indicated by the vertical cliff on Reactive Power) it goes into an even deeper sleep state, and saves another 1.9W,

overeasy · 29 November 2017 22:31

The cycle count is whatever I end up with after two crossings, although I occasionally have to discard a set of samples when the count is low - indicating that the ESP8266 serviced an interrupt while I was collecting samples.

So let me start at the top again:

The shifting in the VTs and CTs is not simply a constant time shift of the entire cycle. It is a convoluted thing that appears to be greater when the slope is steep, and less when the slope is flatter. This can cause introduce error into the measurements. I think we can all agree on that.

The mechanism available for mitigating (correcting not a good term here) the error is to shift one set of data points relative to the other. When we do that, we shift all of the data points, so the net effect will be a change in the power factor by the angle represented by the shift. For instance, if there were 720 data points collected evenly throughout the cycle, shifting one set of points by 7 would represent an angle of 3.5deg.

So we can do this to mitigate the error, but the question is “what is the angle”?

Back to my initial post. By using the power factor calculations (or a phasor diagram - same thing), on the output of the transformer vs the primary (unity load for CT), you get an angle that, if used to shift an entire cycle evenly, will have the effect of mitigating the error.

This number is not the same as the phase shift that you will get from the traditional phase shift measurement methods. It is always smaller.

The phase correction number in IoTaWatt is not the phase difference in the traditional EE sense. It is the angle that the data must be shifted by, to mitigate the error introduced by the waveform distortion of the transformer.

To carry this a step further. There are two transformers involved. The VT and CT. Both exhibit leading shift. The mitigation angles for both components are provided in the configuration tables, and the net shift is computed for each CT/VT pair. That net shift is applied to the data after each sampling when the data is crunched for real power.

It’s not a question of whether phase correction is used, it’s where you get the data to determine how to do it.

You’re right, the Solar Edge inverter has a high power factor. The closer to unity, the less the effect of the phase distortion. Nevertheless, it is important to mitigate the distortion correctly.

My heat pump runs at pf .89 - 27 degrees. I have a revenue grade meter on it (EKM) and also monitor it with IoTaWatt. Over the last5 weeks the EKM meter says I used 514.3 kWh. IoTaWatt says I used 514.0 kWh. This is without any calibration or adjustment. The correction for the VT and CT are straight out of the tables.

If the shift needed to mitigate the VT error were off by 2 degrees, the difference between the traditional phase measurement and the measurements I’m advocating, the pf would be 25 degrees or .91. That would be a 2% difference, yet the two agree.

This isn’t about the importance of mitigating the phase distortion, that, that’s a given, it’s about how you mitigate it, and my point is that you have to do power-factor type calculations to determine how much simple phase shifting is needed. Time will tell with IoTaWatt. There are quite a few out there now. I’m confident the accuracy will surpass the legacy equipment without the need to reprogram and calibrate.

If you have any more questions about how it works, the firmware is on GitHub.

dBC · 29 November 2017 23:00

As far a I know, that’s how everyone does it.

Those few words there demonstrate your misunderstanding of this. You can’t swap between a PF of 0.89 and a measurement of degrees (except in a spreadsheet or a textbook where pure sinewaves rule). Unless your heat pump is extremely old, I bet it has an inverter in it. It’s current signature is going to look a lot more like that of switch mode power supply, probably like the DC brushless motor in my washing machine. Here’s how it looks on it’s spin cycle:

My monitor clocked that signal at:

Apparent Power: 559VA, Real Power: 395W, Reactive Power: +115 VAR, Distortion Power: 378 VAR.

arccos(395/559) is 45 degrees, but that’s a completely meaningless quantity in this situation.

Did you read and understand: Measuring reactive power to build NILM system - #7 by dBC ?

overeasy · 30 November 2017 02:47

Right back at you DB.

Sure I can.

Here’s my heat pump running at a pretty steady rate with a CT that has a phase correction of 0.45 and a VT that has a phase correction of 1.9. (so net 1.45 degrees used to mitigate the effects of phase distortion)
V1_1

The pf is .7997 which I think we both agree is properly computed by dividing real power by the product of Irms and Vrms. The real power is 746 watts, so you can compute the apparent power as 933 VA.

Now I add 2 degrees to the phase correction factor of the VT making it 3.9 instead of 1.9. The net correction is now be 3.9-.45 = 3.45 degrees. Here’s what happens:

V2_2

The measured PF went up by 2.3 degrees (cos .8234 vs cos .7997)
The Real Power went up 3% to 769 watts, yet the Apparent Power stayed the same at 933 VA.

You sure can express power factor as an angle and if you add or subtract to the phase you can expect it to be reflected in the power factor.

This is the same heat pump that IoTaWatt has tracked perfectly for 5 weeks. The extra 2 degrees of shift that would be reflected in using the oscilloscope method would have it off by 3%. So the correction method and particularly the methods I use to develop the correction factors for phase distortion seem to work well hand-in-hand.

If you don’t understand how this works, don’t be changing the subject to your washing machine. I really don’t appreciate your condescending explanation about how that works. At nearly five times the sample rate, my instrument takes that in stride and samples it better in 20ms vs the 20 seconds that you were talking about to get viable samples.

dBC · 30 November 2017 08:32

Not only do we agree that’s how you compute it, that’s the definition of PF.

Actually the PF went up by 0.0237. PF is always a number in the range 0 to 1, it’s never measured in degrees. What went up by 2.3 degrees is the arccos of the PF. Picky perhaps, but you can’t just go changing the definition of PF. We’ve already agreed on its definition above. There is simply no way that definition can produce something measured in degrees.

But since you calculated the arcos’s of the before and after PFs, are you not even the slightest bit curious as to the discrepancy between 2 degrees and 2.3 degrees? It’s explained in the 3D vector diagram you’ll find at the reference above. Think carefully about which of those vectors you’re changing when you deliberately introduced your 2 degree phase error. Think carefully about which angle you’re calculating when you do an arccos (Real / Apparent).

Do you know of anyone who uses the “oscilloscope method” to calibrate their energy monitor? The reports on VT and CT phase errors use oscilloscopes to demonstrate the fundamental issues. The techniques to calibrate your meter are described in the Learn section and as far as I can tell, use the exact same PF techniques you use. I explained above why the numbers you see on the oscilloscope can’t be plugged straight into your calibration (and I’m pretty sure it’s also explained in the Learn section).

I think you completely misunderstood my point there. I sample all 18 channels continuously at 8kHz. There are 8000 x 18 x 24-bit (V, I) pairs arriving every second forever. That’s a lot of data. I don’t attempt to get anywhere near it (apart from when I do wave captures). The IC accumulates energy (Real and Reactive) and I can read the energy registers at my leisure. From memory, I think I’ve set it up so the LSB of the energy registers represents 1/10th of a WattSecond.

Also from memory, I calibrate at 230V, 2A 60 degree phase shift. To calibrate phase shift I do an arctan(Reactive/Real) and tune the IC’s filter until it reads 60 degrees. (Sidebar: the only reason I can do that is because I’m feeding it two perfectly formed sinewaves). At calibration time, when I do the Reactive/Real division, I want those numbers to be relatively big to avoid quantisation errors. For that reason, I let the energy values accumulate for a full 20 seconds. If I let it run for a minute, the division would be even more accurate. I’m talking solely about calibration measurements here, not normal ops.

Very little of that is relevant to your setup except for the basic principle that when calculating calibration coefficients that you plan to lock in forever, averaging is your friend. Keep in mind that at the time I was banging on about that, I only had the following descriptions of your methods:

That sounded pretty reckless to me, but you later revised it to:

overeasy · 30 November 2017 15:50

People may be wondering why I’m having this argument with you. My policy of late is to ignore people like you when it doesn’t infringe on my story. But as you know, this all started a week ago when an IoTaWatt user inquired about voltage swings and whether this would adversely impact accuracy in his device.

I responded that the voltage swings he described were in the normal range and that any change in phase caused by those voltage swing should have a second or third order effect on his accuracy at worst.

At this point, you chimed in to point to a report on the VT and to say that it makes a difference when measuring individual circuits. I don’t believe that’s true and it’s not the message I want to leave unanswered on this forum.

I explained that although the report didn’t specify the method used to arrive at those shift numbers, it was implied that it was done measuring the shift at zero crossing and that’s not what the IoTaWatt uses for phase correction. That prompted a quick and biting rebuttal from the author of the report, complete with the usual ad-hominem attacks.

Fast forward a week, and that author publishes a description of the method used, which is indeed the classic oscilloscope method of measuring phase difference. I didn’t say that was wrong, I said that’s an inappropriate way to determine the phase correction factor for IoTaWatt.

So that brings us to this thread where I take a fresh start to explain how it all works and why the phase correction in IoTaWatt is different from the classic measurement of phase shift using an oscilloscope.

I don’t know if you have figured this out yet, but most people don’t read the learn section. It’s a nice reference feature for people who are interested in AC power theory, but most people are just interested in how many kWh they are using for what. My idea is that a lot more people would be interested in managing their energy use, and OEM products, if they were not being directed to find something in the learn section in order to monitor how many kWh they are using. It’s a busy world. Most people want the quick-start guide.

That’s the philosophy of IoTaWatt. You can connect it and be monitoring your usage very accurately in 15 minutes, without knowing anything about power factors or phase shifting. You can use any of the widely available CTs and VTs that have been tested and included in the configuration tables without a running dialogue with someone that involves FTDI adapters, downloading sketches, a commentary on Arduino IDE vs platformIO, and the way CTs and burden resistors work. Those things are a barrier to wider acceptance of the notion of being aware of and monitoring personal energy use.

While I freely admit that I can’t keep up with you and a few others in the EE specialty, I think I have demonstrated an acquired proficiency in hardware design and production, firmware design and publication, user-interface development, and data communications. The end result is a next generation monitor that was not possible a few years ago because of advances in ADCs and the availability of low-cost, and high speed processors with integrated WiFi. It was only a matter of time before someone did this. Things change, I call it progress.

You can pick it apart and talk all day about harmonics and distortion power, but at the end of the day, most folks just hear blah-blah-blah. The bottom line is that it works - well. Between your washing machine and some-kind of 18 channel specialty device I’ve lost track of why you keep beating me with the ignorant stick.

If you’re not part of the solution, you’re part of the problem.