Rms calculations in EmonLib and Learn documentation

Hardware
1

I believe the entire RMS calculation is flawed (filters not adjusted for gain - plus V/I zero point wanders considerably during each cycle) and probably over complicated.

a) there is no need to use a filter to calculate the mean on the fly (unless you really think it is fluctuating each cycle - in which case the hardware has issues) - it is easy to calculate at the end (after some tiresome maths) - VRMS = sum( (V-V0)^2)/n = sum(V^2)/n -V0^2

b) trying to adjust the phase of the voltage seems a wasted effort given that the current transformers alter the phase by a greater amount. If you really want to compensate for the phase error take two measurements (i.e. V1,I,V2) then the correct V is (v1+v2)/2/cosA where A is the phase delta between successive measurements (more maths)

c) while it amounts to the same thing as adjusting the voltage, explicitly adjusting the phase of the current source might make sense, but at least it would be explicit and perhaps configurable for transformer type, but then it really needs to deal with timings (between samples) which I assume are quite different on different machines

d) finally and probably not important, there seems to be a lot of unnecessary floats in the routine - though I’m guessing speed is not an issue.

Anyway - I’m not an arduino user, so I’m not completely sure of optimizations… and have developed my own stuff for PICs, but I came across your stuff while researching some Ideas for myself.

I’m happy to expound on these issues IF anyone is interested

mike

2

a) has, if I understand what you’re getting at, been incorporated into the forthcoming emonLibCM. The effect is discernible at low currents, but not at normal operating values.
b & c) are probably a result of a misunderstanding. It is the difference in the phase errors, plus the timing error, that is compensated for. The interpolation algorithm injects the least distortion when used on a pure sine wave, therefore it makes a lot more sense to manipulate the voltage wave rather than the current wave, which is not necessarily anywhere near a sine wave (think of a light dimmer, or many ‘electronic’ power supplies).
d) integer maths are used where appropriate where speed is a concern. For the ‘end result’ performed once every 10 s or so, it isn’t.

3

Thank God for the autosave on this board - I am always forgetting to press send/reply whatever!

In my search I was directed to emonlib (reinforced with the information in learn documentation). Clearly much work was done to improve emonlibCM.

I was sufficiency intrigued, that I read the new library and played around with some spreadsheet calculations. All my current sensors are cheap, low current, closed (non-snap-in) with very low phase distortion so I have no experimental data, but some comments - I have split them into 2 responses - they are huge!

mike

4

What you have done for the RMS calculations is better, but not what I was getting at, but then neither did I write it correctly!

I meant VRMS2 = sum( (V-V0)2)/n = sum(V2)/n - V02 where V0 is the zero point and approx = sum(V)/n
Instead of calculating and summing (V-V0)2 each sample you can calculate and sum V2 and V.
At the end of an integral number of cycles and n samples V0 = average of V i.e. V0 = sum(V)/n.
(V-V0)2 = V2 - 2VV0 +V02
sum((V-V0)2) = sum(V2) - 2V0sum(V) +nV02 = sum(V2) -2sum(V)/nsum(V) + nsum(V)2/n2 = sum(V2) - sum(V)2/n = sum(V2) - nV02

All of this is kind of interesting, but assuming that that the actual V0 point doesn’t wander much between measurements, then it really doesn’t matter that you take the “old” V0 point vs the “new” V0 point in your calculation each sample in emonlibCM. It does point out that the low-pass filter isn’t low-pass enough.
At the end of each cycle you have the sum(V-V0’) = sum(V) - n*V0’ (where V0’ is the “old” V0)

and if all measurements were exact V0 = sum(V)/n so you have V0 ≈ V0’ + sum(V-V0’)/n

The original filter was /1024 (.001 not .01) and your new filter at /64 is very fast given that you are aiming at 50 samples per cycle so your effective filter is only .78. I think a slower filter would be much, much better (especially since Emon is always on). I would suggest even slower than the original with a quick calibration at startup (maybe 2-3 cycles). The filter could be many orders of magnitude slower and still track any shifts in the value of V0 day to day.

mike

Edit - fixed superscripts. BT, Moderator

5

My comment on phase adjustment was based on the learn page and emonlib, which just purports to correct the sampling timing error.
As for filtering the voltage instead of the current I think you are fooling yourself when you comment that the filters works better in the case of the “pure” sinusoidal voltage as when the current is non-sinusoidal the phase lead changes drastically. In the LED case you mention I believe you will find the phase lead goes to 0 at the turn off/turn on point of dimmers/LED drivers.
At least if you realize you are phase compensating current you could decide to correct for the non-sinusoidal behaviour (if you really, really wanted to) - of course then I guess you have to compensate for voltage as well …

A problem with the filter used to adjust for phase is that it alters the RMS voltage. I was interested in seeing by how much and made up some spread sheets. The Voltage error created a bigger error that the phase error! (both were tiny)
I trired to find your instructions for calibrating, but was unsuccessful and there is enough confusion in my mind over the various libraries/proposed libraries that I gave up. I don’t want to waste your time (nor mine) so if you have this all figured out great, but according to my spreadsheets with a purely resistive load, trying to maximize calculated power factor is not very successful in discovering the correct PHASECAL. With a phase error of 4° a PHASECAL of 0 resulted in a PF of .999 and getting anywhere near the correct PHASECAL required the PF to be >.99999 and still the corrected real power calculations was worse (marginally) that using no PHASECAL.

All of this to say
a) you need to compensate for the magnitude of the filter.
b) properly detecting the phase may be difficult, but given that you have done code for a PLL I might try doing a PLL for each sensor to calibrate the phase offsets. While there is no real point with UPF loads as the PF goes down the errors will increase (though I suspect even then they are on the order of a few percent.
I would suggest measuring zero crossings, but somewhere I saw someone say there was so much distortion at the zero crossings that it was not very accurate (that may be an artifact of the original V0 filter which did in fact grossly distort the alternate zero crossings)

FWIW (and it may not be much)

mike

PS. I am interested in what work you may have done on calculate phase angles, though as I have said so far I have used low error XFRs and hall effect sensors, and I used PICs with simultaneous sample capability - though it comes at a price 10 vs 12 bit resolution, so I have been considering doing them sequentially (though again they are maybe 50 times faster so the error is really tiny)

6

I don’t think you could have - it has only been released to the administrators and moderators. The available version picked up some serious errors in the process of conversion from a sketch, and should not be used. Anyway, I have no intention of looking at this any time soon, as I’ve got a very bad head cold and can’t think.

7

What has apparently happened, is you looked at the published version of emonLibCM. Immediately it was published, I realised that some serious errors had crept in and I asked for it to be withdrawn, but that did not happen. There are several posts (probably here, but also on the archived forums) warning that it should not be used. I’m sorry if you wasted time and effort looking at it.

What the ‘new’ version does, in place of the low pass filter, is pretty much as you expound, but with some differences due to the implementation. For example, I now initially subtract a constant offset of the nominal half-rail voltage purely to reduce the size of the numbers, and then derive the remaining bias offset over the full reporting period. As you say, hopefully it won’t change much over that period. The high or low pass filter has completely gone.

Phase adjustment doesn’t “just purports to correct the sampling timing error” (not even in emonLib) but attempts a correction for the difference in phase errors between v.t. & c.t. too. It can never succeed completely, because both phase errors depend on the magnitude of the variable being measured.

I take it you’re talking about the fundamental component here. The way real power is calculated (surely you have noticed?) is to multiply individual pairs of voltage and current samples and average. Apparent power comes from multiplying rms voltage and rms current, and power factor thereby comes directly from the definition: the ratio of real to apparent power. None of our sketches do Fourier to extract the fundamentals and thereby phase angle. Clearly, as soon as one of the wave shapes ceases to be a pure sinusoid, phase angle becomes meaningless.

You are referring to the interpolation/extrapolation algorithm for time-shifting the wave? The amplitude error is a known fact and covered in the calibration instructions. More to the point, unless the wave is a pure sinusoid, it distorts the shape too. The distortion is visible even on my ‘flat-topped’ mains voltage wave, and would be wholly unacceptable on a current wave if the load was non-linear. Sadly, as the absolute maximum speed to the ADC is 104 μs per sample, and there is only one ADC in the Atmel 328P, we are stuck with voltage samples that can be in excess of 9° apart, and interpolating between samples to ‘phase shift’ the wave is the only way.

I fully accept that phase correction has little consequence with purely resistive loads, obviously because the top of the cosine curve is a good approximation to flat, but unfortunately as the power factor falls, it becomes more important and in the extreme, the power can end up in the wrong quadrant. The user tends to get very excited when that happens.

A lot of distortion, as well as phase error, comes from the a.c. adapter. Unfortunately, there is no practical (from a regulatory point of view) alternative either for that or for the split-core CT. You must bear in mind that equipment manufactured and sold in the Shop has to be safe for non-technical users. It’s been suggested that a small 1:1 c.t. fed via a multiplier resistor would give much greater fidelity, but we need to find a manufacturer willing to assemble it in a plug-top case (or similar) and go through the regulatory hoops for CE approval, and then anyone opting for that would need a separate 5 V power pack as well. One of my professional colleagues published “GIVMon” - some while ago, in the Archive - but again putting that into production would face the same difficulties. I also put forward a capacitive divider for someone who was unable to have a mains socket in the right place, but I don’t know whether it was built.

For measuring phase errors, I use a computer sound card doing 48 k samples per second, with Fourier analysis in a spreadsheet.

8

Robert,
I hope your head is feeling better.

i) As I said I was “intrigued”, so the value to me was the fun in thinking about this stuff (real, imaginary, doesn’t matter - I’m a complex guy), and I did get several insights and ideas for stuff to play with.

ii) I think there is value in the low pass filter, effectively improving the precision of the A/D and establishing the zero level to a sub-LSB accuracy (assuming random noise)

iii)In the “Learn” section about the “phase shift filter” it specifically talks only about the sampling error and gives a value to accommodate the 2° sample shift, hence my comments.

iv) My comment about phase lead and non-sinusoidal current was that the phase (or really time shift) is dependent on dI/dt and will shrink to 0 from the 4-8° measured for the current sensors for the LED/dimmer transistions. Given that the filter amounts to a resampling by linear approximation (approximately), I believe it is perhaps more valid (and certainly as valid) to filter the current to estimating the “actual” current at the time of the Voltage sample, which while distorted and displaced because to the wall wart, is at least relatively consistent (I assume).
Boating season is just about to start, so it will probably be October before I am sufficiently intrigued again to some physical experiments/measurement, I’m wondering if it is worthwhile or even possible to estimate the time shift due to the current transformer from it sinusoidal performance, and use that in the filter, perhaps with an array or function dependent on dI/dt, but probably not necessary except for “fun”.

v) I would appreciate links to the current (i.e. present not …) calibration as I could not find them. I found a post (of yours) directing to the resources section, but it appears the section has changed and I couldn’t find them.
With my spreadsheet I discovered that it would appear to be very difficult (impossible) to estimate the cumulative phase delay, by observing the measurements with a resistive load, hence the need for a tool to estimate/discover the phase delay - I wasn’t referring to your testing. I am interested in how you suggest people do the calibration

vi) having worked in the electronics industry for many (many) years I understand the compromises you have to make. At home I’m not even legally able to use the yhdc transformers as they are not UL/CSA listed and as far as I can tell I have to by a commercial kit that is approved! Of course what the inspector doesn’t see …
On the other hand I am a great believer in designing in precision and calibrating to achieve accuracy - after all this is still mostly DIY stuff

vII) I found http://www.ti.com/lit/an/slaa122/slaa122.pdf entitled “Current-Transformer Phase-Shift Compensation and Calibration” which seemed promising, though I’m not really sure, I stupidly didn’t save my spreadsheets, computer crashed …, so I have not tested it, but the author seems to saying that by emulating a highly capacitive/inductive load and then PF correcting will make the calibration more sensitive (accurate), I have my doubts - he doesn’t provide any evidence for this, but intriguing none the less.

mike

9

Here you go:
https://learn.openenergymonitor.org/electricity-monitoring/ctac/calibration

I think the executive summary is that you tweak the phase adjustment until RealPower reads the same as ApparentPower while measuring a purely resistive load.

Interesting paper thanks.

Yes, I concur. As you pointed out above, a small phase error is close to unnoticeable with a UPF load but really sticks out at lower PFs.

I’m not particularly familiar with emonlib so I’ll leave you in Robert’s capable hands there, but what your paper describes is not a million miles from how I calibrate away phase errors in my energy IC based monitor. It uses an all-pass filter to add time advances/delays between I and V.

Using a power calibrator (unfortunately too expensive for most home hobbyists) I subject it to two pure 50Hz sine waves precisely 60°out of phase. I then let it accumulate Real and Reactive Energy measurements over an extended period … maybe 20 or 30 seconds depending on how patient I feel. The longer I let it run, the less I’m subjected to quantisation errors. Then I do a tan-1 (Reactive/Real), subtract that from 60° and I have my total phase error (mostly from the CT but also includes various filters etc). That number then goes into a reasonably complex formula (not unlike the ones in your paper) and it spits out the correct coefficient to load into the all-pass filter. Run again with that, and the tan-1 comes out at 60° typically with 2 or 3 decimal places of zeroes after it. I repeat it at 60Hz, and store away the two coefficients, loading them up at runtime based on the observed line frequency.

What I can’t compensate for is varying phase errors due to varying currents. The best I can do there is try to choose CTs that have a fairly flat phase error across the current range. As a rule of thumb, I calibrate at 1/10th the rated current, so for example I calibrate a 20A CT at 2A, i.e. I set the power calibrator to 230V, 2A, 60°phase shift, 50/60Hz.

10

Thanks for the Learn reference. I had read that and thought perhaps it was “old” and there were newer instructions for some of the well, newer stuff.
I (unfortunately) have to admit I missed the iterative step of adjusting voltage level, phase, level …
I may have to reconsider my comments regarding filtering the current as you can probably more accurately measure the voltage for the calibration, but see below.
Also I totally missed the amount of phase error introduced by the voltage transformer, and the comment that they are basically equivalent, so the emphasis on sample phase difference.

As for the paper (http://www.ti.com/lit/an/slaa122/slaa122.pdf), I dreamt about it this morning (sad indictment of my life) and redid the spreadsheets to demonstrate that sure enough it works! (duh!)
As we know phase error makes little difference in a UPF, but creates an increasing difference in low PF loads.
So the valuable part (and the part you may have missed) is that by emulating a large phase delay (i.e. delaying samples internally) the sensitivity of PHASECAL increases significantly, to wit at a 0° phase delay the PF(or calculated real power) changes ≈ .02% - hardly measurable - for each 1° of phase error, but at 60° the PF changes 3% for each 1°.
I.e. you get all the benefits of your power calibrator without the hardware as long as you can accurately determine the actual phase delay of the delayed samples (which should be quite easy in a calibration routine)

By using a similar trick to the RMS calculations one can factor out the filter multiply for each sum and do it once per cycle, at which point one could even use a lookup table for phase calibration values for different amplitudes assuming you could find sufficient different resistive loads to calibrate with (multiple output portable electric heaters come to mind)

Finally, it is reasonably easy to precalculate the magnitude adjustment necessary as a result of different PHASECALs and provide that as a table to a calibration routine so that the user doesn’t have to iterate the calibration.

mike