DIY Current Monitor on Raspberry Pi. My power calculation isn't accurate. Help please?

Hardware
159

Yep, this is where my misunderstanding was stemming from. I am calling a set of data the collection of 2000 samples from each channel (5CTs + 1 voltage)… or 12k samples overall. The terminology of a set in the Learn section means a single reading of one or more CT values paired with a single voltage reading, right?

It all makes sense now. With a purely resistive load, I can assume that the current wave should follow the voltage wave exactly, and the discrepancy between the two when I measure it is the combination of the time shift and transformer phase errors. I can calculate the time shift error, but I don’t think I can calculate the phase error between the voltage and current transformers, so I will have to rely on measurements alone for this error.

Assuming I’m on the right track now, when it comes to using measurements for correction any remaining phase error, should I be looking where the “zeros” of each wave are? For instance, if the zero of ct1 happens in sample number 123, but the zero of the voltage reading happens in sample number 125, there are 2 samples worth of time that I need to correct for, which in my case is 64µs.

So, when generating a new waveform to correct for phase - can I apply the correction technique to the individual current waves, or do I have to generate 5 slightly different voltage waves for each individual current sensor? I’m thinking that perhaps the irregular current waveforms for various electronics won’t take nicely to the phase correction algorithm, whereas the nicely shaped AC sine wave will.

I intend to collect 2k samples from each channel, then go back and perform the phase correction maths for each channel using either: 5 variations of the voltage measurements applied to the respective current measurements, or adjust each of the 5 current wave forms. (To me, it doesn’t matter which method I choose as the number of CPU operations will be the same).

For a bit of miscellaneous stuff… Yesterday I invested the time in adding a debug mode to my software. Previously, in order to visualize the raw data (like for all the screenshots of graphs I shared thus far)…
I would have to set the filename for the CSV data dump in the code, run the program, terminate it, copy the CSV out to my computer, upload it to Plotly, build the chart, name the traces, etc. Now, with my new debug mode, I simply run my program, pass the debug keyword in, followed by a filename, and it dumps an interactive chart with a fresh set of 2000 samples to a local webserver on the Pi that I can view within seconds. Example:

$ python3 power-monitor.py debug "example for OpenEM"

Web server root now has “example for OpenEM.html”:

image

Clicking on the above filename gives me the chart itself, which I can pan and zoom any which way, or turn off individual data series.

image
image1887×946 89 KB

Well worth the time!!

160

Right. Can I call the 2000 of my sets a “batch” of samples?

That’s correct - unless you have the ability to measure each transformer individually, or each pair.

No, that’s unreliable, because the wave shape does get distorted. In your research, did you come across the B-H curve? What it means is the c.t. isn’t linear, so a sine wave in comes out distorted - but worse, the distortion has memory, so the distortion as the current comes down from a peak is different to that on the way up. The only realistic way is to to compare the real power you calculate against the product of voltage and current.

That’s more than likely, and I saw it coming earlier when I wrote

What I’d do, given that you have a store of 2000 samples of voltage and each current, and you’re processing them post-collection, is take a few more samples, then depending on the amount of correction, pick the offset for the pairs of voltage samples that you’ll interpolate between so that you are interpolating (i.e. PHASECAL in the equation is between 0 and 1). That gives the least errors. How many extra samples you need will depend on the offset.

[We can’t do that with the '328P because there isn’t enough memory, so the samples have to be processed on the fly.]

That’s exactly why the shift is always applied to the voltage wave.

I haven’t thought this through, but I don’t think that’s necessary. If you do the maths the way emonLibCM does them, you need what I term the “partial products” for a spread of voltage samples. It’s really hard to explain, so take a look at the maths on the first page of this:
emonLib Maths.pdf (152.8 KB)
and the “partial products” are A & B. Depending on the amount of shift you want, when you apply average power = ... you pick A & B or B & C, or C & D so that PHASECAL is always between 0 and 1.

That indeed looks well worth the investment in time and effort. :+1:

161

No problem! At least now we are on the same page in terminology :laughing:

I did not - but thank you for explaining. So when it comes to relying on measurements for cleaning up the last little bit of phase error, you’re saying to not rely on visual comparisons of the waveform at all, but instead to tune the real power value I’m calculating to that of a meter?

On that note, I don’t think I should be using my electric space heater as the load I’m calibrating against because it has a fan. Should I instead go buy a high power halogen light bulb (~600W), and try to fit as many coils in my clamps as I can? (edit: I bought one)

I had experimented with this in an earlier iteration of my code (here), but at the sampling speed I was using, it was too large of a correction. I also didn’t have the understanding that I have now. Plus, now that I’m sampling over twice as fast as before, I will absolutely try this again… if this works, it will be a lot less work for the CPU than having to rebuild 5 individual wave forms (equiv. to 10K calculations!). If I read the voltage first, then CT0 - CT4 next, maybe this strategy won’t work for CT0, but it might for CT1-4.

I will have to work this out on paper so I can understand it better. My initial thought is that I can think of each set of current samples aka batch is taken either 1, 2, or 3 sample intervals behind or ahead of the voltage sample. To help demonstrate my thought process, I added a Time Shifts Needed column to the table I previously shared:

Sampling Order Time Shift Needed Equivalent Phase Error PHASECAL Time Shifts Needed
ct0 3 * 32µs = 96µs 96µs / 46µs = 2.08696° 1.5 -3
ct4 2 * 32µs = 64µs 64µs / 46µs = 1.3913° 1.3333 -2
ct1 1 * 32µs = 32µs 32µs / 46µs = 0.69565° 1.1667 -1
AC Voltage 0 0 0
ct2 1 * 32µs = 32µs 32µs / 46µs = 0.69565° 1.1667 +1
ct3 2 * 32µs = 64µs 64µs / 46µs = 1.3913° 1.3333 +2

So, I have 5 pre-determined values for phasecal for each sample, depending on when in the sampling order the actual reading happens. For example, ct0’s batch of samples will need to calculate real power using a voltage wave that has been shifted backwards by 3 sample intervals (plus the actual phase error correction).

CT4’s batch will have to use a voltage wave that has been shifted backwards by 2 sample intervals, plus the actual phase error correction.

…Etc…

Maybe I can combine the two correction strategies and create one re-calculated (using phasecal) wave that is 100% phase corrected for a single CT channel, and then compare the rebuilt wave against the other CT input channels and see if the shifting technique works well enough for them.

At least now I have the understanding to experiment with various options! Thank you.

162

Robert, can we revisit the decoupling capacitors between the power supply and the ADC power and ground pins? These are C6 and C7 in the schematic below.

My design has changed a lot since this post (#43), but after reading the Microchip recommendations, I wanted to confirm the values I should be using for my decoupling capacitors.

My prototype board followed exactly this recommendation - but I never looked at the DC signal before and after. I should probably pull a sample set from the 3.3V rail and to one of the Pi’s digital in pins with this decoupling setup to see if it looks ok.

I just want to confirm since Microchip suggests using a value between 10-100µF for the power supply and a 1µF close to the device (if the device frequency is < 1MHz) - and I have no experience in determining which values I should go with, or if it’s even a big deal.

image
image2388×1668 837 KB

My custom PCBs arrived today so I got started on soldering the components I have on hand. I’m still waiting on a few others to arrive. I will clean up the flux when I’m done, but here’s the current status:

image
image4032×3024 1.95 MB

163

100 pF for C7 is too low - it might just help a bit. If you want to go with the Microchip values, that’s fine by me. 220 µF is only twice the value they recommend. For the ‘close’ capacitor, a lot depends on the track lengths - if there’s significant resistance/inductance and the frequency is low, their 1 µF in parallel is probably necessary. If there isn’t much distance between the supply and the ADC (and there isn’t), then 100 nF (not pF) should be enough.

1 Like
164

Thank you. I will use a 100nF capacitor for C7. I’m still waiting for the rest of my parts to arrive before I can finish soldering my PCB and move to testing… hopefully this weekend.

I’ve finished the phase calibration. With a test load of about 3.4kW (several loops of my ~550W halogen lamp), the difference between the calculated real power and RMS power for each CT is only about -0.50W.

Here is the output from two consecutive calculations:

Note: I know I keep on saying that I will have 5 total CTs, but the data I’m sharing only has 4… I will have 5 once I start testing with my PCB. I didn’t see a point in adding the 5th CT circuit to the prototype board.

CT0 Real Power: 3274.90 W | Amps: 28.57  | RMS Power: 3275.64 W | PF: 0.9998
CT1 Real Power: 3265.21 W | Amps: 28.56  | RMS Power: 3265.59 W | PF: 0.9999
CT2 Real Power: 3271.82 W | Amps: 28.55  | RMS Power: 3272.33 W | PF: 0.9998
CT3 Real Power: 3284.61 W | Amps: 28.6   | RMS Power: 3285.13 W | PF: 0.9998


CT0 Real Power: 3271.82 W | Amps: 28.56  | RMS Power: 3272.55 W | PF: 0.9998
CT1 Real Power: 3263.62 W | Amps: 28.55  | RMS Power: 3264.00 W | PF: 0.9999
CT2 Real Power: 3268.18 W | Amps: 28.53  | RMS Power: 3268.66 W | PF: 0.9999
CT3 Real Power: 3279.25 W | Amps: 28.57  | RMS Power: 3279.76 W | PF: 0.9998

The output above is with the phase correction tuned as best I can and the linear correction factor applied to each CT input. I’ll see how much these values change when I move over to my custom PCB.

So, the only major item that remains is the noise problem when the conductors are dead. I’m really hoping this issue goes away or at least lowers in magnitude when I move to the PCB.

165

What do you mean - “as best I can”? Those power factors are excellent. But you must expect them to change as the amplitudes of current and voltage change, which the errors in the transformers are dependent upon. If you want to be really clever (at a later date), you could tweak the correction dynamically according to the average voltage and current for that batch of samples. My thinking about that is you’d need look-up tables for each transformer make & model.

I would not expect the new board to be any different.

I would expect it to reduce significantly. I wouldn’t expect it to go away completely (for the current) although the real power should reduce to insignificant - but expect a lousy power factor.

166

The way I tuned it was adjusting the value of phasecal for each CT sensor so that the real power and RMS power were as close as possible for each CT. Here are the exact variables I was adjusting to reduce the difference in real power and RMS power. There seemed to be a very fine line where I would get the difference down to about 0.50W, and any further adjustments would cause the difference to start rising again. I probably could have gotten the difference below 0.5W if I had tuned beyond 3 decimal places.

Edit: To clarify the above - it might help to know that I started with my pre-determined phasecal values that were supposed to correct for the phase error introduced via time between samples, then I tweaked those pre-determined values to bring the RMS power and real power closer together.

You know, I thought about doing this - not quite for each batch - but at least for the initial correction to save me the effort in the future if I change CTs. Your idea would be really cool to explore in a future version of my code. Instead of hard-coding the parameters for different CT sensors, I was thinking of automating the process I described in the paragraph above - basically a check on the data prior to storing/presenting to the user.

167

The 0.5 W is likely to be due to the amplitude error that the interpolation introduces. The value of power factor you see should follow a curve that looks like the crest of a sine wave - it should be quite hard to find the true top as it flattens out. There’s a better way of doing it: if it were possible to have a load that’s a pure reactance (physically impossible, and hard, expensive and dangerous to get anything close) then you could adjust for zero power factor. That should be very well defined. Or you could do the same thing with your resistive load and move the voltage samples by exactly 90°, which would be equivalent.

You can never do that, because it depends on you having a unity power factor load at the time you make the adjustment - and unless either that is always the case, or you set up a test (as you are doing), you can never know what the actual power factor of your load is.
Putting the parameters in a file that’s accessible easily to you and read by the system before doing the calculations (pretty much as we do the calibration for some Arduino sketches via the serial port) would be nice, but repeating myself, it would need to be done with at least some manual intervention.

168

Ahh, I see. 0.5W is fine enough for me… and with a 0.999# power factor, it seems I’m pretty close to being highly accurate.

I had totally forgotten about that. I guess a way to settle in the middle is to build a “phase correction value finder” function that requires a purely resistive load to be measured. The function would narrow in on the exact phase calibration value to use to bring the power factor to 1 over a series of trial and errors.

So, the last remaining components arrived for my PCB and - to my surprise - nothing really went wrong!

I am very happy to report that the noise problem has improved tremendously! Here is some sample data taken with the CTs around a dead conductor. I’m not concerned with the RMS value… and I’ll probably manually set any amperage reading under 0.20 or so to zero before sending the values off to my db.

CT0 Real Power:  -0.34 W | Amps: 0.16   | RMS Power: 20.34  W | PF: -0.01658
CT1 Real Power:  -0.19 W | Amps: 0.18   | RMS Power: 21.88  W | PF: -0.00863
CT2 Real Power:   0.18 W | Amps: 0.16   | RMS Power: 19.8   W | PF: 0.00906
CT3 Real Power:    0.4 W | Amps: 0.16   | RMS Power: 19.39  W | PF: 0.02072
CT4 Real Power:  -0.19 W | Amps: 0.16   | RMS Power: 19.91  W | PF: -0.0095

There is lots more work to be done on the DB integration, but at least now I’m confident in the accuracy and ready to put the unit back onto my mains (which requires a new electrical box and short conduit back to my main panel to keep the cat5e sensor bundle away from high voltage lines… tomorrow’s task).

169

It’s looking good. :+1:

I was fairly sure the noise would reduce significantly with a custom p.c.b. Anecdotally, most of the complaints about non-zero readings with Arduinos have come from users with ‘breadboard’ analogue front ends (as compared to the emonTx - the same ADC & processor), so as exactly the same thinking applied to you, the chances were good that everything would improve.

1 Like
170

That’s essentially what I do. Specifically, when my PV system output falls below 50 Watts,
the script sets it to zero before passing the measurement data to Influx.

Looks good. Nice work Dave! thumbsupbeer_cheer

1 Like
171

Thanks guys!

Man… Grafana has been tricky to get it exactly how I want it. I’ve come pretty close though… to the point where I’m nearly finished building and tweaking things. I thought I’d never see the day! :laughing:

Here’s what I have so far. I think I’m just about done with my “live view” dashboard. I’ll create another one that’s focused on historical usage.

image
image1920×944 139 KB

I still have lots of work to do with regards to documenting my code, but I am committed to maintaining it in case someone else stumbles upon the project and wants to open the can of worms themselves.

Thank you guys so much for your assistance through all of this! I learned so many invaluable things across a variety of subjects, and to me, that’s more important than the dashboard I ended up with. I wouldn’t have been able to complete this project if not for this forum and your support - especially you, Robert!

One question I had on the fundamentals of the voltage bias is - since we over-estimated the peak current output from the CT sensor in the burden resistor calculation, that means we won’t actually be able to measure up to 100A using a 100A sensor, right? I noticed this when I was doing the phase correction using my 3.3kW load. That was about 26A total, but the ADC was putting out about 763 / 1023 (or ~75%), which should be a measurement of 75A.

172

Did we? What do you actually mean by 763 / 1023? Remember the standing value is about 512, so a 75 % peak (or given a clean sine wave, 53% of full scale) swing is 768 counts peak to peak going from 128 to 896 counts.

Are you still using the 100 A : 50 mA c.t. with a 20 Ω burden and is your ADC input still 0 - 3.3 V?

If so, that should give you (with a maximum 1.1 V rms burden voltage to allow a bit of headroom for component tolerances) a max. current of 110 A rms.

The ADC input will clip on the peak and that’s a bit over 150 A, the relationship peak-peak = rms × 2√2 only holds for pure sine waves, therefore if you have a really spiky current waveform, the rms value will be any number you care to imagine below that! :rofl: and if you’re in a country like mine where the mains voltage is “flat topped” (the peak is slightly less than rms × √2) then with a well-behaved load you’d measure slightly more than your 110 A before it clipped and gave you wrong answers.

173

Oh yeah! In my last post, I forgot to remove the neutral ADC value. I’m back on track now. Peak to peak in my sample data I was referencing was from 258 to 763. I was mistakenly thinking 763 / 1023 * 100 was the amperage value, when it should have been [(763 - 512) / 1023] * 100

For three of my CT sensors - the YHDC 100A : 50mA, I’m using 21 Ω. For my two larger (diameter) CTs, they are 100A : 100mA and I’m using 10 Ω resistors.

It has been fascinating to see the impact of turning on certain appliances in real time!

174

The two 100 mA c.t’s should read 110 A max, the 50 mA one 104.75 A (both with a little bit to spare).

Now you can see where the money goes…

175

That said, you’ve still done one hell of a job! (and not just with Grafana) Your dashboard looks great!
As you continue to tweak it, you’ll “discover” other features that’ll make your graphs easier and/or quicker to read, e.g. table view in the legend.

If you get stuck, by all means ask, but don’t forget there’s an active Grafana forum at your disposal.

You’re very welcome, sir!
It’s quite rewarding to see a project I’ve helped with come to fruition. Your attitude the entire
time was a big part of what made it easy to help. You never gave up, and you turned your errors into
an opportunity to learn. highfive

1 Like
176

Seconded. :+1:

1 Like
177

Hi guys, my project has been running excellently over the past couple of weeks!

I’m stopping by to put a formal close to this thread and say thanks again for all of your help.

I have released my project on my Github page which includes everything a future DIYer will need to implement my project, including source code, the PCB fabrication file, and a BOM for the components.

I’m still working on finishing up the documentation, but I intend to actively maintain the repository for the foreseeable future.

178

Hey, that’s great! beer_cheer
Good to hear it’s worked out so well for you. highfive

1 Like