Help with feed calculation not always correct

I’ve just replaced my EmonPi with an EmonBase and new TX4’s and CT’s.

I have a CT on the solar being generated (the blue line below) and fed to my switchboard and another CT on the grid feed to the switchboard (the red line).

The grid feed goes negative when I have solar to export to the grid.

I have added these 2 feeds in EmonCMS (local) to calculate what the house consumption is (the yellow line).

At times, the house consumption goes negative, which doesn’t make sense.

Has anyone else seen this?

It’s like there is a sudden change in the solar being generated (cloud) and the 2 feeds are being read at slightly different times? Thx.

Welcome back, Matt.

Your new emonTx4s won’t be reading the two powers at different times - provided both go into the same emonTx4, but after being processed by emonHub and then by emonCMS, it’s possible that data averaged over different periods and so coming from different reports sent by the emonTx’s are being added or subtracted.

If the two powers go into separate emonTx4s (or any emonTx for that matter), there’s no mechanism to set or maintain synchronism between two or more, and then you will inevitably be adding or subtracting values averaged over different periods.

I’ve seen on the Inputs page of emonCMS, the time elapsed since the last data being different for data items sent in the same packet of data :roll_eyes:. I don’t know whether there can be a foolproof solution, other than doing what the original emonPi did, which in your case will be to ensure both PV and Grid go into the same emonTx4, and do the subtraction there before the data is even sent to the emonBase. It’s a relatively trivial change if you have a laptop/desktop set up with the Arduino IDE, the necessary plugins and libraries, and you’re happy with coding and familiar with emonHub & emonCMS; otherwise not so simple.

Robert, thanks for your reply.

Both inputs go to the same Tx4 so that’s a good start.

I’m surprised that the value sent from the input would change. Once it’s been sent, isn’t that the time stamped value used in all the calcs? I’m still stuck on the calculation as when I export the data and do the maths in Excel, it is way off at times.

I have some basic Audrino programming skills, so could do what you propose, but I’d need a starting point as to which program I’m editing and the edits I need to make. I’d also need to have the output of the calc at the Tx4 sent on a new input as I use the other inputs for other calcs. I’m not sure how much is possible.

Thanks.

Hi again, I wasn’t sure how much to upload in the first post.

Here’s a new trend and it’s even worse.

Also attached are the calcs done in Excel and compared to the Emon. It’s very wrong more often than right. Something else has to be going on.

Thanks.

No, not if you’re routing through emonHub and using the EmonCMS Fixed Interval Timeseries database. It acquires the time of the time slot when the data makes it into the Feed, having gone through emonHub and whatever it needs in the RPi to get into emonCMS, then there’s the input processing on the Inputs page before it reaches the Feed.
(See my explanation for the mechanism Problem with data from MQTT feed seeming to be "lost" - #13 by Robert.Wall)

All that is possible. The steps are: Get or update the Arduino IDE, and install the plugin for the -DX processor, download the sketch and change it using your favourite editor or the Arduino IDE, compile and upload it to the emonTx4; then edit the emonhub.conf file in your emonBase (via the web browser) to tell it about the new input, then do as you need with the new input in emonCMS.

The instructions for setting up the Arduino IDE and the necessary libraries are in Learn → Electricity Monitoring → Using the Arduino IDE (Using the Arduino IDE — OpenEnergyMonitor 0.0.1 documentation)
You’ll get your sketch from this page (cunningly hidden under ‘Hardware’) Firmware — OpenEnergyMonitor 0.0.1 documentation You’ll need to look to see if you have the ‘6 temperatures’ version - I think the default is 3 temperatures, the top one on the page, and unless you asked for it, you won’t have the 3-phase version.
You’ll also need to install and configure the DxCore plug-in, instructions for which are about half-way down the page (ours, not Github). The instructions for the libraries are repeated just below.

At this point, you might want to download emonLibCM’s documentation file (if you didn’t get it) for reference, though you don’t need it.

Assuming you have the first (6 channel, 3 temperatures) sketch, the files you’ll need in the IDE are
EmonTxV4.ino and EmonTxV4_config.ino but you’ll only edit the first one.

Looking at the first, the powers you want are obtained from emonLibCM in lines (on the Github page) 349 - 364. The values are put into a struct created at lines 88 - 94, so you’ll need to add a new variable (or variables) there. I’d call it P12 if it’s the sum/difference of powers P1 & P2 - but the name is your choice. If you want the accumulated energy as well, it’s the E values.
(Don’t add any more than one power and one energy - 6 bytes - or the radio will foul up, you’re limited to 61 bytes - an integer is two, a long integer is 4)
Going back down the code to where you get the powers, add somewhere convenient & relevant (below emonTx.P2 = ...?)

    emontx.P12 = EmonLibCM_getRealPower(0) -  EmonLibCM_getRealPower(1);

(for the difference, using whichever power or energy values you want, of course).
If you’re sending the data by ISM-band radio, this is all you need to change here. Save, compile and upload it - and the numbers will stop appearing in emonCMS. As long as the sketch runs, don’t panic.

Note where you added the new value(s) in the ‘struct’, and go to your web browser and emonCMS, and Setup → Emonhub. You’ll need the “Edit config” tab on the right. Scroll down to the block for your emonTx4 - probably Node 17:

[[17]]
  nodename = emonTx4_17
  [[[rx]]]
    names = MSG, Vrms, P1, P2, P3, P4, P5, P6, E1, E2, E3, E4, E5, E6, T1, T2, T3, pulse
    datacodes = L,h,h,h,h,h,h,h,l,l,l,l,l,l,h,h,h,L
    scales = 1,0.01,1,1,1,1,1,1,1,1,1,1,0.01,0.01,0.01,1
    units = n,V,W,W,W,W,W,W,Wh,Wh,Wh,Wh,Wh,Wh,C,C,C,p

You’ll need to add your new power (and energy if you did this too) into the same POSITION in the lines of “names = …”, “datacodes = …”, “scales = …” and “units = …” as it was in the ‘struct’
so if you added P12 as the 5th value following P2, then add it as the 5th item in the names = ... list,
add h, as the 5th item in datacodes = ..., 1.0 in the scales = ... and W, in the units = ... lists.
And save it. It should restart automatically, but if not, click ‘Restart’.

Your data should now appear on the Inputs page of emonCMS.
(You could do emonHub first - the data stops flowing while this definition and the data sent are inconsistent.)

I won’t test it, but I think this will work. Don’t hesitate to come back if you have a problem - unless you use Platfrmio - then I can’t help you, cuz it screwed my system up when I tried to use it.

Robert, thx again for the detailed reply. I’m away this week and will take some time to digest all this.

Still have an underlying disappointment that the calcs don’t work and I have to edit the back end.

I was also hoping with the upgrade to the EmonBase and new VMS that I’ll get closer to what my electricity retailer measures but the first 2 days of kWhd aren’t very close. I’ll have to ask them if they reset their accumulators at midnight too.

I think in general, your problem might be that you’re subtracting two numbers that are close in value, when of course the percentage error in the difference could be very large.

Looking at the very first graph you posted, at around 16:00 you have Solar infeed of say 2.8 kW and Grid export of 2.4 kW. If those numbers are accurate, house consumption is 400 W. However, with a 1% error in each (the law of natural perversity prevailing to give the greatest error) and the PV is 2828 W and export is 2377 W, consumption is 451 W, in error by 12¾%.

Continuing this line of thinking, how are you subtracting one from the other – are you using -input or -feed? The advice, on timing considerations (and I don’t know so I’ve never thought through the intimate details of it), has always been to use -Feed; again I’m assuming the maths should done in the processing of the input lower down the inputs page, but I’ve no idea whether this is right or wrong.

Emonhub was written by @pb66 (who we haven’t seen for a while) and heavily modified by Trystan, who wrote emonCMS.
@TrystanLea
Do you have a comment regarding where to put the subtraction calculation (and why), and regarding what appears in this particular use case to be a major problem with timing with the data recorded in the Feeds? I think I know that a (say) 10 s FITS Feed is created with a timestamp of the round 10 s. What I’m getting at is how close do the writes to two feeds need to be in order that they have the same timestamp when the feed is accessed later - or how far apart can they be to have the same timestamp? Or indeed, can they be forced to have the same timestamp?

I realise emonCMS doesn’t work this way and a change would appear to need close to a full rewrite, but the thought is, could writing the data to the Feeds be synchronised in some way so that all data originating from a designated group of inputs (in this case it would be just the two in question) could be dropped into synchronised ‘slots’ in the database?

Or maybe the realistic answer is streamline the processing steps and software in emonHub & emonCMS as much as possible and for Matt to change to the fastest RPi available in the emonBase?

If you are just trying to get something to your actual house electricity consumption, you only need the grid feed, restrict it to being positive and use one of the more kWh accumulators to generate daily usage etc.

I have a virtual feed for my three PV systems which all have CTs on the same TX3. For fun, I’ve just created a new virtual feed with those three and the grid feed CT clamp (which is on another TX3).

I don’t see any huge spikes anywhere, but I also have batteries so they alter what’s in the calculation since they’ll either be charging or discharging - getting the actual base load is nigh impossible. I’ll create another virtual feed which puts the battery inverter into the mix too.

Hello @mgull

This doesn’t quite look right, would you be able to share your input processing? are you subtracting/adding feeds or input values?

When using input processing + - / * input using inputs from the same emonTx the timing on maths across channels will be correct.

But that likely won’t apply to the data extracted from the Feeds and processed in Excel. And this might be confusing the issue.

Trystan, thanks.

Input calculations below. All inputs on the same Tx4.

Nick, thanks.

To get the house consumption, I cannot restrict just to positive as when I’m exporting solar, I also have house consumption I want to record, being the difference between the solar being generated and the solar being exported.

Robert, thanks again.

I’m not sure that your 1% error explains the negative values I see.

I’m using ‘- input’ to do the calc. Refer my screen show below to Trystan.

Maybe the issue is that the timestamps on the 2 inputs is different enough, but I’d need to export the feeds separately and see.

My EmonBase was purchased earlier this year. It appears to be a RPi 4.

Hello @mgull I wonder if the skip next processes are somehow causing the issue here. Could you try moving the -input calc to a position before those processes and then if you need to use the ‘reset to original’ at the end to continue with those skip next processes?

I think this is what @TrystanLea has in mind:

  1. Log to feed: House consumption incl solar (negative is export)
  2. - input: P4 Solar generation watts
  3. Log to feed: House consumption only watts
  4. Power to kWh/d: House consumption only kWh/day
  5. Power to kWh: House consumption only kWh
  6. Reset to Original
  7. Allow negative
  8. Log to feed: Solar export watts
  9. Power to kWh/d: Solar export kWh/day
  10. Power to kWh: Solar export kWh

It’s not quite identical to your original, because it fails to allow a zero value to go to Solar export, instead the feeds will get nothing (NULLs will be recorded).

Trystan, ok, we’ll give this a try.

Thanks.

Matt

Robert, many thanks for your work here. There are functions that I haven’t seen or know how they work.

Is there a doc that explains each of these?

Matt

Robert, changes made to calcs as per your work. Thx again.

Let’s give it a few days and I’ll report back.

Matt

No - all you get is the text in the blue banner below the entry line after you have selected the function. There’s not what I call “proper & complete” documentation you can print out and read. If you want more - the exact details, you have to find the code and work you way through it.

@Robert.Wall Robert, changes made to calcs as per above.

I’m still seeing negative values for ‘house consumption’ :frowning:

@TrystanLea has asked me about remote access so I’m thinking that I do this and allow him to have a look around.

Unless there are some other ideas to try first?

Thanks.

Matt