Data from feeds does not add up properly

Hi, I have been trying to add and subtract feeds through input processing on emoncms.org with some less than satisfying results because the sum of the feeds is similar but not exactly the same as the individual feeds stacked up (e.g. in a graph). Please have a look at the graph below where the filled yellow line has been calculated as the sum of the blue and red lines. In my view the yellow line should be identical to the two stacked-up lines, but unfortunately it isn’t. This is a problem for me.

How can I get the two feeds added up without any difference?

the input processing into feeds can be seen here:

What feed types are they? fixed interval? what interval?

Where does the input data originate? emonTx’s? via rfm, wifi or serial? What update interval?

I imagine you are falling foul of a quirk of timing as the traces do follow quite tightly but just not exactly. I imagine the discrepancy is just from comparing or summing adjacent datapoints because of a time anomaly.

First you need to determine if the discrepancy is in the saved data or just in the way it is displayed, Or both, by looking a little closer at the data, does it change if you zoom in?

Hi Paul, the data comes from two identical, separate emonhubs (on Raspberry Zero W’s), each equipped only with one SMA Bluetooth interfacer. They are connected via Wifi to the internet. The feeds are Realtime and using engine PHPFINA. Each posts the data about once every 10 seconds.

I did look at the data itself (see end of this post), and while admittedly some of the data values are NULL, even when there are all three values present they don’t add up. Maybe you are right that some interpolation of the data values due to different timings in the database is actually causing this. However, the difference can be very significant, as can be see here in 5 sec data (around 11:30:05):

The frustrating bit is that I think it has nothing to do with the way the data is gathered, or displayed. I suspect the values don’t add up as a result of the inputs/feeds being processed. I also tried to add up the inputs, albeit with similar results.

Do you have any tips for me how to get the timing aligned and the difference removed, or at least explained?

2019-06-07 11:30:25, 1591, 1094, 497
2019-06-07 11:30:30, 1591, 1094, 497
2019-06-07 11:30:35, 1578, null, 484
2019-06-07 11:30:40, 1578, null, 484
2019-06-07 11:30:45, null, 1023, null
2019-06-07 11:30:50, null, 1023, null
2019-06-07 11:30:55, 1564, null, 470
2019-06-07 11:31:00, 1564, null, 470
2019-06-07 11:31:05, 1557, 997, 463
2019-06-07 11:31:10, 1557, 997, 463
2019-06-07 11:31:15, 1479, null, 456
2019-06-07 11:31:20, 1479, null, 456
2019-06-07 11:31:25, null, null, null
2019-06-07 11:31:30, null, null, null
2019-06-07 11:31:35, 1409, 966, 443

Whilst there are many aspects to this, there are 2 things for you to consider first.

The fact you are using 2 separate Pi’s and 2 separate emonhub’s isn’t helping much. both buffer and deliver to emoncms.org in batches, these batches may or may not be simultaneous, but they will never be as reliable as coming via a single emonhub, where are they physically installed? could you use just the one? even with 2 bluetooth dongles if needed?

Assuming they can’t, you could alleviate the issue a little by setting the http interfacer update interval to 0s or 1s ( it’s been a long time since I played with this so I can’t be sure 0 works) to remove any “batch sending” thus the inputs should be sent immediately to emoncms.org. This isn’t anything to do with buffering (for network outages etc, although that too can be an issue) it is purely where the sends to emoncms.org are limited to one every 30s (by default) to ease traffic at emoncms.org.

The problem you are seeing could be due to (say) one emonhub posting 30s of data (3x sets of 10s data) and then 3s later the other emonhub sends the corresponding 30s of data (3x 10s sets) but it’s too late the total has been written in the first emonhub input updates OR if the totalling in in the second emonhub’s input processing it will use the very latest values from the first emonhub to calculate the totals for each of the second set of 3 updates. Because of the “batched” sending the inputs are not being updated entirely in chronological order.

The second is the start times of all the feeds in question. use can query that info via the feed api with https://emoncms.org/feed/getmeta.json?id=1. Even if you sort the inputs, if the feeds are misaligned, the results will still be skewed. I would try to make sure all feeds are using a start time that aligns the slots, I tend to only use timestamps that are a multiple of 10 (I have written about “harmonised” intervals and timestamps else where on the forum but it isn’t something that draws much interest). You cannot edit feed start times, but you can create feeds with an exact controlled timestamp by purposefully ensuring the first recorded data uses the timestamp of your choosing before starting to log data too it, the first data determines the start time, not the time of creation.

Knowing all your feeds are aligned means that you will get consistent data when graphing because it will always report the corresponding values for that same timeslot not the sometimes the adjacent slot, depending on the exact second the query is run each time.

Thanks for your comprehensive reply, which gave me some useful additional insight into the workings of the data flow.

I shall try setting the batch sending interval to a low value to remove the uncertainty associated with that…

I am less sure about placing the two SMA interfacer onto one single pi because the solar inverters are quite far apart and, even if I did manage that, I would still not know how to synchronise the two interfacers. Or is there there a way to sync the interfacers to collect and send the data at the same time? I thought they are doing their thing completely independently, don’t they?

I have read your post about "harmonised” intervals and timestamps but unfortunately could not fully understand it. My feeds are in 10s intervals, whereby the intervals have automatically been aligned with the epoch (i.e. last digit of epoch is always zero). I do not understand why feed starting times would matter.

So the last thing for me to do is how to align the interfacers to read at (almost) the same time, and ideally at a 10-second-rounded epoch. Can that be done?