I have tried via the graphs, download from feed and directly via the API call for power to get the power in 30 min increments but each hour (e.g 01/12/2024 07:00:00) value is null whereas the 30min (e.g. 01/12/2024 07:30:00) has a value. If I get the 10s feed then all values come except the one on the exact hour e.g 2024-12-01T07:00:00+00:00 which is null (the 10s values either side are there. So this erroneous null seems to be throwing the aggregations out. All values for an exact hour show null.
I am not a programmer I tend to get data in to excel via csv or json to get the values I want. The reason I want 30min intervals is so I can tie up the energy cost per 30min as I use Octopus.
Has any one any ideas on the cause/solution.
Thanks
Where is your data originating?
As you have a NULL value, it means that no data arrived in emonCMS during that feed interval. If you are sending data at exactly 1 hour intervals at 5 mins before the hour (OK unlikely but it serves to explain what happens) then you’ll always have a value in the feed when the window closes on the hour (by Internet time). If you’re sending the data at 10 ms before the hour and it gets delayed for some reason and arrives 1 ms after the hour, you have a NULL value in the data. For this reason, we always send the data from an emonTx, the emonPi itself and the emonTH at a rate that’s slightly faster than you set the Feed Interval in emonCMS. This way, a value is occasionally overwritten (one arrives just after the window opens and another arrives and overwrites it just before the window closes), but you don’t get NULL values.
I don’t have enough information about your setup to know this is what’s happening, but it seems a likely explanation.
Apologies I should have given further info. The open energy monitor Kit3 is installed which was all preconfigured with my login etc and all the feeds were pre set up on the emoncms web site. I have not done any changes to the set up and the MyHeatpump preconfigured page shows the heatpump data (flow, return, power, energy generated etc) as it should. Looking at the Inputs and feeds pages the values configured seem to be updating between 30-40 seconds, the side bar is green up to 30 seconds then turns yellow but before reaching 40 seconds it updates and turns green again counting up the seconds.
Based on the above the values are being updated to the website within 40 second feed cycle so I think there is another issue causing every reading that is for an exact hour e.g 07:00:00 being null. Any other ideas or is it a bug in the data recording? here is a 10 second interval example from the API page using electric_power feed.
[
“2024-12-01T12:59:40+00:00”,
13.899999618530273
],
[
“2024-12-01T12:59:50+00:00”,
13.699999809265137
],
[
“2024-12-01T13:00:00+00:00”,
null
],
[
“2024-12-01T13:00:10+00:00”,
13.899999618530273
],
[
“2024-12-01T13:00:20+00:00”,
13.800000190734863
],
If I request 30 min cycle i get
[
“2024-12-01T11:00:00+00:00”,
null
],
[
“2024-12-01T11:30:00+00:00”,
10.399999618530273
],
[
“2024-12-01T12:00:00+00:00”,
null
],
[
“2024-12-01T12:30:00+00:00”,
13.399999618530273
],
[
“2024-12-01T13:00:00+00:00”,
null
],
This is quite normal.
What do you mean by “feed cycle”?
There are two times of importance here: The interval (frequency) at which data is sent to emonCMS - you say between 30 - 40 s and this sounds quite reasonable, and the “Feed Interval”, the time between samples in the database. You can find what this is by hovering your mouse pointer over the name of the Feed on the Feeds page.
(Incidentally, this is locked to Internet time, so it happens on the round 10 s, ¼ m, 20 s etc.) If you request (or someone has set) a Feed Interval less than 40 s, you’ll obviously get NULL values because the Feed is moving on faster than data is arriving.
Hi Robert, Thanks for patience on this one, so my feed interval is 10s as in your screenshot. Looking more closely at far right column on the Feeds Tab (see screenshot) the seconds count goes 6, 11,16,21,26 which are all green, then 31 (yellow) then the value to the left updates and the seconds count goes back to 6 11, etc.
From what you are indicating ‘something’ is happening at exactly each hour on the emonTH that means that the 10s interval value for T00:00 doesn’t get written to the database. ALL the 10s interval values between exact hours T00:00 get written correctly (3,599 values) then the T00:00 fails. Does the emonTH do something on the hour to cause this or something at the website end?
Thanks again for your responses
Have you really got an emonTH, or have you got confused?, because as far as I’m aware, an emonTH is not normally part of a heat pump monitor installation.
Hi Robert, I do have a an EmonTH (this was an extra) but that is for the indoor temperature 
the ‘black box’ in the monitoring kit is an emonhp . Sorry for the confusion over the name.
1 Like
Hello @sarnold814 If you want average power you can put a tick in the average box on the graph tool for that feed. Are you seeing null values on the Energy feed? That shouldnt have null values as it should be recorded using Log to Feed (join).
There’s a bit more on averages here: Calculating Averages — OpenEnergyMonitor 0.0.1 documentation
@TrystanLea Surely the original problem is it’s a 10 s feed with data arriving every 30 - 40 s?
@sarnold814 If you set up a Feed for your emonTH, make sure you set the interval to 60 s.
Hi Trystan, looks like setting the Interval to 1800 and yes to average gives me the power in watts for a 30 min timeslice and doesn’t give any nulls. Still can’t understand why when I look at the most granular interval of 10s it gives a NULL whenever the Time element is 00:00 when all the other 10second slices (359 of them between the hours) have a value e.g.
[
“2024-12-01T12:59:50+00:00”,
13.699999809265137
],
[
“2024-12-01T13**:00:00+00:00**”,
null
],
[
“2024-12-01T13:00:10+00:00”,
13.899999618530273
],
I will test the average out a bit more tomorrow but on the face of it that works
Thanks
1 Like
Hi. I’ve been pulling data from the API and have noticed what seems to be a similar issue.
- at 10s to the hour (xx:59:50), 7 feeds are missing more often than not (both power readings; flow/return/rate; diverter and HW)
- on the hour (xx:00:00), these feeds are missing almost every single time so you only get outside_temp, temp, electric_Energy and heatmeter_Energy
- at 10s part the hour (xx:00:10), they all (mostly) reappear. But diverter value and HW are missing almost every other day.
Since I’m looking at data each minute for now, the simple workaround is to take e.g. the 30s value each minute. But it is a curious pattern.
My other query on the API data is why heatmeter_Energy is rounded to whole kWh (almost always, but not in absolutely every case) whereas electric_Energy is always reported to 3 dp. Even with hourly frequency, that’s quite a loss of detail.
Hello @NickB Could you send me some example API requests where you are seeing this so that I can see exactly what you are seeing?
Here’s a couple of examples of what I would expect, half hourly heat pump kWh consumption data (I’ve turned on delta=1 here to return the total for the half hour rather than the cumulative kWh total)
HI @TrystanLea.
I’ll send the code by email - extracting data with 10s intervals for a few hours. But these are the records from on the hour:
And these are the records for 10s past the hour:
You can see the missing values on the hour, far fewer at 10s past. You can also see the heatmeter_Energy values rounded to whole kWh, in contrast to electric_Energy.
Nick
@TrystanLea - and just looking at heatmeter_Energy, if I use delta = 1 with hourly intervals, I still get values rounded to whole kWh
Nick
@NickB could you share with me the specific API calls you are making, or steps to reproduce if it’s via the emoncms user interface?
The call is produced by an R programme - I emailed you a copy. But this is an example which produces missing values on the hour for electric_Power (with api_key removed):
Produces: [[1740531600,null],[1740535200,null]]
But if I nudge the start time on by 10s, I get valid responses:
Produces: [[1740531610,10.100000381469727]]
1 Like
Thanks @NickB I realise what’s probably happening here, there’s a cron task every hour on the emonHP that restarts emonHub, that’s probably causing this null value. We have more recently improved the a few things that should negate the need for this cron task. Shall I try turning it off on your emonHP and see if that clears this issue?
The 1 kWh resolution on the heat meter kWh readings is a result of this being the resolution that is reported by the heat meter. I can add a calculated kWh feed with higher resolution calculated from your heat meter heat feed data if you like?
Thanks @TrystanLea . I’ve got workarounds but great to try fixes.
Hello @NickB. I’ve applied both of these. There’s now a new feed called heatmeter_heat_kwh which has been generated from the power data. You will notice a slight difference in integration, the total energy calculated from the power data is slightly less due to both a gap in the heat power data and also the effect of defrosts. The Energy accumulator on the heat meter does not go down when heat is lost to defrosts as you might expect (it’s a heat only register). To account for defrosts on HeatpumpMonitor.org and in the Emoncms app, all of the total energy values are calculated from the power data and there are additional checks to verify that both electric power and heat power values are present simultaneously for these integrations.
1 Like