I am receiving gas meter consumption data (In real time) into NodeRed via MQTT.
I publish this to Emoncms using MQTT. When I first created a feed I accepted the standard 10 second timing. The result was I was reading far less than the meter. I looked at the feed graph and when I checked missing data box I realised there were considerable gaps. I created a new feed with 60 second timing and all seemed well.
I recently did another check as my daily readings where again coming up about 5% lower than the actual meter. On checking the 60 second feed graph with missing data box checked there are a few gaps.
What does missing data actually mean? Is it a value missing? I have retain set to on when I publish MQTT to emoncms. I could do more checking in NodeRed if needed.
Yes, it means emoncms did not receive a datapoint for the timestamp in question. Can you think of any reason why datapoints may have gone missing somewhere? outages on hardware, radio coms issue, timing of the updates?
Does the missing data appear periodic by any chance? If you have say a 10s feed but post every 11s it should show a missing datapoint every 100s…
Does the measured time gaps drift slightly? You can see form the EmonCMS logs (if you change the log level to debug I think) the time the MQTT messages are picked up. If the gap is more than the 60s, you will get empty slots.
How is the consumption for each 60s reading measured / calculated?
Unless you are using an mqtt topic that includes a timestamp, anything but QoS 1 will be an issue as the data only has value when delivered immediately so that the timestamp is correctly added by emoncms at arrival. If your (say) 10s data gets held up somewhere (eg the broker) and arrives late it will be given a later timestamp so it can miss it’s fixed interval time slot. QoS2 may send multiple copies of the same data, QoS 3 will guarantee delivery no matter how late, so if 5 datapoints are delayed, then all 5 will get delivered asap and the time differences between them will only be millis and therefore only one datapoint will survive. “Fire and forget” QoS1 is best when not passing timestamps as it will simply move on to the next datapoint rather than buffer, delay or repeat datapoints, resulting in time shifted or overwritten datapoints.
If you can, send an ever increasing total rather than “used this period” values, that way if a few datapoints go missing, everything still adds up as soon as the next packet arrives, lost packet wouldn’t then result in missing consumption.
I will change log level and see what it shows. I don’t know how often the data arrives from the meter (via a 3rd party). Debug in NodeRed shows a new message every 5 to 6 seconds but it does vary. The data for gas only changes every half hour. I assume that’s because the gas meter is asleep all other times to save battery life. At the moment I strip the gas data out of the json mqtt message from the meter, convert hex to integer and publish to emoncms with no further processing.
QOS is set to 1
Are you suggesting I only send changed data? In that case it will publish every 30 minutes. What feed timing should I then use? I really want to update emoncms as soon as the latest consumption data is available.
I wasn’t saying that directly, but matching your feed to the data updates is key to using fixed interval feeds.
If the data can only change half hourly (HH data is a common metering standard) then there isn’t much value in sampling or saving more frequently. A 30min feed would be ideal if you only update the inputs every 30mins.
Maybe you can use the HH to trigger the publish to emoncms? Or just arrange things so that the 30min update happens every 30mins around 10s after the data is due, if the HH data is reliable enough.
I believe there is a emoncms Mqtt api that you can submit a timestamp with the data (I’m sure @borpin will be able to confirm) if you used that option you then have absolute control over the feed data as you can dictate the timestamp to align exactly with the feed time slots regardless of when the data lands or takes to reach emoncms, ie every datapoint timestamp could be an exact multiple of 1800 secs, even if it arrives 30s later than expected. I have used this method loads and discussed it here on the forum. I call it “harmonised” timestamps.
I have created a new feed with a half hour interval and now have graphs with no missing data. I have yet worked out how to ensure my feed is updated as soon as the half hourly data changes.
There is a similar missing data issue with electricity data from the smart meter via mqtt which updates every few seconds but that is not a problem as I have the electricity monitoring covered by an emonTX and CT’s
Another option if these are cumulative kWh readings is to use the new Log to feed (Join) input process, see: Calculating Daily kWh - Guide | OpenEnergyMonitor this will join across missing datapoints with a straight line - giving better results in the daily kWh graphs…
