Problem with missing data on graph (second post)

Hardware emonTH
1

Must have posted this incorrectly the first time as I can’t seem to find it and haven’t received any replies.
Sorry for the double post.

+++++
I have attached a file showing a graphing problem from my emonTH V2. config and log are also included.

First, I realize that I have a very weak signal from the unit to the emonBase. However, the temperature seems to be coming through and being graphed (more or less), while the humidity is only shown in small sections of the graph and is missing most of the data (show missing data checked). The log portion shows that humidity data is being received every 60 seconds even though it’s not being graphed most of the time.

Since I’m very new at this, I may be overlooking something obvious. Thanks for any help.

Bob
23.pdf (347.9 KB)

2

It’s here: Problem with missing data on graph
(I’ve now locked that thread so as to avoid having two sets of replies.)

I’m not an emonCMS expert, but to save somebody else asking, how are the feeds in emonCMS set up, viz which feed engine, what interval?

3

Thanks, Robert. See attached file with the info you requested. humidity.pdf (470.8 KB)

I set up the humidity input “log to feed” with the defaults. The other feeds are logging correctly from the emonTH. The file contains another screen shot of the on/off humidity logging plus setup.

Much appreciated for any suggestions.
Bob

4

Can you please put the images in your posts rather than attaching them in pdf’s?

I’m guessing that means you have created a 10s fixed interval feed

If the emonTH is “stock” then it will be posting at 60s intervals, This means 5 x 10s datapoints will be null and every 6th datapoint will contain data. Every datapoint of every graph derived from that feed (or others like it) has only a 1 in 6 chance of not being “missing”. It is probably just a case of luck of the draw that the others appear to be better. But I take a second look when I can see the images.

5

Paul,

Is there a difference in the humidity posting and the other feeds from the emonTH? I ask this because I didn’t change the 10s interval on any temp, batt, or rssi and they all seem to plot correctly. It’s just humidity that is missing so much data.
AT first I thought it was the weak signal but then the other feeds were ok. Everything seems to be posting every 60s as required. Thanks.
Bob

6

No. All the data is sent as a single byte stream: temperature, external temperature, humidity, battery voltage and pulsecount. All is protected by a checksum, and the entire packet is rejected if that fails. So there’s nothing to distinguish the humidity during transmission.

You will probably have a better result if you redo the feed with a 60 s interval. But you’ll lose your existing data if you do that - it’s not possible to change the interval, you must delete the feed and make it again.

7

I read you have an emonBase but there is no mention of the software you are using.

What Robert says about a single byte stream is correct for the journey to the emonBase. A similar situation applies to any data being passed via http from emonhub to emoncms. But on the emonSD image the data is passed from emonhub to emoncms via individual MQTT topics, however in real terms the “batch” of data values are all processed in the same way, at the same time and there is generally no noticeable difference between them.

I previously outlined the expected behavior of having 60s data saved to a 10s feed, this has come up many times and the symptoms are usually the same, data missing in one (or more) feeds but rarely all from the same packet.

The influences in whether a graph will plot with or without missing data will depend on when the feed was created and when the graphing query is run.

The feed start times are not linked or sync’d and when the data is saved it is at multiples of the fixed interval (10s) from the start time, regardless of the actual time received. Those 10s timeslots can be filled with data or set to null.

Every time your 60s data lands it is creating a hole of 5 null datapoints in the 10s feed and only occupying every 6th datapoint.

Whether there is anything peculiar occurring to the humidity feed alone or not, we wouldn’t be able to tell without looking at the raw csv because we cannot rely on the data shown in the graphs, it is expected to be inconsistent in this instance.

Something else that just occurred to me that you could test for. I believe the graph start, stop and sample times might be linked to the first selected feed, so try selecting the feeds in a different order or individually in separate windows, I’m not sure of this so it’s unlikely to firmly prove anything, but if you manage to stumble across a combination that shows the humidity in a good light and/or one of the others with more missing data, it may help.

In short, as Robert says you need to recreate the emonTH feeds with a 60s fixed interval, the sooner you do, the sooner you will start accumulating historic data again and we can revisit this to check for any further abnormalities to the humidity feed. The longer you leave it the more data historic you are discarding.

The new feeds will consume 1/6th of the disk space of the old ones with no loss of resolution.

8

Paul,
I’m not certain which software you are referring to, however I am running an RPI3 with the latest updates along with the latest software from OEM.
Moving to 60s intervals (with data loss) is no problem since I have only had the emonTH for about a week and am still playing around with where I want to put it. It’s now in the crawl space under my house but the signal is fairly weak so I’ll need to reposition it somewhat.
I have reset to 60s so will let you know after it’s been running a while. So far it all looks good.
Thanks to you and Robert for the help.

Bob

9

It sounds like you are using the emonSD image, in which case we know what you have if it’s fully updated. The comments I made about the use of MQTT do apply to your setup, but that is just info, I do not think that is a factor in what we are discussing here.

Cool! let us know if it still doesn’t look right, if debugging/monitoring a weak signal, you have definitely done the right thing, in theory there should be very few if any null datapoints now.

10

Everything seems to be plotting ok with only a few breaks. Thanks again for the help!
Bob