Another graph oddity

[djh (Dave Howorth)]

Here’s another oddity of graph behaviour. Lets start with a graph of temperatures:

It shows three ‘flavours’ of the external temperature where I live. Wattisham is the nearest met office station; weather is scraped from a neighbour’s online weather station; and pizerow is my own external monitor. All three feeds are PHPTIMESERIES. The weather feed goes down for several hours quite often, you may be able to make out straight line sections in the red graph.

So here’s a version showing missing data:

You can clearly see the gaps in the red line where the feed failed. But even more noticeable, and worrying, is that the yellow line has disappeared altogether! Bug say I!

To see what’s going on, here’s the same thing but shown as points:

Now it’s a bit clearer. pizerow supplies data every minute. weather supplies data every 15 minutes. And Wattisham supplies data every hour.

Whatever algorithm the graph is using to decide whether data is missing is clearly broken. It’s deciding data is missing just because the reporting frequency is lower, not because any data is missing!

In fact since they’re all PHPTIMESERIES one could argue that there can be no such thing as missing data, although ‘weather’ provides an interesting test case.

[Timbones (Tim)]

Does it look better if you tick the ‘Average’ option on each feed?

[Robert.Wall]

A note that in the present emonCMS version, PHPTIMESERIES (Variable Interval No Averaging) is called Variable Interval Timeseries, and the old PHPFINA (Fixed Interval No Averaging) is now Fixed Interval Timeseries.

[djh (Dave Howorth)]

I’m just using the words that appear in the feed view Robert. Don’t blame me if the interface is inconsistent!

[djh (Dave Howorth)]

@Timbones

Does it look better if you tick the ‘Average’ option on each feed?

In my limited experience, averaging doesn’t work properly, especially when there is more than one feed on the graph. But I just tried it and the answer is resoundingly NO. In fact with averaging turned on it results in errors when I try to change the time of display. So another problem.

[Robert.Wall]

If someone with the current version of emonCMS reads your post, it won’t make sense unless they know the names have changed.

[djh (Dave Howorth)]

Then they won’t be able to understand their own system, since it displays the words I used. But enough of this distraction, please!

[TrystanLea]

@djh

In fact since they’re all PHPTIMESERIES one could argue that there can be no such thing as missing data, although ‘weather’ provides an interesting test case.

Hello @djh

Bear with me, while it may seem faulty the above behaviour is not a bug. It is actually by design.

The reason you are seeing points but not lines for the yellow plot is because the API has requested a data point every 900s but there are quite a few timeslots that do not have a datapoint within 900s of the requested timeslot and so a null value is returned. So you will have something like : null, null, 6.5 null 3.2 null, null 2.3 etc. The graph does not draw a line across null timeslots if you tick missing data. There is no data point within the interval of the requested time and so it is missing from the API’s perspective.

Can you paste an example of the CSV export of the graph your looking at there with points enabled?

Returning data at fixed intervals from the API is a key feature of the emoncms API’s in order to make processing and combining of data across multiple feeds in the emoncms app’s possible. E.g calculating carnot COP from an outside and flow temperature feed in the MyHeatpump app.

Perhaps the best way to improve on the case where there are not sufficient datapoints for the API request interval would be to build in some interpolation into the PHPTimeSeries feed engine.

[djh (Dave Howorth)]

Hi Trystan, yes I agree about the cause of the behaviour. It’s quite predictable as you change the timescale of the graph. But whether that is the correct behaviour or not is the issue as I see it. If the feed is hourly then even if there are regular samples every hour the graph will claim there is missing data. There isn’t! It should take acount of the feed frequency when assessing things.

My quip about PHPTIMESERIES was meant as a lighthearted comment, since such a feed could have a completely arbitrary sequence of times and still be perfectly valid.

Here’s a sample of the data as you asked:

"Date-time string", "Wattisham:temperature", "pizerow:temperature", "weather:temperature"
2023-12-30 17:15:00, null, 9.8, 9.9
2023-12-30 17:30:00, null, 9.8, 9.9
2023-12-30 17:45:00, null, 9.8, 10.1
2023-12-30 18:00:00, 9.7, 9.9, 10.2
2023-12-30 18:15:00, null, 9.9, 10.3
2023-12-30 18:30:00, null, 10.0, 10.3
2023-12-30 18:45:00, null, 10.0, 10.2
2023-12-30 19:00:00, 10.0, 10.0, 10.2
2023-12-30 19:15:00, null, 10.1, 10.3
2023-12-30 19:30:00, null, 10.1, 10.3
2023-12-30 19:45:00, null, 10.2, 10.3
2023-12-30 20:00:00, 10.1, 10.2, 10.5
2023-12-30 20:15:00, null, 10.2, 10.5
2023-12-30 20:30:00, null, 10.3, 10.6
2023-12-30 20:45:00, null, 10.3, 10.4
2023-12-30 21:00:00, 9.9, 10.3, 10.3
2023-12-30 21:15:00, null, 10.3, 10.4
2023-12-30 21:30:00, null, 10.4, 10.3
2023-12-30 21:45:00, null, 10.4, 10.3
2023-12-30 22:00:00, 10.0, 10.4, 10.3
2023-12-30 22:15:00, null, 10.4, 10.4
2023-12-30 22:30:00, null, 10.4, 10.4
2023-12-30 22:45:00, null, 10.4, 10.6
2023-12-30 23:00:00, 9.8, 10.5, 10.3

What I don’t like is that graph treats PHPFINA as some kind of special citizen. Here’s another graph:

and here’s an extract from the feed page:

The two feeds are of the same data. “radio signal strength” is a PHPTIMESERIES whilst “rssi” is a PHPFINA. pizerow supplies data every minute. And the feed page says rssi has a Feed interval of 60s. But the graph somehow manages to invent data every ten seconds for that feed whilst not doing the same for the other feed. So just the PHPTIMESERIES disappears, because it’s the only graph showing actual truthful data!

[nchaveiro (Chaveiro)]

I must comment that the current behaviour even if by design is an issue and can be misleading, data exist but is not getting showed correctly.

Failing to show data on some scale and averaging time series data was a change in recent versions, previously PHPTIMESERIES showed the same output as MYSQL feeds before the later was discontinued.