Strange graph display disparity between 24 hours and 1 week

djh · 17 November 2023 21:08

I recently bought an emonTH and I’m looking at a graph of its output. I noticed what looks like an error in the display. Here’s the 24 hour graph:

and here’s the 1 week view:

Now the 24 hour view is plausible. The house is heated from 00:30 until 07:30 (E7 tariff) and then the heating is off through the day.

But the one week view shows something completely different over the last 24 hours. The peak is at 07:15 on the 16th, going down to a minimum 18:30-21:00 that day and then up a bit to the present time. You can see lots of nulls in the CSV data.

What’s going on?

Timbones · 17 November 2023 22:11

It looks like you have a mixture of intervals on your feeds, whereby “Node pizerow” is much more frequent that the others. This might have confused the graph tool.

Something to try, which may help

Press the Reload button - this seems to fix some interval issues sometimes
Tick the Average option on each feed - this can smooth out the plot and remove some nulls

If you’re using PHPFINA engine for your feeds, make sure that they have been created with the same interval as the data is incoming to Emoncms. The default interval is 10s, so if the temperatures are only being read every 30 minutes then there’ll be a lot of nulls in between readings. You may find PHPTIMESERIES is better suited to this kind of feed.

Robert.Wall · 17 November 2023 22:21

The emonTH default rate for sending data is a little less than 60 s, so you should choose a Feed Interval of 60 s or greater.

djh · 18 November 2023 12:07

Thanks guys. It occurred to me overnight that it might be some bug to do with different feed frequencies.

Where’s the documentation about all this these days? And how do I examine the details of the feeds? I find the new interface pretty confusing.

Robert.Wall · 18 November 2023 12:34

Good question - it seems to have disappeared some while ago. Since then, several times here in the forum I’ve written a description of how the Fixed Interval Timeseries database works.

Hover your mouse over the name on the Feeds page. You can’t change a fixed interval feed once it has been created, you must either abandon it or import the data from the old into the new - and I can’t tell you any more about this because I don’t know the details.

N.B. Tim meant “Emoncms Variable Interval Timeseries” as it appears in the drop-down list.

djh · 18 November 2023 13:25

Hmm, looking as Robert told me shows pizerow at around 56 s intervals and the emonTH around 5 s. I thought I’d set the emonTH to be 1 minute but I can’t remember exactly. Looking at all the inputs on various devices they seem to have starnge intervals with lots of decimal places, and the numbers for various feeds from the same device seem to have different intervals (which AFAIK should be impossible). So I’m thoroughly confused.

That just removed the last day’s data.

Doesn’t seem to make any difference.

The emonTH are PHPFINA. How do I check/set the intervals? Which figure is shown using Robert’s ‘trick’?

djh · 18 November 2023 13:26

That seems like a major problem. @TrystanLea ?

djh · 18 November 2023 13:31

Looking at the times displayed in the feed list, the times for the emonTH do go from 0s to 60s, so I don’t know what that means.

Robert.Wall · 18 November 2023 13:32

Explain exactly what you’re looking at? This is what I was referring to (mouse cursor is over “MSG”):

Gwil · 18 November 2023 13:41

github.com

emoncms/emoncms/blob/master/docs/timeseries/Fixed-interval.md

## Fixed interval no averaging (PHP Fina)

PHPFina source code can be found here: [PHPFina.php](https://github.com/emoncms/emoncms/blob/master/Modules/feed/engine/PHPFina.php)

In many if not most applications time series data is recorded at a fixed interval. A temperature or power measurement is made every 10 seconds, minute or hour. Given this highly regular nature of the time series data we can do away with the need to record every datapoint's timestamp and instead just record the start time of the time series and the time series interval in a meta file and then only record the datapoint values in the datafile. The timestamp for any given datapoint can easily be worked out by the start time, interval and the position of the data point in the file.

There are two main advantage of this approach versus the variable interval approach.

1. The first advantage is that if we want to read a datapoint at a given time we don’t need to search for the datapoint as we can calculate its position in the file directly. This reduces  the amount of reads required to fetch a datapoint from up to 30 reads down to 1 giving a significant performance improvement.

2. The second advantage is that in a time series where the data is highly reliable the disk use can be up to half that of the disk use used by a variable interval engine, due to not needing to record a timestamp for each datapoint.

The disadvantage is that when you have a gap in your data, that gap needs to be padded with NAN values, those times need to be allocated even if not used. This isn’t a big a disadvantage as it may first seem because even if you had a gap of 6 months out of a year you would still only consume as much disk space as the variable interval method. Another disadvantage is that you need to set the interval of the feed when you create the feed.

The current PHPFina timeseries implementation uses two files for each fixed interval feed, a meta file to store the start time and interval and the datafile to store the 4 byte data values. It would be possible to reduce this to a single file if the first 8 bytes of the data file contained the start_time and interval meta data, this would reduce flexibility to add further meta data in the future. There is not necessarily a right way to do it, it's just a design decision to bear in mind.

**Metafile:**

Meta file: $id.meta:

This file has been truncated. show original

Is this the documentation you are looking for?

Robert.Wall · 18 November 2023 13:47

From memory, the write-up in the old ‘Learn’ wasn’t exactly that - it looks as if the Github stuff is a rewrite.

It begs the question: why couldn’t I find it in ‘Docs’? It seems like I’m not the only one who is mystified by that badger-sett called Github.

Robert.Wall · 18 November 2023 14:02

Timeseries etc data storage: Now gone, was https://learn.openenergymonitor.org/electricity-monitoring/timeseries/History

Here are my crib-notes on Timeseries Feeds:

A Feed in emonCMS is the database - the data storage mechanism. emonCMS offers two types of permanent Feed: EmonCMS Fixed Interval timeseries (also called PHPFINA) and EmonCMS Variable Interval Timeseries (also called PHPTIMESERIES). The first is used for data that arrives at regular intervals without a break. It operates by storing the creation time and interval as metadata, with the advantage that storing only each data item is more economical in storage. The second is used when it is necessary to record the exact arrival time of the data, when data arrives irregularly or unpredictably. Each record contains both the arrival time of the data and the data itself, so this uses approximately twice the storage space compared to the fixed interval.

Explanation of Fixed Interval TimeSeries.

Think of the emonCMS FITS Feed as a line of boxes moving along a conveyor. The speed of the conveyor - the time it takes a box to go past a fixed point - is the “Feed Interval”. If there’s a box going past every 10 s but data falls onto the conveyor every 30 s, two boxes out of three will move on empty, with no data in them (a NULL value). The device sending data to emonCMS needs to send it just slightly more frequently than the Feed interval you choose when you create the Feed. If there’s already data in the box and it hasn’t moved on, it will overlay the data already in there, which is lost. But you won’t have null values in the Feed.

The same applies to any Feed and any timebase - and the boxes move on the Internet minute, or multiples or sub-multiples thereof. i.e., a 10 s feed happens on the minute and at 10, 20 etc seconds past.

If you can always guarantee that the source will send the data in the middle of a time slot by Internet time, you can safely match the rates exactly.

Where the sender, the emonTx for example, depends on a crystal tolerance for its timebase, or there might be transmission delays, we set it to send every 9.96 seconds and use a 10 s feed interval. Similarly, the emonTH is set to send every 58 s approximately and uses a 60 s interval.

djh · 18 November 2023 15:37

In your image, you have cut off the right hand side, which shows times (in green seconds for this device).

djh · 18 November 2023 15:50

Not really no. That just describes the mechanics of PHPFINA. I’m looking for a more naive user viewpoint; explaining how to choose and use the various possibilities and how to use the interface (e.g. describing the facility Robert mentioned to hover over the name of the feed and what all the little icons at the left and top of the screen do)

Robert.Wall · 18 November 2023 16:18

That is the time elapsed since the browser (I believe) received an update from emonCMS. So, knowing your emonTH sends every 58 s or so, if you see a number bigger than 60, an update has gone missing.

Nothing like this has ever existed, as far as I know. I found out almost everything I know about emonCMS by experimenting and observing the effect. Where it’s not obvious, many things seem to have the pop-up Help. Quite a few need a proper explanation though, and some facilities/features I haven’t looked at simply because they’re not documented and explained.

All the software I’ve written comes with documentation, and I reckon it takes at least as much effort to document it as it does to write it. Not everyone does the same.

djh · 20 November 2023 15:09

There certainly used to be something explaining the differences between PHPFINA and PHPTIMESERIES and when to use each, for example.

I’ve got a new issue today: " Request error: Received data not in correct format. Check the logs for more details" at the top of the graph. What does that mean? Which logs is it talking about? I haven’t found anything in /var/log/emoncms/*

Robert.Wall · 20 November 2023 15:24

I didn’t say that there wasn’t - in fact I quoted the dead link:

Read what I quoted from you. This is what’s never existed, as far as I know.

Was the explanation of Timeseries I wrote helpful, or was I wasting my time finding and quoting it?

djh · 20 November 2023 16:05

Ah sorry, I didn’t realize you were being specific.

Sorry again, but that particular stuff I understand already.

Does anybody have a pointer or any explanation about the error I’m seeing today? What logs is it talking about?

The problem is specifically when the timescale is ‘1 week’. It’s fine for ‘24 hours’ or ‘2 weeks’.

djh · 21 November 2023 17:59

OK a little progress. Most of the issues look like bugs in the graph module to me. The graph I’ve been looking at has three lines from my new emonTH and one green one from a separate pi ‘pizerow’. The three feeds from the emonTH are all PHPFINA whilst the pizerow one is a PHPTIMESERIES. I believe the difference is significant.

There seem to be some unexpected features of the way the graph is loading data from a PHPFINA feed. As seen in the first post, it seems to miss whole bunches of data sometimes for some reason I don’t as yet understand. I’ve got another case that indicates what I suspect is the same problem now. First here’s a screenshot similar to the first couple of today’s data:

Now here’s the same graph with one difference - I clicked on the ‘Show missing data’ button:

Note that the PHPFINA lines have disappeared, whilst the points are still there. That’s clearly wrong since the previous view shows there’s data there. In fact moving the mouse over the lines shows that the blocky appearance of the lines is due to the fact that not all readings are shown. There are readings every minute available from the feeds, but there are big gaps in the data shown on the graph. Which apparently appear ‘missing’ when asked to show them. Moving the mouse around reveals that there are actually some points from the ‘missing’ lines still on the graph, bizarrely.

Then we come to the issue I mentioned in post 16. This time it’s the PHPTIMESERIES data behaving strangely. There was a gap in the pizerow data from 2023-11-10 12:00 until 2023-11-14 12:45. It seems that if that gap should appear at the left hand end of a graph view then the graph fails with the “Request error: Received data not in correct format. Check the logs for more details” error instead of showing a straight line or a gap as it does in other circumstances.

And I still have no idea which log or logs I’m supposed to be looking at? I haven’t found anything so far. Somebody must know which logs the message is referring to!

djh · 21 November 2023 19:53

Here’s another oddity. On one of my graphs, the sidebar is blank, except for the expansion button at the bottom. And if I press that, the expanded sidebar is blank as well, so I can’t edit the graph. I can’t take a screenshot of that, because it closes automatically before I can. So here’s a picture:

Any thoughts as to what the problem is?