Strange graph display disparity between 24 hours and 1 week

I recently bought an emonTH and I’m looking at a graph of its output. I noticed what looks like an error in the display. Here’s the 24 hour graph:

and here’s the 1 week view:

Now the 24 hour view is plausible. The house is heated from 00:30 until 07:30 (E7 tariff) and then the heating is off through the day.

But the one week view shows something completely different over the last 24 hours. The peak is at 07:15 on the 16th, going down to a minimum 18:30-21:00 that day and then up a bit to the present time. You can see lots of nulls in the CSV data.

What’s going on?

It looks like you have a mixture of intervals on your feeds, whereby “Node pizerow” is much more frequent that the others. This might have confused the graph tool.

Something to try, which may help

  • Press the Reload button - this seems to fix some interval issues sometimes
  • Tick the Average option on each feed - this can smooth out the plot and remove some nulls

If you’re using PHPFINA engine for your feeds, make sure that they have been created with the same interval as the data is incoming to Emoncms. The default interval is 10s, so if the temperatures are only being read every 30 minutes then there’ll be a lot of nulls in between readings. You may find PHPTIMESERIES is better suited to this kind of feed.

The emonTH default rate for sending data is a little less than 60 s, so you should choose a Feed Interval of 60 s or greater.

1 Like

Thanks guys. It occurred to me overnight that it might be some bug to do with different feed frequencies.

Where’s the documentation about all this these days? And how do I examine the details of the feeds? I find the new interface pretty confusing.

Good question - it seems to have disappeared some while ago. Since then, several times here in the forum I’ve written a description of how the Fixed Interval Timeseries database works.

Hover your mouse over the name on the Feeds page. You can’t change a fixed interval feed once it has been created, you must either abandon it or import the data from the old into the new - and I can’t tell you any more about this because I don’t know the details.

N.B. Tim meant “Emoncms Variable Interval Timeseries” as it appears in the drop-down list.

Hmm, looking as Robert told me shows pizerow at around 56 s intervals and the emonTH around 5 s. I thought I’d set the emonTH to be 1 minute but I can’t remember exactly. Looking at all the inputs on various devices they seem to have starnge intervals with lots of decimal places, and the numbers for various feeds from the same device seem to have different intervals (which AFAIK should be impossible). So I’m thoroughly confused.

That just removed the last day’s data. :frowning:

Doesn’t seem to make any difference.

The emonTH are PHPFINA. How do I check/set the intervals? Which figure is shown using Robert’s ‘trick’?

That seems like a major problem. @TrystanLea ?

Looking at the times displayed in the feed list, the times for the emonTH do go from 0s to 60s, so I don’t know what that means.

Explain exactly what you’re looking at? This is what I was referring to (mouse cursor is over “MSG”):


Is this the documentation you are looking for?

1 Like

From memory, the write-up in the old ‘Learn’ wasn’t exactly that - it looks as if the Github stuff is a rewrite.

It begs the question: why couldn’t I find it in ‘Docs’? It seems like I’m not the only one who is mystified by that badger-sett called Github.

1 Like

Timeseries etc data storage: Now gone, was

Here are my crib-notes on Timeseries Feeds:

A Feed in emonCMS is the database - the data storage mechanism. emonCMS offers two types of permanent Feed: EmonCMS Fixed Interval timeseries (also called PHPFINA) and EmonCMS Variable Interval Timeseries (also called PHPTIMESERIES). The first is used for data that arrives at regular intervals without a break. It operates by storing the creation time and interval as metadata, with the advantage that storing only each data item is more economical in storage. The second is used when it is necessary to record the exact arrival time of the data, when data arrives irregularly or unpredictably. Each record contains both the arrival time of the data and the data itself, so this uses approximately twice the storage space compared to the fixed interval.

Explanation of Fixed Interval TimeSeries.

Think of the emonCMS FITS Feed as a line of boxes moving along a conveyor. The speed of the conveyor - the time it takes a box to go past a fixed point - is the “Feed Interval”. If there’s a box going past every 10 s but data falls onto the conveyor every 30 s, two boxes out of three will move on empty, with no data in them (a NULL value). The device sending data to emonCMS needs to send it just slightly more frequently than the Feed interval you choose when you create the Feed. If there’s already data in the box and it hasn’t moved on, it will overlay the data already in there, which is lost. But you won’t have null values in the Feed.

The same applies to any Feed and any timebase - and the boxes move on the Internet minute, or multiples or sub-multiples thereof. i.e., a 10 s feed happens on the minute and at 10, 20 etc seconds past.

If you can always guarantee that the source will send the data in the middle of a time slot by Internet time, you can safely match the rates exactly.

Where the sender, the emonTx for example, depends on a crystal tolerance for its timebase, or there might be transmission delays, we set it to send every 9.96 seconds and use a 10 s feed interval. Similarly, the emonTH is set to send every 58 s approximately and uses a 60 s interval.

In your image, you have cut off the right hand side, which shows times (in green seconds for this device).

Not really no. That just describes the mechanics of PHPFINA. I’m looking for a more naive user viewpoint; explaining how to choose and use the various possibilities and how to use the interface (e.g. describing the facility Robert mentioned to hover over the name of the feed and what all the little icons at the left and top of the screen do)

That is the time elapsed since the browser (I believe) received an update from emonCMS. So, knowing your emonTH sends every 58 s or so, if you see a number bigger than 60, an update has gone missing.

Nothing like this has ever existed, as far as I know. I found out almost everything I know about emonCMS by experimenting and observing the effect. Where it’s not obvious, many things seem to have the pop-up Help. Quite a few need a proper explanation though, and some facilities/features I haven’t looked at simply because they’re not documented and explained.

All the software I’ve written comes with documentation, and I reckon it takes at least as much effort to document it as it does to write it. Not everyone does the same.

There certainly used to be something explaining the differences between PHPFINA and PHPTIMESERIES and when to use each, for example.

I’ve got a new issue today: " Request error: Received data not in correct format. Check the logs for more details" at the top of the graph. What does that mean? Which logs is it talking about? I haven’t found anything in /var/log/emoncms/*

I didn’t say that there wasn’t - in fact I quoted the dead link:

Read what I quoted from you. This is what’s never existed, as far as I know.

Was the explanation of Timeseries I wrote helpful, or was I wasting my time finding and quoting it?

Ah sorry, I didn’t realize you were being specific.

Sorry again, but that particular stuff I understand already.

Does anybody have a pointer or any explanation about the error I’m seeing today? What logs is it talking about?

The problem is specifically when the timescale is ‘1 week’. It’s fine for ‘24 hours’ or ‘2 weeks’.

OK a little progress. Most of the issues look like bugs in the graph module to me. The graph I’ve been looking at has three lines from my new emonTH and one green one from a separate pi ‘pizerow’. The three feeds from the emonTH are all PHPFINA whilst the pizerow one is a PHPTIMESERIES. I believe the difference is significant.

There seem to be some unexpected features of the way the graph is loading data from a PHPFINA feed. As seen in the first post, it seems to miss whole bunches of data sometimes for some reason I don’t as yet understand. I’ve got another case that indicates what I suspect is the same problem now. First here’s a screenshot similar to the first couple of today’s data:

Now here’s the same graph with one difference - I clicked on the ‘Show missing data’ button:

Note that the PHPFINA lines have disappeared, whilst the points are still there. That’s clearly wrong since the previous view shows there’s data there. In fact moving the mouse over the lines shows that the blocky appearance of the lines is due to the fact that not all readings are shown. There are readings every minute available from the feeds, but there are big gaps in the data shown on the graph. Which apparently appear ‘missing’ when asked to show them. Moving the mouse around reveals that there are actually some points from the ‘missing’ lines still on the graph, bizarrely.

Then we come to the issue I mentioned in post 16. This time it’s the PHPTIMESERIES data behaving strangely. There was a gap in the pizerow data from 2023-11-10 12:00 until 2023-11-14 12:45. It seems that if that gap should appear at the left hand end of a graph view then the graph fails with the “Request error: Received data not in correct format. Check the logs for more details” error instead of showing a straight line or a gap as it does in other circumstances.

And I still have no idea which log or logs I’m supposed to be looking at? I haven’t found anything so far. Somebody must know which logs the message is referring to!

Here’s another oddity. On one of my graphs, the sidebar is blank, except for the expansion button at the bottom. And if I press that, the expanded sidebar is blank as well, so I can’t edit the graph. I can’t take a screenshot of that, because it closes automatically before I can. So here’s a picture:

Any thoughts as to what the problem is?