Stored feed data wrong

pb66 · 2018-03-31T11:20:11.214Z

bart.ellast:

But how can the data come arrive in the feed with a delay?

What is sending the data? How does it know what time it is?

bart.ellast · 2018-03-31T11:22:52.868Z

I’m sending the data using node-red with the emoncms node’s.
The node-red time is correct.
It has been working for a few years, it’s only recently starting this behaviour.

On the dashboard you can see the dial showing the temperature from this very feed! Notice the different value.
I don’t understand how the feed graph can show value’s from a few hours ago?
I’m sure that tonight i’ll be seeing the same data as on the newly created feed on the old feed.

pb66 · 2018-03-31T11:26:55.908Z

bart.ellast:

On the dashboard you can see the dial showing the temperature from this very feed!

There is no dial

image2701×1788 326 KB

bart.ellast · 2018-03-31T11:30:27.556Z

ugh… @pb66 that brings me to this issue…

Error displaying widgets (last 3-4 days?) Emoncms.org

No one else having this problem lately?

There is a dial, just not showing any more when not logged in.

Anyway, it seems ALL data is showing in delay on the graphs.
The heatpump data is also off…(because it is off now, and still showing energy usage on the gaph…) but why not this newly created feed?

The emoncms server time is correct…

pb66 · 2018-03-31T11:42:41.005Z

bart.ellast:

I’m sending the data using node-red with the emoncms node’s.

The node-red time is correct.

have you recently updated the emoncms node?

bart.ellast:

Anyway, it seems ALL data is showing in delay on the graphs.

Does ALL data come from the same source?

bart.ellast:

but why not this newly created feed?

Are these old and new feeds next to each other in the input processing?

Are they both the same type/spec of feed?

bart.ellast:

The emoncms server time is correct…

Do you have a timezone set?

Does the right hand edge of the graph always line up with the current time where you are when you first open it?

Does “last updated” times on the inputs and feeds page seem correct?

bart.ellast · 2018-03-31T11:55:30.780Z

Yes I have updated the node recently BUT no not all data is coming from this node-red instance. Some users with dashboards also affected have nodes that were not updated in a while.
It must be server related.

These feeds are phpfiwa and phpfina. What do you mean next to each other?
They are from different users also.

Timezone is set. The date command shows correct timezone and date. Also the administration page shows correct date and time.

The time does line up correctly on the right edge.

The updated times show correct.

pb66 · 2018-03-31T12:13:14.438Z

bart.ellast:

They are from different users also.

Now you’ve thrown me, why would comparing the data from 2 separate users have any meaning?

bart.ellast:

I tried logging to a new feed, this shows the correct data!

This statement suggests (to me) you have set up another feed on the same data and that is reading correcly

bart.ellast:

What do you mean next to each other?

You are telling me feed A is wrong, but new feed B is correct using the same data, to rule out any possibility of differences caused by other input processes I ask to confirm they are next to each other in the same input process list, of the same input, belonging to the same user. eg

input xyz

log to feed “Feed A”
log to feed “Feed B”

must be the same data where as

input xyz

log to feed “Feed A”
some other process(es)
log to feed “Feed B”

may have different data for each feed.

bart.ellast · 2018-03-31T12:18:26.000Z

Sorry if I’m not clear, English isn’t my native language.

I’m not comparing the data of 2 seperate users. Just saying the data from other users and their graphs is also shifted in time.

I did set up e next feed on the same input to test. You can see this in the solar graph. And that one is reading correctly.
Both log to feed lines are next to each other on the input setting, no process between them.

pb66 · 2018-03-31T12:31:56.431Z

bart.ellast:

Sorry if I’m not clear, English isn’ty native language.

No problem, your English is good, this isn’t easy stuff to discuss even between users of the same native language.

bart.ellast:

Just saying the data from other users and their graphs is also shifted in time.

Some or all users?

bart.ellast:

I did set up e next feed on the same input to test. You can see this in the solar graph. And that one is reading correctly.

Both log to feed lines are next to each other on the input setting, no process between them.

Are they the same feed type and spec? Or

bart.ellast:

These feeds are phpfiwa and phpfina.

are you saying the original feed is phpfiwa and the new test feed is phpfina?

By any chance are all the users with issues using phpfiwa and those that are ok using phpfina?

phpfiwa is no longer supported and not many users still use phpfiwa. So if there was a recent change in emoncms that effected phpfiwa differently, we might not know about it yet. But that’s just a guess based on what you’ve told me this far, I have not heard of any such issue previously.

bart.ellast · 2018-03-31T12:42:15.293Z

Not all affected feeds are phpfiwa. I have a user with all phpfina feeds that’s also affected.

I have users that do not seem affected at first look…

The new feed I created from the same input is phpfina, the old one phpfiwa.

I did ask a while ago if I can migrate data from phpfiwa feeds to phpfina , but never got response. I can’t lose data.

pb66 · 2018-03-31T13:17:32.587Z

Can you open the 2 feeds in the graphs module and try scrolling right to see “into the future” does the faulty feed show data beyond now?

Can you confirm that the timezone setting for each of the troubled accounts is set? To do that please try to set it again even it you think it is correct. There was a change in the way emoncms uses the timezone data quite some time ago which resulted in errors if you had the correct timezone set using the old method.

bart.ellast · 2018-03-31T13:25:09.080Z

The faulty feed does indeed show data up until 23h this evening!

I set the timezone again in the account ( and saved ), but no change.

pb66 · 2018-03-31T13:37:27.634Z

bart.ellast:

The faulty feed does indeed show data up until 23h this evening!

Is that static or is it moving along as time passes?

Is it exactly 23:00? is it Belgium time? What time is it in GMT or unixtime?

pb66 · 2018-03-31T13:57:55.484Z

A post was merged into an existing topic: Convert phpfiwa to phpfina

bart.ellast · 2018-03-31T14:46:11.135Z

It’s static, doesn’t move beyond 23h35. It’s Belgium time indeed.you mean the 23h35 in GMT?

pb66 · 2018-03-31T14:48:45.093Z

If you look at the last value and time for the 2 feeds in question

http://etherm.ddns.net:32500/feed/timevalue.json?id=11

http://etherm.ddns.net:32500/feed/timevalue.json?id=260

you can see they are identical so I suspect it can only be 1 of 2 things, either it’s a display only issue, ie good data with the correct time is being saved but displayed incorrectly, but I fail to see how that would only affect some feeds and not others on the same multigraph OR there was an error that resulted in some data being written in the future, and now the data arriving is correct, but it cannot overwrite what is already there, but in this scenario I wonder why the last value and time would be reported “correctly” perhaps it returns the last “received” rather than the last “on file”.

Ok so I think it’s definitely the latter having looked closer

http://etherm.ddns.net:32500/feed/data.json?id=12&start=1522497600000&end=1522584000000&interval=10

That url returns 10s data from the “faulty feed” from 12:00 (midday) GMT 31/03/18 to 12:00 (midday) GMT 01/04/18.

The last data point in that returned data is [1522532130000,65] which is a timestamp of 21:35:30 31/03/18 and it doesn’t change.

So something went wrong to cause the data to be posted with the wrong timestamp and normal service will most likely resume after that final datapoint has been passed with new data at 21:35:40 GMT.

pb66 · 2018-03-31T14:58:28.184Z

bart.ellast:

It’s Belgium time indeed.you mean the 23h35 in GMT?

Belgium is 1 hour ahead of the UK so, your current time is 2hrs ahead of GMT. The actual timestamp of the end of the data you see is 1522532130 (or 1522532130000 unixmillis) which is 21:35:30 31/03/18 GMT so 23:35:30 31/03/18 localtime in Belgium

bart.ellast · 2018-03-31T15:07:23.380Z

Ok, I’ll wait and see what happens this evening.
Thank you a lot for the help Paul!

pb66 · 2018-03-31T15:18:31.676Z

No Problem, although that doesn’t tell you how it happened and therefore offers no reassurance that it won’t reoccur.

bart.ellast · 2018-04-01T12:05:33.839Z

Paul,

Everything switched back to normal at 23h30 last night. You were right!
You can see the old and new feed run the same data now.

I do think now all wrong data came from a recently updated set of emoncms nodes.