Emoncms Timezone and daily feed issue

I am sending data to the EmonPi at 01:00 pm. For example Days Running count which I send every morning at 01.00.

The problem is that it gets recorded on EmonPi as the day before at “Wed, 11 JAN 2023 19:00”.

And to record this I am using PHPFINA with fixed interval of 1d.

To diagnose this I am logging all “Post to emoncms” out the serial port and into a computer running Termite where I can turn on time stamps.

And I see last nights update timestamped at Jan 12 01:15:06 which is correct.

But the last entry I see in the graph is Wed, 11 Jan 2023 19:00. So, I believe I am sending the end of day stuff correctly at 01:00 but EmonPi time stamps it as the day before at 19:00.

Now I verified the Pi has the correct time and I have setup Emoncms > Setup > My Account > Timezone to my timezone of Toronto, Ontario, Canada.

Meanwhile, the same ESP is sending out Watts every 60 seconds and on EmonPi I see the latest entry of Thu, 12 Jan 2023 19:22 which is up to date time and date.

What am I missing?

From where and how (MQTT or HTTP) are you sending this data? Is this timestapmed data?

That could well be your problem as the interval (1d) starts/ends relative to when you created the feed. At the end of the interval, it looks for the last (I think) data on the Input and records that to the Feed. @TrystanLea am I correct in that?

Thank you borpin, yes sorry, I am using HTTP and it is not time stamped.

And you second comment might explain my problem as I have a number of ESP’s and some of them seem to record the time it was sent.

Hopefully Tystan will reply.

Now if you are correct than how do I send it so that the time it is sent is recorded.

Do I change the interval from once per day to every hour?

Or should I use Variable Interval Time Series?

I still have this problem.

I have confirmed through the serial port that I am sending the end of day data at 1:00am.

But EmonPi records it as the day before at 19:00.

Can anyone help me?

Are you sure that’s correct? I’ve just looked at a kWh/d feed and the Feed start time is Wed Jan 04 2023 00:00:00 GMT+0000 - too much of a coincidence, surely, for that to be the creation time? I know other Feeds increment their time slot on the exact multiple of 10 s, 30 s, minute etc by machine time, so it seems reasonable to think the “slot” into which the data drops starts and ends at midnight, machine time. And further evidence, I’ve just created one at 22:55 local and it shows no Feed start time.

[Edit: At 00:04 on Wednesday 18th, it shows the Feed start time as 00:00:00 17th January - Tuesday]

It amounts to the same thing, but as PHP (the language used) is stateless, I prefer to think of a Feed (the database) as boxes on a conveyor, in this case the box is a day long and the back of one box and the front of the next pass under the input at exactly midnight. Any data arriving before midnight is handled immediately by the Input and dropped onto the conveyor, to be overlaid by the next. Only the last data to arrive and drop into the box before midnight, when the box moves on, remains visible.

Graphing an old kWh/D feed of mine shows data points at exactly 12:00 am (Midnight - it looks as if midnight counts as the next day, so data posted at midnight is the previous day’s data, which is correct). If your data for Monday is sent at 01:00 local time on Tuesday, then it will appear on the bar chart as Tuesday’s data. (Or, that’s what my bar chart on a dashboard does, with the data coming from a Power to kWh/d process.)

I can’t help thinking that, despite your checks @MrGadget, it is a timezone/timesetting discrepancy.

But what’s puzzling me about this is, if it’s using London time/UTC it should be the next day, not the day before, because I’m 5 hours ahead of you (posting this at around 23:30 local on Tues, 17th, you should see this post timed at 18:30 on Tues, 17th).

No, I’m not sure :slight_smile: (which is why I said I think).

However, when I mouse over my Feed names to get the metadata, the start times are pretty random.

image

I don’t actually have any kW/h day feeds and that may be the system creating & treating them in a special way (which seems reasonable).

However, on that basis, perhaps the creation of a 1d interval feed should be handled the same way (allowing for TZ differences).

[edit]
@MrGadget the documentation for the databases can be found here

@TrystanLea could you comment on this, please?

The test to establish the behaviour is relatively simple:
Create a 1hr feed well away from the round hour, check immediately to see if it shows a creation time, then check again at a few minutes past the hour.
Do the same with a 30 min feed, check a few minutes past the half-hour.
Repeat for each available option.

I suspect you’ll find the creation times are the round hour before, the round half-hour before, etc.
The 5 s feed could be a little tricky to prove. :grinning:

Hello, sorry for my delay, have been working a lot lately.

And I am a little confused reading these responses. I am not sure if borpin has the same problem and asking the same question?

Anyway I thought I would set the time to midnight (instead of 1:00) as Robert suggested.

So I am sending data out the serial port of the ESP32 and capturing it on a terminal program. Below is my boot sequence where the time is initially from 1970, then I set the time from a time server and print the time again which you see in the “after sync” and it is the correct time when it booted last night.

Clock before sync: Thu Jan  1 00:00:00 1970
Connecting to WiFi.
2023/01/17 19:40:39: ...
2023/01/17 19:40:40: .
2023/01/17 19:40:41: ..
2023/01/17 19:40:42: ...
2023/01/17 19:40:44: ..
2023/01/17 19:40:45: .
Clock after Sync: Tue Jan 17 19:41:55 2023
Connected to IP Address: 192.168.11.30

Next is the end of day event captured below. It is supposed to be posted at midnight and the ESP32 time confirms this as well the last line is the terminal time stamp confirming it happened at midnight.

####################### ENTERING END OF DAY EVENT AT: Wed Jan 18 00:26:04 2023
GET  /emoncms/input/post.json?apikey=c403e27b63cbdf3e17d311d0e4614104&node=1_HydroMeter2&json={DaysRunning:1,yesterday_kWhrs:8457,yesterdayMinWatts:1638,yesterdayMaxWatts:1893,MsgCounter:271,RSSI:-53
Post to emoncms : success
%%%%%%%%%%%%%%%%%%%%%1	1638	1893
2023/01/18 00:25:55:

Next I am now ssh’ed in to verify the EmonPi (which is a Pi3) time and date which is shown below and is correct:

dataplicity@emonpi:/$ date
Wed 18 Jan 19:04:41 EST 2023
dataplicity@emonpi:/$ ^C
dataplicity@emonpi:/$ 

Finally I am showing the time zone is correct:


And after reading the above post I am now highlighting this feed which I think is showing that it was received at midnight.
FeedTime
But when I look at the graph it shows yesterday at 19:00: Oh, please ignore the fact it is showing a zillion entries. That is another problem I have posted elsewhere and I can only handle one problem at a time.

Interesting - we have had this issue before, but I thought we had solved it. Timezones!

You insert data at 00:00 local (i.e. UTC-5), but the graph says 19:00 (UTC-5). In the CSV output, what does that say for the data point in seconds since epoch? It should match 00:00 local. If so the data is right, it is how it is displayed.

However, the Feed metadata says start time of 00:00 UTC. This doesn’t work for a daily feed (in any timezone actually) I suspect as the start/end of the day can never be right.

Try using an hourly feed but only insert the data once. For yesterday’s data, I think it has to be done as 23:59 local else it will end up in today’s time slot.

I’ll edit the title to be more representative of the issue.

This really is one for @TrystanLea

I have copied the CSV data below. Now I do not know why it plots it a bunch of times. I have proven to myself I only send it once. This is another problem I mentioned earlier that I still need to figure out.

Now about what you said on start time of 00:00 UTC I had changed it to 01:00 UTC because for some reason which I now forget (this has been a problem for a long time and much time was spent for example in the beginning I assumed it was my ESP board) and EmonPi still showed it on the graph as 19:00 from the day before.

So setting it to either 00:00 local or 01:00 local both result in a timestamp of 19:00?

Anyway, I will try 23:59 local and set EmonPi for hourly feed.

Some more info which may help. I am running this on a Pi3.
And, I work at a college and have a setup here running on a Pi4. And I have this same issue at the college.

1673910900, null
1673911800, null
1673912700, null
1673913600, 1
1673914500, 1
1673915400, 1
1673916300, 1
1673917200, 1
1673918100, 1
1673919000, 1
1673919900, 1
1673920800, 1
1673921700, 1
1673922600, 1
1673923500, 1
1673924400, 1
1673925300, 1
1673926200, 1
1673927100, 1
1673928000, 1
1673928900, 1
1673929800, 1
1673930700, 1
1673931600, 1
1673932500, 1
1673933400, 1
1673934300, 1
1673935200, 1
1673936100, 1
1673937000, 1
1673937900, 1
1673938800, 1
1673939700, 1
1673940600, 1
1673941500, 1
1673942400, 1
1673943300, 1
1673944200, 1
1673945100, 1
1673946000, 1
1673946900, 1
1673947800, 1
1673948700, 1
1673949600, 1
1673950500, 1
1673951400, 1
1673952300, 1
1673953200, 1
1673954100, 1
1673955000, 1
1673955900, 1
1673956800, 1
1673957700, 1
1673958600, 1
1673959500, 1
1673960400, 1
1673961300, 1
1673962200, 1
1673963100, 1
1673964000, 1
1673964900, 1
1673965800, 1
1673966700, 1
1673967600, 1
1673968500, 1
1673969400, 1
1673970300, 1
1673971200, 1
1673972100, 1
1673973000, 1
1673973900, 1
1673974800, 1
1673975700, 1
1673976600, 1
1673977500, 1
1673978400, 1
1673979300, 1
1673980200, 1
1673981100, 1
1673982000, 1
1673982900, 1
1673983800, 1
1673984700, 1
1673985600, 1
1673986500, 1
1673987400, 1
1673988300, 1
1673989200, 1
1673990100, 1
1673991000, 1
1673991900, 1
1673992800, 1
1673993700, 1
1673994600, 1
1673995500, 1
1673996400, 1
1673997300, 1
1673998200, 1
1673999100, 1
1674000000, 1
1674000900, null
1674001800, null
1674002700, null

On the graph options, have you selected limit to data interval. and untick fix

image

This limits the plots to the single point of data you have. The Histogram option might work as well for daily data.

I think this is because 19:00 is UTC-5 which is your local timezone. i.e. the start of the timeseries window.

Did you read the database documentation link above to see how Timeseries works?

[edit]
Because it is timeseries, it doesn’t matter when you drop the data into the window, it still is recorded as being at the start of the window.

On the graph options, have you selected limit to data interval . and untick fix

Yes, I have always had it set up as you show above but see many data points.

I think this is because 19:00 is UTC-5 which is your local timezone. i.e. the start of the timeseries window.

  1. I see what you are saying but when I set it to 01:00 UTC-5 why wouldn’t it plot as 20:00 UTC?

  2. Why is it plotting UTC and how do I get it to plot UTC-5.

Did you read the database documentation link above to see how Timeseries works?

[edit]
Because it is timeseries, it doesn’t matter when you drop the data into the window, it >still is recorded as being at the start of the window.

I have in the past but I did again after you suggested it. Now I think I understand how it samples data within a window. Actually, for a long time I thought it averaged my readings within each windows.

Now an update, I did make the changes last night but this morning I do not see any update so I do not know what went wrong. I did test it a few times last night by setting it for five minutes in the future and it worked but for some reason it did not work. This has been a common theme for me.

I hope this is not taken wrong but there have been and still are a number of things EmonPi does and some of it is me and some of it I have not figure out yet (like this issue).

My point is that I can’t seem to build much confidence in EmonCMS and it feels like it is not yet mature enough to be used. I moved over to this from years of using https://blynk.io/. I am now perhaps a year into EmonCMS and I still don’t have something that will just run.

Am I wrong? Because I am approaching this assuming it works and therefore if it doesn’t work it must be me? But, if I know it is still a work in progress then I think it will help me and I can decide if it is the right tool for me.

Please don’t get me wrong, I love what it can do, especially the pre and post processing (which blynk does not do) and I really want to make it work but I am getting frustrated at how little progress I have made for the time I have spent.

I really hope I haven’t ghosted anyone and ideally I would like it to be my problem but I just can’t figure it out.

Is someone willing to help me?

Hello @MrGadget I think the issue may be relating the combination of the use of PHPFina with daily data. Can you try PHPTimeSeries for this? PHPTimeSeries is designed for use with daily data where you can have summer time etc and the interval does not always align to 86400 seconds.

1 Like

I tried this last night on one of my inputs and today it shows the correct time. And the multiple plotting that I mentioned above is gone.

So that sounds very promising, thank you @TrystanLea.

I plan to change my other devices with end of day to the PHPTimeSeries.

Thank you for every ones help.

2 Likes