Having trouble with dates and times in data export (is the options dialogue in BST while the backend in UTC?)

SteffanCook · 2020-04-20T10:08:34.950Z

Hi
I do a lot of data exporting from emoncms (both online and locally).

A recent problem is that if I ask data to start at midnight, I actually get data from 11pm the previous day. I think that the front-end export is using BST, while the backend is still on GMT/UTC?

Can you add a check box (“Tick to use GMT.UTC”) to make the front end specify local time or UTC time, please.

emonPi is working awesome otherwise
Many Thanks.

borpin · 2020-04-20T11:05:10.040Z

Oh there are big discussions on GitHub about timezones

When you say midnight, do you mean midnight local time? How do you specify the time in the request?

Have you set your timezone in the user profile?

Internally, emonCMS uses UNIX time - i.e. number of seconds since the epoch.

borpin · 2020-04-20T12:24:27.974Z

2 posts were split to a new topic: EVESE and Demandshaper time issue. Timezone?

SteffanCook · 2020-04-20T12:24:57.203Z

Hi,

Thanks for the reply.
My timezone is UTC.

Even when I ask to export the data in Unix time, it still gives me an export file that starts one hour earlier. For instance 00:00 on Thursday morning gives me a file that starts at 23:00 Wed night.

As I mentioned it feels like the front end time/date chooser is using BST but the data is in UTC.

Thanks

borpin · 2020-04-20T12:36:02.287Z

borpin:

How do you specify the time in the request?

Can you answer this? There are many interfaces to export data through.

SteffanCook:

For instance 00:00 on Thursday morning gives me a file that starts at 23:00 Wed night.

00:00+01:00 (BST) is the same as 23:00+00:00 (UTC) - it is the same number of seconds since the epoch.

SteffanCook · 2020-04-20T14:14:18.868Z

Thanks for the reply. This is how I usually do it.

Screenshot 2020-04-20 at 15.09.521236×795 90.2 KB

Screenshot 2020-04-20 at 15.09.331236×795 62.4 KB

SteffanCook · 2020-04-20T14:17:56.806Z

borpin · 2020-04-20T14:18:02.948Z

As there is no TZ here, I would expect it to use the user timezone.

SteffanCook:

My timezone is UTC.

Which is why it is doing exactly what you ask it to .

Set your user timezone correctly (just click on it - not very clear you can do that) and you might get what you expect.

SteffanCook · 2020-04-20T14:21:06.278Z

Hi,
My timezone is UTC, and I set the clock on the interface to start from 2am. But it gives me data from 1am UTC?

borpin · 2020-04-20T14:53:08.588Z

SteffanCook:

My timezone is UTC

Your profile says Cardiff so the local time (TZ) to you is UTC+1

It looks as if the export is using the browser TZ. Why not set the user TZ correctly? If you do, what do you get?

This just demonstrates the problem with any system when you display times without a TZ. The string is meaningless without it.

github.com/emoncms/emoncms

Consider emoncms using user account TZ rather than browser TZ when viewing data and dashboards etc

opened 02:18PM - 02 Oct 18 UTC

closed 11:12AM - 06 Apr 23 UTC

pb66

enhancement

See https://community.openenergymonitor.org/t/timezone-in-emoncms/7569?u=pb66 … In short (for example) when we look at a solar plot for any given day it should start to increase AM and tail off PM as a rule. it should not look like the sun rose at 6pm and set again at 6am and peaked at midnight just because it is UK data viewed from Australia. In most cases the data is related to the user account TZ and not the TZ that the data is being viewed from at any given time. If users in 5 different TZ's view the same data that relates to a single location at the same time, it should appear the same to all users, not be TZ adjusted for each user.

nchaveiro · 2020-04-20T15:45:55.538Z

What timezone is on your user profile?

SteffanCook · 2020-04-20T15:57:40.203Z

Screenshot 2020-04-20 at 16.56.57

SteffanCook · 2020-04-20T16:00:24.862Z

I am still reading the threads but I agree that an indication of the time zone in question is needed. Perhaps time/date data export could be three columns ‘EpochTime/TZ time/Feed’. At least this way the EpochTime is always attached.

nchaveiro · 2020-04-20T16:12:20.963Z

Everything is working as expected.

UI is in your computer TZ. eg. BST (UTC+1)
Your profile is in UTC TZ.

1- Export converted to YMDhms comes from profile TZ hence you select in UI TZ and have export data in profile TZ.
2 - Export with no conversion comes always in UTC seconds + offset on the UI. This offset field is automatically filled with the seconds difference for you profile TZ to UTC.

UTC is not a physical location, clocks do not change in UTC.
Why is your profile in UTC?

SteffanCook · 2020-04-20T16:37:09.487Z

'cos UTC sounds cool

I see what you are saying that the browser javascript time setting and my profile on the server time setting are out of sync. Hence one asks for data at 2am and the other correctly sends data from 1am. Nothing wrong there, except the human in the middle. I changed my profile now to London time.

Could be worth adding a local TZ time setting to the dialog box for the Feed.Download dialog box also just like you did for the graphs (although that’s more work ;-(.

Thanks again for the explanation.

borpin · 2020-04-20T19:16:43.109Z

nchaveiro:

Export converted to YMDhms comes from profile TZ hence you select in UI TZ and have export data in profile TZ.

That isn’t obvious to a user of the UI and there is no means of selecting if the export is converted or not.

As I said, the export should indicate it is UTC by including the timezone as per ISO 8601. Without it, it is deemed to be local time.

SteffanCook:

Could be worth adding a local TZ time setting to the dialog box for the Feed.

Unfortunately, the date/time the feed is collected at is related to the user timezone not the sensor generating the data. You may find there is an hours gap in your data, depending on how it is collected.