Daily totals reset at 10am

I am having troubles with my daily kwh totals resetting at 10am

I think this is time zone related
emoncms is running on pi desktop
the OS is set to sync with my time ntp tine server the the time zone Australia/Brisbane (UTC +10)

From the admin screen

OS Linux 4.19.0-9-amd64
Host emonpi | emonpi | (172.31.222.222)
Date 2020-06-23 17:46:27 AEST

I notice the SQL section shows

MYSQL
Version 5.5.5-10.3.22-MariaDB-0+deb10u1
Host localhost:6379 (127.0.0.1)
Date 2020-06-23 17:46:26 (UTC 10:00‌​)

My meter data us fed from estron SDM meters via MQTT
kwh readings are lifetime readings

Is this happening because the data us translated to UTC and is then reset at midnight UTC = 10am?

I assume if I set emoncms and the OS to UTC and feed the local time I will get the kwh to kwhd resetting at midnight. is this the only work around or am I missing something?

image

First thing to know is that all data is stored with a unix timestamp which is equal to UTC (as that is what a timestamp is). So no matter where you are in the world, right now(), is the same unix timestamp.

Have you set the timezone under your user profile?

I managed to track it down last night and it was the php settings

the install instructions have you set the php timezone.
This guide assumes php 5 and the current version is now 7.3

For 7.3, you need to set the timezone in
/etc/php/7.3/apahe2/php.ini and
/etc/php/7.3/cli/php.ini

then restart apache

This seems to have resolved the issue as the feeds reset at midnight local time last night

Did you try and set the timezone for your user?

yes it was always set to local time zone however i cant see how this would ever be related.
You may have users with many timezones but the feed would reset at the same time

Will confirm tomorrow after it resets correctly a second time but as I indicated php 7.3 has two php.ini files that need the local timezone set.

Because internally emoncms looks for that. You can edit it to suit your timezone. It isn’t very obvious you can edit it, but place the mouse over it and a pen icon appears

What you have done may have unexpected consequences.

Thanks Brian,
My user account was always set to the local time zone (Australia/Brisbane)

Just trying to understand and work though what you are saying.

Lets assume I setup multiuser and

User1 has timezone UTC and
User2 has Timezone UTC +10

Are you saying
User1 would see the kwh/d feeds reset the total at midnight and
User2 would see the kwh/d feeds reset the total at 10am

Even though the server OS, SQL and PHP is UTC+10

I would expect this behaviour if the OS, SQL and PHP is UTC not UTC+10

With OS, SQL and PHP as UTC+10
I would expect to see the feed reset at UTC +10.
In other words 2pm for User1 and midnight for user2

What am I missing?

Ok good. There have been issues before over this problem and you may have hit upon the solution that needs to be fixed as part of the install process @TrystanLea.

There is a big discussion on GitHub over timezones. Basically, the system was designed for a single timezone (UK) and user. Outside of that it has some issues.

Conceptually I favour storing the timezone of the Feed in the meta data but others have different views.

The other thing is when should the feed reset? Is it midnight local time at the location of data collection (i.e. where the sensor is), or midnight for the viewer? This relates to my concept above!

For your users, are they looking at the same feed?

That’s exactly my take on the problem, for what it’s worth: the timezone (and don’t forget daylight saving time offsets - and these can change year on year so the history as well) where the use happens needs to be stored in the database, and that implies it’s ‘per feed’.

The browser or whatever is used to view and analyse the data needs to be able to do that using any reasonable timezone - the obvious three are local to the data, local to the viewer, or UTC. A free choice (enter a number ±hh:mm) would be a nice 4th option.

What is the purpose of the reset? If it’s to synchronise with the supplier’s billing, it needs to be the same as that. (e.g. Some old UK “Economy7” billing periods don’t change with daylight saving time.) Otherwise, it’s likely to be midnight local to the usage, but for a shift worker, it could be 6 am local. :bomb: [I can’t find the :minefield: emoticon.]

No you just need the timezone as in Europe/London so the system can look up the offset.

Completely OT, but I am going to be interested in what happens when the EU stops using DST. What will happen to historical data? Will they introduce a new set of timezones?

So you know what has been used within any given period, usually a ‘day’. e.g. the My Electric app

image

and a daily graph

image

Not reset as in a hardware reset just when one accumulation stops and another starts.

No, it’s not OT at all - that’s exactly why you need to store the local timezone and offset as the data is gathered. There’s no guarantee the look-up will work in the future. There’s no guarantee that the user will apply the changing time offsets anyway, but that’s their option.
It’s not as if it’s a massive amount of data: date & time of change and the name of the new offset and its value.

So what is the definition of a “day” - that’s what I was getting at. A shift worker on permanent nights might regard his day as starting at 6 or 7 am.

Do you have a link please?

It’ll take you to a search result.

There have been other discussions here IIRC.

Personally I think it should be sunset or sunrise.
This shifts over the year but for me is most relevant to solar production and billing

We are net metered here in Australia.
However the meter records import and export not net import.
We pay roughly 3 times more for imported power vs exported power
So the name of the game here is maximising self consumption not exports

You also pay a daily fee for connection to the grid this means I need to export just under 10kwh to offset the daily fee.

The only way to negate the night time cost of imported power is to use battery.
So for me it all comes back to sunrise/sunset

But isn’t that the same as midnight? Any time between sunset and sunrise is equivalent from a solar generation point of view. Midnight is a nice fixed time and is shared with lots of other definitions of the change of day.

Not really when combined with billing.

I have 3 Phase main power plus a single phase off peak.
Off Peak is on a separate meter, fed from L1 (before main meter) and switched on using a ripple control relay.

Generally off peak begins around 10pm and ends around 4pm the next day.
Time may shift at the will of the energy supplier but is for a minimum of 18 hours per 24

With midnight solar production is then split between 2 cycles of off peak.
Sunset/Sunrise aligns solar with off peak.

Aligning with sunset/sunrise simplifies controlling imports/exports and battery.

IE Has my exported power covered the cost of peak and off peak imports.
Does the battery need to be topped up from off peak and is so is it better to run the battery lower or pay for the off peak

I have the first one done processing data externally, working on the second one.

You seem to have a very unusual billing arrangement - I’ve never heard of anything like it before. Certainly I wouldn’t say that it represented the mainstream when considering how end of day should be defined. In any case sunset can be before 4pm where I live, so would still lead to the same problem if I had the same billing as you.

In short, I think your argument is very specific to your particular situation, and is not a good argument for a general solution.

Does not seem that unusual to have off peak power
Take a look at the “Time of use - flexible CL” app

In any case the original issue was solved by editing both ini files
Kwh/D in EmonCMS now resets at midnight as expected
I am getting Kwh/D starting at sunset externally and feeding that to Domoticz which handles load control and the Selectronic Inverter/Charger
All the data from all sources ends up in influxdb and graphing is done in Grafana

Indeed no, I have it myself. But mine and all other cases I’ve heard of are pretty low tech - mine has fixed hours and doesn’t even change for daylight saving. The fixed hours vary around the country a bit to minimise the switch-on surge I suppose.

It’s only with modern digital metering that we’re beginning to see flexible pricing, of which the best known here is Octopus Agile. Based on the wholesale half-hour interval pricing used in commercial supply.

But I’m glad you’ve got your system sorted out.

Rather off-topic, I came across this post in another forum. It raises some very interesting points I for one would never have even thought about:

Yet
 In a Western framework, we have very limited understanding of calendar issues in a global framework. Date.ToString() may be far from sufficient. I was working with some Koreans: If you asked them about their age, they first had to convert their lunar calendar to a sun calendar, and then correct for their convention of giving your age as the year of your living: At birth, you are starting your first year, not your zeroth.

Another case: When I was working with web archiving, there were native Americans who wanted access control to their archived web pages to be restricted by their concepts of season: Some documents should be accessible during the seeding season only, others only during the harvesting season. (They also wanted some documents to be available to males only, others to females only - but that is not calendar related.)

Some Asian cultures restart their year count from some astronomical or social event, indicating the base as part of their year indication - similar to our BC/AC, but on a lot more dynamic scale.

And so on. Date.ToString() does not cut it. MS has done a lot to support various calendars, but you can’t take for granted that it is supported in all applications. (In those I devlop, there is certainly not support for all sorts of Asian calendars!)

So there may be a market for the library that the TS (or some “coworker/friend” of his) has created - but mostly in Asian markets. Western software developers generally ignore such issues, taking for granted that the only calendar adaptations required is adapting to the time zone and 24/12 hour clocks.

Lots of Western software have no chance of succeeding in Asia, because the developers have no understanding of Asian culture and the requiremnts that the software should satisfy.

[Source: The Lounge - CodeProject]