Treating of time in Emoncms - UTC or BST

Emoncms
1

I am importing Octopus Agile data into EMONCMS into a PHPTimeseries feed directly. I have setup a dummy input and then created the feed from it. ( I am bulk loading data in the future so I can see the price changes during the day )

I am using -

http://xxx.xxx.xxx.xxx/emoncms/feed/insert.json?id=44&time=" + timevalue[i] + "&value=" + costvalue[i] + "&apikey=....

timevalue is unixtime and UTC.
importing via a python script using requests.
It loads directly into the feed fine.

The csv output from the feed ( via the graph) compared with the raw input from Agile shows ( a sample ) -

CSV Output from feed			raw output from Agile		
2020-06-03 14:00:00	4.9		2020-06-03T14:00:00Z 	4.893	
2020-06-03 14:30:00	5.3		2020-06-03T14:30:00Z 	5.292	
2020-06-03 15:00:00	17.5		2020-06-03T15:00:00Z 	17.472	16:00 in Agile app
2020-06-03 15:30:00	18.2		2020-06-03T15:30:00Z 	18.228	
2020-06-03 16:00:00	18.7		2020-06-03T16:00:00Z 	18.69	
2020-06-03 16:30:00	20.1		2020-06-03T16:30:00Z 	20.055	
2020-06-03 17:00:00	20.1		2020-06-03T17:00:00Z 	20.097	
2020-06-03 17:30:00	20.1		2020-06-03T17:30:00Z 	20.097	
2020-06-03 18:00:00	7.3		2020-06-03T18:00:00Z 	7.2765	
2020-06-03 18:30:00	6.8		2020-06-03T18:30:00Z 	6.8355

The graph display is showing in UTC zone but the Agile App is showing in BST zone.

When I import via the input ( every 30 minutes in the past) Emoncms appears to convert to BST - so is this happening in the logic between an input and a feed?

Any advice on what I should do - should I convert prior to loading into the feed? I want to input values in the future not in the past hence the input field does not really work for me.

2

All data in emoncms is stored with a UTC timestamp.

All data should therefore be imported as UTC (string or unix timestamp) or UTC equivalent time string i.e. a timestring with a timezone. i.e. 2020-06-03T07:03:30+00:00 === 2020-06-03T08:03:30+01:00

emoncms will then show you that data in a graph in your browser timezone.

There are a number of long discussions both here and on GitHub around how time is treated by emoncms.

For this problem, are you using the time string on the feed URL or are you converting it to a timestamp? Try converting it to a timestamp in the python code, try seconds first (there is something about emoncms sometimes expecting milliseconds).

Are there any errors in the log?

3

First of all it is good to hear that emoncms displays in browser timezone. In my normal day job we always use UTC as the base and then convert on output to cope with multi-timezone sensors/data input etc etc

The Agile feed as you can see does not contain accuracy down to a millisecond ( it is 00 all the time) so I converted it into a unixtime and removed the .00 - casted it to an int.

Tonight when the next set of data comes in I will leave the .00 on and see what difference it makes.
This is the conversion that I am doing

dtthis = datetime.datetime(int(results['valid_from'][0:4]),  int(results['valid_from'][5:7]), int(results['valid_from'][8:10]),int(results['valid_from'][11:13]),int(results['valid_from'][14:16]))  

Utimethis = time.mktime(dtthis.timetuple())

timevalue.append(str(int(Utimethis)))

results[‘valid_from’] is 2020-06-02T21:30:00Z the timevalue inside emoncms is 1591131600
note : there are a few lines of code in between the statements.

I can extend it by adding the secs column in - its always 0.

There are no errors in the log files

4

The time is still in seconds - you would need to multiply by 1000 to get the value as miliseconds (as opposed to a precision of miliseconds) - I’d leave it for now.

Using this site, that timestring converts to 1591133400.

The beauty of timeseries is you can simply run it again & again as it will simply overwrite the time slots.

On the graph, if you select view as timestamp (in the CSV output) what does it say?

5

Why not use strptime?

datetime.datetime.strptime("2007-03-04T21:08:12Z", "%Y-%m-%dT%H:%M:%SZ")

(How do I translate an ISO 8601 datetime string into a Python datetime object? - Stack Overflow)

or even fromisoformat?

https://docs.python.org/3.7/library/datetime.html#datetime.datetime.fromisoformat

Both are untested :grinning:.

6

I have re-looked at the code tried your suggestions but it made no difference.

However I believe I may have solved it and will check back later and let you know.

To answer your question the values I showed came from the CSV output.

I have been developing using the WSL system ( Ubuntu bash shell running under Windows 10).

When I go onto emonpi and enter the command date it returns with : Wed 3 Jun 16:36:37 UTC 2020
When I enter the command date on the WSL it returns with : Wed Jun 3 17:37:13 BST 2020
note the difference in time being the time taken for me to move from one environment to the other.

When I load the data as described directly into the feed from the emonpi the time looks like it is now properly adjusted for BST when you run the graph. So the fact that one shell is in BST and the other in UTC seems to have made a difference.

I will check this when the Agile App produces its figures sometime later today.

If this is correct it is interesting how it effects the load as I would not have expected this.

Just one last item - you can’t not reload the data in same time slots as the error logs says it has later entries and so blocks it.

7

Yes I can confirm this is the same as the Agile App output

8

If you are correctly generating a unix timestamp it makes no difference. Those are both the same times.

To prove it type

date '+%s'

into both. The difference will be the number of seconds it took you to switch windows.

I still think the way you are creating the timestamp is the source of your error. Use the builtin tools for the job. YMMV.

9

As you say the date ‘+%s’ gives the same result bar key presses for all as you would expect

Time zone for POP_OS is BST
Timezone for WSL is BST
Timezone for emonpi is UTC

other results:
POP_OS on laptop
Python 2.7 : 2020-06-03T22:00:00Z’, ‘,’, ‘1591218000’, ‘,’, 1591218000.0
Python 3 : 2020-06-03T22:00:00Z , 1591218000 , 1591218000.0

WSL on Windows 10
Python 3.8 : 2020-06-03T22:00:00Z , 1591218000 , 1591218000.0

emonpi
Python 2.7 : 2020-06-03T22:00:00Z’, ‘,’, ‘1591221600’, ‘,’, 1591221600.0

The emonpi produces the right result in the application.

If I change the POP_OS timezone to UTC I get the same as the emonpi.

So the conclusion is that python ignores the UTC derivation in the time and applies the local timezone and then produces the unixtime. Something I will need to look into.

The first column time is :

dtthis = datetime.datetime(int(results['valid_from'][0:4]),  int(results['valid_from'][5:7]), int(results['valid_from'][8:10]),int(results['valid_from'][11:13]),int(results['valid_from'][14:16]))    

    Utimethis = time.mktime(dtthis.timetuple())

The second column is -

difftimevalue.append(datetime.datetime.strptime(results['valid_from'], "%Y-%m-%dT%H:%M:%SZ"))
time.mktime(difftimevalue[i].timetuple())

What are the built in tools for the job?
My aim is to load Agile date from 22:00 today to 21:30 tomorrow in bulk at 18:00 today. These are in 30 minute steps.

10

Not processing Z might still be a bug is a feature Issue 35829: datetime: parse "Z" timezone suffix in fromisoformat() - Python tracker. You need to test it yourself. Potentially simply replace the Z for +00:00 in the string first.

[edit]

datetime.fromisoformat(my_date.replace('Z', '+00:00'))

[edit Nov 2023]
It occurs to me this will create erroneous results if DST is in effect. I think the Z should just be removed so the time is assumed to be local time when converted.

11

Thanks explains it and will try it out. I am new to python my primary area is C++,php and golang

1 Like