Time shift problem publishing SolCast forecast in JSON

Dear All
I wrote a python script, executing each day at midnight to grab the SolCast PV power forecast for the next two days and publishing it to emonpi with MQTT.
Solcast gives this forecast format in JSON for the next two days each 30 minutes.
Of course the timestamp of the SolCast JSON is in UTC, with the Z at the end.
Actually with the summer time, these forecasts are shifted 2 hours viewing them in Emoncms runnign locally on the Raspberry.


On this graph the time is clearly Local Time, i.e. CET when observing the PV production.
I do not understand why the forecast is shifted + 2 hours, because if the JSON timestamp is taken as CET rather than UTC, it should be 2 hours ahead no ?
As I parse the Solcast JSON already because SolCast allows only two roofs and I have three so I compute the three roofs forecast from one with sine/cosine equations, I can also modify the timestamp to align the forecast to the production.
But before doing this kind of patch it is always better to understand the root cause, no ?
Thanks for your support !
Regards
Christophe

For info the raw Solcast JSON is like this (for now 19h28 CET)
{“forecasts”:[{“pv_estimate”:0.1323,“pv_estimate10”:0.0831,“pv_estimate90”:0.1723,“period_end”:“2024-07-31T17:30:00.0000000Z”,“period”:“PT30M”},{“pv_estimate”:0.1062,“pv_estimate10”:0.0735,“pv_estimate90”:0.1212,“period_end”:“2024-07-31T18:00:00.0000000Z”,“period”:“PT30M”},{“pv_estimate”:0.03,“pv_estimate10”:0.0172,“pv_estimate90”:0.0486,“period_end”:“2024-07-31T18:30:00.0000000Z”,“period”:“PT30M”},{“pv_estimate”:0.0089,“pv_estimate10”:0.0045,“pv_estimate90”:0.0118,“period_end”:“2024-07-31T19:00:00.0000000Z”,“period”:“PT30M”}, …

And before publishing to mqtt I already remove the “period”:“PT30M” as well as some zeroes on the time stamp, and rename the “period_end” as “time”

And the emonpi datetime is
image

Correct to the local time here.

Beware, Python treats a timestamp with a Z in it as local time not UTC.

My script simply push the JSON to emon through MQTT after recomputing the power. I do not touch the timestamp.
And if it is handled as local time, the forecast would be 2 hours before the production, not after, no?
Shall I replace the Z by +00:00 ?

So the data is saying 17:30Z when your local time is 19:30 i.e. UTC+2. emoncms (python) is interpreting the time 17:30Z as 17:30 local i.e. 2 hours later (so when it is 19:30 local, it thinks it is only 17:30 local).

Yes try and replace the Z with 00:00 and I expect it to work correctly.

However, that is not guaranteed!

Hi Brian
Thanks for your support.
I changed the Z by +00:00 and no change
Peak solar forecast still at 15h instead of 13h.
In github it seems there is only php files in emoncms… Where can I find the python that handles the mqtt json with timestamp to have a look?
Regards
Christophe

If you look at the data you are passing in, are you sure the peak is at the right time?

Three more things

  1. How are you preprocessing the data?
  2. How are you sending the data for emoncms? A single JSON structure?
  3. What does the emoncms Log say is received?

I might be misremembering how this is processed - might be the PHP processor :frowning:

@TrystanLea

  1. How are you preprocessing the data?

I investigate this point because as Solcast provides free forecast for only two roofs and I have three I have registered an horizontal roof of 1kWh peak, and then I recompute the power for the three roofs using tilt and azimuth with the sun position, which is computed using the time… so there is place also for some aleas here. Next forecast at midnight will also log the factor computed with the sun position so we will be sure there is no errors here.

  1. How are you sending the data for emoncms? A single JSON structure?

Yes a single structure with 48 elements for a one day forecast

  1. What does the emoncms Log say is received?

log says nothing, it is stucked in WARN level despite I set it to INFO in example-settings.ini in /var/www/emoncms
Did I miss something on this debug level settings ?

OK I got it.
Problem was effectively in my sun position computation.
seems that datetime.fromisoformat treat the time as local. I added +00:00 on it too and now it is OK


And I also managed to find the settings.ini to change the log level to INFO.
In fact I send the data one JSON per timestamp one by one. The MQTT buffer seems to handle this correctly.
emoncms_log.txt (5.0 KB)
Thanks for your support.

Example settings are just that. Use the settings.ini file to add any overrides to the defaults settings. (I think is says that at the top of the file).

Yep.

If you read my other post it is because the fromiso function is the inverse of the toiso function. As the toiso can never generate a timestamp with Z in it, it then cannot convert it back.

That is why you should never use Z in an API but +00:00!

1 Like