Emoncms Demand Shaper module

johnbanks · 30 May 2021 12:21

Here’s the script that polls the Tesla EV …

# PURPOSE: To poll a Tesla EV periodically to discover whether it is online and if so - what is it's battery % charge, battery range & odometer reading

# ACKNOWLEDGEMENTS: This script is built around a core: carson 1.0.2 https://pypi.org/project/carson/ by Michael Loyd who in turn acknowledges Tim Dorr and dozens of contributors to his tesla-api project

# INITIAL REQUIREMENTS ...
# Do   sudo pip3 install carson    # NOTE  sudo
# Do   sudo pip3 install aiohttp   # NOTE  sudo

# The script will create an INPUT node to receive the data

# Copy this script file to  /home/pi  and make it executable with:  chmod +x /home/pi/tesla_JB+carson.py  # using the correct script name

# Run the script with: /usr/bin/python3 /home/pi/tesla_JB+carson.py  # using the correct script name

# All being well, data will appear in the emoncms webpage Inputs refreshing every 30 secs

# Create FEEDS using Log to Feed and add UNITS (pencil drop-down) to each

# IMPORTANT NOTE ...
# Data from this script is input to emoncms via an http API.  This bypasses emonhub ...
# The EmonHubEmoncmsHTTPinterfacer is not required ...

# IMPORTANT NOTE ...
# FEB 2021 - Tesla changed the process for authentications - script changes have been made - labelled FEB 2021

# FINALLY ONCE THE SCRIPT RUNS OK: Create the tesla-EV.service 
# A SERVICE (tesla-EV.service) should be set up as described at the end/below

# MOST IMPORTANT NOTE ...
# If emon is updating but EV asleep ... This may happen when the script is first run ... Wake up the EV by using the Mobile App

# REF:  https://pypi.org/project/carson/

# ===============================================================================================

import asyncio, json, datetime, time, requests, aiohttp
from time import sleep
from carson import Session 

# Start of User inputs section ===============


# This section commented in FEB 2021 ... Tesla changed the process for authentication

# Tesla EV data 

ev_name = 'XXXXXXXXXXXXXX'

# Location of current access token ...
token_file = "/home/pi/tesla-scripts/access_token.txt"

# Emoncms server configuration - Node 15
emoncms_apikey = "XXXXXXXXXXXXXXXXXXXXX"    # For Node 15 - adjust as appropriate
emoncms_server = "http://127.0.0.1"
node = "Tesla EV"  # Name of the Input(tag) that has ALREADY BEEN CREATED to receive the INPUT data
read_interval = 30   # Polling interval in secs	

# Get latest Access Token from file ... added FEB 2021
f = open(token_file, "r")
acc_tok = f.read()

# Name each of the data inputs associated with the newly created emoncms Input(tag)
di1 = "ONLINE is 1 and  ASLEEP is 0"                  
di2 = "Battery Charge percent"      
di3 = "Battery Range miles"
di4 = "Odometer miles" 

# End of User inputs section ===============


async def main():

	# Initialise ...
	prior_data2 = 1
	prior_data3 = 1
	prior_data4 = 1
	starttime = time.time()
		
	# Request Tesla EV data at user specified read_interval
	while True: 
	
		# async with Session(email, password) as session:      # JB change FEB 2021
		async with Session(access_token=acc_tok) as session:
			car = await session.vehicles(ev_name)
			
			state_string = str(car.state)
			print(state_string)   # ######

			# Is the EV ONLINE?		
			if state_string == "online":
				state = 1
			else:
				state = 0
				
#			print(state)   # ########

			# If ONLINE - update values but if NOT then use prior values 		

			if state is 1:
				json_response = await car.vehicle_data()
				jsonstr = json.dumps(json_response, default=str)
				data = json.loads(jsonstr)
#				print(data['response']['charge_state']['battery_level'])   # #####
#				print(data['response']['charge_state']['battery_range'])    # ####
#				print(data['response']['vehicle_state']['odometer'])    # ####
			
				data2 = data['response']['charge_state']['battery_level']
				prior_data2 = data2
				data3 = data['response']['charge_state']['battery_range']
				prior_data3 = data3
				data4 = data['response']['vehicle_state']['odometer']
				prior_data4 = data4
				

			# Send data to emoncms (Note - prior values = current values if ONLINE)
			
			data1 = state

			dev_data = {di1: data1, di2: prior_data2, di3: prior_data3, di4: prior_data4}

			data = {
				'node': node,
				'data': json.dumps (dev_data),
				'apikey': emoncms_apikey
			}

			response = requests.post(emoncms_server+"/input/post", data=data)
			

		# Sleep until time to next poll the Tesla EV
		
		time.sleep(read_interval - ((time.time() - starttime) % read_interval))

asyncio.run(main())

# FINALLY ONCE THE SCRIPT RUNS OK: Create the tesla-EV.service as follows:
# Do: CTRL-C to stop the script then - Do: sudo nano /etc/systemd/system/tesla-EV.service  and copy & paste in the following (using the correct script name) ...
# NOTE User=root ... this is a Carson requirement

"""

[Unit]
Description=Tesla EV Charge Status
After=network.target
After=mosquitto.service
StartLimitIntervalSec=0

[Service]
Type=simple
Restart=always
RestartSec=1
User=root
ExecStart=/usr/bin/python3 /home/pi/tesla_JB+carson.py

[Install]
WantedBy=multi-user.target

"""
# Then save & exit and to ensure the tesla-EV.service runs on boot up - Do:  sudo systemctl start tesla-EV.service  AND THEN  sudo systemctl enable tesla-EV.service

# AS A VERY LAST CHECK - Do: sudo reboot then SSH in again and check the service is active with:  systemctl status tesla-EV.service

# Finally close the SSH terminal. The script/service will continue to run surviving any future reboots

Here’s the Input Processing for the EV soc …

Here’s the Input Processing for the openevse soc that was created …

Here’s the Demand Shaper settings that are used …

Finally here’s the script that is run every evening at 11pm to set up Demand Shaper …

# PURPOSE ...
# To be run as a cronjob prior to midnight

# Regardless of whether the EV is connected - to switch openEVSE from DISABLED state to WAITING state
# And to set up the Demand Shaper parameters

# This includes adjusting the parameters which control the Demand Shaper SMART schedule timing 
# And this is achieved WITHOUT needing to access the Demand Shaper webpage - hence can be incorporated into a cron run script

# Facility available from emon ver 10.2.7 and later
# Ref: https://community.openenergymonitor.org/t/emoncms-demand-shaper-module/9097/170

# NOTE - Select Battery Charge Level (input)in the 'spanner dropdown' section of the Demand Shaper webpage (Tesla EV)
# The option (set-device-settings api) is used when current_soc is input manually

# ---------------------------------------------------------------

import requests, json, os, time


# User info ...
# SEASON VARIABLE PARAMETER ... target_soc
# Dependent on the Season and the need to leave 'space' in the EV battery for next days ANTICIPATED charge from excess solar gen
# Max is 0.9 - recognises the limit set to conserve EV battery life
# WINTER season = 0.9    There's close to zero excess solar gen because PowerWall captures what there is

# The primary objective is to MINIMISE EXPORT - less than full EV charge maybe a 'price to pay' - subject to a precautionary minimum limit
# Reliable predictions of solar gen are NOT feasible
# ALSO - House & ASHP loads are variable
# ALSO - PowerWall takes first call on any solar gen in excess of House & ASHP loads but only until it is fully charged
# The primary objective is to MINIMISE EXPORT - less than full EV charge maybe a 'price to pay' - subject to a precautionary minimum limit
# The above does assume that the EV is connected during the day in the MID and Pool seasons

# Select the value of the parameter target_soc from one of the following ...
# Then insert in line 101 below ...

# 0.9 for Winter season = Nov > Feb
# 0.7 for MID season = mid Sep > Oct and Mar > Apr
# 0.5 for Pool season = May > mid Sep


# emoncms server configuration ...
emoncms_apikey = "XXXXXXXXXXXXXXXXXXXX"   # For Node 15 - use appropriate key
emoncms_server = "http://127.0.0.1"

# PART 1 ###############################################
# Set up openEVSE ...
# Switch Divert Mode (> Normal Fast then > ECO mode) in order to change openEVSE from DISABLED to the WAITING State - THIS MAY NOT BE CORRECT !!!!!
# Note - openEVSE will have been set to DISABLED by another script earlier in the evening ...

os.system("mosquitto_pub -h localhost -r -u emonpi -P emonpimqtt2016 -t emon/openevse/divertmode/set -m 1")    # Note:  -r  to retain message
time.sleep(10)
os.system("mosquitto_pub -h localhost -r -u emonpi -P emonpimqtt2016 -t emon/openevse/divertmode/set -m 2")    # Note:  -r  to retain message


# PART 2 ###############################################

# Get the Demand Shaper key MODES ...

# Demand Shaper MODES are ...
# divert_mode = 1 = ECO mode ON
# divert_mode = 2 = ECO mode OFF
# Other modes: 'smart'  'off'  'on'  'timer'

params = (
    ('device', 'openevse'),
    ('apikey', emoncms_apikey)
)

reply = requests.get(emoncms_server+"/demandshaper/get", params=params)
jsonstr = reply.text.rstrip()
ds_data = json.loads(jsonstr)

print(reply.status_code)    # Debugging only

# Set the Demand Shaper key MODES ...
# divert_mode = 1 = ECO mode ON
# divert_mode = 2 = ECO mode OFF
# Other modes are: 'smart'  'off'  'on'  'timer'

ds_data['schedule']['settings']['ctrlmode'] = 'smart'
ds_data['schedule']['settings']['on_completion'] = 'smart'
ds_data['schedule']['settings']['divert_mode'] = 1         # ECO mode ON

# Now save the changed key MODES ...
reply = requests.post(emoncms_server+"/demandshaper/save", params=params, data={'schedule':json.dumps(ds_data['schedule'])})

print(reply.status_code) 


# PART 3 ###############################################

# Adjust the parameters which control the Demand Shaper SMART schedule timing 

params = (     
    ('apikey', emoncms_apikey),
    ('device', 'openevse'),
# Insert target_soc value in the following line ...	
    ('settings', '{"target_soc":0.5, "battery_capacity":75, "charge_rate":6.9}'),   # XXXX  Adjust values as appropriate ...
)

reply = requests.get(emoncms_server+"/demandshaper/set-device-settings", params=params)

print(reply.text)           # Debugging only
print(reply.status_code)   


# FOR VALIDATION ONLY - print the complete & final Demand Shaper json ...

params = (
    ('device', 'openevse'),
    ('apikey', emoncms_apikey)
)

reply = requests.get(emoncms_server+"/demandshaper/get", params=params)
jsonstr = reply.text.rstrip()
ds_data = json.loads(jsonstr)

print (ds_data)        
print(reply.status_code)

Formatted to remove large bold font. For future reference, when posting code or output, please put 3 ‘backticks’ (normally found at the top left of the keyboard) on a line of their own before the code, and 3 more backticks also on a line of their own after the code:

```
code
```

If it is something like php you can add a language identifier after the first 3 backticks: ```php or even ```text if you don’t want any language markup applied.

Moderator (RW)

johnbanks · 30 May 2021 12:27

@TrystanLea

Apologies - I really don’t know why the scripts in my previous post have large bold fonts where the original script had commented lines.
I copied & pasted using Notepad++

Bill.Thomson · 31 May 2021 18:53

That happens because “# text”
(that’s the # character, a space, then some text)
at the beginning of a line, is a signal to Discourse (the forum software) the text should be
large and bold.

e.g. # text
as the first charcters in the line, produces the effect,

text

but the same string anywhere else in the line doesn’t. e.g. # text
Note if the space isn’t included, even for text at the beginning of a line, e.g.
#text
the text isn’t affected.

johnbanks · 31 May 2021 19:50

@Bill.Thomson
Thanks for that clear explanation.

johnbanks · 3 June 2021 21:12

@TrystanLea

I’d like to report another bizarre SMART charging event on 3 June.

The Tesla EV started charging at 4:50am and stopped at 7:00am. And these times are by co-incidence? exactly the same charging times as those for the ‘rogue’ charge I reported a couple of days ago.

My Demand Shaper script (see post above) had run at 11pm and set the target_soc to 0.5

This screenshot shows what next happened …

In the period before the charge:

openevse:soc FEED was 44%

Tesla EV SOC was 1% (the default when the vehicle is asleep)

So it was correct/as intended for a charge to start.

However it did not stop when the target 50% was reached – it continued to 65% which begs the additional question – why did it stop at 65% and not continue beyond that?

Any suggestions?

Thx

TrystanLea · 9 June 2021 15:32

Hello @johnbanks

Sorry for the delay. I think I follow the logic of how you are getting the stage of charge into emoncms, I wonder if you could move the step where you check if the soc is less than 4% to the script that polls the tesla EV?

Something like:

async def main():

  # Initialise ...
  dev_data = {}
  
  starttime = time.time()
    
  # Request Tesla EV data at user specified read_interval
  while True: 
  
    # async with Session(email, password) as session:      # JB change FEB 2021
    async with Session(access_token=acc_tok) as session:
      car = await session.vehicles(ev_name)
      
      state_string = str(car.state)
      print(state_string)   # ######

      # If ONLINE - update values but if NOT then use prior values     
      if state_string == "online":
        state = 1
      else:
        state = 0


      if state_string == "online":        
        json_response = await car.vehicle_data()
        jsonstr = json.dumps(json_response, default=str)
        data = json.loads(jsonstr)
      
        if data['response']['charge_state']['battery_level']>4.0:
            dev_data['battery_level'] = data['response']['charge_state']['battery_level']
            
        dev_data['battery_range'] = data['response']['charge_state']['battery_range']
        dev_data['odometer'] = data['response']['vehicle_state']['odometer']

      dev_data['state'] = state
      
      # Send data to emoncms
      response = requests.post(emoncms_server+"/input/post", {
        'node': node,
        'data': dev_data,
        'apikey': emoncms_apikey
      })
      

    # Sleep until time to next poll the Tesla EV
    time.sleep(read_interval - ((time.time() - starttime) % read_interval))

asyncio.run(main())

I havent tested the above, not sure if all the changes are valid.

Interesting, can you see if there is anything in the demandshaper log, it can be found in /var/log/emoncms/demandshaper.log

johnbanks · 10 June 2021 23:10

@TrystanLea
Thx for the time you’ve spent on this.

I’ve tried splicing yr code suggestions into my original script but got into such an indent mess that I abandoned that approach.

But it has caused me to look at my code more closely. When the EV is sleeping, the polled EV soc info is unreliable. So it is necessary to use a prior valid value but I don’t think I’ve handled that correctly. And I take yr suggestion to do that in the script rather than in Input Processing.

I will work more on this and let you know if I still have the problem – namely, SMART charging not happening when it should (EV actual soc being below target soc).

My second problem was that a SMART charge was correctly started but it continued beyond the target EV soc.

You suggested looking at /var/log/emoncms/demandshaper.log herewith:


pi@emonpi-node-15:/var/log/emoncms $ cat demandshaper.log

2021-06-06 07:01:00.969 | openevse schedule complete

2021-06-06 07:01:00.970 | openevse next ctrlmode: smart

2021-06-06 07:01:00.974 | Recalculating EVSE schedule based on emoncms current soc input: 1

2021-06-06 07:01:00.977 | EVSE timeleft: 0

2021-06-07 07:02:00.554 | openevse schedule complete

2021-06-07 07:02:00.555 | openevse next ctrlmode: smart

2021-06-07 07:02:00.558 | Recalculating EVSE schedule based on emoncms current soc input: 0.95

2021-06-07 07:02:00.560 | EVSE timeleft: 0

2021-06-08 07:01:00.429 | openevse schedule complete

2021-06-08 07:01:00.434 | openevse next ctrlmode: smart

2021-06-08 07:01:00.437 | Recalculating EVSE schedule based on emoncms current soc input: 0.74

2021-06-08 07:01:00.439 | EVSE timeleft: 0

2021-06-09 07:01:00.189 | openevse schedule complete

2021-06-09 07:01:00.189 | openevse next ctrlmode: smart

2021-06-09 07:01:00.192 | Recalculating EVSE schedule based on emoncms current soc input: 0.73

2021-06-09 07:01:00.194 | EVSE timeleft: 0

2021-06-10 00:01:10.957 | emon/openevse switch off

2021-06-10 00:01:15.843 | emon/openevse set timer 00 31 04 00

2021-06-10 07:01:00.718 | openevse schedule complete

2021-06-10 07:01:00.718 | openevse next ctrlmode: smart

2021-06-10 07:01:00.721 | Recalculating EVSE schedule based on emoncms current soc input: 0.17

2021-06-10 07:01:00.723 | EVSE timeleft: 12913.043478261

2021-06-10 07:02:00.126 | emon/openevse set timer 00 24 04 00

2021-06-10 20:15:13.728 | emon/openevse set timer 00 00 03 35

pi@emonpi-node-15:/var/log/emoncms $

The log does not go back far enough to see the problem in the early hours of 3 June.

However, I am intrigued that something seems to happen at 07:01/07:02 each day. I do not have a cronjob that runs at this time.

It may be a co-incidence but both the problem charges that I’ve reported stopped at that time.

Is something happening in emomcms at that time?

Thx

johnbanks · 16 June 2021 22:53

@TrystanLea

Another bizarre SMART charge occurred in the early hours of 15 June – see screenshot:

openevse soc was 55% at 12:00am when the charge started and it stopped at 3:36am when the EV charge reached 89%

All this despite my script which had run just before midnight setting target_soc to 0.5. So the charge should not have started.

This time I’ve captured some log file detail.

The first is for /var/log/emoncms/demandshaper.log

pi@emonpi-node-15:/var/log/emoncms $ cat demandshaper.log
2021-06-13 07:01:00.434 | openevse schedule complete
2021-06-13 07:01:00.434 | openevse next ctrlmode: smart
2021-06-13 07:01:00.456 | Recalculating EVSE schedule based on emoncms current soc input: 0.72
2021-06-13 07:01:00.458 | EVSE timeleft: 0
2021-06-14 07:01:00.073 | openevse schedule complete
2021-06-14 07:01:00.074 | openevse next ctrlmode: smart
2021-06-14 07:01:00.082 | Recalculating EVSE schedule based on emoncms current soc input: 0.59
2021-06-14 07:01:00.084 | EVSE timeleft: 0
2021-06-14 14:58:00.970 | emon/openevse divert mode 2
2021-06-15 07:01:00.152 | openevse schedule complete
2021-06-15 07:01:00.153 | openevse next ctrlmode: smart
2021-06-15 07:01:00.157 | Recalculating EVSE schedule based on emoncms current soc input: 0.89
2021-06-15 07:01:00.160 | EVSE timeleft: 0
2021-06-16 07:01:00.061 | openevse schedule complete
2021-06-16 07:01:00.061 | openevse next ctrlmode: smart
2021-06-16 07:01:00.064 | Recalculating EVSE schedule based on emoncms current soc input: 0.87
2021-06-16 07:01:00.066 | EVSE timeleft: 0
pi@emonpi-node-15:

There are no Errors or Warnings but it seems strange that on successive days, openevse schedule complete is shown each day at 7:01am. Several of the bizarre SMART charges I’ve reported earlier in this thread completed at this time – a coincidence?

The second log file detail is for /var/log/emoncms/emoncms.log

pi@emonpi-node-15:/var/log/emoncms $ cat  emoncms.log
2021-06-15 07:01:00.287|ERROR|demandshaper_model.php|Saved to disk
2021-06-16 00:14:45.748|WARN|PHPTimeSeries.php|post_bulk_prepare() data in past, nothing saved.  feedid=42 timestamp=1623802185 last=1623802185 value=87
2021-06-16 00:14:45.867|WARN|PHPTimeSeries.php|post_bulk_prepare() data in past, nothing saved.  feedid=62 timestamp=1623802185 last=1623802185 value=3
2021-06-16 00:34:46.507|WARN|PHPTimeSeries.php|post_bulk_prepare() data in past, nothing saved.  feedid=62 timestamp=1623803386 last=1623803386 value=3
2021-06-16 00:39:46.595|WARN|PHPTimeSeries.php|post_bulk_prepare() data in past, nothing saved.  feedid=42 timestamp=1623803686 last=1623803686 value=87
2021-06-16 00:39:46.629|WARN|PHPTimeSeries.php|post_bulk_prepare() data in past, nothing saved.  feedid=62 timestamp=1623803686 last=1623803686 value=3
2021-06-16 00:44:46.730|WARN|PHPTimeSeries.php|post_bulk_prepare() data in past, nothing saved.  feedid=42 timestamp=1623803986 last=1623803986 value=87
2021-06-16 00:44:46.762|WARN|PHPTimeSeries.php|post_bulk_prepare() data in past, nothing saved.  feedid=62 timestamp=1623803986 last=1623803986 value=3
2021-06-16 00:49:46.848|WARN|PHPTimeSeries.php|post_bulk_prepare() data in past, nothing saved.  feedid=42 timestamp=1623804286 last=1623804286 value=87
2021-06-16 01:19:47.596|WARN|PHPTimeSeries.php|post_bulk_prepare() data in past, nothing saved.  feedid=42 timestamp=1623806087 last=1623806087 value=87
2021-06-16 01:19:47.626|WARN|PHPTimeSeries.php|post_bulk_prepare() data in past, nothing saved.  feedid=62 timestamp=1623806087 last=1623806087 value=3
2021-06-16 07:01:00.233|ERROR|demandshaper_model.php|Saved to disk
pi@emonpi-node-15:

There’s an ERROR at the ‘magic’ time of 7:01am which I do not understand.

And this ERROR repeats on 16 June also at 7:01am.

Re the WARNINGS – Feed 42 is the openevse soc (Feed 62 is quite unrelated).

Your suggestions would be most welcome.

TrystanLea · 18 June 2021 10:41

Hello @johnbanks

Do you have the schedule set to complete by 7am?

Yes that should be unrelated

Do you have any of the demandshaper log from midnight to 4am for that period?

johnbanks · 18 June 2021 12:33

@TrystanLea

I have never consciously set the schedule to finish at any time assuming that is the job/purpose of Demand Shaper.

Could I have done it unwittingly? How can I check if a schedule finish time is set?

In my post above there is the complete demandshaper.log for the period 13 to 16 June.

jpalo · 9 November 2021 14:18

The demandshaper background job demandshaper_run.php doesn’t seem to be able to reschedule Nordpool Spot portion correctly, but instead it always sets the schedule in OpenEVSE from 4 am to 8 am. What is curious is that this 4am to 8am matches with the Run period and Complete by settings?!

If I go to the web page, it correctly calculates the schedule and sets it on OpenEVSE, but the next day it again reverts to 4-8 am.

Any idea where to start investigating this? Is it the scheduler2.php that is responsible for this?

UPDATE:
The core reason why this happens is somehow related to the fact that there isn’t always 24h price data available for the future, but in the morning for example, you see that there is no data available for the following night. This leads the scheduler to assume prices after midnight are 0, so it always selects the last 4 hours, so from 4 to 8. Somewhere in program logic we should not allow these “empty” hours to be included in the calculation.

This hasn’t always been like this, the previous code did work correctly and this issue only started after the “big” refactoring that introduced multiple sources, etc.

UPDATE2: PR submitted at Fix Nordpool Spot scheduling by jpalo · Pull Request #65 · emoncms/demandshaper · GitHub

MyForest · 13 January 2022 03:13

I’ve had a look around and I don’t see an example of someone using Demand Shaper to get the hot water heated up whilst the power is cheap.

I’m on Octopus Go and so I have a very simple tariff to consider. Even so, I’m still using a fixed-price in my calcs so I’m not seeing the right numbers here (it should be 12p, not 35p).

Anyway, here’s me bumping up my desired DHW temp which causes my heat pump algorithm to think it’s worth nudging the DHW up. That means the tank is warmer in the morning so we get a shorter DHW heat up during the day. Maybe if the sun ever shines again it’ll even do it from the solar.

Trystan, did I miss something where you have your Ecodan hooked up to Demand Shaper for DHW?

Of course I’m also pondering the lower efficiency of doing it in the cold night time rather than the warmer day time. And yes, mine is way less efficient than yours because my flow temp ramps up like it’s on a mission to create lava.

mjr · 13 January 2022 09:54

No “eco” setting for hot water on your FTC?

I am on economy7 (so 0-7h is about two-thirds price of the rest) and currently think optimal water heating is ideally overnight, earlier in the night before minimum temperature is reached, but late enough that everyone is sound asleep so the pumps (more the water one inside than the heat pump outside) don’t disturb them, and late enough that the water tank doesn’t lose too much heat before morning. It is a balancing act.

MyForest · 13 January 2022 15:45

I’m using Eco mode but it’s still way too enthusiastic

You’re right about the balancing act.