Community
OpenEnergyMonitor

OpenEnergyMonitor Community

Python script to import Octopus Agile consumption data into emoncms

Edit 10 Oct This is now in Usefulscripts follow this post 30 to use Python script to import Octopus Agile consumption data into emoncms.

After a long wait, Im now finally on Octopus Agile :slight_smile: I know @borpin has a nice tool for importing smart meter consumption data from the Octopus API here Import Octopus consumption data but as I dont use node-red, I thought I’d put together a small python script to do the same.

This script can be ran daily from cron to pull in the latest data automatically. The script requests new data since the last valid reading:

import sys,requests, json
from datetime import datetime

# Emoncms server configuration
emoncms_apikey = "APIKEY"
feedid = FEEDID
server = "http://emonpi.local"

# Agile API configuration
mpan = MPAN
serial_number = ""
agile_apikey = ""

# Agile request parameters
params = {'page':1,'order_by':'period','page_size':25000}

# Step 1: Create feed via API call or use input interface in emoncms to create manually
# http://IP-ADDRESS/feed/create.json?tag=agile&name=consumption&datatype=1&engine=5&options={"interval":"1800"}&unit=kWh

# Step 2: Fetch feed meta data to find last data point time and value
result = requests.get(server+"/feed/getmeta.json",params={'id':feedid,'apikey':emoncms_apikey})
meta = json.loads(result.text)
print("Feed meta data:\t\t"+result.text)

if meta['npoints']>0:
    end_time = meta['start_time'] + (meta['interval'] * meta['npoints'])
    params['period_from'] = datetime.fromtimestamp(end_time).astimezone().isoformat()
    print("Request from:\t\t"+params['period_from'])

# Step 3: Request history from Octopus
url = "https://api.octopus.energy/v1/electricity-meter-points/%s/meters/%s/consumption/" % (mpan,serial_number)
result = requests.get(url,params=params,auth=(agile_apikey,''))
data = json.loads(result.text)

if not data: sys.exit(0)
if not 'results' in data: sys.exit(0)

dp_received = len(data['results'])
print("Number of data points:\t%s" % dp_received);

# Step 4: Process history into data array for emoncms
data_out = []
for dp in data['results']:
    time = int(datetime.timestamp(datetime.strptime(dp['interval_start'],"%Y-%m-%dT%H:%M:%S%z")))
    value = dp['consumption']
    print(str(time)+" "+str(value))
    data_out.append([time,value])

# Step 5: Send data to emoncms
if len(data_out):
    print("Posting data to emoncms")
    result = requests.post(server+"/feed/insert.json",params={'id':feedid,'apikey':emoncms_apikey},data={'data':json.dumps(data_out)})

Example crontab entry to run at 7am every morning:

0 7 * * * /usr/bin/python3 /home/pi/agile.py > /dev/null 2>&1

The script uses a new bulk feed data upload API currently available in the emoncms master branch, see: New Emoncms feed data bulk upload API.

My Meters are being done on Monday.

Looks good!

1 Like

The flaw here @trystan, is it will not load historical data whereas my NR node did/does.

I think because the first call to emoncms uses a hardcoded node name, which if it doesn’t exist, emoncms creates. This then creates the data from the earliest record (max 25000 IIRC).

You could fix that, by not creating the Feed first and just looking for all data first time so getting the oldest data.

Again IIRC by default the newest record is returned first so you need to request it in reverse.

I’ve tested it quite a few times and it does work :slight_smile: it loads the full history, note order_by paramter is ‘period’ which means its ordered by oldest data first.

1 Like

Ah Ok, as long as no data is inserted when first created the start is not set - makes sense!

1 Like

Yes that’s it, it assumes an empty feed to start with, Id like to build this into the emoncms agile app somehow so that you dont have to create the feed and setup this script, I need to think through the best approach.

@TrystanLea

Thx for this script and for moving it to stable.

The site I monitor moved to Agile in early July and I have all the necessary configuration credentials so I decided to try it – running the script in an SSH terminal on an RPi running the latest emon ver 10.2.6.
The Rpi named test-RPi is a fresh install and is running on my home network and accessed from my Windows laptop.

This is what I found …

server = “http://192.168.0.32” worked as did “http://127.0.0.1

I could not create the agile feed by running http://192.168.0.32/feed/create.json?~~~etc in the script (gave a syntax error) and so I created the agile feed by putting the API in a browser bar - per screen shot

Then, I got:

[email protected]:~ $  python octopus_consumption_perTL.py
Feed meta data:         {"success":false,"message":"Feed does not exist"}
Traceback (most recent call last):
  File "octopus_consumption_perTL.py", line 32, in <module>
    if meta['npoints']>0:
KeyError: 'npoints'
[email protected]:~ $

As well a the KeyError, this also looks ominous … “message”:“Feed does not exist”

In my script I had feedid = “agile” (as I had already created it via a browser)

Then I commented out the if meta[‘npoints’]>0: stanza as it was a first time run and got the following:

[email protected]:~ $  python octopus_consumption_perTL.py
Feed meta data:         {"success":false,"message":"Feed does not exist"}
Number of data points:  2640
Traceback (most recent call last):
  File "octopus_consumption_perTL.py", line 51, in <module>
    time = int(datetime.timestamp(datetime.strptime(dp['interval_start'],"%Y-%m-%dT%H:%M:%S%z")))
AttributeError: type object 'datetime.datetime' has no attribute 'timestamp'
[email protected]:~$

2640 looks correct as the Agile tariff started in early July.

But I’d now reached the limit of my skills. And time in Linux/Python is very confusing. One link I stumbled across had a concluding footnote “If, that is, we ignore the effects of Relativity”.

My python is Python 2.7.16

Your suggestions would be most welcome.
Thx

Note the crontab is Python3. The script should really have a #! as well. Suspect it needs python3.

Python 2.7 went EOL in Jan which means no updates I believe. You need to migrate off it as it will soon go from the repos - not installed by default on Raspberry OS I believe.

Hello @johnbanks

Yes that was intended to be ran in the browser, great!

You need to enter the emoncms feedid, it will be a number, hover your mouse over the feed name and it should show it.

I think you do need the latest python3 for this to work. Can you run the script on a Pi with a recent emonSD image installed?

1 Like

@TrystanLea

Yes - I finally figured out what was needed for feedid.

And with @borpins’s steer to Python3, I got the script to run OK. It also worked 24 hrs later just doing an incremental update.

Octopus sends kWh data to 2 decimal places which emoncms rounds (or truncates?) and so I’ve multiplied the value line by 1000 and show the units as Wh’s – this preserves all the Octopus data.

Because of the time I spent, I’ve added comments in my copy of the script:


# Script by Trystan Lea ref: https://community.openenergymonitor.org/t/python-script-to-import-octopus-agile-consumption-data-into-emoncms/15230

# Requires Python 3

# To run the script in an SSH terminal do: /usr/bin/python3 /home/pi/script-name.py (octopus_consumption_perTL.py)

# To run the script as a daily cronjob do: exactly the same

# Unlike other 'devices' which provide INPUT data as it occurs at n secs intervals, Octopus is providing a FEED of historical 30mins consumption data

# In consequence, it is necessary to FIRST create an emoncms FEED with a chosen tag, name & interval. The ID NUMBER for this FEED must then be entered in the script below at feedid =

These may/may not be of help to others.

Thx for yr work on this.
Yr new feed/insert.json functionality is a great addition which I think may have many uses.

That shouldnt be the case, mine have all imported to 3 decimal places (watt hour resolution). Are you looking at the values via the graph module? There’s a decimal place option to show up to 3 dp in the graph UI.

Great thanks!

never noticed that!

@TrystanLea

Indeed there is - just found it - one lives & learns …

It’s in the Feeds in View box under the graph ‘hidden’ under a column heading DP - pls give us oldies a break - there’s enough space there for a column heading Dec pts or something similar.

But banter to one side - Octopus appears to send the data at 10 Wh resolution - so more trailing zero’s is just spurious accuracy.

So it’s down to user choice (add *1000 to one script line). And personally I favour Wh’s have just invested heavily in PowerWalls and am hoping the Octopus numbers will be low.

:slight_smile:

thanks tristan, another user of your script, just got it working, plus discovered your app as well, thanks again

1 Like

Hi @TrystanLea

I think I have found an issue with this where you have a solar feed.

My first attempt did not show Agile Import Actual data.

Settings:-

Second test attempt leaving out solar feed works as expected.

settings:-

j

Thanks @ian yes correct, it does not yet support showing the smart meter data in Solar PV mode. Do your agile consumption half hourly values otherwise show both import and export values? If you look at them outside the app…

Its not a problem to me, it just confused me first time I tried it. The real value to me was the ability to get correction factors and I can run the app without solar to get those.
I have all the data outside the app in legacy dashboards but the app is much slicker and what I tend to look at.

@TrystanLea, I now have some Agile data - only taken 6 months to get meters etc!

Just playing with the script.

  1. Is it possible to auto create the feed (on first run) then check to see if the Feed exists on each run rather than having to create the feed manually?
  2. One disadvantage of this method (over my Node-Red version) is it doesn’t create a cumulative feed. Can that be done via post process?
  3. How do I compare a measured feed with this data?

I think this script should go into the useful scripts repository with an install script to make it a bit more accessible to others.

Is this now in stable?

1 Like

Would love to try this but not a clue how to add Node Red to my my EmonPi looked at the instructions but have no idea how to get to the prompt to type in the command to load it. Would it be possible to put a quick idiots guide to putting all of this great functionality onto a EmonPi latest software version.

John

For this you do not need to install Node Red, it just uses Python3.

How comfortable are you with SSH and the command line?