Community
OpenEnergyMonitor

Community

Python script to import Octopus Agile consumption data into emoncms

After a long wait, Im now finally on Octopus Agile :slight_smile: I know @borpin has a nice tool for importing smart meter consumption data from the Octopus API here Import Octopus consumption data but as I dont use node-red, I thought I’d put together a small python script to do the same.

This script can be ran daily from cron to pull in the latest data automatically. The script requests new data since the last valid reading:

import sys,requests, json
from datetime import datetime

# Emoncms server configuration
emoncms_apikey = "APIKEY"
feedid = FEEDID
server = "http://emonpi.local"

# Agile API configuration
mpan = MPAN
serial_number = ""
agile_apikey = ""

# Agile request parameters
params = {'page':1,'order_by':'period','page_size':25000}

# Step 1: Create feed via API call or use input interface in emoncms to create manually
# http://IP-ADDRESS/feed/create.json?tag=agile&name=consumption&datatype=1&engine=5&options={"interval":"1800"}&unit=kWh

# Step 2: Fetch feed meta data to find last data point time and value
result = requests.get(server+"/feed/getmeta.json",params={'id':feedid,'apikey':emoncms_apikey})
meta = json.loads(result.text)
print("Feed meta data:\t\t"+result.text)

if meta['npoints']>0:
    end_time = meta['start_time'] + (meta['interval'] * meta['npoints'])
    params['period_from'] = datetime.fromtimestamp(end_time).astimezone().isoformat()
    print("Request from:\t\t"+params['period_from'])

# Step 3: Request history from Octopus
url = "https://api.octopus.energy/v1/electricity-meter-points/%s/meters/%s/consumption/" % (mpan,serial_number)
result = requests.get(url,params=params,auth=(agile_apikey,''))
data = json.loads(result.text)

if not data: sys.exit(0)
if not 'results' in data: sys.exit(0)

dp_received = len(data['results'])
print("Number of data points:\t%s" % dp_received);

# Step 4: Process history into data array for emoncms
data_out = []
for dp in data['results']:
    time = int(datetime.timestamp(datetime.strptime(dp['interval_start'],"%Y-%m-%dT%H:%M:%S%z")))
    value = dp['consumption']
    print(str(time)+" "+str(value))
    data_out.append([time,value])

# Step 5: Send data to emoncms
if len(data_out):
    print("Posting data to emoncms")
    result = requests.post(server+"/feed/insert.json",params={'id':feedid,'apikey':emoncms_apikey},data={'data':json.dumps(data_out)})

Example crontab entry to run at 7am every morning:

0 7 * * * /usr/bin/python3 /home/pi/agile.py > /dev/null 2>&1

The script uses a new bulk feed data upload API currently available in the emoncms master branch, see: New Emoncms feed data bulk upload API.

My Meters are being done on Monday.

Looks good!

1 Like

The flaw here @trystan, is it will not load historical data whereas my NR node did/does.

I think because the first call to emoncms uses a hardcoded node name, which if it doesn’t exist, emoncms creates. This then creates the data from the earliest record (max 25000 IIRC).

You could fix that, by not creating the Feed first and just looking for all data first time so getting the oldest data.

Again IIRC by default the newest record is returned first so you need to request it in reverse.

I’ve tested it quite a few times and it does work :slight_smile: it loads the full history, note order_by paramter is ‘period’ which means its ordered by oldest data first.

1 Like

Ah Ok, as long as no data is inserted when first created the start is not set - makes sense!

1 Like

Yes that’s it, it assumes an empty feed to start with, Id like to build this into the emoncms agile app somehow so that you dont have to create the feed and setup this script, I need to think through the best approach.

@TrystanLea

Thx for this script and for moving it to stable.

The site I monitor moved to Agile in early July and I have all the necessary configuration credentials so I decided to try it – running the script in an SSH terminal on an RPi running the latest emon ver 10.2.6.
The Rpi named test-RPi is a fresh install and is running on my home network and accessed from my Windows laptop.

This is what I found …

server = “http://192.168.0.32” worked as did “http://127.0.0.1

I could not create the agile feed by running http://192.168.0.32/feed/create.json?~~~etc in the script (gave a syntax error) and so I created the agile feed by putting the API in a browser bar - per screen shot

Then, I got:

[email protected]:~ $  python octopus_consumption_perTL.py
Feed meta data:         {"success":false,"message":"Feed does not exist"}
Traceback (most recent call last):
  File "octopus_consumption_perTL.py", line 32, in <module>
    if meta['npoints']>0:
KeyError: 'npoints'
[email protected]:~ $

As well a the KeyError, this also looks ominous … “message”:“Feed does not exist”

In my script I had feedid = “agile” (as I had already created it via a browser)

Then I commented out the if meta[‘npoints’]>0: stanza as it was a first time run and got the following:

[email protected]:~ $  python octopus_consumption_perTL.py
Feed meta data:         {"success":false,"message":"Feed does not exist"}
Number of data points:  2640
Traceback (most recent call last):
  File "octopus_consumption_perTL.py", line 51, in <module>
    time = int(datetime.timestamp(datetime.strptime(dp['interval_start'],"%Y-%m-%dT%H:%M:%S%z")))
AttributeError: type object 'datetime.datetime' has no attribute 'timestamp'
[email protected]:~$

2640 looks correct as the Agile tariff started in early July.

But I’d now reached the limit of my skills. And time in Linux/Python is very confusing. One link I stumbled across had a concluding footnote “If, that is, we ignore the effects of Relativity”.

My python is Python 2.7.16

Your suggestions would be most welcome.
Thx

Note the crontab is Python3. The script should really have a #! as well. Suspect it needs python3.

Python 2.7 went EOL in Jan which means no updates I believe. You need to migrate off it as it will soon go from the repos - not installed by default on Raspberry OS I believe.

Hello @johnbanks

Yes that was intended to be ran in the browser, great!

You need to enter the emoncms feedid, it will be a number, hover your mouse over the feed name and it should show it.

I think you do need the latest python3 for this to work. Can you run the script on a Pi with a recent emonSD image installed?

1 Like

@TrystanLea

Yes - I finally figured out what was needed for feedid.

And with @borpins’s steer to Python3, I got the script to run OK. It also worked 24 hrs later just doing an incremental update.

Octopus sends kWh data to 2 decimal places which emoncms rounds (or truncates?) and so I’ve multiplied the value line by 1000 and show the units as Wh’s – this preserves all the Octopus data.

Because of the time I spent, I’ve added comments in my copy of the script:


# Script by Trystan Lea ref: https://community.openenergymonitor.org/t/python-script-to-import-octopus-agile-consumption-data-into-emoncms/15230

# Requires Python 3

# To run the script in an SSH terminal do: /usr/bin/python3 /home/pi/script-name.py (octopus_consumption_perTL.py)

# To run the script as a daily cronjob do: exactly the same

# Unlike other 'devices' which provide INPUT data as it occurs at n secs intervals, Octopus is providing a FEED of historical 30mins consumption data

# In consequence, it is necessary to FIRST create an emoncms FEED with a chosen tag, name & interval. The ID NUMBER for this FEED must then be entered in the script below at feedid =

These may/may not be of help to others.

Thx for yr work on this.
Yr new feed/insert.json functionality is a great addition which I think may have many uses.

That shouldnt be the case, mine have all imported to 3 decimal places (watt hour resolution). Are you looking at the values via the graph module? There’s a decimal place option to show up to 3 dp in the graph UI.

Great thanks!

never noticed that!

@TrystanLea

Indeed there is - just found it - one lives & learns …

It’s in the Feeds in View box under the graph ‘hidden’ under a column heading DP - pls give us oldies a break - there’s enough space there for a column heading Dec pts or something similar.

But banter to one side - Octopus appears to send the data at 10 Wh resolution - so more trailing zero’s is just spurious accuracy.

So it’s down to user choice (add *1000 to one script line). And personally I favour Wh’s have just invested heavily in PowerWalls and am hoping the Octopus numbers will be low.

:slight_smile:

thanks tristan, another user of your script, just got it working, plus discovered your app as well, thanks again

1 Like

Hi

I think I have found an issue with this where you have a solar feed.

My first attempt did not show Agile Import Actual data.

Settings:-

Second test attempt leaving out solar feed works as expected.

settings:-