Integrating Octopus Go API in to emonCMS

I may be pushing the bounds of the OEM forum here, but I’ll give it a go :grinning:

Using the Octopus API I am able to see (somewhat delayed) half hourly consumption figures by just typing a url in to my browser and then filling in a Username & Password. (See output below)
https://api.octopus.energy/v1/electricity-meter-points/REDACTED/meters/REDACTED/consumption/

I would like to be able to record & graph this data within emonCMS. If I use Node-RED to grab data every 30 minutes, how would co-ordinate this? Perhaps poling the data at Times xx:01 and xx:31 and limiting the data via
Parameters: **period_from (datetime, optional)** Show consumption from the given datetime (inclusive). This parameter can be provided on its own. **period_to (datetime, optional)** Show consumption to the given datetime (exclusive). This parameter also requires providing the period_from parameter to create a range.

And then logging the data to a feed & then a graph. Would that work, or am I barking up the wrong tree? There is more here, but as you know this is at the edge of my comfort zone! Octopus Energy APIs — Octopus Energy API 1.0 documentation

TIA

{"count":816,"next":"https://api.octopus.energy/v1/electricity-meter-points/REDACTED/meters/REDACTED/consumption/?page=2","previous":null,"results":[{"consumption":0.0,"interval_start":"2020-03-15T23:00:00Z","interval_end":"2020-03-15T23:30:00Z"},{"consumption":0.0,"interval_start":"2020-03-15T22:30:00Z","interval_end":"2020-03-15T23:00:00Z"},{"consumption":0.0,"interval_start":"2020-03-15T22:00:00Z","interval_end":"2020-03-15T22:30:00Z"},{"consumption":0.001,"interval_start":"2020-03-15T21:30:00Z","interval_end":"2020-03-15T22:00:00Z"}, SNIP {"consumption":0.0,"interval_start":"2020-03-13T22:00:00Z","interval_end":"2020-03-13T22:30:00Z"},{"consumption":0.0,"interval_start":"2020-03-13T21:30:00Z","interval_end":"2020-03-13T22:00:00Z"}]}

I suggest you use a NR function to return the value of the most recent element plus a timestamp.

If you look at the emonCMS Node help, you can include a timestamp which should then be used.

Do a bit of work to ensure it only gets sent once and you should be fine.

My plan is to extend my node to include a consumption component once my switch is completed and I have the access to play :unamused:

so

{
	"consumption": 0.0,
	"interval_start": "2020-03-13T22:00:00Z",
	"interval_end": "2020-03-13T22:30:00Z"
}

make the msg.payload value the consumption and set msg.time as the end time (the ISO format should be fine - check emoncms logs), there will be a debug message along lines of ‘valid time found’.

HTH

[edit]
When I create the Octopus NR node, I’ll store the last value received and then look for the next one after 30 mins to ensure no overlap. I might even attempt to recover missed data.

Also, as the emonCMS node uses the HTTP API, you could use the ‘full JSON’ option and construct the message payload passed to the emonCMS node as

{
    "consumption":0.0,
    "time":"2020-03-17T16:28:02+00:00"
}

Thanks @borpin ; I have now created a flow which works to an extent - it grabs the data & only passes on one set of consumption / interval_end values to emonCMS push, but an error shows up below the emonCMS node, and no data seems to arrive at my emonPi.local (unlike the Powerwall data which does).

Do I need to re-format / tidy up the data that gets sent to emonCMS, or is this OK as is?

I’m making progress, but still very much in the learning stage of Node-RED ! For example, I don’t even know how / where to insert this …

TIA

What is in the function node?

Use the debug node at each stage so you can see what is happening to the data e.g.

image

Configure the debug node so it extracts the whole msg object.

What is the data coming out of the ‘Octopus’ node?

@borpin; herewith some screenshots. The Octopus node does collect “sensible” data, the Function node strips out all but the most recent consumption data (c/w times), the Delete Interval_start mode strips out one of the two time stamps (I thought this worthwhile but could be superfluous), but then there is an error with the emonCMS node.

emonCMS Node 0 is the one I use for the Powerwall, but I am aware that I have not configured this properly…


.
.

.
.

.
.

If you look at the emonCMS Input API,

it is looking for a JSON object that looks like this.

{"power1":100,"power2":200,"power3":300,"time":"2020-03-25T09:54:38+00:00"}

If you look at the data you are sending the emoncms node i.e. this output
image
Does it look like that? (hint - it doesn’t)

The node-red node does some of the hard work, but it still has to lokk familiar to it.

Delete the switch node and play with the function to construct a suitable msg to input to emonCMS node.

There is detail in the emonCMS Node Help panel on the data format. I suggest you use a slightly different format to feed into the emoncms node. i.e. a separate time and payload and just send a value.

so e.g.

msg.payload = different formats
msg.time = an ISO time string

Try this in the function - untested you may need to debug it.

var newmsg = {};

newmsg.payload = msg.payload.results[0].consumption;
newmsg.time = msg.payload.results[0].interval_end;

return newmsg;

Remember your Feed on the emonCMS end needs to be a 30 minute feed and you only want to trigger this every 30 minutes. However, by using the ‘time’ function you should not muck up the Feed data. There is a risk you don’t get the last period data for a max of 30 mins (depending on when it starts, but that is the best you can do without some fancy processing.

If you wanted to post all the historical data to emonCMS, you could actually loop through the array and load all the API will give you.

Amazing help @borpin - thank you. I have set the Timestamp node to action once every 30 minutes.

I needed to tweak a setting within the emonCMS node (to accept “simple” JSON), but then it worked; the Octopus consumption figures are only every 30 minutes, and delayed by 18 hours, so it’ll be a slow process to get the graph to show much - all the more so as I hope to only import anything between 00:30 - 04:30 anyway. But I’ll monitor it over the next 24 hours.

This is all very clever! :blush:

1 Like

I spoke too soon :crazy_face:. It seems that the data is published once a day, about 25 hours delayed. (ie up until 23:30 the night before last night). I therefore want to grab the first 48 thirty minute data points once every 24 hours at, say, 05:00, each day.

So rather than the Function node showing …
var newmsg = {};
newmsg.payload = msg.payload.results[0].consumption;
newmsg.time = msg.payload.results[0].interval_end;
return newmsg;

… it needs to be something along the lines of …
var newmsg = {};
newmsg.payload = msg.payload.results[0-47].consumption;
newmsg.time = msg.payload.results[0-47].interval_end;
return newmsg;

What might the correct syntax for this be? I have googled it, but not found what I am after.

I tried this by way of an experiment …
var newmsg = {};
newmsg.payload = msg.payload.results[0].consumption;
newmsg.time = msg.payload.results[0].interval_end;
newmsg.payload = msg.payload.results[1].consumption;
newmsg.time = msg.payload.results[1].interval_end;
return newmsg;

but only the “0” data point came back (which surprised me).

On that basis, you will need some form of loop and the emoncms Node will not help you as you will need to use the Feed API to insert the data.

this won’t work as you are overwriting the first value with the second assignment.

this is too difficult to do without testing and I won’t have data to test on for a while as merter installs have been halted :sob:

Is this working now? I want to do similar import into emonCMS but I’m not a nodered user.

I may just create a new emonhub interface for it.

Import what? Powerwall Battery (this thread got a bit OT)? It only did not work because of an old version of Node-RED.

Sorry @borpin I meant the Octopus meter readings using the API

I thought I’d I posted the solution I sorted out with @haffle but it appears not.

My solution does require Node-RED though. It is not live, comes through about 18hrs later in 24Hr chunks IIRC.

There are some python solutions out there.

If anybody is interested, I’ve put together some simple Python code to import the Octopus energy meter readings (both gas and electric).

The code will automatically create the necessary Inputs and Feeds in emoncms so you don’t need to pre-configure emoncms.

The code automatically finds the first meter reading from Octopus and imports the historic data and then just the incremental data after that (uses the last write date from emoncms to determine the checkpoint)

Schedule this code to run once per day using cron or similar.

Code is here…

Its nice to see how accurate the emonpi is…

Red line is the import recorded by emoncms and the green actual from smart meter

5 Likes

Noob question - but how do I install this code, where does it run etc?