Import Octopus consumption data

Updated 25/09 - problem was the returned time format from Octopus had changed.

Continuing the discussion from Integrating Octopus Go API in to emonCMS:

I’ve written a Node-Red flow to import your Octopus consumption data into emonCMS.

Short version

Import this flow into Node Red - follow the embedded instructions.

[{"id":"294d21ac.2850ae","type":"debug","z":"9ae82de6.3d714","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","targetType":"full","x":770,"y":180,"wires":[]},{"id":"414a095d.7f2e98","type":"http request","z":"9ae82de6.3d714","name":"Octopus Get Data","method":"GET","ret":"obj","paytoqs":"ignore","url":"","tls":"","persist":false,"proxy":"","authType":"basic","x":327,"y":60,"wires":[["8d8c8096.9b3f9"]]},{"id":"58561661.49bee8","type":"inject","z":"9ae82de6.3d714","name":"","repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"","payloadType":"date","x":100,"y":60,"wires":[["b657a771.70b908"]]},{"id":"7e0b0ad9.132b44","type":"http request","z":"9ae82de6.3d714","name":"Put data to emoncms","method":"GET","ret":"txt","paytoqs":"query","url":"","tls":"","persist":false,"proxy":"","authType":"","x":740,"y":120,"wires":[["294d21ac.2850ae"]]},{"id":"70cf228a.32aa9c","type":"delay","z":"9ae82de6.3d714","name":"","pauseType":"rate","timeout":"3","timeoutUnits":"seconds","rate":"1","nbRateUnits":"2","rateUnits":"second","randomFirst":"1","randomLast":"5","randomUnits":"seconds","drop":false,"x":520,"y":120,"wires":[["7e0b0ad9.132b44"]]},{"id":"b657a771.70b908","type":"function","z":"9ae82de6.3d714","name":"Octopus - build consumption URL","func":"var newmsg = {};\n\nvar host = \"https://api.octopus.energy/v1\";\nvar mpn = \"\";\nvar meter = \"\";\n\n// number of half hour periods max 25000 (will take a long time to load)\n// run with a small number of items to create the input so you can add\n// the processing to it.\n//\n// For ongoing loading of data, set to 48 and run once a day.\n\nvar page_size = 48;\n\nnewmsg.url = host;\nnewmsg.url += \"/electricity-meter-points/\" + mpn;\nnewmsg.url += \"/meters/\" + meter;\nnewmsg.url += \"/consumption/?page_size=\" + page_size;\n\nreturn newmsg;","outputs":1,"noerr":0,"initialize":"","finalize":"","x":200,"y":120,"wires":[["414a095d.7f2e98"]]},{"id":"8d8c8096.9b3f9","type":"function","z":"9ae82de6.3d714","name":"Send data as emonCMS Input","func":"var arrayobj = msg.payload.results;\nvar newmsg = {};\n\nvar apikey =\"12345\";\n\n// for host if you get a 404 error, this might require 'http://x.x.x.x/emoncms/input/post?'\n// should also work for emoncms.org\nvar host = \"http://x.x.x.x/input/post?\";\nvar node_name = \"octopus\"\n\narrayobj.reverse();\narrayobj.forEach(myFunction)\n\nfunction myFunction(item, index, arr) {\n    newmsg.url = host;\n    newmsg.url += \"node=\" + node_name;\n    newmsg.url += \"\\&fulljson={ \\\"time\\\": \\\"\" + encodeURIComponent(arr[index].interval_start) + \"\\\", \";\n    newmsg.url += \"\\\"consumption\\\": \" + arr[index].consumption + \"}\";\n    newmsg.url += \"\\&apikey=\" + apikey;\n    node.send(newmsg);\n}\n","outputs":1,"noerr":0,"initialize":"","finalize":"","x":610,"y":60,"wires":[["70cf228a.32aa9c"]]},{"id":"30c1ce17.0526a2","type":"comment","z":"9ae82de6.3d714","name":"Instructions for use","info":"### Preparation\n\nModify the 2 functions with your own details from Octopus (MPN, MPRN) and from emonCMS (API).\n\nModify the *Octopus Get Data* node and add the Octopus API Key\n\nPage size should be just a few first time to create the Input.\n\nOnce the input has been created in emonCMS, you can add the processing to it. Create 2 Feeds;\n\n* Just Log to Feed\n* PHPFINA Fixed Interval\n* Period of 30 Mins.\n\nRun once again and check the data is received.\n\nModify page_size to a maximum of 25000.\n\nEach consumption period will take 1s to import (to ensure emonCMS can keep up - this may be too conservative).","x":390,"y":180,"wires":[]}]

Longer version;

In Node red go to the hamburger menu (top right) click on import and paste the copied text into the box there. Click Import. Deploy the nodes that appear. Click on the Comment box for Instructions.

image

Notes

I have found that to view the data on the Graphing page you mat need these settings

image

Change to 1800 and click reload.

It seems the first few data points get loaded immediately, but once feedwriter has cycled, it will buffer for the length of the feedwriter sleep so in this case you usually only see an update of data every 300s

On Admin page

image

On the Inputs page, because the time used is very old, it will say inactive although the values will change.

You might need to add emoncms to the host variable in the function so

192.168.2.50/emoncms/input

HTH

[edit 06/09/20]

In the Node-Red http_rquest node, add this (colon is not needed);

image

2 Likes

Fantastic work, Brian. Thank you so much for doing it. :grinning:

1 Like

Nice work @borpin :+1:

I’ve just tried to give this a go, however I can’t see anywhere to put the Octopus API key in the Octopus function, am I missing something?

I’ve been on Agile for over a year now, I gonna look into getting this historic data into Emoncms to compare with my energy monitor data.

The only place I seem to have put my API is in the “Send data as emonCMS Input” node; have you done that?

Add the API key to the http request node? - Can’t remember TBH.

image

I also have a feeling you do not need the API key for reading the data.

Thanks, I’ve put the API key in the HTTP request bearer authentication Token field, however I’m getting error response: {"detail":"Authentication credentials were not provided."}

To test I’ve put the URL is directly into the HTTP request node, can you see what I’m missing?

Really? I would be surprised (and worried!) if personal consumption data could be viewed without the API key!

Yes, however that’s the Emoncms API key to post the data to and Emoncms account

I was just wondering if putting it there was the answer.

The API docs do not state the need for the key for consumption data.

https://developer.octopus.energy/docs/api/#list-consumption-for-a-meter

The first node builds the whole URL so no need to put it in the HTTP Request node (leave it blank).

Again according to the docs, the bearer request should work. It does say add a : though.

[edit]
I cannot test as my meters were only installed today but the above worked for @haffle.

I think they must have changed this from my first use.

In the Node-Red http_rquest node, add this;

image

Yes! I think that works. Do you just leave the password blank?

Although the consumption seems to return as 0?

That is what the API docs say to do.

Did you try it from the API page on your account? If you go to Sign in | Octopus Energy and then the API key page, there are a couple of pre-setup curl commands you can try.

Hi guys. Thanks @borpin for getting me started with this. I finally got connected to my Agile as of Friday night/Saturday morning, so I’m keen to get a logged version of what my meter has been telling Octopus. I’m already monitoring meter pulses (and have been for a year or so, off and on) so I have pretty good ‘by-the-minute’ realtime watt data which works well with the Agile App, but having the Octopus API data I reckon would let me confirm everything is all square.

I’ve managed to get your node-red flow working - to a degree - but I’m struggling with the concept of how it’s meant to be displayed and interrogated later.

I’m getting data back from the Octopus API all fine (btw - you don’t need that colon symbol on the Octopus API key), and that’s going out to emoncms (I’m running mine on a standalone Pi) and seems ok.

In emoncms, the arrival of that data has created two inputs: octopus:time and octopus:consumption. I’ve tried to follow your instructions and set up two feeds, but although I was able to set the consumption feed to have a feed unit of kWh, I couldn’t find anything logical to set the time feed unit to. As a result, when I look at the time feed in the CSV view on a Graph, I just see ‘2020’ for all the time values. Any ideas?

Moreover, I’m struggling to understand how having two separate feeds - one of times and one of power-consumption spot figures, logged in two separate PHPFina feeds each with its own data-posting timestamps, actually enables me to see that on a graph with the Octopus time on one axis and the consumption on the other. I seem to end up with time and consumption on two y-axes, and the data-posting timestamp on the x-axis, which isn’t a lot of use. How do I get the Octopus time on the horizontal axis? Is there supposed to be ONE input, not two - with the consumption as the payload, and a timestamp derived from the Octopus data? Or are the two inputs meant to be somehow integrated into a single feed, with the time component replacing the timestamp of the data-posting (i.e. the point where it was fetched).

I’m sure I’m doing something wrong, or I’ve failed to grasp some key concept here. Any help gratefully received. I feel I’m so close, but so far, and no cigar! LOL Thanks.

The time is pretty irrelevant as a feed. All it tells you is the UNIX time the last data was tagged at.

The other feed is the consumption in kWh for the 30 minute period. Just graph that against time.

Ok, now I’m definitely confused… :slight_smile:

The payload/data part of the ‘octopus:time’ feed that’s coming in from Node-RED is surely the interval_end timestamp that signals the end of the half-hour period for which the other input feed data (consumption in kWh) tallies. Looking at a debug msg.payload on the json node, I’m seeing ‘time’ data like: interval_end: “2020-09-13T00:30:00+01:00” which all seems sensible and relevant, and these appear to being passed correctly into EmonCMS.

Of course, because they are passed in when I do the grand fetch, they also possess a PHPFina automatically-added UNIX timestamp component, presumably, and this is the time of the grand fetch and means nothing to us in terms of useful calculation.

To be clear, it’s not the UNIX timestamp I’m having trouble with - it’s the Octopus ‘interval_end’ time - i.e. the actual data received, denoting the half-hour of interest. How is that supposed to be marked, in the ‘Feed Unit’ box? (see pic)

In case it’s helpful, I’m seeing the two INPUTS circled in red on the next pic:

Is this correct?

And in terms of feeds, I’ve done as you indicated and told both those inputs to Log to Feed with a 30min data interval (pic 3)

Is that correct too?

When I try and view this in a graph, I see the following:

- As you can see, the consumption (kWh) is recorded correctly (value-wise) but it is graphed on the x-axis according to the data-gathering Unix timestamp of when it was fetched, rather than as the more meaningful interval_end time which we really want (the consumption is the blue bar in this pic).

Meanwhile the point (with the tooltip) shows the interval_end component, but only partially recorded, as ‘2020’ as the second feed (time), also placed on the graph at the point of the data-gathering. What I’m sure we want, is for that time component to become the x-axis value for the consumption component - and I’m not sure what I’m doing wrong, or how to make that happen.

Is there a step in the Feed creation that I’ve missed?

Sorry if I’m being stupid or dim… Although I’ve had EmonCMS a while, I’m still finding some of the finer points a bit of a ‘learning cliff’ :slight_smile:

Is the misunderstanding that you want to be sending the data into emonCMS with the time attached to the value, as per the JSON format option as specified in the Input API:

{“power1”:100,“power2”:200,“power3”:300,“time”:“2020-09-14T11:54:35+01:00”}

which (as I understand it) inhibits emonCMS from attaching its own time stamp. Therefore you need to combine the two “Feeds” before they become Feeds? I know nothing about the process you’re using to transfer the data, so I can’t help any further.

1 Like

Hi @Robert.Wall, yes, I think that’s it. I believe I’ve just cracked it, though… I changed the contents of the myFunction to the following:

function myFunction(item, index, arr) {
    
    //convert Octopus's ISO8601 time to Unix Epoch timestamp
    var epoch = Date.parse(arr[index].interval_end)/1000;
    
    newmsg.url = host;
    newmsg.url += "node=" + node_name;
    newmsg.url += "\&time=" + epoch;
    newmsg.url += "\&fulljson={\"consumption\": " + arr[index].consumption + "}";
    newmsg.url += "\&apikey=" + apikey;
    node.send(newmsg);
} 

This basically just takes Octopus’s interval_end figure which is an ISO8601 datetime, and stuffs it into JS’s Date.parse function, which you can divide by 1000 to turn into a UNIX epoch timestamp. (This may be unnecessary - I was working on the API guide here https://emoncms.org/site/api#input which shows a manual time input using a UNIX timestamp, so I simply followed suit. It may be that it’ll quite happily take an ISO8601 native time in there too - I’ll test that in a bit!)

I then just re-jigged @borpin 's original code slightly, to put the time element as a separate part of the querydata, rather than inside the fulljson (again, as per the API guide, with a bit of combining two methods into one).

Seems to have worked! I’ve managed to get my four bits of initial Octopus data into a single input, and in turn, a single feed (again, PHPFina with a 30m interval) and it’s all displaying as I would expect now.

Thanks for the response, @Robert.Wall - it confirms I’m on the right track, I think (and learning!) Cheers! :smile:

1 Like

No, I don’t think it does. As I understand it, some times are UTC (or Unix Internet time) and some are that with milliseconds - and they don’t mix.

No it should be the start of the period.

Remember this is a timeseries Feed, so the data value simply drops into a slot that, in this case, is 30 minutes long. The time is then used to set the datestamp of the data as it arrives. After that, it is meaningless. I really should not have included it in the data recorded.

It does, but this is the same as the HTTP interface where it gets data and the timestamp of that data. So no

you do not.

I’m pretty sure that was what happened anyway but I am too busy to look. When I actually have the data setup for me I’ll check.

Ah, that could be the problem, I’m not sure the emoncms.org site handles timestamps in the same way.

The point I was trying to get across was that sending the two data items to emonCMS and recording them as separate “Feeds” was where the misunderstanding arose. That’s why I called them “Feeds” - with the inverted commas - because the data item carrying the value and the time is destined to become a single Feed, and the two parts should never have been separated.

Earlier you wrote:

Time might be irrelevant as a feed, but it’s wholly and completely relevant as a label to the data. I think the omission at that point was to say that the timestamp should have been passed with the data so that it became attached to the data in the Timeseries feed, and it needed an explanation as to how that should have been achieved.