Yes, however that’s the Emoncms API key to post the data to and Emoncms account
I was just wondering if putting it there was the answer.
The API docs do not state the need for the key for consumption data.
https://developer.octopus.energy/docs/api/#list-consumption-for-a-meter
The first node builds the whole URL so no need to put it in the HTTP Request node (leave it blank).
Again according to the docs, the bearer request should work. It does say add a : though.
[edit]
I cannot test as my meters were only installed today but the above worked for @haffle.
I think they must have changed this from my first use.
In the Node-Red http_rquest node, add this;
Yes! I think that works. Do you just leave the password blank?
Although the consumption seems to return as 0?
That is what the API docs say to do.
Did you try it from the API page on your account? If you go to Sign in | Octopus Energy and then the API key page, there are a couple of pre-setup curl commands you can try.
Hi guys. Thanks @borpin for getting me started with this. I finally got connected to my Agile as of Friday night/Saturday morning, so I’m keen to get a logged version of what my meter has been telling Octopus. I’m already monitoring meter pulses (and have been for a year or so, off and on) so I have pretty good ‘by-the-minute’ realtime watt data which works well with the Agile App, but having the Octopus API data I reckon would let me confirm everything is all square.
I’ve managed to get your node-red flow working - to a degree - but I’m struggling with the concept of how it’s meant to be displayed and interrogated later.
I’m getting data back from the Octopus API all fine (btw - you don’t need that colon symbol on the Octopus API key), and that’s going out to emoncms (I’m running mine on a standalone Pi) and seems ok.
In emoncms, the arrival of that data has created two inputs: octopus:time and octopus:consumption. I’ve tried to follow your instructions and set up two feeds, but although I was able to set the consumption feed to have a feed unit of kWh, I couldn’t find anything logical to set the time feed unit to. As a result, when I look at the time feed in the CSV view on a Graph, I just see ‘2020’ for all the time values. Any ideas?
Moreover, I’m struggling to understand how having two separate feeds - one of times and one of power-consumption spot figures, logged in two separate PHPFina feeds each with its own data-posting timestamps, actually enables me to see that on a graph with the Octopus time on one axis and the consumption on the other. I seem to end up with time and consumption on two y-axes, and the data-posting timestamp on the x-axis, which isn’t a lot of use. How do I get the Octopus time on the horizontal axis? Is there supposed to be ONE input, not two - with the consumption as the payload, and a timestamp derived from the Octopus data? Or are the two inputs meant to be somehow integrated into a single feed, with the time component replacing the timestamp of the data-posting (i.e. the point where it was fetched).
I’m sure I’m doing something wrong, or I’ve failed to grasp some key concept here. Any help gratefully received. I feel I’m so close, but so far, and no cigar! LOL Thanks.
The time is pretty irrelevant as a feed. All it tells you is the UNIX time the last data was tagged at.
The other feed is the consumption in kWh for the 30 minute period. Just graph that against time.
Ok, now I’m definitely confused… 
The payload/data part of the ‘octopus:time’ feed that’s coming in from Node-RED is surely the interval_end timestamp that signals the end of the half-hour period for which the other input feed data (consumption in kWh) tallies. Looking at a debug msg.payload on the json node, I’m seeing ‘time’ data like: interval_end: “2020-09-13T00:30:00+01:00” which all seems sensible and relevant, and these appear to being passed correctly into EmonCMS.
Of course, because they are passed in when I do the grand fetch, they also possess a PHPFina automatically-added UNIX timestamp component, presumably, and this is the time of the grand fetch and means nothing to us in terms of useful calculation.
To be clear, it’s not the UNIX timestamp I’m having trouble with - it’s the Octopus ‘interval_end’ time - i.e. the actual data received, denoting the half-hour of interest. How is that supposed to be marked, in the ‘Feed Unit’ box? (see pic)
In case it’s helpful, I’m seeing the two INPUTS circled in red on the next pic:
Is this correct?
And in terms of feeds, I’ve done as you indicated and told both those inputs to Log to Feed with a 30min data interval (pic 3)
Is that correct too?
When I try and view this in a graph, I see the following:
Meanwhile the point (with the tooltip) shows the interval_end component, but only partially recorded, as ‘2020’ as the second feed (time), also placed on the graph at the point of the data-gathering. What I’m sure we want, is for that time component to become the x-axis value for the consumption component - and I’m not sure what I’m doing wrong, or how to make that happen.
Is there a step in the Feed creation that I’ve missed?
Sorry if I’m being stupid or dim… Although I’ve had EmonCMS a while, I’m still finding some of the finer points a bit of a ‘learning cliff’
Is the misunderstanding that you want to be sending the data into emonCMS with the time attached to the value, as per the JSON format option as specified in the Input API:
{“power1”:100,“power2”:200,“power3”:300,“time”:“2020-09-14T11:54:35+01:00”}
which (as I understand it) inhibits emonCMS from attaching its own time stamp. Therefore you need to combine the two “Feeds” before they become Feeds? I know nothing about the process you’re using to transfer the data, so I can’t help any further.
1 Like
Hi @Robert.Wall, yes, I think that’s it. I believe I’ve just cracked it, though… I changed the contents of the myFunction to the following:
function myFunction(item, index, arr) {
//convert Octopus's ISO8601 time to Unix Epoch timestamp
var epoch = Date.parse(arr[index].interval_end)/1000;
newmsg.url = host;
newmsg.url += "node=" + node_name;
newmsg.url += "\&time=" + epoch;
newmsg.url += "\&fulljson={\"consumption\": " + arr[index].consumption + "}";
newmsg.url += "\&apikey=" + apikey;
node.send(newmsg);
}
This basically just takes Octopus’s interval_end figure which is an ISO8601 datetime, and stuffs it into JS’s Date.parse function, which you can divide by 1000 to turn into a UNIX epoch timestamp. (This may be unnecessary - I was working on the API guide here https://emoncms.org/site/api#input which shows a manual time input using a UNIX timestamp, so I simply followed suit. It may be that it’ll quite happily take an ISO8601 native time in there too - I’ll test that in a bit!)
I then just re-jigged @borpin 's original code slightly, to put the time element as a separate part of the querydata, rather than inside the fulljson (again, as per the API guide, with a bit of combining two methods into one).
Seems to have worked! I’ve managed to get my four bits of initial Octopus data into a single input, and in turn, a single feed (again, PHPFina with a 30m interval) and it’s all displaying as I would expect now.
Thanks for the response, @Robert.Wall - it confirms I’m on the right track, I think (and learning!) Cheers! 
1 Like
No, I don’t think it does. As I understand it, some times are UTC (or Unix Internet time) and some are that with milliseconds - and they don’t mix.
No it should be the start of the period.
Remember this is a timeseries Feed, so the data value simply drops into a slot that, in this case, is 30 minutes long. The time is then used to set the datestamp of the data as it arrives. After that, it is meaningless. I really should not have included it in the data recorded.
It does, but this is the same as the HTTP interface where it gets data and the timestamp of that data. So no
you do not.
I’m pretty sure that was what happened anyway but I am too busy to look. When I actually have the data setup for me I’ll check.
Ah, that could be the problem, I’m not sure the emoncms.org site handles timestamps in the same way.
The point I was trying to get across was that sending the two data items to emonCMS and recording them as separate “Feeds” was where the misunderstanding arose. That’s why I called them “Feeds” - with the inverted commas - because the data item carrying the value and the time is destined to become a single Feed, and the two parts should never have been separated.
Earlier you wrote:
Time might be irrelevant as a feed, but it’s wholly and completely relevant as a label to the data. I think the omission at that point was to say that the timestamp should have been passed with the data so that it became attached to the data in the Timeseries feed, and it needed an explanation as to how that should have been achieved.
Yes and no, sometimes it is useful to see it in the inputs. I’ll see if I can work out how to exclude it if the FeedAPI is used as that doesn’t make any sense.
Well that is exactly what happens if you include a time element in the JSON data. So in this instance, I send the time tagged data at a rate of 1 item (a 30m consumption slot) per second (I discovered the HTTP API borked if sent too quickly). This was before the bulk upload API was done by Trystan.
The error is to then process that element (time) of the JSON in the FeedAPI such that it becomes a feed.
Hi guys,
I think I have a working solution, and tested it pretty hard today.
Here’s my updated version of @borpin 's Node-RED flow (and of course much kudos to him for getting me on the right track and started on this process. I knew nothing of Node-RED before Sunday! 
)
And at the bottom of this post, if anyone is interested to test and have a play, is the code for it.
Bear in mind that I used the emoncms node, available from https://flows.nodered.org/node/node-red-node-emoncms which simplifies pushing data to the local emoncms platform a tad, imho.
I also dropped the JSON convert node earlier in the flow, opting to simply have the http-request node return its data as a JSON object, which it can do automatically.
Finally, I changed the request to Octopus so that the interval_start figure was used as the timestamp, not the interval_end as was originally set. This means that on my EmonCMS, Octopus’s data lines up precisely with locally-computed use_kwh half-hourly graphs and is thus useful for sanity checking. YMMV!
Hope this is useful to someone. Thanks again for an interesting challenge - I’ve learned tons today!
Here da code…
[ { "id": "380a5c78.95b144", "type": "tab", "label": "Octopus API Data-Fetch", "disabled": false, "info": "" }, { "id": "d4d25902.975ec8", "type": "http request", "z": "380a5c78.95b144", "name": "", "method": "GET", "ret": "obj", "paytoqs": "ignore", "url": "", "tls": "", "persist": false, "proxy": "", "authType": "basic", "x": 190, "y": 220, "wires": [ [ "1bcf6cc0.397f03", "e9452b0c.0b1ac8" ] ] }, { "id": "a4c504b.b2e80f8", "type": "inject", "z": "380a5c78.95b144", "name": "", "props": [ { "p": "payload" }, { "p": "topic", "vt": "str" } ], "repeat": "", "crontab": "", "once": false, "onceDelay": 0.1, "topic": "", "payload": "", "payloadType": "date", "x": 120, "y": 100, "wires": [ [ "2cda6207.39451e" ] ] }, { "id": "12a67b3b.c888c5", "type": "delay", "z": "380a5c78.95b144", "name": "", "pauseType": "rate", "timeout": "3", "timeoutUnits": "seconds", "rate": "2", "nbRateUnits": "1", "rateUnits": "second", "randomFirst": "1", "randomLast": "5", "randomUnits": "seconds", "drop": false, "x": 270, "y": 340, "wires": [ [ "1512001.ccb42", "f0e942d1.d8bfc" ] ] }, { "id": "2cda6207.39451e", "type": "function", "z": "380a5c78.95b144", "name": "Octopus - build consumption URL", "func": "var newmsg = {};\n\nvar host = \"https://api.octopus.energy/v1\";\nvar mpan = \"INSERT_YOUR_MPAN\";\nvar meter = \"INSERT_YOUR_METER_NUMBER\";\n\n// number of half hour periods max 25000 (will take a long time to load)\n// run with a small number of items to create the input so you can add\n// the processing to it.\n//\n// For ongoing loading of data, set to 48 and run once a day.\n\nvar page_size = 48;\n\nnewmsg.url = host;\nnewmsg.url += \"/electricity-meter-points/\" + mpan;\nnewmsg.url += \"/meters/\" + meter;\nnewmsg.url += \"/consumption/?page_size=\" + page_size;\n\nreturn newmsg;", "outputs": 1, "noerr": 0, "initialize": "", "finalize": "", "x": 220, "y": 160, "wires": [ [ "d4d25902.975ec8" ] ] }, { "id": "252c1c24.6dbc14", "type": "comment", "z": "380a5c78.95b144", "name": "Instructions for use", "info": "### Preparation\n\nModify the Octopus function node with your meter details (MPAN, MPRN). \n\nInput your Octopus APIKey into the Username field in the http request node. Leave password blank.\n\nModify the EmonCMS node, with your emonCMS Server address and API write key.\n\n### EmonCMS Input Creation\n\nIn the Octopus function node, set the page size to a low number (e.g. 4), deploy and hit the trigger once to run the flow for the first time to create the 'octopus:consumption' Input in EmonCMS.\n\nThen, in EmonCMS, you can add the processing to that octopus:consumption Input in order to create a single feed. Choose:\n\n* Log to Feed\n* PHPFINA Fixed Interval\n* Period of 30 Mins.\n\nThis should now create an octopus:consumption Feed in EmonCMS.\n\nRun once the Node-RED flow once again and check the data is received into the Feed. If you can see the same number of received elements on a graph of the feed, as you had set on page_size earlier, all is well so far.\n\n### EmonCMS backfilling from Octopus API\n\nModify page_size to Octopus's maximum of 25000 (depending on when your smart meter went live, there may be much less data than this in practice, however).\n\nIn EmonCMS, delete the data from the octopus:consumption Feed, but don't delete the feed itself. (The reason for this is because once the feed is created, it seems EmonCMS won't allow data with earlier timestamps than the first given set to be added, so clearing the feed-data allows you to backfill all available historic data from the Octopus API).\n\nOnce you have an empty feed, run the Node-RED flow again.\n\nThis operation will take some time because each consumption period will take 0.5s to import (to ensure emonCMS can keep up).\n\nYou should now have as much history data as Octopus have!\n\n### Daily data download\n\nOnce you have your backfilled history, set the page_size to 48 (24hrs times half-hour periods), and configure the Trigger node to repeat once every 24 hours sometime after 2am or thereabouts (to avoid any daylight savings confusion). \n\nRun the flow one last time, and it should now automatically populate your EmonCMS with daily data from the Octopus API, soon after it is made available.\n\n### NB - Interval tagging choice!\n\nI have selected to use the Octopus 'interval_start' figure for each of the half-hour data-blocks, so that Octopus derived-data can be overlaid directly on to my own computed half-hourly use_kwh data, in order to spot discrepancies. However, if you wish, you can choose to index your Octopus data by the 'interval_end' tag instead. Bear in mind if you do this, the data will appear half-an-hour later on your graphs (but this may be exactly what you want!)", "x": 430, "y": 60, "wires": [] }, { "id": "e9452b0c.0b1ac8", "type": "debug", "z": "380a5c78.95b144", "name": "Octopus data", "active": false, "tosidebar": true, "console": false, "tostatus": false, "complete": "payload", "targetType": "msg", "statusVal": "", "statusType": "auto", "x": 550, "y": 220, "wires": [] }, { "id": "1bcf6cc0.397f03", "type": "function", "z": "380a5c78.95b144", "name": "Send data as emonCMS Input", "func": "var arrayobj = msg.payload.results;\n\narrayobj.reverse(); //iterate the Octopus data oldest first (to start the EmonCMS input at the beginning)\narrayobj.forEach(sendMessage);\n\nfunction sendMessage(item, index, arr) {\n var newmsg = {};\n \n newmsg.nodegroup = \"octopus\";\n newmsg.time = arr[index].interval_start;\n newmsg.payload = {\"consumption\":arr[index].consumption};\n \n node.send(newmsg);\n}\n", "outputs": 1, "noerr": 0, "initialize": "", "finalize": "", "x": 290, "y": 280, "wires": [ [ "12a67b3b.c888c5" ] ] }, { "id": "1512001.ccb42", "type": "emoncms", "z": "380a5c78.95b144", "name": "EmonCMS Push", "emonServer": "6c76c71b.24b448", "nodegroup": "", "datatype": "fulljson", "x": 320, "y": 400, "wires": [] }, { "id": "f0e942d1.d8bfc", "type": "debug", "z": "380a5c78.95b144", "name": "emoncms-message", "active": false, "tosidebar": true, "console": false, "tostatus": false, "complete": "true", "targetType": "full", "statusVal": "", "statusType": "auto", "x": 530, "y": 340, "wires": [] }, { "id": "6c76c71b.24b448", "type": "emoncms-server", "z": "", "server": "http://emonpi.local", "name": "" } ]
I discovered what the problems were (now I have enabled smart meters), a couple of changes had been made to the API I think.
- As previously noted the API key needs to be in the HTTP Get node
- The time returned from Octopus is now in a genuine ISO format so the
zat the end is replaced by+00:00. I was onlyescapeing that string so the received string at emoncms had a space rather than a+. - As @BigJacko notes, it seems to be that the period_start needs to be used to make the data align.
- As also noted by @BigJacko, the http node can now return a JSON object directly so there is no need for the JSON node.
All fixed.
The advantage of this method over the Python version is it uses the Input API so the input can be processed and a cumulative value determined as well as the periodic value.
1 Like
I agree CUM consumption data is desirable if only to save a lot of subsequent PITA analysis steps.
Trystan’s Python script works well but it only provides 30 min consumption data …
I’ve made a few simple script mods and so now also have CUM consumption data per these screenshots …
.
.
A cronjob run each day early AM updates history with the previous day data.
1 Like
Care to share (on the original Script thread please - not here).
I’ve built a Python client for the Octopus API that may be handy for some of these tasks, the documentation can be found here.
