I think I have a working solution, and tested it pretty hard today.
And at the bottom of this post, if anyone is interested to test and have a play, is the code for it.
I also dropped the JSON convert node earlier in the flow, opting to simply have the http-request node return its data as a JSON object, which it can do automatically.
Hope this is useful to someone. Thanks again for an interesting challenge - I’ve learned tons today!
[
{
"id": "380a5c78.95b144",
"type": "tab",
"label": "Octopus API Data-Fetch",
"disabled": false,
"info": ""
},
{
"id": "d4d25902.975ec8",
"type": "http request",
"z": "380a5c78.95b144",
"name": "",
"method": "GET",
"ret": "obj",
"paytoqs": "ignore",
"url": "",
"tls": "",
"persist": false,
"proxy": "",
"authType": "basic",
"x": 190,
"y": 220,
"wires": [
[
"1bcf6cc0.397f03",
"e9452b0c.0b1ac8"
]
]
},
{
"id": "a4c504b.b2e80f8",
"type": "inject",
"z": "380a5c78.95b144",
"name": "",
"props": [
{
"p": "payload"
},
{
"p": "topic",
"vt": "str"
}
],
"repeat": "",
"crontab": "",
"once": false,
"onceDelay": 0.1,
"topic": "",
"payload": "",
"payloadType": "date",
"x": 120,
"y": 100,
"wires": [
[
"2cda6207.39451e"
]
]
},
{
"id": "12a67b3b.c888c5",
"type": "delay",
"z": "380a5c78.95b144",
"name": "",
"pauseType": "rate",
"timeout": "3",
"timeoutUnits": "seconds",
"rate": "2",
"nbRateUnits": "1",
"rateUnits": "second",
"randomFirst": "1",
"randomLast": "5",
"randomUnits": "seconds",
"drop": false,
"x": 270,
"y": 340,
"wires": [
[
"1512001.ccb42",
"f0e942d1.d8bfc"
]
]
},
{
"id": "2cda6207.39451e",
"type": "function",
"z": "380a5c78.95b144",
"name": "Octopus - build consumption URL",
"func": "var newmsg = {};\n\nvar host = \"https://api.octopus.energy/v1\";\nvar mpan = \"INSERT_YOUR_MPAN\";\nvar meter = \"INSERT_YOUR_METER_NUMBER\";\n\n// number of half hour periods max 25000 (will take a long time to load)\n// run with a small number of items to create the input so you can add\n// the processing to it.\n//\n// For ongoing loading of data, set to 48 and run once a day.\n\nvar page_size = 48;\n\nnewmsg.url = host;\nnewmsg.url += \"/electricity-meter-points/\" + mpan;\nnewmsg.url += \"/meters/\" + meter;\nnewmsg.url += \"/consumption/?page_size=\" + page_size;\n\nreturn newmsg;",
"outputs": 1,
"noerr": 0,
"initialize": "",
"finalize": "",
"x": 220,
"y": 160,
"wires": [
[
"d4d25902.975ec8"
]
]
},
{
"id": "252c1c24.6dbc14",
"type": "comment",
"z": "380a5c78.95b144",
"name": "Instructions for use",
"info": "### Preparation\n\nModify the Octopus function node with your meter details (MPAN, MPRN). \n\nInput your Octopus APIKey into the Username field in the http request node. Leave password blank.\n\nModify the EmonCMS node, with your emonCMS Server address and API write key.\n\n### EmonCMS Input Creation\n\nIn the Octopus function node, set the page size to a low number (e.g. 4), deploy and hit the trigger once to run the flow for the first time to create the 'octopus:consumption' Input in EmonCMS.\n\nThen, in EmonCMS, you can add the processing to that octopus:consumption Input in order to create a single feed. Choose:\n\n* Log to Feed\n* PHPFINA Fixed Interval\n* Period of 30 Mins.\n\nThis should now create an octopus:consumption Feed in EmonCMS.\n\nRun once the Node-RED flow once again and check the data is received into the Feed. If you can see the same number of received elements on a graph of the feed, as you had set on page_size earlier, all is well so far.\n\n### EmonCMS backfilling from Octopus API\n\nModify page_size to Octopus's maximum of 25000 (depending on when your smart meter went live, there may be much less data than this in practice, however).\n\nIn EmonCMS, delete the data from the octopus:consumption Feed, but don't delete the feed itself. (The reason for this is because once the feed is created, it seems EmonCMS won't allow data with earlier timestamps than the first given set to be added, so clearing the feed-data allows you to backfill all available historic data from the Octopus API).\n\nOnce you have an empty feed, run the Node-RED flow again.\n\nThis operation will take some time because each consumption period will take 0.5s to import (to ensure emonCMS can keep up).\n\nYou should now have as much history data as Octopus have!\n\n### Daily data download\n\nOnce you have your backfilled history, set the page_size to 48 (24hrs times half-hour periods), and configure the Trigger node to repeat once every 24 hours sometime after 2am or thereabouts (to avoid any daylight savings confusion). \n\nRun the flow one last time, and it should now automatically populate your EmonCMS with daily data from the Octopus API, soon after it is made available.\n\n### NB - Interval tagging choice!\n\nI have selected to use the Octopus 'interval_start' figure for each of the half-hour data-blocks, so that Octopus derived-data can be overlaid directly on to my own computed half-hourly use_kwh data, in order to spot discrepancies. However, if you wish, you can choose to index your Octopus data by the 'interval_end' tag instead. Bear in mind if you do this, the data will appear half-an-hour later on your graphs (but this may be exactly what you want!)",
"x": 430,
"y": 60,
"wires": []
},
{
"id": "e9452b0c.0b1ac8",
"type": "debug",
"z": "380a5c78.95b144",
"name": "Octopus data",
"active": false,
"tosidebar": true,
"console": false,
"tostatus": false,
"complete": "payload",
"targetType": "msg",
"statusVal": "",
"statusType": "auto",
"x": 550,
"y": 220,
"wires": []
},
{
"id": "1bcf6cc0.397f03",
"type": "function",
"z": "380a5c78.95b144",
"name": "Send data as emonCMS Input",
"func": "var arrayobj = msg.payload.results;\n\narrayobj.reverse(); //iterate the Octopus data oldest first (to start the EmonCMS input at the beginning)\narrayobj.forEach(sendMessage);\n\nfunction sendMessage(item, index, arr) {\n var newmsg = {};\n \n newmsg.nodegroup = \"octopus\";\n newmsg.time = arr[index].interval_start;\n newmsg.payload = {\"consumption\":arr[index].consumption};\n \n node.send(newmsg);\n}\n",
"outputs": 1,
"noerr": 0,
"initialize": "",
"finalize": "",
"x": 290,
"y": 280,
"wires": [
[
"12a67b3b.c888c5"
]
]
},
{
"id": "1512001.ccb42",
"type": "emoncms",
"z": "380a5c78.95b144",
"name": "EmonCMS Push",
"emonServer": "6c76c71b.24b448",
"nodegroup": "",
"datatype": "fulljson",
"x": 320,
"y": 400,
"wires": []
},
{
"id": "f0e942d1.d8bfc",
"type": "debug",
"z": "380a5c78.95b144",
"name": "emoncms-message",
"active": false,
"tosidebar": true,
"console": false,
"tostatus": false,
"complete": "true",
"targetType": "full",
"statusVal": "",
"statusType": "auto",
"x": 530,
"y": 340,
"wires": []
},
{
"id": "6c76c71b.24b448",
"type": "emoncms-server",
"z": "",
"server": "http://emonpi.local",
"name": ""
}
]