Import Octopus consumption data

Yes and no, sometimes it is useful to see it in the inputs. I’ll see if I can work out how to exclude it if the FeedAPI is used as that doesn’t make any sense.

Well that is exactly what happens if you include a time element in the JSON data. So in this instance, I send the time tagged data at a rate of 1 item (a 30m consumption slot) per second (I discovered the HTTP API borked if sent too quickly). This was before the bulk upload API was done by Trystan.

The error is to then process that element (time) of the JSON in the FeedAPI such that it becomes a feed.

Hi guys,

I think I have a working solution, and tested it pretty hard today.

Here’s my updated version of @borpin 's Node-RED flow (and of course much kudos to him for getting me on the right track and started on this process. I knew nothing of Node-RED before Sunday! :smile: )

And at the bottom of this post, if anyone is interested to test and have a play, is the code for it.

Bear in mind that I used the emoncms node, available from https://flows.nodered.org/node/node-red-node-emoncms which simplifies pushing data to the local emoncms platform a tad, imho.

I also dropped the JSON convert node earlier in the flow, opting to simply have the http-request node return its data as a JSON object, which it can do automatically.

Finally, I changed the request to Octopus so that the interval_start figure was used as the timestamp, not the interval_end as was originally set. This means that on my EmonCMS, Octopus’s data lines up precisely with locally-computed use_kwh half-hourly graphs and is thus useful for sanity checking. YMMV!

Hope this is useful to someone. Thanks again for an interesting challenge - I’ve learned tons today! :smile:

Here da code…

 [
{
    "id": "380a5c78.95b144",
    "type": "tab",
    "label": "Octopus API Data-Fetch",
    "disabled": false,
    "info": ""
},
{
    "id": "d4d25902.975ec8",
    "type": "http request",
    "z": "380a5c78.95b144",
    "name": "",
    "method": "GET",
    "ret": "obj",
    "paytoqs": "ignore",
    "url": "",
    "tls": "",
    "persist": false,
    "proxy": "",
    "authType": "basic",
    "x": 190,
    "y": 220,
    "wires": [
        [
            "1bcf6cc0.397f03",
            "e9452b0c.0b1ac8"
        ]
    ]
},
{
    "id": "a4c504b.b2e80f8",
    "type": "inject",
    "z": "380a5c78.95b144",
    "name": "",
    "props": [
        {
            "p": "payload"
        },
        {
            "p": "topic",
            "vt": "str"
        }
    ],
    "repeat": "",
    "crontab": "",
    "once": false,
    "onceDelay": 0.1,
    "topic": "",
    "payload": "",
    "payloadType": "date",
    "x": 120,
    "y": 100,
    "wires": [
        [
            "2cda6207.39451e"
        ]
    ]
},
{
    "id": "12a67b3b.c888c5",
    "type": "delay",
    "z": "380a5c78.95b144",
    "name": "",
    "pauseType": "rate",
    "timeout": "3",
    "timeoutUnits": "seconds",
    "rate": "2",
    "nbRateUnits": "1",
    "rateUnits": "second",
    "randomFirst": "1",
    "randomLast": "5",
    "randomUnits": "seconds",
    "drop": false,
    "x": 270,
    "y": 340,
    "wires": [
        [
            "1512001.ccb42",
            "f0e942d1.d8bfc"
        ]
    ]
},
{
    "id": "2cda6207.39451e",
    "type": "function",
    "z": "380a5c78.95b144",
    "name": "Octopus - build consumption URL",
    "func": "var newmsg = {};\n\nvar host = \"https://api.octopus.energy/v1\";\nvar mpan = \"INSERT_YOUR_MPAN\";\nvar meter = \"INSERT_YOUR_METER_NUMBER\";\n\n// number of half hour periods max 25000 (will take a long time to load)\n// run with a small number of items to create the input so you can add\n// the processing to it.\n//\n// For ongoing loading of data, set to 48 and run once a day.\n\nvar page_size = 48;\n\nnewmsg.url = host;\nnewmsg.url += \"/electricity-meter-points/\" + mpan;\nnewmsg.url += \"/meters/\" + meter;\nnewmsg.url += \"/consumption/?page_size=\" + page_size;\n\nreturn newmsg;",
    "outputs": 1,
    "noerr": 0,
    "initialize": "",
    "finalize": "",
    "x": 220,
    "y": 160,
    "wires": [
        [
            "d4d25902.975ec8"
        ]
    ]
},
{
    "id": "252c1c24.6dbc14",
    "type": "comment",
    "z": "380a5c78.95b144",
    "name": "Instructions for use",
    "info": "### Preparation\n\nModify the Octopus function node with your meter details (MPAN, MPRN). \n\nInput your Octopus APIKey into the Username field in the http request node. Leave password blank.\n\nModify the EmonCMS node, with your emonCMS Server address and API write key.\n\n### EmonCMS Input Creation\n\nIn the Octopus function node, set the page size to a low number (e.g. 4), deploy and hit the trigger once to run the flow for the first time to create the 'octopus:consumption' Input in EmonCMS.\n\nThen, in EmonCMS, you can add the processing to that octopus:consumption Input in order to create a single feed. Choose:\n\n* Log to Feed\n* PHPFINA Fixed Interval\n* Period of 30 Mins.\n\nThis should now create an octopus:consumption Feed in EmonCMS.\n\nRun once the Node-RED flow once again and check the data is received into the Feed. If you can see the same number of received elements on a graph of the feed, as you had set on page_size earlier, all is well so far.\n\n### EmonCMS backfilling from Octopus API\n\nModify page_size to Octopus's maximum of 25000 (depending on when your smart meter went live, there may be much less data than this in practice, however).\n\nIn EmonCMS, delete the data from the octopus:consumption Feed, but don't delete the feed itself. (The reason for this is because once the feed is created, it seems EmonCMS won't allow data with earlier timestamps than the first given set to be added, so clearing the feed-data allows you to backfill all available historic data from the Octopus API).\n\nOnce you have an empty feed, run the Node-RED flow again.\n\nThis operation will take some time because each consumption period will take 0.5s to import (to ensure emonCMS can keep up).\n\nYou should now have as much history data as Octopus have!\n\n### Daily data download\n\nOnce you have your backfilled history, set the page_size to 48 (24hrs times half-hour periods), and configure the Trigger node to repeat once every 24 hours sometime after 2am or thereabouts (to avoid any daylight savings confusion). \n\nRun the flow one last time, and it should now automatically populate your EmonCMS with daily data from the Octopus API, soon after it is made available.\n\n### NB - Interval tagging choice!\n\nI have selected to use the Octopus 'interval_start' figure for each of the half-hour data-blocks, so that Octopus derived-data can be overlaid directly on to my own computed half-hourly use_kwh data, in order to spot discrepancies. However, if you wish, you can choose to index your Octopus data by the 'interval_end' tag instead. Bear in mind if you do this, the data will appear half-an-hour later on your graphs (but this may be exactly what you want!)",
    "x": 430,
    "y": 60,
    "wires": []
},
{
    "id": "e9452b0c.0b1ac8",
    "type": "debug",
    "z": "380a5c78.95b144",
    "name": "Octopus data",
    "active": false,
    "tosidebar": true,
    "console": false,
    "tostatus": false,
    "complete": "payload",
    "targetType": "msg",
    "statusVal": "",
    "statusType": "auto",
    "x": 550,
    "y": 220,
    "wires": []
},
{
    "id": "1bcf6cc0.397f03",
    "type": "function",
    "z": "380a5c78.95b144",
    "name": "Send data as emonCMS Input",
    "func": "var arrayobj = msg.payload.results;\n\narrayobj.reverse(); //iterate the Octopus data oldest first (to start the EmonCMS input at the beginning)\narrayobj.forEach(sendMessage);\n\nfunction sendMessage(item, index, arr) {\n    var newmsg = {};\n    \n    newmsg.nodegroup = \"octopus\";\n    newmsg.time = arr[index].interval_start;\n    newmsg.payload = {\"consumption\":arr[index].consumption};\n    \n    node.send(newmsg);\n}\n",
    "outputs": 1,
    "noerr": 0,
    "initialize": "",
    "finalize": "",
    "x": 290,
    "y": 280,
    "wires": [
        [
            "12a67b3b.c888c5"
        ]
    ]
},
{
    "id": "1512001.ccb42",
    "type": "emoncms",
    "z": "380a5c78.95b144",
    "name": "EmonCMS Push",
    "emonServer": "6c76c71b.24b448",
    "nodegroup": "",
    "datatype": "fulljson",
    "x": 320,
    "y": 400,
    "wires": []
},
{
    "id": "f0e942d1.d8bfc",
    "type": "debug",
    "z": "380a5c78.95b144",
    "name": "emoncms-message",
    "active": false,
    "tosidebar": true,
    "console": false,
    "tostatus": false,
    "complete": "true",
    "targetType": "full",
    "statusVal": "",
    "statusType": "auto",
    "x": 530,
    "y": 340,
    "wires": []
},
{
    "id": "6c76c71b.24b448",
    "type": "emoncms-server",
    "z": "",
    "server": "http://emonpi.local",
    "name": ""
}
]

I discovered what the problems were (now I have enabled smart meters), a couple of changes had been made to the API I think.

  1. As previously noted the API key needs to be in the HTTP Get node
  2. The time returned from Octopus is now in a genuine ISO format so the z at the end is replaced by +00:00. I was only escape ing that string so the received string at emoncms had a space rather than a +.
  3. As @BigJacko notes, it seems to be that the period_start needs to be used to make the data align.
  4. As also noted by @BigJacko, the http node can now return a JSON object directly so there is no need for the JSON node.

All fixed.

The advantage of this method over the Python version is it uses the Input API so the input can be processed and a cumulative value determined as well as the periodic value.

1 Like

@borpin

I agree CUM consumption data is desirable if only to save a lot of subsequent PITA analysis steps.

Trystan’s Python script works well but it only provides 30 min consumption data …

I’ve made a few simple script mods and so now also have CUM consumption data per these screenshots …


.


.

A cronjob run each day early AM updates history with the previous day data.

1 Like

Care to share (on the original Script thread please - not here).

I’ve built a Python client for the Octopus API that may be handy for some of these tasks, the documentation can be found here.