Community
OpenEnergyMonitor

OpenEnergyMonitor Community

Python script to import Octopus Agile consumption data into emoncms

I wouldn’t regard it as such - it definitely falls within the remit of Open Energy Monitoring.

It was more the angle of ‘posting something in a different coding paradigm to someone else’s thread about their own work/different codebase’. I did this on another thread and wasn’t sure, and then later saw someone else on another thread being told to post their stuff in a new thread… so I figured I’d ask! :wink:

If a better place emerges, we can always move it (even if you can’t!).

Fair play… here goes then!

Click to view Octopus API specific-day backfill Node-RED flow code
[
    {
        "id": "dda3a2d6.12063",
        "type": "tab",
        "label": "Octopus Agile Specific Day",
        "disabled": false,
        "info": ""
    },
    {
        "id": "731abbac.7c1e84",
        "type": "http request",
        "z": "dda3a2d6.12063",
        "name": "Octopus API http-request",
        "method": "GET",
        "ret": "obj",
        "paytoqs": "ignore",
        "url": "",
        "tls": "",
        "persist": false,
        "proxy": "",
        "authType": "basic",
        "x": 230,
        "y": 220,
        "wires": [
            [
                "5be2a24d.fb9ccc",
                "4656ec43.52e5f4"
            ]
        ]
    },
    {
        "id": "2e3bf84d.b0e6c8",
        "type": "inject",
        "z": "dda3a2d6.12063",
        "name": "Trigger",
        "props": [
            {
                "p": "payload"
            },
            {
                "p": "topic",
                "vt": "str"
            }
        ],
        "repeat": "",
        "crontab": "00 08 * * *",
        "once": false,
        "onceDelay": 0.1,
        "topic": "",
        "payload": "",
        "payloadType": "date",
        "x": 120,
        "y": 100,
        "wires": [
            [
                "eb641d81.0f7dc"
            ]
        ]
    },
    {
        "id": "eb641d81.0f7dc",
        "type": "function",
        "z": "dda3a2d6.12063",
        "name": "Build Octopus API URL",
        "func": "var newmsg = {};\n\nvar host = \"https://api.octopus.energy/v1\";\nvar mpan = \"YOUR_MPAN_HERE\";\nvar meter = \"YOUR_METER_NUM_HERE\";\n\nnewmsg.url = host;\nnewmsg.url += \"/electricity-meter-points/\" + mpan;\nnewmsg.url += \"/meters/\" + meter;\nnewmsg.url += \"/consumption\";\nnewmsg.url += \"?period_from=2020-10-03T23:30Z\";\nnewmsg.url += \"&period_to=2020-10-04T23:30Z\";\nnewmsg.url += \"&order_by=period\";\n\nreturn newmsg;",
        "outputs": 1,
        "noerr": 0,
        "initialize": "",
        "finalize": "",
        "x": 190,
        "y": 160,
        "wires": [
            [
                "731abbac.7c1e84"
            ]
        ]
    },
    {
        "id": "962d3ef1.417c7",
        "type": "comment",
        "z": "dda3a2d6.12063",
        "name": "Instructions for use",
        "info": "### First-time Preparation\n\n1 - Modify the 'Build Octopus API URL' function node with your meter details (MPAN and meter number). \n\n2 - Input your Octopus APIKey into the Username field in the 'Octopus API http-request node. Leave password blank.\n\n3 - Modify the 'Backfill data to chosen period' node with your EmonCMS write API key, and the feed id number of the feed you are backfilling data to (this step is crucial to get correct, or the new data you fetch could wreck a different feed than intended!!). \nYou can determine your chosen feed's correct ID number by viewing it in a Graph display - it'll be listed at the bottom panel where the colours and charting features are selected, preceding the feed name.\n\n4 - Also in the 'Backfill data to chosen period' node, change the hostname of the newmsg.url, so that it points to your local EmonCMS installation correctly. (NB. May work with emonCMS.org, but currently untested. Use at your own risk!)\n\n### Performing a 'run'\n\n1 - in the 'Build Octopus API URL' node, set the 'period_from' and 'period_to' items correctly. Best to just change the year/month/day parts accordingly; the current time settings will fetch what Octopus generally considers 'day-end' and 'day-start' limits, and handle (ignore) any ongoing daylight-savings in force at the time.\n\n2 - Select any debug nodes you wish to view/suppress, according to your taste.\n\n3 - Deploy the NodeRed Flow!\n\n4 - Press the button on the Trigger node to begin the run.\n\n",
        "x": 310,
        "y": 40,
        "wires": []
    },
    {
        "id": "5be2a24d.fb9ccc",
        "type": "debug",
        "z": "dda3a2d6.12063",
        "name": "Octopus return-data debug",
        "active": true,
        "tosidebar": true,
        "console": false,
        "tostatus": false,
        "complete": "payload",
        "targetType": "msg",
        "statusVal": "",
        "statusType": "auto",
        "x": 520,
        "y": 220,
        "wires": []
    },
    {
        "id": "4656ec43.52e5f4",
        "type": "function",
        "z": "dda3a2d6.12063",
        "name": "Backfill data to chosen period",
        "func": "var apikey = \"YOUR_EMONCMS_WRITE_KEY\"; //your EmonCMS write APIkey\nvar feedid = YOUR_FEED_ID; //change this to the correct Feed ID!!\nvar host = \"http://YOUR_EMONCMS_HOST\"; //your local EmonCMS server address\n\nvar fetchedarr = msg.payload.results;\nfetchedarr.forEach(dataParse);\n\nfunction dataParse(item,index,arr) {\n\n    var newmsg = {};\n    \n    newmsg.payload  = {};\n    newmsg.payload.apikey = apikey;\n    newmsg.payload.id = feedid; \n    newmsg.payload.time = Date.parse(arr[index].interval_start)/1000;\n    newmsg.payload.value = arr[index].consumption;\n    newmsg.payload.skipbuffer = 1; //what does this do??\n    \n    newmsg.url = host + \"/feed/update.json\";\n    \n    node.send(newmsg);\n}",
        "outputs": 1,
        "noerr": 0,
        "initialize": "",
        "finalize": "",
        "x": 260,
        "y": 280,
        "wires": [
            [
                "2aa7f578.b6f46a"
            ]
        ]
    },
    {
        "id": "78b2d951.0095d8",
        "type": "http request",
        "z": "dda3a2d6.12063",
        "name": "send to EmonCMS",
        "method": "GET",
        "ret": "txt",
        "paytoqs": "query",
        "url": "",
        "tls": "",
        "persist": false,
        "proxy": "",
        "authType": "",
        "x": 270,
        "y": 400,
        "wires": [
            [
                "5e8837bd.9fe058"
            ]
        ]
    },
    {
        "id": "2aa7f578.b6f46a",
        "type": "delay",
        "z": "dda3a2d6.12063",
        "name": "",
        "pauseType": "rate",
        "timeout": "5",
        "timeoutUnits": "seconds",
        "rate": "1",
        "nbRateUnits": "2",
        "rateUnits": "second",
        "randomFirst": "1",
        "randomLast": "5",
        "randomUnits": "seconds",
        "drop": false,
        "x": 240,
        "y": 340,
        "wires": [
            [
                "78b2d951.0095d8",
                "e8e5e44f.eee338"
            ]
        ]
    },
    {
        "id": "5e8837bd.9fe058",
        "type": "debug",
        "z": "dda3a2d6.12063",
        "name": "emoncms debug",
        "active": true,
        "tosidebar": true,
        "console": false,
        "tostatus": false,
        "complete": "true",
        "targetType": "full",
        "statusVal": "",
        "statusType": "auto",
        "x": 530,
        "y": 400,
        "wires": []
    },
    {
        "id": "e8e5e44f.eee338",
        "type": "debug",
        "z": "dda3a2d6.12063",
        "name": "backfill data debug",
        "active": false,
        "tosidebar": true,
        "console": false,
        "tostatus": false,
        "complete": "true",
        "targetType": "full",
        "statusVal": "",
        "statusType": "auto",
        "x": 530,
        "y": 340,
        "wires": []
    }
]

Should look vaguely like this when imported successfully:
image

(and kudos to @borpin for inspiration in the first place!)

1 Like

Clearing all the data is a PITA.

@TrystanLea I have modified the original script to pull in a specified number of days of data with the intention of overwriting the data that is there and thus filling in the gaps.

Unfortunately it looks like the bulk upload API doesn’t overwrite the existing data. I’m sure the Node-Red module I tried did, but that used the single insert API.

Is this a bug or a feature of the bulk API?