Python script to import Octopus Agile consumption data into emoncms

But it doesn’t ‘compare’ it just uses it instead? I was hoping for a means to check the accuracy of the measured figures. I couldn’t work out how to do it on the Graphs.

I have worked out that the easiest way to check between 2 accumulating feeds, is to graph one on the left scale, one on right and compare the diff figure in statistics. Once I have that I can then apply a scaling multiplier in the Input Process. Has been quite successful.

Perhaps this could be done as an app where you can select 2 (or more) different feeds for comparison (e.g. Agile, Pulse, CT)?

1 Like

If you open the browser console window on the emoncms app, it prints this information out:

1 Like

Good idea as well!

I seem to be getting quite a few data outages on my half hourly data from the smart meter, i often have full days missing, but then clearing the feed and re-running the script from the start seems to pull the missing historic data in again… do octopus query for this missing data at the end of the month/week?

At the moment Im missing new data since the September 24th and am missing both 21nd and 22nd of September but older missing days have now appeared after I cleared the feed and re-imported from the start…

Processing a cumulative feed from this would require going back and reprocessing when the missing days become available… will have to think how best to do this…

I haven’t tried the script yet but FYI my Octopus (Go) data has frozen
on the website since the 24th September too so there may be a wider issue.

1 Like

Oh that is obvious… :rofl:.

1 Like

I think Octopus are having issues. I have an open ticket re missing data and they say it isn’t just me.

1 Like

I did mention it in the other thread on the agile app Octopus Agile emoncms app - #111 by TrystanLea but I dont blame you for not seeing it. This and the comparison feature needs to be added to the guide documentation…

1 Like

Should probably just include the info below the tickbox selector…

1 Like

I’m noticing a few gaps too - it’s at the Octopus end, usually. They sometimes need a chase, but they will have to re-fetch it before a bill is issued. It is the subject of some debate on the Octopus Agile forum atm.

FWIW I wrote a Node-RED flow that will backfill data for the missing slots, if that’s useful to anyone. It uses the same method as the editrealtime visualisation (ie. pokes it in via the update url, rather than the input one). Works well for me so far. Needs moderately careful handling insofar as you need to put the period from/to in correctly, and I’m longing for a decent ‘calendar style date-picker’ to refine the Node-Red flow somewhat - but it works…

Shout if wanted. Not sure whether to just post it here or whether that’s a breach of local etiquette.

1 Like

I wouldn’t regard it as such - it definitely falls within the remit of Open Energy Monitoring.

It was more the angle of ‘posting something in a different coding paradigm to someone else’s thread about their own work/different codebase’. I did this on another thread and wasn’t sure, and then later saw someone else on another thread being told to post their stuff in a new thread… so I figured I’d ask! :wink:

If a better place emerges, we can always move it (even if you can’t!).

Fair play… here goes then!

Click to view Octopus API specific-day backfill Node-RED flow code
[
    {
        "id": "dda3a2d6.12063",
        "type": "tab",
        "label": "Octopus Agile Specific Day",
        "disabled": false,
        "info": ""
    },
    {
        "id": "731abbac.7c1e84",
        "type": "http request",
        "z": "dda3a2d6.12063",
        "name": "Octopus API http-request",
        "method": "GET",
        "ret": "obj",
        "paytoqs": "ignore",
        "url": "",
        "tls": "",
        "persist": false,
        "proxy": "",
        "authType": "basic",
        "x": 230,
        "y": 220,
        "wires": [
            [
                "5be2a24d.fb9ccc",
                "4656ec43.52e5f4"
            ]
        ]
    },
    {
        "id": "2e3bf84d.b0e6c8",
        "type": "inject",
        "z": "dda3a2d6.12063",
        "name": "Trigger",
        "props": [
            {
                "p": "payload"
            },
            {
                "p": "topic",
                "vt": "str"
            }
        ],
        "repeat": "",
        "crontab": "00 08 * * *",
        "once": false,
        "onceDelay": 0.1,
        "topic": "",
        "payload": "",
        "payloadType": "date",
        "x": 120,
        "y": 100,
        "wires": [
            [
                "eb641d81.0f7dc"
            ]
        ]
    },
    {
        "id": "eb641d81.0f7dc",
        "type": "function",
        "z": "dda3a2d6.12063",
        "name": "Build Octopus API URL",
        "func": "var newmsg = {};\n\nvar host = \"https://api.octopus.energy/v1\";\nvar mpan = \"YOUR_MPAN_HERE\";\nvar meter = \"YOUR_METER_NUM_HERE\";\n\nnewmsg.url = host;\nnewmsg.url += \"/electricity-meter-points/\" + mpan;\nnewmsg.url += \"/meters/\" + meter;\nnewmsg.url += \"/consumption\";\nnewmsg.url += \"?period_from=2020-10-03T23:30Z\";\nnewmsg.url += \"&period_to=2020-10-04T23:30Z\";\nnewmsg.url += \"&order_by=period\";\n\nreturn newmsg;",
        "outputs": 1,
        "noerr": 0,
        "initialize": "",
        "finalize": "",
        "x": 190,
        "y": 160,
        "wires": [
            [
                "731abbac.7c1e84"
            ]
        ]
    },
    {
        "id": "962d3ef1.417c7",
        "type": "comment",
        "z": "dda3a2d6.12063",
        "name": "Instructions for use",
        "info": "### First-time Preparation\n\n1 - Modify the 'Build Octopus API URL' function node with your meter details (MPAN and meter number). \n\n2 - Input your Octopus APIKey into the Username field in the 'Octopus API http-request node. Leave password blank.\n\n3 - Modify the 'Backfill data to chosen period' node with your EmonCMS write API key, and the feed id number of the feed you are backfilling data to (this step is crucial to get correct, or the new data you fetch could wreck a different feed than intended!!). \nYou can determine your chosen feed's correct ID number by viewing it in a Graph display - it'll be listed at the bottom panel where the colours and charting features are selected, preceding the feed name.\n\n4 - Also in the 'Backfill data to chosen period' node, change the hostname of the newmsg.url, so that it points to your local EmonCMS installation correctly. (NB. May work with emonCMS.org, but currently untested. Use at your own risk!)\n\n### Performing a 'run'\n\n1 - in the 'Build Octopus API URL' node, set the 'period_from' and 'period_to' items correctly. Best to just change the year/month/day parts accordingly; the current time settings will fetch what Octopus generally considers 'day-end' and 'day-start' limits, and handle (ignore) any ongoing daylight-savings in force at the time.\n\n2 - Select any debug nodes you wish to view/suppress, according to your taste.\n\n3 - Deploy the NodeRed Flow!\n\n4 - Press the button on the Trigger node to begin the run.\n\n",
        "x": 310,
        "y": 40,
        "wires": []
    },
    {
        "id": "5be2a24d.fb9ccc",
        "type": "debug",
        "z": "dda3a2d6.12063",
        "name": "Octopus return-data debug",
        "active": true,
        "tosidebar": true,
        "console": false,
        "tostatus": false,
        "complete": "payload",
        "targetType": "msg",
        "statusVal": "",
        "statusType": "auto",
        "x": 520,
        "y": 220,
        "wires": []
    },
    {
        "id": "4656ec43.52e5f4",
        "type": "function",
        "z": "dda3a2d6.12063",
        "name": "Backfill data to chosen period",
        "func": "var apikey = \"YOUR_EMONCMS_WRITE_KEY\"; //your EmonCMS write APIkey\nvar feedid = YOUR_FEED_ID; //change this to the correct Feed ID!!\nvar host = \"http://YOUR_EMONCMS_HOST\"; //your local EmonCMS server address\n\nvar fetchedarr = msg.payload.results;\nfetchedarr.forEach(dataParse);\n\nfunction dataParse(item,index,arr) {\n\n    var newmsg = {};\n    \n    newmsg.payload  = {};\n    newmsg.payload.apikey = apikey;\n    newmsg.payload.id = feedid; \n    newmsg.payload.time = Date.parse(arr[index].interval_start)/1000;\n    newmsg.payload.value = arr[index].consumption;\n    newmsg.payload.skipbuffer = 1; //what does this do??\n    \n    newmsg.url = host + \"/feed/update.json\";\n    \n    node.send(newmsg);\n}",
        "outputs": 1,
        "noerr": 0,
        "initialize": "",
        "finalize": "",
        "x": 260,
        "y": 280,
        "wires": [
            [
                "2aa7f578.b6f46a"
            ]
        ]
    },
    {
        "id": "78b2d951.0095d8",
        "type": "http request",
        "z": "dda3a2d6.12063",
        "name": "send to EmonCMS",
        "method": "GET",
        "ret": "txt",
        "paytoqs": "query",
        "url": "",
        "tls": "",
        "persist": false,
        "proxy": "",
        "authType": "",
        "x": 270,
        "y": 400,
        "wires": [
            [
                "5e8837bd.9fe058"
            ]
        ]
    },
    {
        "id": "2aa7f578.b6f46a",
        "type": "delay",
        "z": "dda3a2d6.12063",
        "name": "",
        "pauseType": "rate",
        "timeout": "5",
        "timeoutUnits": "seconds",
        "rate": "1",
        "nbRateUnits": "2",
        "rateUnits": "second",
        "randomFirst": "1",
        "randomLast": "5",
        "randomUnits": "seconds",
        "drop": false,
        "x": 240,
        "y": 340,
        "wires": [
            [
                "78b2d951.0095d8",
                "e8e5e44f.eee338"
            ]
        ]
    },
    {
        "id": "5e8837bd.9fe058",
        "type": "debug",
        "z": "dda3a2d6.12063",
        "name": "emoncms debug",
        "active": true,
        "tosidebar": true,
        "console": false,
        "tostatus": false,
        "complete": "true",
        "targetType": "full",
        "statusVal": "",
        "statusType": "auto",
        "x": 530,
        "y": 400,
        "wires": []
    },
    {
        "id": "e8e5e44f.eee338",
        "type": "debug",
        "z": "dda3a2d6.12063",
        "name": "backfill data debug",
        "active": false,
        "tosidebar": true,
        "console": false,
        "tostatus": false,
        "complete": "true",
        "targetType": "full",
        "statusVal": "",
        "statusType": "auto",
        "x": 530,
        "y": 340,
        "wires": []
    }
]

Should look vaguely like this when imported successfully:
image

(and kudos to @borpin for inspiration in the first place!)

1 Like

Clearing all the data is a PITA.

@TrystanLea I have modified the original script to pull in a specified number of days of data with the intention of overwriting the data that is there and thus filling in the gaps.

Unfortunately it looks like the bulk upload API doesn’t overwrite the existing data. I’m sure the Node-Red module I tried did, but that used the single insert API.

Is this a bug or a feature of the bulk API?

18 months later…

I now appear to have reliable Gas and Electric data from Octopus (they have eventually fitted a dual band meter system).

I’d really like to import the Gas data.

@TrystanLea, I realise Gas data is of no interest to you, but for the wider community could we update the script to do both?

The tricky bit is the fact the electricity-meter-points string is hardcoded :frowning:

The rest just works.

1 Like

Sure, do we just need to break out that part of the URL as another setting?

I think it will need a bit more surgery than that. These are my thoughts :grinning:

The URLs diverge as follows;

GET /v1/electricity-meter-points/{mpan}/meters/{serial_number}/consumption/
GET /v1/gas-meter-points/{mprn}/meters/{serial_number}/consumption/

The gas and electric bits could be inserted from a fuel_type config entry.

The tag (node) is associated to the emoncms part as is the measurement name. I suggest they need to be part of the meter config.

I’d also like to collect both sets of data on a single run and send to multiple emoncms instances as well :slight_smile:

Another factor is calorific value of gas - this varies and I have just noticed that the SMETS1 gas meters report the data as kWh and SMETS2 as m³.

As this data is passed directly to a feed (so no processing), I suggest SMETS2 meters need to upload the m³ and do the calorific value calculation as well and upload that separately (add m3 & kWh to end of the name entry perhaps). Calorific value (CV) | National Grid Gas

image

Perhaps not quite as simple as I first imagined :frowning:

I was thinking of a config file something like this. It possibly needs a V2 script so as not to break existing setups.

[emoncms]
[[server1]]
url =
emoncms_apikey =
[[server2]]
url =
emoncms_apikey =

[octopus]
octopus_apikey = 

[[meter1]]
fuel_type = gas
meter_type = SMETS1
feed_node = 
feed_name = 
mpan = 
serial_number = 

[[meter2]]
fuel_type = gas
meter_type = SMETS2
feed_node = 
feed_name = 
mpan = 
serial_number = 
calorific_value =
volume_correction =

[[meter3]]
fuel_type = electric
feed_node = 
feed_name = 
mpan = 
serial_number = 
1 Like

Thanks @borpin sounds good. Will hopefully get back to you on this soon.

1 Like

This is great, thanks.

When I edited my new agile.conf file, I changed the tag and name of the target feed (because I’m on Go, not Agile) however it ignored this and created agile/consumption anyway.

I can live with that tag/name but if you would like me to modify the python script to use the values from agile.conf I’ll happily do that?

Someone would just have to point me at the steps required to check out/in.

p.s. I don’t have Gas so can’t do that change.