Octopus energy : Smart Meter Feed

That is really interesting, especially,

Not sure even Octopus have implemented that level of access. “Any information that is capable of being stored…”

Hi John

Yes, in the end I used nodered running on a pi and a simple nodered setup - image below

Once mqtt enabled on hildebrand glowstick and you have credentials from them, this just works.

Nodered setup on pi is simple as well.

giving the following type of granular data

Happy to share the nodered setup/code if you wish

Julian

Just to confirm, is the data directly from the stick or are you retrieving it after it has been posted to the cloud?

What are you able to access; is it the current usage or can you see history? Is it only electricity or gas you see gas (if you have it). Is it only import or can you see export (even if your export is zero)?

From my reading of the regulations, you should be able to see everything but acknowledge that Hildebrand are facilitating access but aren’t the licensee so don’t themselves hold the responsibility to make the data available.

Julian

I’d be interested in some samples: I need an input into my emonpi/ emoncms from my SMETS2: This seems to be an interesting way to get rates and useage into the system using glowmarkt, your nodered should give me some useful hints (just being a bit lazy, guess I could work it out with lots of trial and error!).

Unless someone has done all the work to do what I want - to get a feed into the emonpi!

Many thanks!

1 Like

My node red flow looks like this … you’ll need to sort out your IP addresses, glowstick mac address etc … cut and paste between the ‘[]’ into your nodered
GUI

[
    {
        "id": "a31e89f5.a102e8",
        "type": "tab",
        "label": "glowstickMQTTEmoncms",
        "disabled": false,
        "info": ""
    },
    {
        "id": "fbe2b82f.8d1248",
        "type": "mqtt in",
        "z": "a31e89f5.a102e8",
        "name": "",
        "topic": "SMART/HILD/xxxxxxxxxxxx", ### the unique id / mac for your glowstick ###
        "qos": "2",
        "datatype": "auto",
        "broker": "87923771.022e08",
        "x": 170,
        "y": 280,
        "wires": [
            [
                "4729f673.a2afd8",
                "21dcf206.96247e"
            ]
        ]
    },
    {
        "id": "a0638792.bb68f8",
        "type": "debug",
        "z": "a31e89f5.a102e8",
        "name": "",
        "active": false,
        "tosidebar": true,
        "console": false,
        "tostatus": false,
        "complete": "true",
        "targetType": "full",
        "statusVal": "",
        "statusType": "auto",
        "x": 490,
        "y": 280,
        "wires": []
    },
    {
        "id": "4729f673.a2afd8",
        "type": "json",
        "z": "a31e89f5.a102e8",
        "name": "",
        "property": "payload",
        "action": "",
        "pretty": false,
        "x": 190,
        "y": 360,
        "wires": [
            [
                "93d4ac29.88172"
            ]
        ]
    },
    {
        "id": "93d4ac29.88172",
        "type": "function",
        "z": "a31e89f5.a102e8",
        "name": "extract elements",
        "func": "msg.meterSerial = msg.payload.elecMtr[\"0702\"][\"03\"][\"08\"]\nmsg.timeStamp = msg.payload.gmtime\nmsg.todayConsumptDelivered0401 = parseInt(msg.payload.elecMtr[\"0702\"][\"04\"][\"01\"],16)\n//msg.instantDemand0400 = parseInt(msg.payload.elecMtr[\"0702\"][\"04\"][\"00\"],16) - value is hex representation of an unsigned INT 24 bits\nmsg.currSummReceived0001 = parseInt(msg.payload.elecMtr[\"0702\"][\"00\"][\"01\"],16)\nmsg.currSummDelivered0000 = parseInt(msg.payload.elecMtr[\"0702\"][\"00\"][\"00\"],16)\n\nmsg.instantDemand0400 = parseInt(msg.payload.elecMtr[\"0702\"][\"04\"][\"00\"],16)\nif ((msg.instantDemand0400 & 0x800000) > 0) {\n    msg.instantDemand0400 = msg.instantDemand0400 - 0x1000000\n}\n\nreturn msg;",
        "outputs": 1,
        "noerr": 0,
        "initialize": "",
        "finalize": "",
        "x": 330,
        "y": 420,
        "wires": [
            [
                "a0638792.bb68f8",
                "9ddc9a28.91f898"
            ]
        ]
    },
    {
        "id": "9ddc9a28.91f898",
        "type": "function",
        "z": "a31e89f5.a102e8",
        "name": "Send data as emonCMS Input",
        "func": "var arrayobj = msg.payload.results\nvar newmsg = {}\nvar apikey =\"xxxxxxxxxxxxxxxxxxxxxxxx\"\n\n// for host if you get a 404 error, this might require 'http://x.x.x.x/emoncms/input/post?'\n// should also work for emoncms.org\n//var host = \"http://192.168.1.xxx/emoncms/input/post?\"\n//http://192.168.1.xxx/input/post?time=1598996829&node=1&csv=100,200,300\n\nvar host = \"http://192.168.1.xxx/emoncms/input/post?\"\nvar node_name = \"glowstick\"\n\n    newmsg.url = host\n    newmsg.url += \"time=\" + msg.timeStamp\n    newmsg.url += \"\\&node=\" + node_name\n    newmsg.url += \"\\&csv=\" + msg.instantDemand0400\n    newmsg.url += \"\\&apikey=\" + apikey\n    \n//    newmsg.url += \"\\&fulljson={ \\\"time\\\": \\\"\" + escape(arr[index].interval_end) + \"\\\", \"\n//    newmsg.url += \"\\\"consumption\\\": \" + arr[index].consumption + \"}\"\n//    newmsg.url += \"\\&apikey=\" + apikey\n// bulk?data=[[-2,%22octopusGo%22,1000],[0,%22octopusGo%22,1000]]&time=1598596010\n    //newmsg.url += \"&data=[[0,\\\"\" + node_name + \"\\\",\" + arr[index].consumption + \"]]\"\n    //msg.dateconv = new Date(arr[index].interval_end);\n    //msg.millisec = Date.parse(arr[index].interval_end);\n    //newmsg.url += \"\\&time=\" + msg.millisec;\n    ////newmsg.url += \"\\&csv=\" + arr[index].consumption;\n    //newmsg.url += \"\\&fulljson={ \\\"time\\\": \\\"\" + escape(arr[index].interval_end) + \"\\\", \"\n    //newmsg.url += \"\\\"consumption\\\": \" + arr[index].consumption + \"}\"\n\nreturn newmsg;\n",
        "outputs": 1,
        "noerr": 0,
        "initialize": "",
        "finalize": "",
        "x": 630,
        "y": 420,
        "wires": [
            [
                "b07cc45e.0b4938",
                "bf9d361f.ca5a08"
            ]
        ]
    },
    {
        "id": "b07cc45e.0b4938",
        "type": "http request",
        "z": "a31e89f5.a102e8",
        "name": "",
        "method": "GET",
        "ret": "txt",
        "paytoqs": "query",
        "url": "",
        "tls": "",
        "persist": false,
        "proxy": "",
        "authType": "",
        "x": 710,
        "y": 520,
        "wires": [
            []
        ]
    },
    {
        "id": "bf9d361f.ca5a08",
        "type": "debug",
        "z": "a31e89f5.a102e8",
        "name": "",
        "active": false,
        "tosidebar": true,
        "console": false,
        "tostatus": false,
        "complete": "true",
        "targetType": "full",
        "statusVal": "",
        "statusType": "auto",
        "x": 790,
        "y": 340,
        "wires": []
    },
    {
        "id": "21dcf206.96247e",
        "type": "debug",
        "z": "a31e89f5.a102e8",
        "name": "",
        "active": false,
        "tosidebar": true,
        "console": false,
        "tostatus": false,
        "complete": "false",
        "statusVal": "",
        "statusType": "auto",
        "x": 420,
        "y": 200,
        "wires": []
    },
    {
        "id": "87923771.022e08",
        "type": "mqtt-broker",
        "z": "",
        "name": "glowstick hildebrand",
        "broker": "glowmqtt.energyhive.com",
        "port": "8883",
        "tls": "",
        "clientid": "",
        "usetls": true,
        "compatmode": false,
        "keepalive": "60",
        "cleansession": true,
        "birthTopic": "",
        "birthQos": "0",
        "birthPayload": "",
        "closeTopic": "",
        "closeQos": "0",
        "closePayload": "",
        "willTopic": "",
        "willQos": "0",
        "willPayload": ""
    }
]
1 Like

@cbrookson

Here’s the Python script that I’m using:

# PURPOSE: To input Smart Meter data into emoncms (data obtained via a Hildebrand Glow Stick https://www.hildebrand.co.uk/our-products/glow-stick-wifi-cad/ )

# With due acknowledgement to ndfred - a contributor to the Glowmarkt forum
# https://gist.github.com/ndfred/b373eeafc4f5b0870c1b8857041289a9

# Developed and tested on a Raspberry Pi running the Oct 2019 emon image updated to ver 10.2.6

# Electricity only - not Gas

# HOW TO ...

# IMPORTANT - Install mosquitto-clients with: sudo apt-get install mosquitto-clients    XXXXXXXXXXXXX !!!!!!!!!

# The script will create an INPUT node to receive the data

# Copy this script file to  /home/pi  and make it executable with:  chmod +x /home/pi/JB-glowmqtt.py  # using the correct script name

# Run the script with: /usr/bin/python3 /home/pi/JB-glowmqtt.py  # using the correct script name

# All being well, Smart meter data will appear in emoncms webpage Inputs refreshing every 10 secs - Power Now(W) and Daily & CUM Energy(kWh)

# Create FEEDS using Log to Feed and add UNITS (pencil drop-down) to each

# IMPORTANT NOTE ...
# Data from this script is input to emoncms via an http API.
# The EmonHubEmoncmsHTTPinterfacer is not required - it cannot send this script data to watchman

# FINALLY ONCE THE SCRIPT RUNS OK: Create the glow.service and enable it so the script runs on boot up as follows:
# Do: CTRL-C to stop the script then - Do: sudo nano /etc/systemd/system/glow.service  and copy & paste in the following BUT adjust the script name ...

"""

[Unit]
Description=Glow Stick service
After=network.target
After=mosquitto.service
StartLimitIntervalSec=0

[Service]
Type=simple
Restart=always
RestartSec=1
User=pi
ExecStart=/usr/bin/python3 /home/pi/JB-glowmqtt.py

[Install]
WantedBy=multi-user.target

"""
# Then save & exit and to ensure the glow.service runs on boot up - Do:  sudo systemctl enable glow.service

# AS A VERY LAST CHECK - Do: sudo reboot then SSH in again and check the service status with:  systemctl status glow.service

# Finally close the SSH terminal and the script/service will continue to run surviving any future reboots

# ===============================================================================

import datetime
import logging
import json
import paho.mqtt.client as mqtt    # paho-mqtt is already installed in emon
import requests    

# Glow Stick configuration
GLOW_LOGIN = "xxxxxxxxxxxxxxxxxx"
GLOW_PASSWORD = "xxxxxxxxxxxxxxxxx"
GLOW_DEVICE_ID = "xxxxxxxxxxxxxxxxx"


# Emoncms server configuration
emoncms_apikey = "xxxxxxxxxxxxxxxxxxxxxxxx"   # Change as appropriate
emoncms_server = "http://127.0.0.1"
node = "Glow Stick"  # Name of the node created to receive the INPUT data 

# Name each of the data inputs associated with the newly created node
di1 = "Power Now"      # ref E_NOW below
di2 = "Daily Energy"   # ref E_DAY below
di3 = "CUM Energy"     # ref E_METER below


def on_connect(client, _userdata, _flags, result_code):
    if result_code != mqtt.MQTT_ERR_SUCCESS:
        logging.error("Error connecting: %d", result_code)
        return

    result_code, _message_id = client.subscribe("SMART/HILD/" + GLOW_DEVICE_ID)

    if result_code != mqtt.MQTT_ERR_SUCCESS:
        logging.error("Couldn't subscribe: %d", result_code)
        return

    logging.info("Connected and subscribed")

def on_message(_client, _userdata, message):
    payload = json.loads(message.payload)
    current_time = datetime.datetime.now().strftime("%H:%M:%S")
	
    electricity_consumption = int(payload["elecMtr"]["0702"]["04"]["00"], 16)

    if electricity_consumption > 10000000: electricity_consumption = electricity_consumption - 16777216 # Added JB - hex FFFFFF is 16777215 in unsigned 24 bit but -1 in signed 24 bit
    E_NOW = electricity_consumption   # Added JB

    electricity_daily_consumption = int(payload["elecMtr"]["0702"]["04"]["01"], 16)
	
    # electricity_weekly_consumption = int(payload["elecMtr"]["0702"]["04"]["30"], 16)  # Data not provided by GLOW
    # electricity_monthly_consumption = int(payload["elecMtr"]["0702"]["04"]["40"], 16)  # Data not provided by GLOW
    electricity_multiplier = int(payload["elecMtr"]["0702"]["03"]["01"], 16)
    electricity_divisor = int(payload["elecMtr"]["0702"]["03"]["02"], 16)
    electricity_meter = int(payload["elecMtr"]["0702"]["00"]["00"], 16)

    electricity_daily_consumption = electricity_daily_consumption * electricity_multiplier / electricity_divisor
    E_DAY = electricity_daily_consumption     # Added JB
	
    # electricity_weekly_consumption = electricity_weekly_consumption * electricity_multiplier / electricity_divisor
    # electricity_monthly_consumption = electricity_monthly_consumption * electricity_multiplier / electricity_divisor
    electricity_meter = electricity_meter * electricity_multiplier / electricity_divisor
    E_METER = electricity_meter     # Added JB                               
	
    assert(int(payload["elecMtr"]["0702"]["03"]["00"], 16) == 0) # kWh
    
    logging.info("Reading at %s", current_time)
    logging.info("electricity consumption: %dW", electricity_consumption)
    logging.info("daily electricity consumption: %.3fkWh", electricity_daily_consumption)
    # logging.info("* weekly electricity consumption: %.3fkWh", electricity_weekly_consumption)
    # logging.info("* monthly electricity consumption: %.3fkWh", electricity_monthly_consumption)
    logging.info("electricity meter: %.3fkWh", electricity_meter)
    
    # logging.info("Full payload: %s", json.dumps(payload, indent=2))   # Don't need this info printed
	                    	
    data1 = E_NOW
    data2 = E_DAY
    data3 = E_METER

    # Send data to emoncms

    dev_data = {di1: data1, di2: data2, di3: data3}

    data = {
      'node': node,
      'data': json.dumps (dev_data),
      'apikey': emoncms_apikey
    }

    response = requests.post(emoncms_server+"/input/post", data=data)
	
	
	
def loop():
    logging.basicConfig(level=logging.DEBUG, format='%(message)s')
    client = mqtt.Client()
    client.username_pw_set(GLOW_LOGIN, GLOW_PASSWORD)
    client.on_connect = on_connect
    client.on_message = on_message
    client.connect("glowmqtt.energyhive.com")
    client.loop_forever()

if __name__ == "__main__":
    loop()

This shows the INPUTS to emoncms:

Here are my FEEDS:

I’m a big fan of GlowStick.
Hope this helps

1 Like

Is that a SMETS 1 stick?

@borpin
Mid 2020, Octopus Energy fitted a SMETS1 SECURE Sprint 211 because SMETS2 3 phase meters are not yet available.

When I made my first enquiry to Hildebrand/GlowStick, I got the following response:

That is good news as we do have the ability to remotely do the Open/Join HAN instruction with Octopus.

We are about to start testing with SMETS2 polyphase meters and have never connected to a SMETS1 variant (as they are so rare!). What I’d like to propose, if that is OK with you, is that we send you one of our GlowSticks which works with a single phase SMETS1 Secure meter and see what happens with the polyphase - so no risk for you.

1 Like

@johnbanks for your script, did you ever create an extended version that included gas data as well as electric? I’ve just found out about this project, have a Hildebrand CAD and wanted to try monitoring gas as well as electric

@Townsmcp
The installation I’m involved with is rural with no gas supply and so I have no plans to extend my script.

A couple of comments tho’ …
ndfred’s script does include gas

https://gist.github.com/ndfred/b373eeafc4f5b0870c1b8857041289a9

So it should be possible for you to add gas to my script.

Also, I do not have a Hildebrand CAD but their GlowStick and so I don’t know whether my script works with a CAD. You could ask Hildebrand?

Regards

Ah ok. Thanks for clarifying that about the script

Going back to the original topic, has anyone got a finished method to get the above data into emoncms, noting that it delivers a day’s worth of half-hourly data a day late?

I suspect I need to do something with input/bulk but I don’t understand the help on time offsets and how time=1664194418 makes -6 mean 1387730121 seconds since 0 second and not 166194412.

It looks as if Trystan’s tried to be far too clever - he’s picked up the Time Now and put it into time=1664194418, whereas I think “1387730121 seconds since 0 second” is when he wrote the help page - December 22, 2013

@TrystanLea An elephant-sized oopsie here!

1 Like
1 Like

Thanks but I’m not on Octopus Agile so I think I need to pull from the regular API? Hopefully it’s a starting point for the emoncms side, though.

The script above is only reading consumption data from Octopus, and doesn’t appear to require the Agile tariff at all. I think it’ll do what you need. Replace the word agile with octopus if it helps.

@mjr Tim is correct, although labelled Agile, it just pulls in the consumption data.

Mine has been dead for nearly a year, a long and ugly Smart Meter issue that is now fixed with my 3rd set of meters and Comms Hub.

I’ll get the script up and running again and see what happens.

I know I’m having one of those days, but why’s it labelled Agile then? :frowning:

Would a patch be welcomed?

Because this is was the target audience.

You are correct, it shouldn’t be.

I’ve written my own script to get data from Octopus, which adds the consumption into a cumulative kWh feed. This makes is easier to plot the delta on a daily, weekly or monthly basis:

I also have secondary feeds for the cumulative cost, based on whatever the unit rate is at the time, which could optionally also support Agile pricing.

Arguably, for absolute accuracy, one should be using the time from interval_end rather than interval_start, as the data represents energy consumption to the end of the half-hour.

1 Like