A taste of things to come? - UK smart meter data access

Hi all, sorry we can’t monitor all channels as much as we’d like. Contact us at [email protected] or Twitter @n3rgy for better response times!!

Occasionally meters don’t give up their data so we need to give them a kick (or worse). If there’s no data within 48 hrs of signing up ping us the MPAN via email or Twitter and we’ll have a look.

Regarding CADs, building your own will be a challenge. You’ll need to get a ZigBee chipset running ZigBee smart energy 1.4 (from memory). The more challenging part will be to get it whitelisted and paired with your meter. We have a commercial service to do this today, I hope we can offer a consumer service to support open source communities in future as we do with the half hour data. If someone is making progress with open CADs feel free to reach out!

Matt

Hello there,

I honestly almost forgot I still have an n3rgy account, as I was getting blank reading (blank csv files). Well, just checked and they are still blank (almost). I downloaded it again (between now and 1/07) and the only reading there was, was on the 1/07, gas standing charge with 30mins intervals. Gas meter reading was empty, as well as whole electricity. I sent you guys an email with MPAN. Let’s give it a try once more.

Hi Ben,

Are you sure it is acting as a CAD and getting its data from the meter, or is it getting the data from a web server somewhere else? I was with Bulb and last time I looked the way their service worked was the latter. But I’ve since moved to Octopus for other reasons so I’m not familiar with the current status.

PS It just took an absolute age to publish my post. Other things work fine at my end. Is there something wrong with the community.openenergymonitor.org website or ISP?

I’ve just (at 00:26 GMT+1 approx) seen a delay of some tens of seconds. Maybe Trystan is updating something?

If its reading higher frequency power usage data from the meter and then transmitting this outside the smart meter HAN over home wifi to somewhere else then it is acting as a CAD. Local and/or unfettered access to data is unfortunately not a requirement of being a CAD, I wish it was!

and it may well be that Samsung SmartThings app is getting it via a web server but it had to have come from the meter via the IHD and home wifi in first place because this type and resolution of data is not available through the central smart metering system.

Also, maybe I have missed something obvious… couldnt I just get the
data from the Smart Things API?

Has anyone tried this with the Bulb integration?

@beaylott
Have you seen this post? I don’t have smart things, so I don’t know how it works?

https://community.bulb.co.uk/t/apis/45027/42

No! Sounds very promising! Thanks. Will give it a go at the weekend and report back!

@DanielKenning

My supplier is Octopus but they did not have to be involved.

The GlowStick connects to a Raspberry Pi running emon via a Python script.

Try this link and enter yr MPAN

https://glowmarkt.com/register.html

Oherwise email - Hildebrand Support [email protected]
I’ve always found Jane Wilson to be very helpful

Thank you John, I have done this, and yes, Jane Wilson was the person there and extremely helpful!

What a shame the UK meters do not have a P1 port (Energy Management in Home Assistant - Home Assistant). What an absolute mess we have made of the whole Smart Meter design and deployment; it is depressing.

For me, the meter is effectively inside a faraday cage (house has a ‘teacozy’ of PIR insulation) so is unlikely to ever work consistently.

@borpin
That’s why I went for second option (Home Assistant Glow), 2xphotodiode, 2xesp32 and 2xRGB led - for electric and gas. £12,70 delivered. I don’t know how I am getting it installed nor if it is going to work (especially with gas meter), but I am not breaking a bank here. Probably another HA project that will end up unfinished, oh well.

But that is a poor substitute and will not work for a Gas meter - no flashing diode. I’ve got a photo measurement (as well as other OEM kit) but that is irrelevant.

Agreed

I thought the installers were supposed to test for correct operation of both the meter and IHD when they started installing, and in the case of problems to refer it for a more advanced solution?

OEM already offers a sensor to read the diode on meters; no need to use something else. The only problem is what the light means can depend on the configuration of the meter. At least with old-fashioned meters, I’ve no idea if things are any better with smart meters. How do they distinguish between import and export?

@borpin @djh
Did not notice, gas meter doesn’t have flashing led. Make sense, since it would decrease battery life I guess. But I was also hoping leds on HAN device indicate electricity AND gas. I know it is far from average solution (to say at least). Having port would make things so much easier.
And yes, I know, in 99% I spent those 12 quid for nothing.

@DanielKenning

My GlowStick has been running without problems for 12 months.
It’s located in a utility room close to the smart meter on an outside wall.
It’s plugged into a USB port of a Raspberry Pi just for convenience and to power it.
The data comes via wifi & MQTT.
The RPi runs the emon system.

I spent many hours getting things running as I was new to MQTT - you may find my Python script useful …

# PURPOSE: To input Smart Meter data into emoncms (data obtained via a Hildebrand Glow Stick https://www.hildebrand.co.uk/our-products/glow-stick-wifi-cad/ )

# With due acknowledgement to ndfred - a contributor to the Glowmarkt forum
# https://gist.github.com/ndfred/b373eeafc4f5b0870c1b8857041289a9

# Developed and tested on a Raspberry Pi running the Oct 2019 emon image updated to ver 10.2.6

# HOW TO ...

# IMPORTANT - Install mosquitto-clients with: sudo apt-get install mosquitto-clients    XXXXXXXXXXXXX !!!!!!!!!

# The script will create an INPUT node to receive the data

# Copy this script file to  /home/pi  and make it executable with:  chmod +x /home/pi/JB-glowmqtt.py  # using the correct script name

# Run the script with: /usr/bin/python3 /home/pi/JB-glowmqtt.py  # using the correct script name

# All being well, Smart meter data will appear in emoncms webpage Inputs refreshing every 10 secs - Power Now(W) and Daily & CUM Energy(kWh)

# Create FEEDS using Log to Feed and add UNITS (pencil drop-down) to each

# IMPORTANT NOTE ...
# Data from this script is input to emoncms via an http API.  This bypasses emonhub ...
# The EmonHubEmoncmsHTTPinterfacer is not required

# FINALLY ONCE THE SCRIPT RUNS OK: Create the glow.service and enable it so the script runs on boot up as follows:
# Do: CTRL-C to stop the script then - Do: sudo nano /etc/systemd/system/glow.service  and copy & paste in the following BUT adjust the script name ...

"""

[Unit]
Description=Glow Stick service
After=network.target
After=mosquitto.service
StartLimitIntervalSec=0

[Service]
Type=simple
Restart=always
RestartSec=1
User=pi
ExecStart=/usr/bin/python3 /home/pi/JB-glowmqtt.py

[Install]
WantedBy=multi-user.target

"""
# Then save & exit and to ensure the glow.service runs on boot up - Do:  sudo systemctl enable glow.service

# AS A VERY LAST CHECK - Do: sudo reboot then SSH in again and check the service status with:  systemctl status glow.service

# Finally close the SSH terminal and the script/service will continue to run surviving any future reboots

# ===============================================================================

import datetime
import logging
import json
import paho.mqtt.client as mqtt    # paho-mqtt is already installed in emon
import requests    

# Glow Stick configuration
GLOW_LOGIN = "~~~~~~~~~"
GLOW_PASSWORD = "~~~~~~~~~~~"
GLOW_DEVICE_ID = "~~~~~~~~~~~"   # This is the MAC address of the GlowStick

# Emoncms server configuration
emoncms_apikey = "~~~~~~~~~~~~~~~"   # for Node 15 - change as appropriate
emoncms_server = "http://127.0.0.1"
node = "Glow Stick"  # Name of the node created to receive the INPUT data 

# Name each of the data inputs associated with the newly created node
di1 = "Power Now"      # ref E_NOW below
di2 = "Daily Energy"   # ref E_DAY below
di3 = "CUM Energy"     # ref E_METER below


def on_connect(client, _userdata, _flags, result_code):
    if result_code != mqtt.MQTT_ERR_SUCCESS:
        logging.error("Error connecting: %d", result_code)
        return

    result_code, _message_id = client.subscribe("SMART/HILD/" + GLOW_DEVICE_ID)

    if result_code != mqtt.MQTT_ERR_SUCCESS:
        logging.error("Couldn't subscribe: %d", result_code)
        return

    logging.info("Connected and subscribed")

def on_message(_client, _userdata, message):
    payload = json.loads(message.payload)
    current_time = datetime.datetime.now().strftime("%H:%M:%S")
	
    electricity_consumption = int(payload["elecMtr"]["0702"]["04"]["00"], 16)

    if electricity_consumption > 10000000: electricity_consumption = electricity_consumption - 16777216 # Added JB - hex FFFFFF is 16777215 in unsigned 24 bit but -1 in signed 24 bit
    E_NOW = electricity_consumption   # Added JB

    electricity_daily_consumption = int(payload["elecMtr"]["0702"]["04"]["01"], 16)
	
    # electricity_weekly_consumption = int(payload["elecMtr"]["0702"]["04"]["30"], 16)  # Data not provided by GLOW
    # electricity_monthly_consumption = int(payload["elecMtr"]["0702"]["04"]["40"], 16)  # Data not provided by GLOW
    electricity_multiplier = int(payload["elecMtr"]["0702"]["03"]["01"], 16)
    electricity_divisor = int(payload["elecMtr"]["0702"]["03"]["02"], 16)
    electricity_meter = int(payload["elecMtr"]["0702"]["00"]["00"], 16)

    electricity_daily_consumption = electricity_daily_consumption * electricity_multiplier / electricity_divisor
    E_DAY = electricity_daily_consumption     # Added JB
	
    # electricity_weekly_consumption = electricity_weekly_consumption * electricity_multiplier / electricity_divisor
    # electricity_monthly_consumption = electricity_monthly_consumption * electricity_multiplier / electricity_divisor
    electricity_meter = electricity_meter * electricity_multiplier / electricity_divisor
    E_METER = electricity_meter     # Added JB                               
	
    assert(int(payload["elecMtr"]["0702"]["03"]["00"], 16) == 0) # kWh
    
    logging.info("Reading at %s", current_time)
    logging.info("electricity consumption: %dW", electricity_consumption)
    logging.info("daily electricity consumption: %.3fkWh", electricity_daily_consumption)
    # logging.info("* weekly electricity consumption: %.3fkWh", electricity_weekly_consumption)
    # logging.info("* monthly electricity consumption: %.3fkWh", electricity_monthly_consumption)
    logging.info("electricity meter: %.3fkWh", electricity_meter)
    
    # logging.info("Full payload: %s", json.dumps(payload, indent=2))   # Don't need this info printed
	                    	
    data1 = E_NOW
    data2 = E_DAY
    data3 = E_METER

    # Send data to emoncms

    dev_data = {di1: data1, di2: data2, di3: data3}

    data = {
      'node': node,
      'data': json.dumps (dev_data),
      'apikey': emoncms_apikey
    }

    response = requests.post(emoncms_server+"/input/post", data=data)
	
	
	
def loop():
    logging.basicConfig(level=logging.DEBUG, format='%(message)s')
    client = mqtt.Client()
    client.username_pw_set(GLOW_LOGIN, GLOW_PASSWORD)
    client.on_connect = on_connect
    client.on_message = on_message
    client.connect("glowmqtt.energyhive.com")
    client.loop_forever()

if __name__ == "__main__":
    loop()

The Input Processing I use to create Feeds is …

And here’s one view of the Feeds …

(The zero import thru’ the night is because the system hs a large battery installed)

My understanding of how things work is that the GlowStick intercepts the consumption data that is sent by mobile phone signal to the ‘central authority’ that then sends this data to yr supplier who bills you.
Hildebrand is authorised to do this by the ‘central authority’ but only with yr permission - hence the request for yr MPAN.

I recall struggling to get MQTT certification sorted - so this may also help …

I found I needed first to run …

$ curl "https://letsencrypt.org/certs/isrgrootx1.pem.txt" > cert.pem
Then …

$ curl "https://letsencrypt.org/certs/letsencryptauthorityx3.pem.txt" >> cert.pem
The cert.pem file ended up in   /home/pi   and so the connect statement then needed was …

mosquitto_sub --cafile /home/pi/cert.pem -h glowmqtt.energyhive.com -u ~~yr username~~ -P
~~~yr password~~~ -t SMART/HILD/~~~MACaddress~~~ -p 8883

My bottom line/editorial comment is - GlowStick is much easier than tryiing to use uni-directional CT clamps and the data it provides is revenue grade.

Hope this helps

1 Like

Thanks you - I’m sure it will help when I receive the Glowstick. I think this looks like a brilliant challenge for my son who’s into python… a good homework project!
Cheers,
Daniel