A taste of things to come? - UK smart meter data access

Integrations
141

Hi All,

n3rgy just beta launched a new feature for consumers wishing to access their own smart meter data, simply sign up and download your own data! Most of SMETS1 and now and SMETS2 are available!

For the technically minded / “prosumer” there’s also a RESTful API. See a demonstration at n3rgy data Consumer Smart Meter Demonstrator

Note. This is a free service offered to con/prosumers, for personal use only, to encourage knowledge and innovation around smart meters and the data they hold. If you wish to access multiple meters please sign up as a business account

2 Likes
142

Is the API documented anywhere?

143

Documentation is still being completed (beta service) for the consumer API but it’s pretty straight forward. If you take a look at the homebrew site linked to above you’ll find some initial guidance.

144

@Matthew_Roderick
Great stuff, thanks for sharing.
I have a couple of questions, if you don’t mind.
I just switched from SSE to Bulb and I just lost my smart meter functionality (I have SMETS1 - Elster models). I would be interested in using API, to pull data from my meters and display them inside Home Assistant.

  1. To sign up, I need to enter either MPRN or MPAN (I found 10-digit MPRN on my last SSE GAS bill and last 13-digits MPAN on my SSE Electricity bill). Entering both numbers results in error:
    “UNSUCCESSFUL CONSENT. You have not granted consent. Please check the values you submitted and retry again later. If you keep having issues, please contact support”.
    Is it because MPAN and MPRN are from SSE and now I am with Bulb and those numbers change with new supplier? (I don’t have a bill from Bulb yet, so I don’t know if I can check for new numbers)
  2. You said:

Note. This is a free service offered to con/prosumers, for personal use only, to encourage knowledge and innovation around smart meters and the data they hold. If you wish to access multiple meters please sign up as a business account

So if I would like to have data from both gas and electric meters (within same premises), do you class that as multiple? Or by multiple, you mean multiple premises?

Regards.

UPDATE: Just found Welcome Pack from Bulb and MPRN and MPAN seem to be the same, so I don’t get it why I received that error.

145

Hi @Lucas_Ch

Alas we don’t yet have access to all the smart meters and it’s partly the same reason why they go dumb. The main reason a smart meter go dumb is because the energy suppliers that originally installed it used their own proprietary system that they will not share. The n3rgy platform currently has access to all the meters that don’t go dumb (and some of the ones that do). We also tried to get access to some of the other proprietary systems out there but we were refused by the energy supplier :frowning:

You have one of these proprietary meters therefore, for the moment, there’s typically no access to anyone other than your original installing energy supplier. But there is good news as Government is currently forcing all energy suppliers to start migrating the proprietary systems the new network (provided by DCC) which will happen over the next few years. And sorry, no, there’s currently no easy way of telling when your meter will be migrated.

If you really really want to get access to the data sooner rather than later you’ll probably need to sign with an energy supplier & tariff that requires a smart meter to work (e.g. one that has different prices at different times of the day/week). In this scenario they will have no choice but to swap your meter for a SMETS2

Regarding charge, we offer domestic customers access to their own data for free, so gas & electricity from one address. We have a business plan that allows these businesses to access multiple properties data (yes we mean premise or property, will need to change that, ty) which is pay for. But before that business can access your data they will need to get your consent (which we enforce).

Hope this helps a little

Matt

2 Likes
146

@Matthew_Roderick
Thank you for such comprehensive answer.
Everywhere I go for “my usage data” I seem to hit a brick wall, even when I was still with SSE. I have been in contact with new supplier Bulb and they mentioned 2 dates for migration (from SMETS1 to SMETS2), end of this year or beginning of next year. I hope they know what they are saying, although I am not holding my breath. And because I just swapped supplier, I guess I am gonna have to wait (at the end of the day it is not that far away). Thanks again.

147

Lucas - your MPAN (elec) and MPRN (gas) are fixed and identify your supply ‘point’ to your energy supplier

1 Like
148

OK, its been a while so I figured time to revisit all of this as I have just switched energy suppliers to Octopus. Gone with a fixed tariff for the moment, but the agile one looks interesting.

It seems to me that the best option for real time data is to use the glow HID and MQTT. Whilst it does bounce the data from their servers its not going to add any noticeable delay into the picture. The only downside of using somebody else’s cloud like that is if they ever decide to turn it off, or like some other companies have done decide to add a monthly charge to it. I’d like to think in those situations there might be some hack one can do, but its not worth spending any time looking at that whilst its all working.

The only other alternative seems to be using a local device with a clamp meter and I guess for gas you could use a flow type meter inline (which would be better than the half hour reads, but a lot of effort). I’m not sure such a device exists with the simplicity and price point of the glow.

If anybody knows any other solutions then please do speak up before I commit :slight_smile:

This is a question for glow really, @clive maybe one you have some insight into as I see above you mentioned knowledge of your product. I see that it has a 3.5" TFT touch screen built in, and on the web page it talks about Applications and SDK available. I run Home Assistant (HASS) and having a remote touch screen device that could act as a display to provide data other than just gas and electricity would be really good. There are various DIY projects to do stuff, but if we could write a application for this device to interact with it a lot it would end up looking like a much more professional product than a home made one does.

So the question really is what sort of spare processing power / capacity doe the device have to run Applications, and can I view the documentation and SDKs for it that are mentioned on the website?

1 Like
149

I’ve installed a device to capture SMETS1 smart meter data every 10 secs and have been authorised to access the data via MQTT – a free service.

I’ve now got it working to the point where if I issue the following command in an SSH terminal window on an RPi running emon then I can see the data changing every 10 secs …

mosquitto_sub --cafile /home/pi/cert.pem -h glowmqtt.energyhive.com -u UserName -P PassWord -t SMART/HILD/MACaddress -p 8883.

It’s json data that looks like this …

{"elecMtr":{"0702":{"03":{"01":"000001","05":"43","04":"9B","02":"0003E8","07":"XXXXXXXXXX","08":"XXXXXXXXXXX","03":"43","00":"00","06":"00"},"00":{"07":"26D2DEF6","01":"000000000000","00":"00000009E91C","14":"02","02":"0000000019A0"},"04":{"01":"0000C8","00":"FFFFD8"},"02":{"00":"00"}}},"gasMtr":{"0702":{"00":{}}},"ts":"2020-08-21 19:37:30","hversion":"EHZBWIFI0v4","time":"5F40227A","gmtime":1598038650,"pan":{"status":"joined","nPAN":"00","join":"0","lqi":"EC"},"ets":"2020-08-21 19:37:26","error":{"lastCommand":"00","errorCode":"00"},"gid":"70B3D521E0003F41"}

Which, when interpreted, looks like this …

{
  "elecMtr": {
    "0702": {
      "03": {
        "01": "000001",
        "05": "43",
        "04": "9B",
        "02": "0003E8",
        "07": "XXXXXXXXXXXXX",
        "08": "XXXXXXXXX",
        "03": "43",
        "00": "00",
        "06": "00"
      },
      "00": {
        "14": "02",
        "07": "26D2DF00",
        "01": "000000000000",
        "00": "00000009E91C",
        "02": "0000000019A0"
      },
      "04": {
        "01": "0000C8",
        "00": "000189"
      },
      "02": {
        "00": "00"
      }
    }
  },
  "gasMtr": {
    "0702": {
      "00": {}
    }
  },
  "ts": "2020-08-21 19:37:39",
  "hversion": "EHZBWIFI0v4",
  "time": "5F402283",
  "gmtime": 1598038659,
  "pan": {
    "status": "joined",
    "nPAN": "00",
    "join": "0",
    "lqi": "EC"
  },
  "ets": "2020-08-21 19:37:36",
  "error": {
    "lastCommand": "00",
    "errorCode": "00"
  },
  "gid": "70B3D521E0003F41"
}

Buried in the json data is an error code, date & time, instantaneous power and energy over a period. And as an aside - this info is neatly provided in the Hildebrand BRIGHT mobile App updating about every 6 secs.

The question is – how to grab the data as Inputs to emoncms?

Using the PowerWall interfacer as an example – it grabs json data via an http/api call and then selects what is relevant from the json to build a data cargo that is finally published as Inputs to emoncms.

Can this approach be adapted? – an Interfacer that does MQTT connect rather than making http/api calls with the update frequency controlled by the source rather than by emon?

Another possible approach is MQTT. My trawl of the Forum has not helped much with other examples even suggesting that it might not be too reliable. And I’ve fallen at the first hurdle trying the simple ‘Hello World’ test suggested in Resources.
The response I get is …

Connection Refused: not authorised.
Error: The connection was refused.

And the mosquito log shows …

1598052121: OpenSSL Error: error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed

Any suggestions would be welcome.

150

There is no such thing as ‘MQTT Connect’. When the data is published to the broker, it then pushes (publsihes) it out to the clients that subscribe to that data topic. I’d suggest using node-red to extract the data you need from the JSON and format it and then use the emoncms node-red node (or just outgoing MQTT) to send it on to emoncms.

151

@beaylott @JaneatGlow Just had an electric ‘smart’ meter installed and have an IHD. I wonder if you know if the meter reports true instantaneous consumption, or is it simply derived from the time between pulses?

152

I’ve installed a device to capture SMETS1 smart meter data every few secs

https://www.hildebrand.co.uk/our-products/glow-stick-wifi-cad/ 12

It’s USB powered so I just stuck it into a Raspberry Pi (just for the power/no data flows) – very convenient.

The Hildebrand BRIGHT mobile App provides comprehensive smart meter data updating about every 6 secs.

Having been registered to also access the data via MQTT – a free service - I’ve now cobbled together a script to integrate 10 secs data into emoncms – power now, energy today and CUM energy.

It owes a lot to the efforts of others. And this handy tool helped …

https://curl.trillworks.com/#python

The script has HOW TO notes embedded.

glow.zip (2.6 KB)

It’s a standalone Python script not relying on another bash script being called.

And the data is revenue grade accurate – our bills are based on it – unlike CTs.

Screenshot (247)
Screenshot (247)1920×1080 129 KB

It may be of interest.

2 Likes
153

As far as I can see the following still applies.

  1. The only manufacturer to make CAD devices and to sell them direct to home users in the UK is Hildebrand they also have it appears the most flexible API access
  2. There are some other manufacturers but the rest only sell to energy companies
  3. Most energy companies do not supply CAD devices to home owners, in fact I have only seen one who does which if I remember correctly is Bulb and it is built-in to their IHD meaning it is a combined IHD/CAD and is made by Chameleon
  4. Of the CAD devices made for the UK all of them only provide data via the CAD makers own cloud services, none of them allow local access, this means if your Internet goes down you lose all access to your data and as some/you have mentioned if the CAD maker either starts charging for accessing their cloud service or turns it off or goes bust you’re screwed
  5. I have so far only seen one CAD device which supports local access which is the RainForest Eagle-200, sadly it is not certified for UK use and therefore will not be authorised for connecting to your UK smart meter, it is also not fully SMETS2 compliant

Here is some newer information.

  1. There is a ‘free’ software only solution from n3rgy, however I would expect that this is only accessing the normal SMETS data which is not updated as fast/frequently as a CAD originated data stream which can update every 30 seconds
  2. SMETS meters are supposed to be able to provide tariff information as well as consumption information, the Hildebrand can access this and even cope with the Octopus Agile tariff
  3. Reading a different thread apparently smart meters in some other European countries also have an RJ11 USB port, this port can be used with a suitable USB serial cable to read data from the meter, this approach is of course a local only feed, does anyone know if any UK SMETS meters have such a port?
154

I though an IHD was a CAD. What is the technical difference? I have a Chameleon device (IHD/CAD) from Octopus.

155

An IHD - In Home Display is a device that locally communicates with your smart meters via the HAN - Home Area Network and displays locally readings from the meters. A CAD - Consumer Access Device also links to the smart meters via the HAN and downloads data from the smart meters and can hypothetically make it available locally via WiFi. (In reality as I mentioned all UK available CAD devices upload via WiFi to the Internet to a makers cloud servers.)

The HAN uses Zigbee, it however uses a different Zigbee profile to that normally used by smart home systems. There are also plans to move to a different frequency to normal Zigbee so as to enable longer ranges to cope with a meter being some distance from say the living room especially in a block of flats meaning the meter and IHD may be a long distance from each other. There are also plans that appliances like washing machine be connectable to the smart meters so that they can intelligently schedule their operation at lower energy use and hence cheaper times, again being potentially some distance from the smart meters this is dictating the move to a different Zigbee frequency. It is also the case that now at least and likely always there will be fewer devices connected to the HAN meaning you will not benefit from the ‘mesh’ networking normally possible with Zigbee.

The IHD only displays data and gives you no access to it, the CAD makes it available to the consumer. The Hildebrand Glow USB stick is a pure CAD device as is the RainForest Eagle-200. The Chameleon IHD3 and IHD6 as the model names suggest are IHD devices but can contain a CAD as an optional module inside them. I believe Bulb are the only provider to include one, I am not convinced Octopus include this module in their Chameleon IHD. Octopus do provide a web feed of energy consumption data but this is the standard smart meter data from the DCC network and is not real time. To get real time data you need a CAD. The smart meters do not upload data in real time to the DCC to save costs, energy consumption and to reduce the load on the DCC network.

Some IHDs can have a CAD as an optional module in the IHD but as the makers do not sell direct and as many energy companies choose not to include the module to save costs and refuse to sell one as well this make life difficult to put it mildly.

1 Like
156

Just a brief update on the Carbon Co-op side as I have seen some things I recognise being mentioned… we are currently working with Salford Uni to try to test the Rainforest Eagle-200 in their smart meter sandbox. If that works we will try to work with them and Rainforest to get it put onto the CPL so it can then be paired to the system ‘normally’.

One other issue to be aware of is that you need to a DCC user to facilitate CAD pairing. As part of that they need to check its on the CPL. Note that you dont have to be a supplier in theory, you can be an ‘other user’ party. Carbon Co-op long term are hoping to become an ‘Other user’ SEC party and plug into DCC ourselves, but at the moment we are just a paying n3rgy customer which gives us access to an API for the same data they provide to consumers (as discussed above). We have also discussed the possibility of n3rgy offering CAD pairing as a service (to paying business customers) so something like that might happen in future which would make it much easier to connect CADs.

1 Like
157

John, thanks for that summary. In conclusion, we are some way off from getting an open source zigbee connection to the SMETS2 meters?

It seems that someone way back decided to put zigbee into the spec, but the manufacturers have done nothing to make this accessible. I can plug in a £20 smart plug and start reading power flow instantly over a zigbee link, but for a £420 smart meter that isn’t the case.

In July 2019, on this thread, Ben Aylott estimated £150K to develop software to access SMETS2, which suggests there is something very wrong with SMET2.

If a home owner were to use software to access their smart meter, would they need some certification?

As with a lot of IoT, there seems to be a trend to force users to go via cloud links to get their own data. This can be access via “n3rgy”. Does this provide semi “Real-Time” data?

Meanwhile, the OEM EmonTX is quick to install and reads all this data without hassle. Are we stuck with that for the next decade?

158

SMETS meters use standard Zigbee wireless frequencies (currently) but use a different Zigbee ‘profile’ to that normally used by smart home products. I believe the specs should be available but in order to be allowed to be connected to the SMETS HAN i.e. the Zigbee network used by the meter, IHD and optionally CAD the device has to be officially tested and certified. Chameleon as an example will have done this but RainForest have not.

The legitimate justification for requiring devices to be tested and certified is security and probably safety, again there does also seem to be some use of this as an obstacle.

This testing and certification has a cost and designing and building a CAD has a cost. There probably is some case of these costs being inflated to keep people out but to be fair there will be some legitimate costs.

My belief is that n3rgy is using data uploaded by all SMETS meters via the DCC network and this is far less frequently updated than data obtainable via a CAD.

In some European countries smart meters have a special serial port called a P1 port and it is possible to get an appropriate USB cable and plug that in to a computer and read the data and this approach does not require approval or certification. I don’t yet myself have a UK smart meter so I don’t know if any UK smart meters have this same port.
https://www.amazon.co.uk/Kaifa-Dutch-Smart-Meter-ft232r-ISKRA-ME-382-EN-MT3/dp/B07DFM2H6S

159

Thing is, I don’t mind the HAN being private and the devices needing paired. I do object to that data not being made available out of the CAD/IHD. Even a simple USB serial interface would suffice (and cost would be minimal). But making the data totally inaccessible is plain wrong. This of course is a result of the regulations being written to favour the supplier and not the customer.

3 Likes
160

I totally agree with you, the fact that only one manufacturer will even consider selling you hardware shows that the rest are not interested in end users and are operating a cartel.

2 Likes