Octopus energy : Smart Meter Feed

JaneatGlow · 2020-02-08T11:33:25.095Z

Excuse me for not being able to do inline replies.

It does not look as though the regulation about mandating the IHD will be changed soon unfortunately from what we are aware of. The ability to monitor your energy anytime, anywhere is invaluable in our view. There is some interpretation possible

We hope to offer the GlowStick as SMETS2 as well but need enough orders to justify the manufacturing cost and minimum order quantities.

Sorry, hadn’t picked up the other thread - just had a very quick skim. So we already have the service the n3rgy offer and have been an approved Other User with SECAS (the Smart Energy Code Administrator) since July 2019.

Our SMETS1 devices (prior to the GlowStick we had a wired CAD) have been on the market since 2016 and are in thousands of homes across the UK. We are happy to give our customers API access to their data, and MQTT if required. We also support a number of research projects - and are like to have people develop their own Apps and algorithms that can be added to the ecosystem.

fyi, the reason Octopus can read your mum’s SMETS1 meters as smart may well be because they and First Utility are on the Secure SMSO (does she have Secure Meters?).

The Zigbee protocol is in the public domain, it is the required security protocols (based on keys) which are the limitation.

borpin · 2020-02-08T11:50:06.941Z

JaneatGlow:

Excuse me for not being able to do inline replies.

If you highlight some text, it will offer to add it as a quote to the reply (even from different threads).

JaneatGlow:

The ability to monitor your energy anytime, anywhere is invaluable in our view.

Yes definitely, but anyone buying a CAD/IHD other than that provided by the supplier is 99% wanting to do something else with the data .

I presume you can have 2 CADs connected to a meter at once (that is an assumption on my part )

Do you supply your CADs to specific suppliers & can the API be accessed or do they lock it down? Knowing the CAD from a specific supplier is one of yours and the API is available (if not advertised) might influence the choice of supplier.

JaneatGlow:

We are happy to give our customers API access to their data, and MQTT if required.

@TrystanLea might be worth an integration with emoncms?

JaneatGlow · 2020-02-08T12:01:23.737Z

borpin:

Yes definitely, but anyone buying a CAD/IHD other than that provided by the supplier is 99% wanting to do something else with the data .

I presume you can have 2 CADs connected to a meter at once (that is an assumption on my part )

Do you supply your CADs to specific suppliers & can the API be accessed or do they lock it down? Knowing the CAD from a specific supplier is one of yours and the API is available (if not advertised) might influence the choice of supplier.

Thanks Brian for teaching me this technique! The good news is that we also sell our devices direct to the public - as these choices shouldn’t be limited to who your supplier is. There are suppliers who bought our GlowStick for SMETS1 meters but obviously they can’t install them anymore - they included Octopus, Usio, Eversmart. Anyone with Secure Meters should be able to connect our GlowStick. Our SMETS2 IHD/CAD is now being sold by a few suppliers, including Igloo and some non-domestic suppliers.

Actually we find a lot of our direct customers don’t take the API - they just want the benefit of using our Bright app to monitor their consumption as it suits them, not confined to their home.

JaneatGlow · 2020-02-08T12:04:19.901Z

borpin:

@TrystanLea might be worth an integration with emoncms?

We’d be happy to discuss this with whoever is the right person at emoncms - with the proviso that I’m not a techie so can only take the conversation so far before I’d need to get colleagues involved.

If people with SMETS2 meters are happy to have their data on an historic basis (as in not real-time), we can support that at no charge - although, as mentioned above, as a DCC Other User, we need to have the individual go through a verification and validation / consent process first.

JaneatGlow · 2020-02-08T13:04:51.408Z

borpin:

I presume you can have 2 CADs connected to a meter at once (that is an assumption on my part )

Sorry - forgot to answer this. Yes, you can have more than one CAD connected to a meter - for SMETS2 it is apparently a maximum of 6 devices. SMETS1 varies by the meter and its related Head End. Secure Meters that are SMETS1 can have 5 devices connected.

ian · 2020-02-08T15:05:38.980Z

Hi Brian

It is the Hildebrand Glow device for SMETS2 I am using that is referred to in this thread:-

Graph Missing Data - What does it mean? Emoncms

That should be ok then. I wasn’t saying that directly, but matching your feed to the data updates is key to using fixed interval feeds. If the data can only change half hourly (HH data is a common metering standard) then there isn’t much value in sampling or saving more frequently. A 30min feed would be ideal if you only update the inputs every 30mins. Maybe you can use the HH to trigger the publish to emoncms? Or just arrange things so that the 30min update happens every 30mins around 10…

@JaneatGlow clearly is cautious about self promotion. I have to say I am very pleased with the device.
The support is first class with email response frequently the same day and sometimes within minutes!

I was interested in the first place to have something that displays up to date current cost of energy even when on Octopus Agile TOU tariff. I also wanted to be able to check my Octopus bills as its fair to say that they have had a few glitches with SMETS2 meters which Octopus fully acknowledge.

With the Glow API and Octopus API I can pull accurate smart meter data and TOU tariff data with a crude python programme that has start and end date and gives a totalled print out within a few seconds (depending on the length of the period). I should probably also say that my Octopus bills have always been correct within a few pence which I guess are rounding errors.

All in all a very satisfied customer of both Octopus and Hildebrand…

borpin · 2020-02-08T15:18:26.646Z

JaneatGlow:

The good news is that we also sell our devices direct to the public - as these choices shouldn’t be limited to who your supplier is.

Ah, but if get one from the supplier it is free!

ian:

It is the Hildebrand Glow device for SMETS2 I am using that is referred to in this thread:-

Ah OK, how are you doing the integration? Python? Care to post it on a new thread? I can’t test, but part of my reluctance on the meter front was the inability to get the data. If that is actually now possible I may be persuaded especially if they come with one of the Glow devices!

TrystanLea · 2020-02-10T10:32:26.453Z

borpin:

@TrystanLea might be worth an integration with emoncms?

Interested to learn more, thanks for your input @JaneatGlow, interested as well to hear about your implementation @ian

TrystanLea · 2020-02-10T10:41:09.261Z

I’ve got a SMETS2 meter installed by Octopus, but there seems to be an issue with communication, still waiting to see half hourly readings and to then be switched on to Agile.

Im not seeing anything on my IHD either. I assume the Hildebrand Glow needs the rest of the SMETS2 meter setup and working before I’d be able to access data?

ian · 2020-02-10T11:27:29.278Z

Hi

As I understand it the Hildebrand Glow is just another IHD on your local zigbee HAN. Therefore until the Octopus IHD works the Glow won’t work. The main IHD difference is that it has a wifi connection as well and Hildebrand being licensed by the DCC (in the south west. I don’t know who covers Wales) are able to read the meter directly which they do every 5 to 6 seconds. The data is then available on Hildebrands Bright App and via API. They also publish data via MQTT

The API details are here:-

docs.glowmarkt.com

GlowmarktApiDataRetrievalDocumentationIndividualUser.pdf

212.50 KB

The MQTT is undocumented and is a json object but with some zigbee documentation I was able to work out what the gas and electric meter fields are.

I will confirm with Hidebrand that the zigbee documentation is OK to publicly share and if so will post details here or maybe @JaneatGlow will respond.

I am processing the MQTT data in nodered but have some issues publishing to emoncms as described in this thread:-

Graph Missing Data - What does it mean? Emoncms

That should be ok then. I wasn’t saying that directly, but matching your feed to the data updates is key to using fixed interval feeds. If the data can only change half hourly (HH data is a common metering standard) then there isn’t much value in sampling or saving more frequently. A 30min feed would be ideal if you only update the inputs every 30mins. Maybe you can use the HH to trigger the publish to emoncms? Or just arrange things so that the 30min update happens every 30mins around 10…

borpin · 2020-02-10T11:46:26.475Z

ian:

The main IHD difference is that it has a wifi connection as well and Hildebrand being licensed by the DCC (in the south west. I don’t know who covers Wales) are able to read the meter directly which they do every 5 to 6 seconds. The data is then available on Hildebrands Bright App and via API.

Am I reading this right, the API does not connect directly with the IHD, but with the central data store once the CAD has read the data and sent it on?

ian · 2020-02-10T11:59:47.977Z

Correct. I understand that the Smart Meter regulations are such that users cannot directly interface with there own HAN unless they are registered with the DCC. Hildebrand have registered and can therefore make the data available. I understand registration costs are in the tens of thousands of pounds.

I personally think lack of user access to their own energy HAN network is wrong but typical of governments. I guess security is an issue but I don’t think my energy usage is likely to be of interest to anyone other than me. (My meters are external and publicly accessible anyway)

borpin · 2020-02-10T12:02:03.716Z

ian:

I personally think lack of user access to their own energy HAN network is wrong but typical of governments.

Agree 100%.

ian:

I guess security is an issue

Giving the data to a third party and then getting it back is likely to be a bigger security risk! Hardly any different to having a home router with the password stamped on the back of it.

Typical of technology illiterate government officials and ministers.

ian · 2020-02-10T12:08:14.491Z

Hildebrand advised that the data from the meter to them is encrypted as part of the licensing provisions.
Presumably also encrypted back to their APP.
However the MQTT data is just subscribed with a username and password. The API is also needs username and password but provides a token for data access which changes every few days.

borpin · 2020-02-10T12:17:38.012Z

ian:

Hildebrand advised that the data from the meter to them is encrypted

I’d have thought it is the IHD/CAD that does the encryption and not the meter itself (I could be wrong). Although, having said that Octopus identified that a CAD can only ever be paired to a meter once - where the pairing was tested in the factory, those devices would never connect to a meter. So it could be a one time key creation at initial pairing so the data between the meter and the CAD is encrypted. If so, the whole thing makes more sense.

I’m going to contradict myself, if the CAD can’t decode the data, how does the IHD know what the current consumption is. I can’t believe the data goes Meter → CAD → central store → IHD.

Having said that, it is quite possibly the way is had been thrown together, so yes I can believe it. I’m going back to my position of ‘over my dead body’ or when financially I have to, to get a decent tariff.

ian · 2020-02-10T19:33:57.809Z

@borpin

The smart security behind the GB Smart Metering System

borpin · 2020-02-10T20:27:08.959Z

Thanks @ian. Not come across that before. If NCSC have been involved then my confidence rises significantly - they do actually know what they are doing. If they say the scurity issues can be managed then no one should question that. It’d be like telling your surgeon what to do.

JaneatGlow · 2020-02-15T10:46:01.315Z

TrystanLea:

I’ve got a SMETS2 meter installed by Octopus, but there seems to be an issue with communication, still waiting to see half hourly readings and to then be switched on to Agile.

Im not seeing anything on my IHD either. I assume the Hildebrand Glow needs the rest of the SMETS2 meter setup and working before I’d be able to access data?

Sorry for the delay in getting back to you Trystan - you are correct, if your underlying meters aren’t performant then we can’t perform the final miracle… much as we’d like to! Once they are working then all is fine - and if you didn’t want real-time and granular, we can also provide half-hourly data on a delay because we are a DCC Other User, approved by SECAS. This is through our own DCC connector. Hope Octopus are able to get you sorted out soon.

JaneatGlow · 2020-02-15T10:58:14.479Z

ian:

I will confirm with Hidebrand that the zigbee documentation is OK to publicly share and if so will post details here or maybe @JaneatGlow will respond.

Hi Ian, you are welcome to share the Zigbee documentation, it is publicly available information and is the Zigbee Smart energy profile specification. To make it easier - here is the link - hosted on our own secure servers https://oc2.energyhive.com/owncloud/s/fGCW460aFmtm3JA.

For those who quite rightly express concerns about our security credentials, we are ISO 27001 certified and also passed the Smart Energy Code Administrator’s Privacy and Security Audits which are mandatory before you can be approved as a DCC Other User. The process is described here: https://smartenergycodecompany.co.uk/about-security-and-privacy-obligations/

clive · 2020-02-15T11:53:09.318Z

Hi Brian
In the order in your post

There are TWO directions of travel for meter data - meter → comms hub → wan (gprs or 3g) → head end (smets1 an smso, smets2 DCC) and then onwards to the supplier - in the smets 2 world not only is the data encrypted over the wan link as you would expect where readings are involved they are further encrypted with individual meter/requesting party specific public/private key pair - so even if the payload in intercepted en route the important data cannot be recovered by anyone other than the requesting party.
The second direction is over the PAN from comms hub to CADs - this is encrypted using SEP with zigbee keys being burned into the CAD, notified via the head end to the coms hub - an initial join takes place using these keys and then new random keys are generated and exchanged between the comms hub and the CAD - this is repeated periodically and it means that the only possible way of sniffing the traffic is to a) know the initial CAD key and b) intercepting traffic during the initial join
CADs can be repaired as often as required - you have to complete the process of removing them from the old meter set and then they are free to join a new set - most unlike Octopus to get that wrong

The upshot is that in either direction your data is secure.