A taste of things to come? - UK smart meter data access

Lucas - your MPAN (elec) and MPRN (gas) are fixed and identify your supply ‘point’ to your energy supplier

1 Like

OK, its been a while so I figured time to revisit all of this as I have just switched energy suppliers to Octopus. Gone with a fixed tariff for the moment, but the agile one looks interesting.

It seems to me that the best option for real time data is to use the glow HID and MQTT. Whilst it does bounce the data from their servers its not going to add any noticeable delay into the picture. The only downside of using somebody else’s cloud like that is if they ever decide to turn it off, or like some other companies have done decide to add a monthly charge to it. I’d like to think in those situations there might be some hack one can do, but its not worth spending any time looking at that whilst its all working.

The only other alternative seems to be using a local device with a clamp meter and I guess for gas you could use a flow type meter inline (which would be better than the half hour reads, but a lot of effort). I’m not sure such a device exists with the simplicity and price point of the glow.

If anybody knows any other solutions then please do speak up before I commit :slight_smile:

This is a question for glow really, @clive maybe one you have some insight into as I see above you mentioned knowledge of your product. I see that it has a 3.5" TFT touch screen built in, and on the web page it talks about Applications and SDK available. I run Home Assistant (HASS) and having a remote touch screen device that could act as a display to provide data other than just gas and electricity would be really good. There are various DIY projects to do stuff, but if we could write a application for this device to interact with it a lot it would end up looking like a much more professional product than a home made one does.

So the question really is what sort of spare processing power / capacity doe the device have to run Applications, and can I view the documentation and SDKs for it that are mentioned on the website?

1 Like

I’ve installed a device to capture SMETS1 smart meter data every 10 secs and have been authorised to access the data via MQTT – a free service.

I’ve now got it working to the point where if I issue the following command in an SSH terminal window on an RPi running emon then I can see the data changing every 10 secs …

mosquitto_sub --cafile /home/pi/cert.pem -h glowmqtt.energyhive.com -u UserName -P PassWord -t SMART/HILD/MACaddress -p 8883.

It’s json data that looks like this …

{"elecMtr":{"0702":{"03":{"01":"000001","05":"43","04":"9B","02":"0003E8","07":"XXXXXXXXXX","08":"XXXXXXXXXXX","03":"43","00":"00","06":"00"},"00":{"07":"26D2DEF6","01":"000000000000","00":"00000009E91C","14":"02","02":"0000000019A0"},"04":{"01":"0000C8","00":"FFFFD8"},"02":{"00":"00"}}},"gasMtr":{"0702":{"00":{}}},"ts":"2020-08-21 19:37:30","hversion":"EHZBWIFI0v4","time":"5F40227A","gmtime":1598038650,"pan":{"status":"joined","nPAN":"00","join":"0","lqi":"EC"},"ets":"2020-08-21 19:37:26","error":{"lastCommand":"00","errorCode":"00"},"gid":"70B3D521E0003F41"}

Which, when interpreted, looks like this …

{
  "elecMtr": {
    "0702": {
      "03": {
        "01": "000001",
        "05": "43",
        "04": "9B",
        "02": "0003E8",
        "07": "XXXXXXXXXXXXX",
        "08": "XXXXXXXXX",
        "03": "43",
        "00": "00",
        "06": "00"
      },
      "00": {
        "14": "02",
        "07": "26D2DF00",
        "01": "000000000000",
        "00": "00000009E91C",
        "02": "0000000019A0"
      },
      "04": {
        "01": "0000C8",
        "00": "000189"
      },
      "02": {
        "00": "00"
      }
    }
  },
  "gasMtr": {
    "0702": {
      "00": {}
    }
  },
  "ts": "2020-08-21 19:37:39",
  "hversion": "EHZBWIFI0v4",
  "time": "5F402283",
  "gmtime": 1598038659,
  "pan": {
    "status": "joined",
    "nPAN": "00",
    "join": "0",
    "lqi": "EC"
  },
  "ets": "2020-08-21 19:37:36",
  "error": {
    "lastCommand": "00",
    "errorCode": "00"
  },
  "gid": "70B3D521E0003F41"
}

Buried in the json data is an error code, date & time, instantaneous power and energy over a period. And as an aside - this info is neatly provided in the Hildebrand BRIGHT mobile App updating about every 6 secs.

The question is – how to grab the data as Inputs to emoncms?

Using the PowerWall interfacer as an example – it grabs json data via an http/api call and then selects what is relevant from the json to build a data cargo that is finally published as Inputs to emoncms.

Can this approach be adapted? – an Interfacer that does MQTT connect rather than making http/api calls with the update frequency controlled by the source rather than by emon?

Another possible approach is MQTT. My trawl of the Forum has not helped much with other examples even suggesting that it might not be too reliable. And I’ve fallen at the first hurdle trying the simple ‘Hello World’ test suggested in Resources.
The response I get is …

Connection Refused: not authorised.
Error: The connection was refused.

And the mosquito log shows …

1598052121: OpenSSL Error: error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed

Any suggestions would be welcome.

There is no such thing as ‘MQTT Connect’. When the data is published to the broker, it then pushes (publsihes) it out to the clients that subscribe to that data topic. I’d suggest using node-red to extract the data you need from the JSON and format it and then use the emoncms node-red node (or just outgoing MQTT) to send it on to emoncms.

@beaylott @JaneatGlow Just had an electric ‘smart’ meter installed and have an IHD. I wonder if you know if the meter reports true instantaneous consumption, or is it simply derived from the time between pulses?

I’ve installed a device to capture SMETS1 smart meter data every few secs

https://www.hildebrand.co.uk/our-products/glow-stick-wifi-cad/ 12

It’s USB powered so I just stuck it into a Raspberry Pi (just for the power/no data flows) – very convenient.

The Hildebrand BRIGHT mobile App provides comprehensive smart meter data updating about every 6 secs.

Having been registered to also access the data via MQTT – a free service - I’ve now cobbled together a script to integrate 10 secs data into emoncms – power now, energy today and CUM energy.

It owes a lot to the efforts of others. And this handy tool helped …

https://curl.trillworks.com/#python

The script has HOW TO notes embedded.

glow.zip (2.6 KB)

It’s a standalone Python script not relying on another bash script being called.

And the data is revenue grade accurate – our bills are based on it – unlike CTs.

It may be of interest.

2 Likes

As far as I can see the following still applies.

  1. The only manufacturer to make CAD devices and to sell them direct to home users in the UK is Hildebrand they also have it appears the most flexible API access
  2. There are some other manufacturers but the rest only sell to energy companies
  3. Most energy companies do not supply CAD devices to home owners, in fact I have only seen one who does which if I remember correctly is Bulb and it is built-in to their IHD meaning it is a combined IHD/CAD and is made by Chameleon
  4. Of the CAD devices made for the UK all of them only provide data via the CAD makers own cloud services, none of them allow local access, this means if your Internet goes down you lose all access to your data and as some/you have mentioned if the CAD maker either starts charging for accessing their cloud service or turns it off or goes bust you’re screwed
  5. I have so far only seen one CAD device which supports local access which is the RainForest Eagle-200, sadly it is not certified for UK use and therefore will not be authorised for connecting to your UK smart meter, it is also not fully SMETS2 compliant

Here is some newer information.

  1. There is a ‘free’ software only solution from n3rgy, however I would expect that this is only accessing the normal SMETS data which is not updated as fast/frequently as a CAD originated data stream which can update every 30 seconds
  2. SMETS meters are supposed to be able to provide tariff information as well as consumption information, the Hildebrand can access this and even cope with the Octopus Agile tariff
  3. Reading a different thread apparently smart meters in some other European countries also have an RJ11 USB port, this port can be used with a suitable USB serial cable to read data from the meter, this approach is of course a local only feed, does anyone know if any UK SMETS meters have such a port?
1 Like

I though an IHD was a CAD. What is the technical difference? I have a Chameleon device (IHD/CAD) from Octopus.

An IHD - In Home Display is a device that locally communicates with your smart meters via the HAN - Home Area Network and displays locally readings from the meters. A CAD - Consumer Access Device also links to the smart meters via the HAN and downloads data from the smart meters and can hypothetically make it available locally via WiFi. (In reality as I mentioned all UK available CAD devices upload via WiFi to the Internet to a makers cloud servers.)

The HAN uses Zigbee, it however uses a different Zigbee profile to that normally used by smart home systems. There are also plans to move to a different frequency to normal Zigbee so as to enable longer ranges to cope with a meter being some distance from say the living room especially in a block of flats meaning the meter and IHD may be a long distance from each other. There are also plans that appliances like washing machine be connectable to the smart meters so that they can intelligently schedule their operation at lower energy use and hence cheaper times, again being potentially some distance from the smart meters this is dictating the move to a different Zigbee frequency. It is also the case that now at least and likely always there will be fewer devices connected to the HAN meaning you will not benefit from the ‘mesh’ networking normally possible with Zigbee.

The IHD only displays data and gives you no access to it, the CAD makes it available to the consumer. The Hildebrand Glow USB stick is a pure CAD device as is the RainForest Eagle-200. The Chameleon IHD3 and IHD6 as the model names suggest are IHD devices but can contain a CAD as an optional module inside them. I believe Bulb are the only provider to include one, I am not convinced Octopus include this module in their Chameleon IHD. Octopus do provide a web feed of energy consumption data but this is the standard smart meter data from the DCC network and is not real time. To get real time data you need a CAD. The smart meters do not upload data in real time to the DCC to save costs, energy consumption and to reduce the load on the DCC network.

Some IHDs can have a CAD as an optional module in the IHD but as the makers do not sell direct and as many energy companies choose not to include the module to save costs and refuse to sell one as well this make life difficult to put it mildly.

1 Like

Just a brief update on the Carbon Co-op side as I have seen some things I recognise being mentioned… we are currently working with Salford Uni to try to test the Rainforest Eagle-200 in their smart meter sandbox. If that works we will try to work with them and Rainforest to get it put onto the CPL so it can then be paired to the system ‘normally’.

One other issue to be aware of is that you need to a DCC user to facilitate CAD pairing. As part of that they need to check its on the CPL. Note that you dont have to be a supplier in theory, you can be an ‘other user’ party. Carbon Co-op long term are hoping to become an ‘Other user’ SEC party and plug into DCC ourselves, but at the moment we are just a paying n3rgy customer which gives us access to an API for the same data they provide to consumers (as discussed above). We have also discussed the possibility of n3rgy offering CAD pairing as a service (to paying business customers) so something like that might happen in future which would make it much easier to connect CADs.

1 Like

John, thanks for that summary. In conclusion, we are some way off from getting an open source zigbee connection to the SMETS2 meters?

It seems that someone way back decided to put zigbee into the spec, but the manufacturers have done nothing to make this accessible. I can plug in a £20 smart plug and start reading power flow instantly over a zigbee link, but for a £420 smart meter that isn’t the case.

In July 2019, on this thread, Ben Aylott estimated £150K to develop software to access SMETS2, which suggests there is something very wrong with SMET2.

If a home owner were to use software to access their smart meter, would they need some certification?

As with a lot of IoT, there seems to be a trend to force users to go via cloud links to get their own data. This can be access via “n3rgy”. Does this provide semi “Real-Time” data?

Meanwhile, the OEM EmonTX is quick to install and reads all this data without hassle. Are we stuck with that for the next decade?

SMETS meters use standard Zigbee wireless frequencies (currently) but use a different Zigbee ‘profile’ to that normally used by smart home products. I believe the specs should be available but in order to be allowed to be connected to the SMETS HAN i.e. the Zigbee network used by the meter, IHD and optionally CAD the device has to be officially tested and certified. Chameleon as an example will have done this but RainForest have not.

The legitimate justification for requiring devices to be tested and certified is security and probably safety, again there does also seem to be some use of this as an obstacle.

This testing and certification has a cost and designing and building a CAD has a cost. There probably is some case of these costs being inflated to keep people out but to be fair there will be some legitimate costs.

My belief is that n3rgy is using data uploaded by all SMETS meters via the DCC network and this is far less frequently updated than data obtainable via a CAD.

In some European countries smart meters have a special serial port called a P1 port and it is possible to get an appropriate USB cable and plug that in to a computer and read the data and this approach does not require approval or certification. I don’t yet myself have a UK smart meter so I don’t know if any UK smart meters have this same port.
https://www.amazon.co.uk/Kaifa-Dutch-Smart-Meter-ft232r-ISKRA-ME-382-EN-MT3/dp/B07DFM2H6S

Thing is, I don’t mind the HAN being private and the devices needing paired. I do object to that data not being made available out of the CAD/IHD. Even a simple USB serial interface would suffice (and cost would be minimal). But making the data totally inaccessible is plain wrong. This of course is a result of the regulations being written to favour the supplier and not the customer.

3 Likes

I totally agree with you, the fact that only one manufacturer will even consider selling you hardware shows that the rest are not interested in end users and are operating a cartel.

2 Likes

It’s also down to economics. The number of people that would actually make use of being able to locally access the data is small, very small. Add in the cost of supporting the small number of people using it, it just doesn’t make sense to do.

Today n3rgy collects data from the meter once a day. i.e. it downloads your data in the early hours of the morning and makes it available to those who have been granted permission to access it. We’re also piloting increasing this collection frequency to every 2 hours (and potentially more frequently in the future) . With regards to CAD pairing, we would be happy to also offer this service when and if a market of CADs becomes available but as previously mentioned, there’s very few CADs available on the market today which aren’t already part of a closed system.

Out of interested, what do you guys believe the imperative for real time data is? If you have the tariffs (which always need to be published the day before) you already have the mechanism to automatically minimise the cost of power hungry appliances!?

This argument is a cop out.

Whilst I agree it might be small, you could build a smaller device, no (pretty useless) display, and allow devices to connect via WiFi (like a massive number of smart switches do) to get the info. You could even then do updates to fix bugs and allowing API access to the data is easy.

Octopus provide API access to probably a small part of their user base. Are you suggesting they shouldn’t?

My point is, it should have been built into the specification to provide local access. Commercially, no one is going to do it unless they have to generally as they have too much self interest.

1 Like

Should I export, import or send to battery/charge EV.

This whole forum is based on users wanting real time energy use gathering!!!

3 Likes

The main interest is to measure exports/imports for renewables generators.

Admittedly that is going to be a bigger market in other countries with better solar resource or subsidies, and it seems each country is implementing something different for its smart meters, with the only common interfaces being the LED display and the pulsing LED.

1 Like

I would tend to agree that the near-real-time data isnt needed for a lot of things. But I think one clear use case for a CAD is if you want to adjust power usage of a smart appliance (battery/EV charger/diverter) according to actual generation output (or some other control signal) then you need near real-time data from the grid feed. A lot of current energy smart appliances come with these grid feed monitors which use CT clips for this purpose (and in some cases G98 compliance purposes). But most are not interoperable and we are increasingly seeing situations in our work now where our members have a battery from one manufacturer, diverter from another, EV charger from yet another and they all have a CT clip watching the grid feed so they can follow PV output etc. but then racing each other to grab any excess with no coordination or prioritisation possible. Seems like an interoperable CAD+energy management system would be better to sit in middle here dealing with the control priority issues as well as maybe reducing costs by not having multiple of these devices clipped on (and one would hope a basic Zigbee SEP CAD would end up being a bit cheaper than a CT monitor).

1 Like