A taste of things to come? - UK smart meter data access

Sadly whilst I agree ‘we’ should be able to access our data, ‘we’ are a small minority and hence not a market worth investing the time / cost in.

Great idea, in theory, sadly USB by its very nature is a bi-directional protocol, once again ‘we’ are looking at if from a practical use perspective, however there are other that would look to exploit this to their own advantage. USB connections introduce an attack vector for any device. If you look back at all the ‘hacks’ of ISP routers to add features etc, it all starts with a serial port.

I’m 100% in agreement, I’d love to get all the juicy goodness out of my smart meters, for free, without having to buy external hardware to replicate the same functionality. But sadly i’m resigned to the fact it’s not going to happen.

Hehe, this reminds me of a personal tale… many moons ago (early 1990s) when I was the Technical Editor of a reasonably popular 16-bit computer magazine, my Editor told me “Neil, this ‘Comms Corner’ stuff at the back of the mag - I’m sure it’s all good, but it’s all very geeky. Are you sure that people are really interested in that whole modem, BBS and Internet stuff you write about?”

Several years later we all met up in our local pub for a mag/publishing-house reunion - all arranged of course, via the Internet, especially ironic because our Ed was now resident in a country literally the other side of the world, and would never have found out about it otherwise.

He bought me many drinks and apologised profusely for his lack of vision (and faith in me) that night. :smiley:

The point I’m trying to make (albeit rather nostalgically and self-indulgently) is that ‘from small acorns, sometimes big oak trees grow’. There is a market for this stuff, just as surely as there is now a market for routers in people’s houses where once-upon-a-time a simple modem would do. And people at home have LANs, and WiFi rather than simply plugging their personal PC directly into the house phone via a comms-port or USB.

This stuff gets built upon, because geeks like me (and others, far, far better at it than me), like to poke around in things, and find strategies and techniques that (sooner or later) enable the creation of other, better standalone devices that… just work.

The market for displays, logging, analysis, charting, etc, is potentially large - possibly even huge… once people are ‘turned on’ to the idea of actually watching what their energy usage is doing. And surely, that’d be good for everyone, especially so, given the whole point of energy monitoring in the first place?

Take the market for home weather stations, for example: no self-respecting device on the market these days comes without some form of PC interface, logging and/or the ability to upload data to a cloud, or the PWC Underground, etc. A few years ago, people like Oregon Scientific said that those sorts of features were not in demand, and made them too expensive - now, they have many more competitors, and there are vast ranges of weather devices ostensibly for home use, many of which have these features in by default, for not very much money.

I can see the same happening to energy home-displays, in the future. Sure, most people will not want to go to the trouble of making them interface, writing code, etc… but they will want the fruits of those labours, and thus the functionality (and perhaps even some standardised, interchangeable communication standards) would be de rigeur in all the ‘decent’ (i.e. best-selling) devices.

A plea to Chameleon, et al: Let’s save some energy… let’s ditch the usual approach of product-marketing where manufacturer X falsely limits product-development, designs-in limitations and milks the market for every damned penny, reselling the same basically-hamstrung features in differently-shaped boxes over-and-over, whilst creating a mountain of environmentally-harmful plastic and electronic junk that some barefoot kid in a scrapyard in Asia will end up wading through pools of electrolyte to dismantle. Let’s just aim for something truly useful, first time, and maybe enable us to keep using it gainfully for more than just one iPhone season, and perhaps even extend the usefulness of, in the future?

Radical, I know… but I can dream, can’t I? :smiley:

I’m not. While the lights are still on, there is always hope. :bulb:

Oh how true, indeed back in 2008 I was one of the merry band of people hacking away at the serial output from CurrentCost in house power meters, sticking it into a round robin database :slight_smile:

1 Like

@MattB, Is there an API for this site?

@borpin Consumer api is still beta but examples can be found at n3rgy · GitHub :smiley:

3 Likes

Yes! We provide consumption (import) data and production (export) data

@markb DCC network relies on deployment to cover the last few percent via a mesh network (from smart meters within range of the main wan)

It amazes me there is no ability to attach an external antenna to the meters. I suspect part of my issue is that I effectively live in a bit of a farday cage (light steel frame & foil backed PIR ‘tea cosy’ insulation). An extenal antenna would make all the difference.

So the Smart meters themselves create a mesh? If so, that is new information and quite interesting :slight_smile:. Presumably it is the WAN interface creating the mesh, rather than the meters themselves as they are on the HAN.

Apparently, some do have this capability… see link lower down.

Latter part is indeed correct. It’s the comms unit (which usually sits atop the electricity meter, so I am told), not the meter per se, which does the mesh side (which can be either local zigbee, I think, or just plain direct-talk to cellular phone network, if there are no nearby houses to form a mesh with.

This article I found quite useful for background: Technical information on SMETS 2 Smart Metering - The full story
and it’s also got a list of all the currently-approved comms units, and which ones use cellular network, vs zigbee, and which ones have antenna capability.

Bear in mind also that there is another ‘north-south divide’ - it appears that comms-hubs in the north (which all communicate via Arqiva systems) do it solely via long-range radio. Whereas southern region hubs use the Telefonica comms-systems, and have access to either cellular (I’m guessing this is piggybacked somewhere down a side-alley of the O2 mobile network but chinese-walled from normal mobile phone comms), or by local mesh.

Only the latter (i.e. southern region) appear to have external antennas, and most of the southern comms units are Toshiba made. Mine is an SKU1, Toshiba pedigree, which surprises me, because I’m in a suburban area in a city of 80k people. Maybe there just aren’t enough other smart-meters locally yet to be able to form a mesh with, so they’ve chosen the cellular only approach. Or maybe it’s just cheaper? If the latter, it’s a bit shortsighted, because no mesh will ever get created if they continue that approach (AES was my installer).

1 Like

Before anyone pulls me up on it, I’ve not actually proven to myself yet that any ‘WAN mesh’ is done with Zigbee… and this may indeed not be the case!

But there is (or can be) a WAN mesh of some kind, in the southern region, it seems. That much is established! :smiley:

EDIT - seems it probably is Zigbee, after all - or at least some variant thereof. The datasheet for the Toshiba SKU2 (which can do cellular or mesh modes) shows IEEE 802.15.4g-2012 as the mesh protocol standard, and that’s basically Zigbee.

More datasheets and useful data on comms hubs from the SmartDCC site here:

I’ve been contacted by Good Energy asking whether I would like to have a smart meter installed. So far I have established that it gets installed by SMS-Plc and they supply a Chameleon IHD3. This has the option of having CAD (IHD3-CAD). I have asked the question whether I can get the CAD version supplied, or whether it can be upgraded. I have also asked Good Energy whether I can get access to the data. SMS have told me that question needs to go to Good Energy, even though the data will probably come from their servers.

there are two regions for WAN - north and south, south is gsm cellular provision (telefonica), north is mesh (arquiva)

Good evening.

Did anybody actually start using n3rgy API and managed to pull data from home smart meters? My SMETS1 meter(s) are finally smart again after 4 months and I was able to sign up at n3rgy website and give consent to collect data by them. I know I will probably sound a little bit like a weirdo here, but still not 100% sure this is really free service for personal use (Nothing is really free these days). End of the day, pricing section goes into thousands of pounds. :wink:
My understanding is, data is being collected once a day and I can view that data using RestAPI?

Hi Lucas,

I did! The steps I used were to call the following

For electricity:

GET https://consumer-api.data.n3rgy.com/electricity/consumption/1?start={startDate}&end={endDate}

And for gas:

GET https://consumer-api.data.n3rgy.com/gas/consumption/1?start={startDate}&end={endDate}

{startDate} - The first date in the range, this is in the form YYYYMMDD
{endDate} - The last date in the range, this is in the form YYYYMMDD

AND an Authorization header with the IHD MAC that you enter on the portal page in capitals

The responses I get are formed like this:

{
    "resource":"/gas/consumption/1",
    "responseTimestamp":"YYYY-MM-DDTHH:mm:SS.Z",
    "start":"YYYYMMDD0030",
    "end":"YYYYMMDD0000",
    "granularity":"halfhour",
    "values":[{
        "value":0.0,
        "timestamp":"YYYY-MM-DD 00:30"
    }, 
    ...,
    {
        "value":0.0,
        "timestamp":"YYYY-MM-DD 00:00"
    }],
    "availableCacheRange":{
        "start":"YYYYMMDD1930",
        "end":"YYYYMMDD0330"
    },
    "unit":"m3"
}

Let me know if this makes sense or if I can help in anyway, it seems clear to me but might not be clear to others so would like to fill in the blanks

1 Like

In fact, it even works if you exclude the date parameters. In that case it shows you the data for the whole of the day before you call it (yesterday’s usage).

1 Like

@Occamatic
Thank you for that. I want to try to get those http requests using nodered (I don’t know if it even possible). Where do you put those requests? In Linux terminal or something? Do you save them to the actual json file?

I’m also with Bulb and recently got a smart meter installed because they were advertising their meters were SMETS2.

Looking at their site: We're installing smart meters for free | Bulb they are still using the same meters and using firmware updates to make it SMETS2 some how.

Anyways, I just tried the API as well still get the same error message and wondered if you had any luck since your original post?

@dannytsang
No, I haven’t. I am actually waiting for @Occamatic to answer some questions :wink:. Preferably, I would like to pull the data and save it as json file.
It is still weird, as I get nothing, when I use their test website to get data.

I would just use Python or Postman to make the call (Usually Python if I wanted to save to a file) but it depends on what you are trying to do. I haven’t used nodered before but from a quick look it seems this can be achieved with a HTTP request node, you can enter the parameters I specified into one of those. If this doesn’t help and I get some spare time I will look into how this would be done in nodered and note down the steps for you

1 Like

From this page you can click on the underlined “gas” or “electricity” and subsequent pages past there, it “builds” up the GET request parameters in the URL at the top of the page

1 Like