Script: importing UK smart meter data into EmonCMS

For a while I’ve been using a self-hosted installation of EmonCMS with a home-built (ESP32-based) Open Energy Monitor.

I’m in the UK, and I’ve just had “smart” electricity and gas meters installed. I was excited about the possibility of accessing my gas consumption, which I hadn’t previously been able to monitor.

I investigated APIs to access the data, and discovered a free third-party API from n3rgy.

I’ve written a PHP script that will download your half-hourly electricity and gas readings via the n3rgy API, convert any gas readings from m3 to kWh via a National Grid API, and then upload them to EmonCMS.

I’ve published and documented my code on GitHub if anyone else is interested.

2 Likes

Hi, welcome :slight_smile:

This is really neat especially the Gas calculation and the fact the electric data should be available for anyone whose supplier passes on the 30min data.

Couple of things, I’d break out the settings into a separate file as a default.settings.ini. Tell users to create a settings.ini from that, which you can then import to the script. Add settings.ini to the .gitignore file (so git ignores it when updating). Then if someone clones your repo to use, and you make changes, a simple git pull will update it. Otherwise, users cannot easily update it without merging changes.

Second thing, use a Unix timestamp (in seconds) for the date-time. Using a date/time string without a timezone (or DST info) can be iffy. A Unix timestamp is always the best way to set and use time. Comparing Unix timestamps is also much simpler.

You can also script creating the Feed and send data directly to a Feed rather than setting that up via an Input. Click on Feeds in emoncms and top right Feed API Help

This is how the Octopus script works (in UsefulScripts) IIRC.

Cheers

1 Like

Good call on the timestamps. I didn’t worry about it for my use as I’m running it on the same server as EmonCMS, so I figured everything would match - but of course when the clocks change there might be a glitch, so I should probably change it to timestamps (or specify timezones on everything). (It also might break if the EmonCMS server was in a different timezone.)

I flipped back and forth on putting the config in a separate file. In the end, for my use case (dumping the script into a cron directory) I decided it was least likely to break if everything was in the single file. I agree that for wider use it would probably be better in a separate file.

I also looked at putting the data directly into a feed. But I decided that the data I’m importing is just an input, and what processing you do to it to turn it into a feed is a separate decision, so it would be more modular if it just inputted the data, leaving the feed processing to EmonCMS. But if it works better for someone else to import it straight into a feed, feel free to take my code and adapt it however best works for you :smiley:

2 Likes

Thanks for the suggestions.

I’ve updated the code on GitHub:

  • Moved the configuration out into a .ini file
  • Ensured all date/times are in UTC throughout
  • (Hopefully) fixed a bug when gas consumption data is available but the National Grid API hasn’t yet published the day’s calorific values
1 Like

A little more information about when data becomes available (for my meter at least):

At about 4am, electricity and gas readings become available up to the 3.00-3.30 slot that morning.

However the National Grid calorific values for those morning slots didn’t become available until about 11am.

I’m not sure if those are typical, but if they are then an appropriate time for a daily cron job might be 12 noon.

Of course, the meters only update once every 24 hours, and I don’t know if they all update overnight, or whether everyone’s meter is set to a random time to spread the load on the servers/network more evenly.

1 Like