Feeling overwhelmed as to how to process my data for Octopus Agile emoncms App

wills106 · 2020-03-21T14:59:32.136Z

I want to start tracking my electrical demands in the Octopus Agile emoncms App to see if it will be cheaper for me to switch to Agile or not.

Am I best at tracking real time info or pulling data historically through the Octopus API?

I am feeding real time data into an Influx Database through Home-Assistant from my SolaX Inverter. This comes from a ModBus SDM230 Energy Meter Directly attached to the Inver and the readings from the Inverter itself.

As I have battery storage as well I don’t know how relevant the PV Power is by itself.

I have the following data points:
(These are real time updated per second)

PV1 Power
PV2 Power
PV Toal Power (Just a simple add the two together)
Battery Power (Positive = Battery Charging, Negative = Battery Discharging)
Measured Power (Positive = Export to Grid, Negative = Pulling from Grid)

I also have Imported Grid Power from previous day. Due to limitation on Octopus API.

How am I best formatting this data into the required fields for the app? As I really don’t know where to start with emoncms.
I have emon running locally on a PI as I can’t get a docker version to work. I don’t have any CT clamps attached to the PI.

TrystanLea · 2020-03-25T20:18:45.283Z

Hello @wills106 are you logging the readings listed in emoncms?
For the agile app (with agile expord calculation) you need:

import (power in watts)
import_kwh (cumulative kWh imported)
use_kwh (cumulative total home consumption kWh)
solar_kwh (cumulative total solar generation kWh)

The creation of these feeds for non battery systems is covered by the Solar PV application guide here: Solar PV - Guide | OpenEnergyMonitor your system is a type 2 configuration.

So for your system, using the emoncms input processing, multiply ‘Measured Power’ by -1 and then use ‘allow positive’ to isolate the import component only. Log the result to a feed called ‘import’ and also add a power_to_kwh process to create a cumulative kWh feed called ‘import_kwh’.

For solar, log PV Total Power to a feed called ‘solar’ and add a ‘power_to_kwh’ process logging to a feed called ‘solar_kwh’.

Im struggling a bit with how to work out ‘use_kwh’ as the battery system is complicating things a bit. Maybe the app needs an option to use an ‘export_kwh’ feed. Or just grid_import_export… it would probably make things a bit easier… I could add this as a development item, but I’ve got a bit of a backlog at the moment so might take a while

borpin · 2020-03-25T21:33:02.765Z

@TrystanLea I’m working on a Node-Red flow that will do the import.

The data is in 30 min consumption periods in kWh.

I have the input to emonCMS working with the consumption data (kWh) and the timestamp however, I’m stuck on the processing side for emonCMS.

First, do I need to convert the kWh consumption figure to Watts? If so I’ll do this on the NR side but it seems rather pointless

Why does the App need it or does it calculate the 30 min kWh on the fly?

What are the right processes please?

TrystanLea · 2020-03-25T21:41:25.388Z

borpin:

First, do I need to convert the kWh consumption figure to Watts? If so I’ll do this on the NR side but it seems rather pointless

Should not need this. I can make import optional here app/apps/OpenEnergyMonitor/octopus/octopus.php at master · emoncms/app · GitHub its just used for live status.

The app only really needs to import_kwh feed and solar_kwh, use_kwh if used for agile export

borpin · 2020-03-25T21:45:25.848Z

TrystanLea:

The app only really needs import_kwh

Ok, so in the processing, I am pushing the input to a Feed PHPFINA 30 mins, and just logging it? What is the best way to accumulate it (or do I not need that for this)?

Just selecting the import_kWh and use_kWh as the imported data feed it does not let me continue - it wants the import as well

TrystanLea · 2020-03-25T21:47:04.583Z

An interim fix is to create a blank PHPFINA feed called import and use that. It will just show 0W top-left of the app but the rest should work

borpin · 2020-03-25T21:49:38.569Z

TrystanLea:

An interim fix is to create a blank PHPFINA feed called import

Is this a bug then? That selection should be optional?

borpin · 2020-03-25T21:53:31.321Z

The things you find.

As the data is coming in quite old, I see this in the Inputs page

Even though the number is changing as the data comes in…

TrystanLea · 2020-03-25T22:50:25.373Z

I’ve pushed a change to master to make ‘import’ optional : make import feed optional · emoncms/app@9360cb5 · GitHub

borpin · 2020-03-25T22:57:04.699Z

TrystanLea:

I’ve pushed a change to master to make ‘import’ optiona

I’ll have to switch branch

TrystanLea · 2020-03-25T22:58:01.690Z

I will do a merge to stable soon, I wanted to get through a few more issues to make it worthwhile with all the changelog and versioning overhead.

miker · 2020-03-29T19:29:37.115Z

@wills106 I wrote a blog post on how I compared the cost of Octopus agile with Ecotricity: Domestic data science – energy use – scottishsnow. As far as I understand it, you only need to compare import to get an estimate of potential savings, this is without behaviour/technological change. If you can reduce/eliminate the peak price load (e.g. with solar/battery) these will be additional savings on top of the straight cost difference.

wills106 · 2020-04-02T10:04:22.944Z

@TrystanLea
I think I have it setup correct now bar the use_kwh

emoncms22576×474 82.4 KB

The results as follows.

emoncms11664×1637 277 KB

@miker
I have had a look at your website and the example code.
I can see where you import the agile price info:

# data

tarrif_2018 = ` `read_excel` `(` `"agile_rates.2018-12-20.xlsx"` `, sheet = ` `"South Scotland"` `) %>%

But where does it pull in your personal usage?
I have my half hourly from 01-10-2019 till 31-03-2020 as a CSV file.

If you can reduce/eliminate the peak price load (e.g. with solar/battery) these will be additional savings on top of the straight cost difference.

When the sun comes out I am pretty much ok at keeping the load very low.
I am considering a second battery though.

energy-use1173×1309 187 KB

miker · 2020-04-02T13:45:24.336Z

My use data is spread across a few files, which I list with list.files. Then the lapply function iterates over those files and reads them.

If you’ve not used R before then there are some good resources online: R 4 data science book, Johns Hopkins R programming course. If you’re pushed for time, the data carpentry lessons give you a good intro.

borpin · 2020-04-02T13:45:38.451Z

wills106:

But where does it pull in your personal usage?

I wrote a Node-Red flow to do that.

Import Octopus consumption data Integrations

Updated 25/09 - problem was the returned time format from Octopus had changed. Continuing the discussion from Integrating Octopus Go API in to emonCMS: I’ve written a Node-Red flow to import your Octopus consumption data into emonCMS. Short version Import this flow into Node Red - follow the embedded instructions. [{"id":"294d21ac.2850ae","type":"debug","z":"9ae82de6.3d714","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","targetType":"full","x":77…

However, if you are measuring your import, why do you need that?

wills106 · 2020-04-02T15:52:49.694Z

@miker
Even though I saw the R Tag at the bottom of your post I didn’t put 2 and 2 together realising R is the language it’s written in!

I’ll have to have a read at some point as it does seem an interesting subject (Data Science in general)

I have to say Home Automation has taken down a Rabbit Hole so far. Having to learn the basics of Python, so I have a vague idea what’s going on and yaml as well.

If I had home automation hardware to play with while at Uni I probably would have a better understanding and ability to program.

@borpin
I only have about 4 Days worth of Data in emoncms so far.
But I have the output of the half hourly data going back to end of Sep 2019.
I’m still learning what to do with this data and how best use it.

All this came about from wanting to control my Central Heating better while working shifts!

miker · 2020-04-02T18:04:07.604Z

I hear your pain. Despite working with data and programming for a living playing with IoT is a painful, bleeding edge frontier!

I should have been more explicit that I was analysing data away from emonpi.