PowerWall Data Integration

[johnbanks]

@bwduncan

Thx for yr suggestions in this thread …

Fetch Tesla Power Wall State of charge using EmonHub

That’s a decent chunk of instantaneous data! I’m surprised that you say you are really only interested in the battery power (presumably the energy imported/exported can be integrated by emoncms or something later). I guess you have other things monitoring your grid and solar flows? Yes, in fact we don’t even need to specify the entire URL in the config. We know the endpoints which have the data we’re interested in. We just need the hostname (or the IP address) to be specified in the conf file. …

To recap - I’m trying to adapt the published EmonHubTeslaPowerWallInterfacer.py so I can grab more data from another Tesla API – eg: battery instant power.

I have no Python expertise but mindful that an infinite number of chimps given an infinite amount of time could type The Bible, like a chimp I set to work. But after days spent on this, I’m now heartily sick of bananas. For example, it took me 2 days to discover that in Python to comment out a line it was necessary to use #space – who’d have thought that?

I finally settled on what I thought was a methodical approach. Below is a version of the original EmonHubTeslaPowerWallInterfacer.py with all the alternatives I can think of as comment lines – I could swap lines in/out easily using nano.

import time, json, Cargo, requests
from emonhub_interfacer import EmonHubInterfacer

"""class EmonHubTeslaPowerWallInterfacer

Fetch Tesla Power Wall state of charge

"""

class EmonHubTeslaPowerWallInterfacer(EmonHubInterfacer):

    def __init__(self, name):
        super().__init__(name)

        self._settings.update(self._defaults)

        # Interfacer specific settings
        self._template_settings = {'name': 'powerwall',
                                   'url': False,
                                   'readinterval': 10.0}

        # FIXME is there a good reason to reduce this from the default of 1000? If so, document it here.
        # set an absolute upper limit for number of items to process per post
        self._item_limit = 250

        # Fetch first reading at one interval lengths time
        self._last_time = 0

    def read(self):
        # Request Power Wall data at user specified interval
        if time.time() - self._last_time >= self._settings['readinterval']:
            self._last_time = time.time()

            # If URL is set, fetch the SOC
            if self._settings['url']:
                # HTTP Request
                try:
                    reply = requests.get(self._settings['url'], timeout=int(self._settings['readinterval']), verify=False)
					# aggregates = requests.get(self._settings['url'], timeout=10, verify=False).json()  # per @bwduncan
                    # aggregates = requests.get(self._settings['url'], timeout=10, verify=False)  # per @bwduncan but no .json()
					# reply = requests.get(self._settings['url'], timeout=10, verify=False).json()  # per @bwduncan & reply
                    # reply = requests.get(self._settings['url'], timeout=10, verify=False)  # per @bwduncan but no .json() & reply
					
					reply.raise_for_status()  # Raise an exception if status code isn't 200
                except requests.exceptions.RequestException as ex:
                    self._log.warning("%s couldn't send to server: %s", self.name, ex)

                jsonstr = reply.text.rstrip()
                self._log.debug("%s Request response: %s", self.name, jsonstr)

                # Decode JSON
                try:
                    data = json.loads(jsonstr)
                except Exception:  # FIXME Too general exception
                    self._log.warning("%s Invalid JSON", self.name)
                    return


					
                # Create cargo object
                c = Cargo.new_cargo()
                c.nodeid = self._settings['name']
                c.names = ["soc"]
                # c.realdata = [data['percentage']]
				c.realdata = [data['battery']['instant_power']]
				# c = Cargo.new_cargo(names=['battery_instant_power'], realdata=[aggregates['battery']['instant_power'])   # per @bwduncan
				# c = Cargo.new_cargo(names=['battery_instant_power'], realdata=[data['battery']['instant_power'])   # per @bwduncan but data
				# c = Cargo.new_cargo(names=['battery_instant_power'], realdata=[aggregates['battery']['instant_power']])  # added ]
                return c

        # return empty if not time
        return

    def set(self, **kwargs):
        for key, setting in self._template_settings.items():
            # Decide which setting value to use
            if key in kwargs.keys():
                setting = kwargs[key]
            else:
                setting = self._template_settings[key]
            if key in self._settings and self._settings[key] == setting:
                continue
            elif key == 'readinterval':
                self._log.info("Setting %s %s: %s", self.name, key, setting)
                self._settings[key] = float(setting)
                continue
            elif key == 'name':
                self._log.info("Setting %s %s: %s", self.name, key, setting)
                self._settings[key] = setting
                continue
            elif key == 'url':
                self._log.info("Setting %s %s: %s", self.name, key, setting)
                self._settings[key] = setting
                continue
            else:
                self._log.warning("'%s' is not valid for %s: %s", setting, self.name, key)

        # include kwargs from parent
        super().set(**kwargs)

But whatever alternative I’ve tried, I always got a popup error message and emonhub stopped (but could be restarted with … sudo systemctl start emonhub.service)

EmonCMS Error

-------------

Message: SyntaxError: unexpected token: identifier

Route: Lib/jquery-1.11.3.min.js

Line: 1

Column: 4

The emonhub log is not informative.

Then, quite by chance, I ran EmonHubTeslaPowerWallInterfacer.py as is but with a url containing /api/meters/aggregates

The log revealed the wealth of data that is available …

2020-07-31 23:12:48,043 DEBUG    PowerWall  PowerWall Request response: {"site":{"last_communication_time":"2020-07-31T23:12:47.889536541+01:00","instant_power":4.8070068359375,"instant_reactive_power":169.26604461669922,"instant_apparent_power":169.33428824341266,"frequency":49.95012283325195,"energy_exported":1101936.5047250446,"energy_imported":876557.3925028223,"instant_average_voltage":124.1665776570638,"instant_total_current":0,"i_a_current":0,"i_b_current":0,"i_c_current":0,"timeout":1500000000},"battery":{"last_communication_time":"2020-07-31T23:12:47.890245881+01:00","instant_power":2130,"instant_reactive_power":-20,"instant_apparent_power":2130.0938946440833,"frequency":49.969,"energy_exported":319410,"energy_imported":393570,"instant_average_voltage":248.33333333333334,"instant_total_current":-50.3,"i_a_current":0,"i_b_current":0,"i_c_current":0,"timeout":1500000000},"load":{"last_communication_time":"2020-07-31T23:12:47.889536541+01:00","instant_power":2147.7866236629907,"instant_reactive_power":151.9973041128399,"instant_apparent_power":2153.158275938683,"frequency":49.95012283325195,"energy_exported":0,"energy_imported":-299539.11222222226,"instant_average_voltage":124.1665776570638,"instant_total_current":17.297622791819002,"i_a_current":0,"i_b_current":0,"i_c_current":0,"timeout":1500000000},"solar":{"last_communication_time":"2020-07-31T23:12:47.890574865+01:00","instant_power":0,"instant_reactive_power":0,"instant_apparent_power":0,"frequency":0,"energy_exported":0,"energy_imported":0,"instant_average_voltage":0,"instant_total_current":0,"i_a_current":0,"i_b_current":0,"i_c_current":0,"timeout":1500000000}}
2020-07-31 23:12:48,044 WARNING  PowerWall  PowerWall Percentage key not found

And the warning – PowerWall Percentage key not found – suggests the code operated OK to that point. And emonhub continued to operate.

I’d more than welcome any further suggestions.

And I’m happy to be a testbed of any ideas suggested by those not having a PowerWall.

TIA

[borpin (Brian Orpin)]

From a browser, what URL do you use, and what do you get back?

Basic steps are

1. Create return object (cargo)
2. Send URL
3. Load cargo with data returned from URL.
4. return cargo

If you want to use 2 URLs, do steps 2 & 3 again so

1. Create return object (cargo)
2. Send URL 1
3. Load cargo with data returned from URL 1.
4. Send URL 2
5. Load cargo with data returned from URL 2.
6. return cargo

[johnbanks]

@TrystanLea

The EmonHubTeslaPowerWallInterfacer works well publishing the Battery % Charge to emoncms.

However there is another Tesla API (unofficially available) with a wealth of additional battery data – most importantly – instant power and total energy imported & exported.

Per below, I’ve managed to adapt my EmonHubTeslaPowerWallInterfacer.py file to capture this additional data and publish it to emoncms …

import time, json, Cargo, requests
from emonhub_interfacer import EmonHubInterfacer

"""class EmonHubTeslaPowerWallInterfacer

Fetch Tesla Power Wall power and energy 

url needs a trailing /

"""

class EmonHubTeslaPowerWallInterfacer(EmonHubInterfacer):

    def __init__(self, name):
        super().__init__(name)

        self._settings.update(self._defaults)

        # Interfacer specific settings
        self._template_settings = {'name': 'powerwall',
                                   'url': False,
                                   'readinterval': 10.0}

        # FIXME is there a good reason to reduce this from the default of 1000? If so, document it here.
        # set an absolute upper limit for number of items to process per post
        self._item_limit = 250

        # Fetch first reading at one interval lengths time
        self._last_time = 0

    def read(self):
        # Request Power Wall data at user specified interval
        if time.time() - self._last_time >= self._settings['readinterval']:
            self._last_time = time.time()

            # If URL is set, fetch the POWER and ENERGY
            if self._settings['url']:
                # HTTP Request
                try:
                    reply = requests.get(self._settings['url'] + 'api/meters/aggregates', timeout=int(self._settings['readinterval']), verify=False)
                    reply.raise_for_status()  # Raise an exception if status code isn't 200
                except requests.exceptions.RequestException as ex:
                    self._log.warning("%s couldn't send to server: %s", self.name, ex)

                jsonstr = reply.text.rstrip()
                self._log.debug("%s Request response: %s", self.name, jsonstr)

                # Decode JSON
                try:
                    data = json.loads(jsonstr)
                except Exception:  # FIXME Too general exception
                    self._log.warning("%s Invalid JSON", self.name)
                    return


                # Create cargo object
                c = Cargo.new_cargo()
                c.nodeid = self._settings['name']
                c.names = ["power and energy"]
                c.realdata = [data['battery']['instant_power'], data['battery']['energy_exported'], data['battery']['energy_imported']]  # All in watts and +ve power=discharging  -ve power=charging
                return c

        # return empty if not time
        return

    def set(self, **kwargs):
        for key, setting in self._template_settings.items():
            # Decide which setting value to use
            if key in kwargs.keys():
                setting = kwargs[key]
            else:
                setting = self._template_settings[key]
            if key in self._settings and self._settings[key] == setting:
                continue
            elif key == 'readinterval':
                self._log.info("Setting %s %s: %s", self.name, key, setting)
                self._settings[key] = float(setting)
                continue
            elif key == 'name':
                self._log.info("Setting %s %s: %s", self.name, key, setting)
                self._settings[key] = setting
                continue
            elif key == 'url':
                self._log.info("Setting %s %s: %s", self.name, key, setting)
                self._settings[key] = setting
                continue
            else:
                self._log.warning("'%s' is not valid for %s: %s", setting, self.name, key)

        # include kwargs from parent
        super().set(**kwargs)

This approach means I can only run one or the other version – not both at the same time.

I am confused about whether a new Child Class should be created or whether this should be just another Object in the same Child Class.

Also what changes need to be made to the Interfacer stanza in emonhub.conf?

@TrystanLea – I would really appreciate yr guidance - thx

PS - There is a small annoyance. Per this screenshot of the Inputs page, the 2nd and 3rd data items have meaningless titles.

[TrystanLea]

Thanks @johnbanks that looks great, certainly useful to be able to access those properties. I’ve reworked your code to make two requests, one for the state of charge and the other for the new properties, these are then combined and passed on to emoncms:

Here’s the read function:

def read(self):
    # Request Power Wall data at user specified interval
    if time.time() - self._last_time >= self._settings['readinterval']:
        self._last_time = time.time()

        # If URL is set, fetch the SOC
        if self._settings['url']:
            # -------------------------------------------------------------------------
            # Part 1: Fetch Power Wall state of charge percentage
            # -------------------------------------------------------------------------
            try:
                reply = requests.get(self._settings['url'] + 'api/system_status/soe', timeout=int(self._settings['readinterval']), verify=False)
                reply.raise_for_status()  # Raise an exception if status code isn't 200
            except requests.exceptions.RequestException as ex:
                self._log.warning("%s couldn't send to server: %s", self.name, ex)

            jsonstr = reply.text.rstrip()
            self._log.debug("%s Request response: %s", self.name, jsonstr)

            # Decode JSON
            try:
                data_p1 = json.loads(jsonstr)
            except Exception:  # FIXME Too general exception
                self._log.warning("%s Invalid JSON", self.name)
                return

            # Check if battery percentage key is in data object
            if not 'percentage' in data_p1:
                self._log.warning("%s Percentage key not found", self.name)
                return
                
            # -------------------------------------------------------------------------
            # Part 2: Fetch Power Wall instant_power, energy_exported, energy_imported
            # -------------------------------------------------------------------------
            try:
                reply = requests.get(self._settings['url'] + 'api/meters/aggregates', timeout=int(self._settings['readinterval']), verify=False)
                reply.raise_for_status()  # Raise an exception if status code isn't 200
            except requests.exceptions.RequestException as ex:
                self._log.warning("%s couldn't send to server: %s", self.name, ex)

            jsonstr = reply.text.rstrip()
            self._log.debug("%s Request response: %s", self.name, jsonstr)

            # Decode JSON
            try:
                data_p2 = json.loads(jsonstr)
            except Exception:  # FIXME Too general exception
                self._log.warning("%s Invalid JSON", self.name)
                return

            # Check if battery key is in data object
            if not 'battery' in data_p2:
                self._log.warning("%s battery key not found", self.name)
                return

            # Create cargo object
            c = Cargo.new_cargo()
            c.nodeid = self._settings['name']
            c.names = ["soc","instant_power","energy_exported","energy_imported"]
            c.realdata = [data_p1['percentage'], data_p2['battery']['instant_power'], data_p2['battery']['energy_exported'], data_p2['battery']['energy_imported']]
            return c

    # return empty if not time
    return 

I haven’t tested this here, if you could get a chance to test that would be great! Cheers for looking into this

[johnbanks]

@TrystanLea
Sorry for the delay in testing.

I substituted yr def read code lines for those in the published Interfacer and had lots of issues with indentation. Also had problems getting emonhub.service to restart. syslog revealed …

Sep 10 15:07:28 watchman systemd[1]: emonhub.service: Service RestartSec=100ms expired, scheduling restart.
Sep 10 15:07:28 watchman systemd[1]: emonhub.service: Scheduled restart job, restart counter is at 5.
Sep 10 15:07:28 watchman systemd[1]: Stopped emonHub service description.
Sep 10 15:07:28 watchman systemd[1]: emonhub.service: Start request repeated too quickly.
Sep 10 15:07:28 watchman systemd[1]: emonhub.service: Failed with result 'exit-code'.
Sep 10 15:07:28 watchman systemd[1]: Failed to start emonHub service description.

However this guy seemed knowledgeable on system.d …

Creating a Linux service with systemd

My problem was fixed by adding RestartSec=1 to:
/opt/openenergymonitor/emonhub/service/emonhub.service

Which begs the question - should this be incorporated in the official image?

The enhanced Interfacer works well on a Raspberry Pi running the Oct 2019 image updated to ver 10.2.6 …
JB+TL-ver2.zip (1.4 KB)

I’ve tweaked the comments and names for clarity and added a leading / to both api calls.

The interfacer entry in emonhub.conf now needs to be …

[[PowerWall]]
    Type = EmonHubTeslaPowerWallInterfacer
    [[[init_settings]]]
    [[[runtimesettings]]]
        pubchannels = ToEmonCMS,
        name = powerwall
        url = http://IP-address  # local network IP address or hostname of PowerWall	
        readinterval = 10

Here’s a screenshot …

I think PowerWall users will find this a worthwhile enhancement.
Thx

[borpin (Brian Orpin)]

@johnbanks

My problem was fixed by adding RestartSec=1 to:
/opt/openenergymonitor/emonhub/service/emonhub.service

Which begs the question - should this be incorporated in the official image?

No. If the unit is causing an error on starting, fix the error.

I suspect this is just masking the problem.

[johnbanks]

@borpin

The problems occurred whilst “developing” the enhanced Interfacer - so a transitory condition.

I needed a fix so I could continue trying things out.

Without masking transitory problems of my own creation - I’d have been stuck.

But, in general/normal running, I agree with you - best to fix problems rather than masking them - motherhood and apple pie.

[borpin (Brian Orpin)]

@johnbanks

The problems occurred whilst “developing” the enhanced Interfacer - so a transitory condition.

So no need to add a fix to a non-existent problem I’d suggest.

[pb66 (Paul)]

@johnbanks

should this be incorporated in the official image?

I’d be inclined to say that would be a good idea, it seems many, if not all service status output’s for emonhub contain these lines.

@johnbanks

Sep 10 15:07:28 watchman systemd[1]: emonhub.service: Start request repeated too quickly.
Sep 10 15:07:28 watchman systemd[1]: emonhub.service: Failed with result 'exit-code'.

There is little point in trying to restart a service before it’s ready. The default delay for restarts is just 100ms, extending that to 1s might result in some meaningful log messages when emonhub doesn’t start rather than just “too fast”. That will be why it help you find your issue and can also assist with other issues so I’ll flag this for @TrystanLea to consider.

Just by way of comment to streamline the settings code. Each interfacer has to have a unique name so the name = powerwall setting is a bit redundant as there is already the “PowerWall” name. Also readinterval = 10 is doubling up on the interval setting inherited by all interfacers from the parent interfacer code. Some time ago Trystan changed the http interfacer to use “sendinterval” rather than use the existing “interval” all interfacers have and it caused some confusion, it has since been changed back to “interval”.

[TrystanLea]

@johnbanks

The enhanced Interfacer works well on a Raspberry Pi running the Oct 2019 image updated to ver 10.2.6 …
JB+TL-ver2.zip (1.4 KB)

Fantastic, would you say this is good to go now if I replace the existing interfacer with the version in your zip?

[borpin (Brian Orpin)]

@TrystanLea, there are a whole host of other things to be added to emonhub in the repository.

[borpin (Brian Orpin)]

@TrystanLea

Fantastic, would you say this is good to go now

Documentation?

[johnbanks]

@TrystanLea

Yes - it’s good to go.

The current interfacer is described in the emonhub documentation. I suggest making the changes shown below in bold italics …

e.) Tesla Power Wall Interfacer

This interfacer fetches the state of charge , current power and cumulative energy imported & exported of a Tesla Power Wall on the local network. Enter your PowerWall IP-address or hostname in the URL section of the following emonhub.conf configuration:
[[PowerWall]]
Type = EmonHubTeslaPowerWallInterfacer
[[[init_settings]]]
[[[runtimesettings]]]
pubchannels = ToEmonCMS,
name = powerwall
url = http://POWERWALL-IP
readinterval = 10