ESPHome Component for EmonTX (ESPx & PiPicoW)

[borpin (Brian Orpin)]

Continuing the discussion from EmonESP - ESPHome Component:

3 years since I first suggested this!

The idea is still to simplify the interface between the EmonTXx and other systems via Wi-Fi rather than RFM69. This is done by using the ESPHome Core and adding on a component specifically for the EmonTX. This approach has a number of advantages:

• The C++ code base required to be maintained by OEM is reduced resulting in bug fixes and security updates to the base libraries being easily passed onto the ESP device.
• Updates & configuration changes can be done OTA
• The interface to HomeAssistant would be simplified and the device work independently from EmonCMS.

ESPHome has changed significantly since I first suggested this, with the core developer now part of the HA family and the integration to HA tighter and the support on a more professional footing.

I have a basic component up and running on a WEMOS D1 Mini and when ESPHome support for the PiPico chip (RP2040) arrives on Wednesday, I intend to try that out. It should ‘just work’. The Component in it’s current form can be found here GitHub - openenergymonitor/esphome_components: ESPHome Components with my working fork here GitHub - borpin/esphome_components: ESPHome Components.

However, for this component to work more widely, the EmonTX firmware needs to be modified to output a JSON string (again discussed elsewhere).

If anyone with some C++ skills wants to chip in, they would be very welcome as I’m reaching the limit of my C++ knowledge.

Things to do:

• Add HTTP interface (possibly including compression)
• Add a configuration item for scale to the Pulse output (currently assumes 1 pulse = 1 watt)
• Investigate the SPI interface to the RFM69 chip
• Investigate being able to modify the EmonTX settings from ESPHome
• Investigate moving the I2C and OneWire sensors to be read directly by this device. This would widen the sensors supported by the EmonTX.

[borpin (Brian Orpin)]

@awjlogan - if you are looking at this, I’d suggest two components

1. A component to read the data from the TX Serial
2. A component to send the data to emoncms. Using a Lambda to send the data by MQTT is horribly clunky. Could provide an option for MQTT or HTTP API then it could go to emoncms.org.

[awjlogan (Angus Logan)]

@borpin - apologies, I’ve only just had a chance to have a look properly at this. Looks great - I’ll start having a play with it!

On the todo items, the emonPi3/Tx++ can easily do the OneWire interface and there’s an expander on board for I2C and I’ve left the temperature API agnostic to the actual sensor being used. We can skip the SPI interface and directly interface with the UART (using the existing header) - I put JSON support in to match the Tx4/Pi2 firmware, and this is clearly where it came from!

As it interfaces with the UART on the Pi3, if ESPHome supports commands being sent to a destination, it could configure it directly.

Currently, I have a USB-C header on the ESP board for programming - this seems a bit clunky (as there’s already a USB-C connector on the Pi3), so I’ll also look at programming the board through the Pi3 as well (I’m not too familiar with Espressif’s stuff yet).

In the meantime, I can make you an ESP board up (probably, have to check my part stash) and after I’m happy it works, could send you one to use as well?

[borpin (Brian Orpin)]

@awjlogan

I’ll start having a play with it!

Great - I’m a real fan of this approach. I think my fork may be more up-to-date with what I am using - I really can’t remember!

@awjlogan

I put JSON support in to match the Tx4/Pi2 firmware, and this is clearly where it came from!

Yes, I got @TrystanLea to add the JSON option

@awjlogan

On the todo items, the emonPi3/Tx++ can easily do the OneWire interface and there’s an expander on board for I2C and I’ve left the temperature API agnostic to the actual sensor being used.

One of the advantages of an ESPHome component is that it then integrates with HA easily.

Secondly, you can configure ESPHome on the device to be ready for a ‘first use’ scenario by a user.

I’m pretty sure ESPHome will communicate over any interface you like

[awjlogan (Angus Logan)]

Quick skim of the ESPHome documentation suggests that the UART Component would be a reasonable place to start. It does support transmit from the ESPHome as well, so could be used to configure the Tx/Pi in situ.

Looking at Brian’s Tx4 module, one question I have would be how to generalise it so the JSON decoding is not static. That is, the Pi3 can support 3 digital (OneWire or pulse) or 1 extra analog and 2 digital (OneWire or pulse) inputs. The number of temperature sensors is not defined but is detected at start up. There is a defined maximum for static allocation, but unlikely to be used in practice so you’d end up with a lot of empty entries (“T5”: 30000, “T6”: 30000 …). See what I can come up with - I also need to set up HA so I can test things

[borpin (Brian Orpin)]

@awjlogan

one question I have would be how to generalise it so the JSON decoding is not static.

I think the JSON Decoding needs to move into the component itself to make it faster (I tried but could not work out how).

You can then create a configuration within the yaml to tell it what interfaces to use, and what sensors to create.

@awjlogan

I also need to set up HA so I can test things

You don’t really. The HA integration is automatic once the sensors are created by ESPHome .

There is the Web Tools that you can use (I think) plus the Web Component.

ESP Web Tools (esphome.github.io)

The Discord Channel is pretty good if you can get them to engage.

[awjlogan (Angus Logan)]

@borpin

There is the Web Tools that you can use (I think) plus the Web Component.

Thanks Brian, that’s very useful

[FredM67 (Fred Metrich)]

Hi all,

Is there still some “demand” for an official ESPHome component to fully support EmontxV4 and other similar devices ?

If yes, I can take what already exists (which repo should I take, the original or the fork ?) and make it to the end with documentation, … and ask the ESPHome team to integrate it officially.

First, I would concentrate on reading the measurements… and if possible, I would implement the possibility to fully configure EmonTx through HA.

Fred

[awjlogan (Angus Logan)]

Hi @FredM67 - yes, I think it would be useful. I’ve got a design made up with an ESP32-C3 that just fits on to the normal Raspberry Pi header with power and UART. I haven’t had time to put much effort into the ESPHome setup for it, so it’s rather languished (and honestly I don’t really enjoy working with frameworks!).

There are two options for the Tx5 and the incoming Tx6 - key:value data, or JSON. I was hoping to use ESPHome to digest the JSON output, seemed most straightforward. Configuration of the Tx is all through the UART.

In terms of documentation, probably the best thing to look at is the source code. For the Tx4/5, it’s here for 6 CT inputs. Line 28 describes the formats. Configuration options are here.

For the upcoming Tx6/Pi3, the output will be the same and you can see the configuration options on line 936 here.

Let me know if you would like one of the ESP32 boards and if you have any other queries/suggestions

[FredM67 (Fred Metrich)]

Well, that sounds all good.

Yes, that’d be great to have one of the ESP32 boards.
I don’t really have the hardware needed for that.

A couple of weeks ago, I’ve implemented an integration for Robin Emley’s mk2pvrouter. That works very fine. This integration not only reads the data (power, diverted energy, …) but it can control the router too (override, stop diversion, …). This integration is in the queue for review to be integrated in ESPHome.

I’ll keep you inform about the progress and for sure, I’ll need a couple of testers

Fred

[FredM67 (Fred Metrich)]

Hi again,

Since the number of output values is very “dynamic” (single phase, three-phase, CTs between 0 and 12, …), I’d go rather with something similar to the mk2pvrouter component which is itself similar to the component Teleinfo.
Here, the user has to select which sensors he wants to read. Providing a gist to give him a complete list of all available sensors, he’ll have to simply comment/uncomment what he wants or what he doesn’t want.
The other way would be to predefine all the sensors, but well because of the huge amount of possibilities, the user would probably have a lot of sensors he doesn’t want or that are simple not available. The method is, I think, much more appropriate for a device which provides a finite well-defined list of sensors.

Fred

[awjlogan (Angus Logan)]

Hi Fred - here’s an image of the adaptor board connected to one of the emonPi3 prototypes.

I don’t know enough about the HA integration side to comment too much on the data that’s available and how it’s parsed. For example, the emonPi3 will always output all the voltage and CT channels, even if they are inactive, as 0 is still a valid value. If the HA integration can filter out values that are known to be unused, that would be best - is that the first option you presented?

[borpin (Brian Orpin)]

Can we assume that the JSON format is always going to be the same @TrystanLea?

If so, then it will only be the sensors themselves that will change. If the Component has a mechanism so you can tell it to look for CT1 and then creates HA sensors called Mains Supplyfor voltage, power etc (as it does now - badly) that should be flexible to cope with any number of inputs. You can then also look for Temperature as well.

One thing to consider is that we also want the component to be able to communicate with emoncms - the OEM software. This communication can be done locally via MQTT or by HTTP (locally and directly to emoncms.org).

Ideally the same data could be sent to multiple end points (HA, MQTT & HTTP).

Data frequency is 10s (generally).

[FredM67 (Fred Metrich)]

Well my idea is to have something fully flexible. The user decides what he wants to see, what he wants to track.
This way, physical unconnected sensor, which show with a value of “0” can simply be skipped.
The user could also track just one part of a sensor. Let say, CT1 monitors my induction plate. If the user don’t care about the instant power, he would simply define in the yaml file the sensor for energy.
It would look something like this:

sensor:
  - platform: emontx
    tag_name: "E1"
    name: "Induction plate"

I would provide a gist with all available sensors, and the user simply have to:

• copy-paste
• comment/uncomment what he doesn’t need/what he needs.

If the user makes a mistake and defines a sensor that does not exist (like “E13”), the sensor would simply appear as “Not available” or something similar.

What do you think about that ?

BTW, I would also let the ESP32 takes control of the temperature sensors.

Fred

[awjlogan (Angus Logan)]

@FredM67

This way, physical unconnected sensor, which show with a value of “0” can simply be skipped.
The user could also track just one part of a sensor.

I think this is good - as 0 is a valid value, you can’t rely on that to indicate a data source’s status.

@FredM67

BTW, I would also let the ESP32 takes control of the temperature sensors.

Less keen on this - the OEM systems typically use 1Wire temperature sensors, so you’d have to expand the adaptor board to either include external connectors or connect to the 1Wire pin on the Pi header - you’d then need a branch of the firmware to not activate the 1Wire pins. I would strongly suggest sticking with the data made available. The Pi3(/Tx6) will also have the facility to use I2C sensors within the unit.

@borpin - I don’t expect the JSON format to change, and if it does, does it matter? It’s just key/value pairs so it will always be available to decode. Sorry if I’ve missed something here?

@borpin

Ideally the same data could be sent to multiple end points (HA, MQTT & HTTP).

Yes, definitely Is this possible with ESPHome? Although, if you have emonCMS running, can you not just use the HA integration for that? Again, sorry if I’m missing something there!

[FredM67 (Fred Metrich)]

@awjlogan

@FredM67

This way, physical unconnected sensor, which show with a value of “0” can simply be skipped.
The user could also track just one part of a sensor.

I think this is good - as 0 is a valid value, you can’t rely on that to indicate a data source’s status.

Indeed, a value of 0 could be simply an installed CT on a “dead” wire, or not installed at all.

@awjlogan

@FredM67

BTW, I would also let the ESP32 takes control of the temperature sensors.

Less keen on this - the OEM systems typically use 1Wire temperature sensors, so you’d have to expand the adaptor board to either include external connectors or connect to the 1Wire pin on the Pi header - you’d then need a branch of the firmware to not activate the 1Wire pins. I would strongly suggest sticking with the data made available. The Pi3(/Tx6) will also have the facility to use I2C sensors within the unit.

It’s only a possibility. On EmonTxv4, I know the expansion board can take control of the Dallas sensors and even the RF module.

@awjlogan

@borpin - I don’t expect the JSON format to change, and if it does, does it matter? It’s just key/value pairs so it will always be available to decode. Sorry if I’ve missed something here?

As long as the JSON stays “flat” (no sub-element, …), the component will work.

@awjlogan

@borpin

Ideally the same data could be sent to multiple end points (HA, MQTT & HTTP).

Yes, definitely Is this possible with ESPHome? Although, if you have emonCMS running, can you not just use the HA integration for that? Again, sorry if I’m missing something there!

I did already try a couple of months ago to “forward” sensor values to IFTTT, through a REST-API. It worked.
So, it should be possible to do that for MQTT too.
BTW, this part won’t be to be implemented in the component itself. That’s pure ESPHome stuff.
To ease all of this, we could provide a gist with a sample config for MQTT and HTTP.

[awjlogan (Angus Logan)]

@FredM67

As long as the JSON stays “flat” (no sub-element, …), the component will work.

Got it - is there any advantage to nesting it? For example, you could represent it as:

{ "CT": [45.4, 56.5] }

In some ways, that is nicer than just padding the K/V format. As far as I know, there are no consumers for the JSON format just now - it was added for use the ESPHome. If you’d prefer it nested, I can do it for the Pi3…

@FredM67

It’s only a possibility. On EmonTxv4, I know the expansion board can take control of the Dallas sensors and even the RF module.

Yes - the onboard microcontroller will handle those when there’s no Raspberry Pi present. All the data and configuration just comes through the UART port.

@FredM67

BTW, this part won’t be to be implemented in the component itself. That’s pure ESPHome stuff.
To ease all of this, we could provide a gist with a sample config for MQTT and HTTP.

Nice - sounds good.

[borpin (Brian Orpin)]

@awjlogan

Yes, definitely Is this possible with ESPHome? Although, if you have emonCMS running, can you not just use the HA integration for that? Again, sorry if I’m missing something there!

My idea is that the ESPHome Component isn’t simply to ease connection to HA, but can also be used as the means of connecting the ESP to emoncms as well. This would negate the need for OEM’s ESP firmware.

This would need it to be written so it can simply be onboarded when first booted. I’ve seen some examples of this type of deployment.

@awjlogan

I don’t expect the JSON format to change, and if it does, does it matter? It’s just key/value pairs so it will always be available to decode.

@awjlogan

Got it - is there any advantage to nesting it?

No. Let’s keep it simple!

@FredM67

BTW, I would also let the ESP32 takes control of the temperature sensors.

As the data is already coming out on the UART in the JSON, there is no need.

@FredM67

BTW, this part won’t be to be implemented in the component itself. That’s pure ESPHome stuff.
To ease all of this, we could provide a gist with a sample config for MQTT and HTTP.

I found this really tricky to do and took a lot of the ESPs cycle time. I think doing this within the component a bit more would be easier.

I never managed to get the HTTP to connect to the emoncms API. The raw JSON can simply be sent to emoncms, I have just never worked out how to make it available to send

[FredM67 (Fred Metrich)]

@awjlogan

In terms of documentation, probably the best thing to look at is the source code. For the Tx4/5, it’s here for 6 CT inputs. Line 28 describes the formats. Configuration options are here.

For the upcoming Tx6/Pi3, the output will be the same and you can see the configuration options on line 936 here.

Something must be wrong in the sources.
On my old Emontx v4 (3 phases), I receive
{"MSG":14004,"V1":232.00,"V2":232.12,"V3":232.15,"P1":0,"P2":0,"P3":0,"P4":0,"P5":0,"P6":0,"E1":1,"E2":-3,"E3":-3,"E4":0,"E5":-3,"E6":-4}

It’s V1-3 instead of Vrms1-3 !

[awjlogan (Angus Logan)]

Ah, sorry @FredM67 - the comment says Vrms as that’s the packed format for the RFM transmission. Your readings are the correct ones. The Pi3 also sends temperature and pulse data - this could potentially be put into the Tx4/5 firmware but it’s already tight for cycles…