Connecting ebusd and emoncms via MQTT

[ramcq (Robert McQueen)]

I’ve just found time to set up my eBUS Adapter and ebusd, meaning I am now able to passively read or actively fetch values from my Vaillant heat pump. When values are updated on the bus I get a message on a topic like /ebusd/hmu/Status01 with some nice JSON contents about the current flow temperatures and pump status:

{
  "0": {
    "name": "temp1",
    "value": 38.5
  },
  "1": {
    "name": "temp1",
    "value": 35
  },
  "2": {
    "name": "temp2",
    "value": null
  },
  "3": {
    "name": "temp1",
    "value": null
  },
  "4": {
    "name": "temp1",
    "value": null
  },
  "5": {
    "name": "pumpstate",
    "value": "on"
  }
}

A lot of these “just show up”, because it seems like the Vaillant controller regularly polls for these values so ebusd sees them going past. Others I can trigger a fetch by publishing a message like ebusd/hmu/CurrentYieldPower/get, or set up polling by publishing to that topic with a message body like ?3.

{
  "0": {
    "name": "",
    "value": 5.8
  }
}

Note as @TrystanLea mentioned on Twitter the JSON structure is a little odd - it’s possible to get these as strings with ; delimiters for the multiple value fields. That might be easier to handle (from the CLI tool but the same values available via MQTT:

hmu CurrentCompressorUtil = 114
hmu CurrentConsumedPower = 1.9
hmu CurrentYieldPower = 5.8
hmu State = 0;757;129;8
700 Hc1PumpStatus = 1
hmu Status01 = 35.0;35.5;-;-;-;off

I’ve got EmonCMS running under Docker with https://github.com/emoncms/emoncms-docker/tree/v2 (although emonhub is temporarily disabled as I’ve not moved my RFM69Pi over so the initialisation fails). I understand that EmonCMS monitors the emon/ topic, but I’m not sure the best way to proceed from here.

So a few questions, with apologies as I’m a complete MQTT novice:

• Should I simply tell ebusd to use emon/ebusd as its topic base? Or can I tell emon (or emonhub) to read some specific additional topics?
• Can I make something in emoncms publish the “get” messages when it wants to update a value to read, or should I set up the polling in ebusd?
• If the latter, is there something that can publish those polling messages or should I do that via the CPI and add a script to switch those on to my ebusd container? (look who learnt how to write a Dockerfile over Christmas! )

Thanks!

[ramcq (Robert McQueen)]

Who needs sleep?! I’ve made some progress just moving the ebusd topic to emon/ebusd and have a small patch to unbox JSON values of the form { name: blarg, value: blarg } which I can submit later after a little more testing.

[TrystanLea]

Great to hear @ramcq sounds like you have got this!?

[ramcq (Robert McQueen)]

Maybe?! https://github.com/emoncms/emoncms/pull/1779 looks promising:

I have some problems with string values - it seems like emoncms is just discarding them - is that expected? There are some which are obviously booleans (on / off / true / false / yes / no) which could be mapped to 1 or 0 with a generic function. Does that make sense to consider adding lower down in the MQTT service before it submits the values, or is there somewhere else that’s worth considering (on the way “in” to emoncms)?

However, some of the fields reported by ebusd are enums (see https://github.com/john30/ebusd-configuration/blob/master/ebusd-2.1.x/en/vaillant/_templates.csv#L86 for example) - is there somewhere in the pipeline I’d be able to configure a mapping for those I cared about?

[modeller (steve)]

I use node-red as the data manipulator between ebusd and emon using MQTT.
Node-red has timers which poll for certain data E.g. yield and help trigger the calculation of CoP

[borpin (Brian Orpin)]

@ramcq

I have some problems with string values - it seems like emoncms is just discarding them - is that expected?

Yes.

@modeller

I use node-red as the data manipulator between ebusd and emon using MQTT.

Yes NR is often the best tool.

Ideally the JSON will be structured such that {"name": value, "name2": value2}. You can include a "time":timestamp_in_seconds as part of the JSON and this will be used for the Feed. Means any drift can be eliminated if you call / poll for the data regularly.

If you use NR then when you send the data as MQTT it automatically translates a Javascript Object into a JSON string. If you want to map booleans to 1/0 then do it there as well.

[glyn.hudson]

Nice work, this looks great. I’m just getting started with ebusd. I’ve just go a basic setup up and running:

eBUS hardware adapter (and ebusd software) Thread

Since I wasn’t getting much luck with the docker version of ebusd and it quite awkward to debug, I decided to install natively and it worked first time, thanks a lot to @Zarch for this help Install direct From https://github.com/john30/ebusd-debian/blob/master/README.md Tested On Bullseye Debian 11 emonSD-10Nov22 with version ebusd 23.2.23.2 wget -qO - https://raw.githubusercontent.com/john30/ebusd-debian/master/ebusd.gpg.key|sudo apt-key add - sudo wget -O /etc/apt/sources.list.d/…

However, the MQTT topics I’m getting are not JSON like yours e.g

pi@emonhp:~/ebusd $ mosquitto_sub -v -u 'emonpi' -P 'emonpimqtt2016' -t 'ebusd/#'
ebusd/global/version ebusd 23.2.23.2
ebusd/global/running true
ebusd/global/scan finished
ebusd/global/signal true
ebusd/global/updatecheck OK, device firmware 1[3a0f] available, vaillant/08.hmu.csv: different version available, vaillant/broadcast.csv: different version available, vaillant/yield3f40.inc: different version available
ebusd/hmu/SetMode auto;28.0;-;-;0;1;1;0;0;0
ebusd/vwzio/SetMode auto;0.0;-;-;1;0;0;0;0;0
ebusd/hmu/Status01 29.0;27.0;-;-;-;on
ebusd/hmu/State 1;13.0;Eco mode HP on;heating
ebusd/vwzio/Status01 -;-;-;-;39.5;off
ebusd/vwz/unknownMPoll10sB512h 7c 02 00 00 80 ff 00
ebusd/hmu/State 1;13.0;Eco mode HP on;heating

How can I get it to return JSON? Assuming JSON is easier to work with than the strings?

[ramcq (Robert McQueen)]

I think I was passing the --mqttjson option to get JSON strings out.

Yeah:

version: '3'

services:
  ebusd:
    container_name: ebusd
    image: john30/ebusd
    restart: always
    ports:
     - "8888:8888"
     - "8880:8880"
    command: -f --scanconfig --httpport=8880 --device=enh:ebus-dd9cd0.broadband:9999 --mqtthost=mqtt --mqttport=1883 --pollinterval=30 --mqtttopic=emon/ebusd --mqttjson
    # --configpath=/etc/ebusd 
    networks:
      - mqtt
    #volumes:
    #  - ./ebusd-configuration/latest/en/:/etc/ebusd

networks:
  mqtt:
    external: true

[borpin (Brian Orpin)]

@glyn.hudson

Assuming JSON is easier to work with than the strings?

Strictly JSON is of course a character string

Depends how you are processing it. One received topic is better than many for emoncms.

Did you suggest at one point an interfacer for emonhub that would read a specific topic and repost onto emoncms?

In most cases like this, I’d process the JSON in Node-RED (I have an instance on my PVE setup just for this) and republish on the emon/ topic as a single JSON.

[ramcq (Robert McQueen)]

I went down the JSON route because it was already an accepted format for emoncms and offered a more generic path to parsing the MQTT topics which have multiple attributes posted in one topic, avoiding the need to write specific parsers into EmonHub to unpack ebusd’s particular string format. I’d much rather see a properly documented/tested/supported path to plug these components together directly than having to rely on “code as configuration” in something like Node-RED, but that’s just how my brain works.

[borpin (Brian Orpin)]

@ramcq

because it was already an accepted format for EmonHub

Do you mean emonhub or emoncms?

The issue is that everything sends data in a different structure and format, so as generic as possible solution (such as Node-RED) is often the path of least resistance.

[ramcq (Robert McQueen)]

@borpin

Do you mean emonhub or emoncms?

Sorry, thinko. Corrected!

@borpin

The issue is that everything sends data in a different structure and format, so as generic as possible solution (such as Node-RED) is often the path of least resistance.

Least resistance for who?! If we want to promote people using ebusd together with the heat pump app, being able to offer a working configuration that doesn’t require third party intermediary software seems a very compelling and reasonable thing to do.

[borpin (Brian Orpin)]

@ramcq

Least resistance for who?!

For some poor smuck trying to maintain something they themselves don’t use

@ramcq

being able to offer a working configuration that doesn’t require third party intermediary software seems a very compelling and reasonable thing to do.

You will always need some intermediary software of some sort to act as a bridge for how ebusd sends the data and how emoncms expects it. How you do that and how generic you make it is the issue. Every device will send the data in a specific format so you’ll need a decoder for each.

What then happens is the devices the ‘devs’ use generally are the ones that get a decoder and the others don’t as the devs don’t use it so creating, debugging and maintaining it has no benefit to them (HomeAssistant is a great example of this - if a dev uses something it gets fixed; if they don’t it doesn’t). Such is open source.

So the more generic your intermediary software is, the better. emonhub is actually a pretty generic intermediary layer, but even there you end up with specific decoders.

[Jaspers156 (JGJ)]

This was actually very helpfull. I have now a raspberry 5 with ebusd shield 5 directly from China and data in emonhub/emoncms via mqtt. Topic emon\ in emonhub conf file, and emon\ebusd in ebusd conf file. Quite a puzzle but data is flowing at least. thanks.