Setting up MQTT for the first time

[haffle (Julian Berry)]

I have an emonPi which runs a local version of emonCMS, and several emonTx/ESP8266’s. I’d like to do a bit more with the data generated, ie using Node-RED automate a smart plug to charge a 12v battery whenever an inverter is producing power.

Is the best way to do this to start using MQTT? I have put some details in the MQTT section of one emonTx (see below), but what else do I need to do to get it so say “Connected: Yes”? I can’t find anything within my emonPi to start an MQTT server (if such a thing is needed), and ssh’ing in to the emonPi and typing sudo service mosquitto restart didn’t do anything

Or can Node-RED (on a Raspberry Pi Zero W) query my local emonCMS data directly?

I don’t have any other MQTT instances on my LAN.

TIA

[borpin (Brian Orpin)]

This is explained in other threads re MQTT and how it works.

The emonPi has the Mosquitto MQTT Broker (rather than server) installed by default and some of the data will pass through that.

The MQTT details on the ESP are for sending (strictly speaking publishing) data to the Broker. emonCMS then subscribes to those topics (i.e. wants and data published to that topic) and uses that an an input. You can subscribe to multiple topics at once in a tree structure and by default, emonCMS subscribes to any data published on the emon/ base topic.

There are 2 ways to get data;

1. you subscribe to the relevant emon topic in Node-RED - effectively look at the input to emonCMS
2. as part of the input processing publish data to a new topic (NOT emon) for Node-RED (NR) to pick up. This is useful where the data has arrived via the HTTP API.

You can simply point NR to the broker (by IP) on the emonPi and subscribe to topics and publish data back (for other systems).

The suggestion is to not run Node-RED on your emonPi (recent thread about a dead emonPi).

Suggest you have a good dig around with Google and understand how MQTT works.

[haffle (Julian Berry)]

Thanks @borpin; I am 99.9% certain that I have set up an MQTT broker on a Raspberry Pi that is also running Homebridge, on 192.168.2.58:1883

I have entered those details in emonCMS on an emonTx/ESP8266 (192.168.2.51), saved it and it changed to “Connected: Yes”.

I have set up a flow within Node-RED (on 192.168.2.57:1880), but the “answer” is coming back blank - whereas I half hoped it would come back with something like “ct1:200 (watts)” or similar.

Is there something very basic that I have omitted, or am I still way off beam?

.
.

.
.

.
.

.
.

[borpin (Brian Orpin)]

@haffle

I am 99.9% certain that I have set up an MQTT broker on a Raspberry Pi that is also running Homebridge, on 192.168.2.58:1883

This would mean all the data goes to that broker and none to the emonCMS it normally goes to.

@haffle

I have entered those details in emonCMS on an emonTx/ESP8266

emonCMS does not run on an ESP8266.

I’m not familiar with how this is setup - The instructions are a little ambiguous

I suggest first box should be emon, second garage`.

In NR, to test, subscribe to emon/garage/# and I expect you will get a number of sub topics.

[haffle (Julian Berry)]

This is such fun when it comes together and works

By slightly changing things within the espwifi3012 that is attached to the emonTx in the barn (thanks for clarifying this, @borpin ), and being more explicit within NR (ie cutting out things like ‘pulse’, ‘spent’, & ‘Vrms’ etc, the 10 second feed value of the specific CT that I am interested in (“real time” pv generation in watts) flows through.

All I now need to do is work out what to do with this data

[borpin (Brian Orpin)]

This data is no longer going to emonCMS - is that what you want?

You can just send the data via MQTT to the emonPi and then connect NR to that Broker. Better to only have one Broker on the network.

Either than or bridge the 2 but that is not usually required on same network.

[haffle (Julian Berry)]

I might be misunderstanding something here, and I did check, but the data (from the only CT I have on this emonTx/ESP8266 combo) is actually arriving in both NR and my local version of emonCMS on the emonPi.

Every 10 seconds, the same digit (PV generation) appears in both the NR debug window AND the emonCMS Feed page.

I certainly don’t want to cut out the data going to emonCMS - I just wanted to effectively create a duplicated branch so that I could then work with it in a different way (c/f control a smart plug if, say, PV generation is >=50w).

[borpin (Brian Orpin)]

@haffle

I might be misunderstanding something here, and I did check, but the data (from the only CT I have on this emonTx/ESP8266 combo) is actually arriving in both NR and my local version of emonCMS on the emonPi.

Ah OK, I thought the ESP was an either or (MQTT/HTTP). I don’t use it.

But does the emonPi receive the data via RFM or is it out of range?

Again, I think the emonTX will still transmit the data by RFM, but if it is out of range of the emonPi you will only get it once.

[haffle (Julian Berry)]

99% sure my system doesn’t use the RFM due to extended distances between the Tx’s and the emonPi. (Back when I initially set things up I even put RF signal blockers on the aerial screw threads).

For a temporary “proof of concept”, I have now been able to cobble together a NR flow that turns on a Hue lamp on when the barns PV are producing >100w, and turn it off when not. Next stage will be to replace the lamp with a smart plug, & replace the generation Input to an export Input, so that I can use otherwise exported generation to charge a 12v battery which will in turn keep a classic car storage “bubble” inflated FOC for much of the year.

I am tempted to think that the espwifi3012 widget is able to transmit via MQTT to NR and to my emonPi via wifi / emonCMS, as data is arriving in both places. Perhaps @TrystanLea could confirm this?

Most excellent so far!

[haffle (Julian Berry)]

I have now included a routine whereby if the panels produce <2 watts NR generates an email saying so. This will be handy to monitor the inverter switching off due unexpectedly which is great. It’ll also send me an email each evening as the sun goes down, but I can cope with that for now.

Is there a way to get an emonPi to send MQTT messages? There isn’t the same type of “Control Panel” that the esp8266 wifi devices have. Not sure where else to look …

TIA

[borpin (Brian Orpin)]

@haffle

Is there a way to get an emonPi to send MQTT messages? There isn’t the same type of “Control Panel” that the esp8266 wifi devices have.

What do you want to send? Look at the input process - you can include a publish to MQTT, but it is to the Broker on the emonPi (as specified in settings - same one in and out - can’t change that - hence only having one on your network) and use a different base topic not emon.

[haffle (Julian Berry)]

Thanks. Whilst I measure my barns pv generation on an emonTx, I measure net in/out on an emonPi; I’d have preferred to have used this net out data to control my smart switch.

I understand that one can / should only have one MQTT broker on a network, and because I am using one on a RaspberryPi I would have liked to get my emonPi to send an MQTT message out to the RPi broker. But it seems like this might be a complication too many, so rather than using a net out figure, I will just use a figure of ~50 at the inverter, which will allow for the constant ~35 draw that the barns & office use 24/7.

One other quick MQTT question … Whilst I understand that it is a very lightweight protocol, where do all of the messages end up? Does one need to clear out a cache every now and again to prevent the SD card from filling up? Or is it all de minimis?

EDITED TO ADD: I’d also like to have an automated email alert for if my panels go offline due to a circuit trip; I just thought that MQTT & NR would be a good solution for this.

[CelsoHenriques]

@borpin

The suggestion is to not run Node-RED on your emonPi (recent thread about a dead emonPi).

Hi @borpin
Can you please elaborate more?
I have been looking for that thread and i am not sure what one you are refering to.
Is this the one?

Thanks

[borpin (Brian Orpin)]

yes it is that one. The partitions etc of the emonSD have been modified to try and reduce wear levels, log files are in ram and a few other tweaks mean that running anything else on an emonSD based system risks the underlying function (running emonCMS).

[borpin (Brian Orpin)]

@haffle

I understand that one can / should only have one MQTT broker on a network, and because I am using one on a RaspberryPi I would have liked to get my emonPi to send an MQTT message out to the RPi

There already is one running on the emonPi (as I said before), so just use that and connect the ESP & NR to that broker.

Then you can use the publish to MQTT process and voila.

NR can of course connect to multiple brokers - another solution.

@haffle

Does one need to clear out a cache every now and again

No.

[CelsoHenriques]

Thanks.
I am looking for another alternative since my idea is to have everything integrated on the same machine.

[borpin (Brian Orpin)]

@CelsoHenriques

I am looking for another alternative since my idea is to have everything integrated on the same machine.

If that machine is a Pi and has emonCMS on it, then I suggest that is a bad idea. Run everything on a VM or on an NUC with an HDD/SSD - different beast but YMMV. Good luck.

[haffle (Julian Berry)]

Not for the first time I have tied myself up in knots

A few days ago …

I installed a Mosquito broker on a headless RPi that is also running Homebridge.
I told an emonTX/ESP8266 to publish data from a CT to that broker
I created a Node-RED flow that uses this MQTT info
All good so far.

I understood - rightly or wrongly - that one should only have one MQTT broker on a LAN. But because I would like to send a Feed data from my emonPi via MQTT so that it can be included in the NR Flow I needed to make some changes to the Broker.

I took it that I ought to delete the MQTT broker on the RPi first. Trouble is, I don’t know how to. I ssh’d in and typed sudo apt-get purge --remove mosquitto* but whilst it seemed to accept the command the MQTT feed within NR is still working, despite a RPi re-boot.

Moving on, I ssh’d in to my emonPi, created a new MQTT password & restarted the Pi following the instructions at https://guide.openenergymonitor.org/technical/credentials/#mqtt but whilst I can log back in to emonpi.local and see the Tx Inputs and Feeds etc, all of the Inputs on the emonPi are showing “n/a” & “NULL”

I am not quite sure what I have done here …

[haffle (Julian Berry)]

OK, I have managed to back out of most of the errors, but I am left with two questions … but one can wait for now.

Getting MQTT data out of a local emonCMS on an emonPi

I have set the emonPi to act as MQTT broker, and it is working well for data arriving from two emonTx/ESP8266’s. But I can’t work out how to publish MQTT out of a CT Input on the emonPi. Does anyone know how to do this?

Within the MQTT section in the espwifi page of an emonTx/ESP8266 I have this.

Within a Node-RED flow I have this (with “MQTT emon” being on my emonPi).

Both work well and do what I want. But the lack of the MQTT “page” when logging in to my emonPi on 192.168.2.50 means that I can’t work out how to tell the emonPi what Topic to use etc. I have added a Process (see below) to one of the CT’s on the emonPi, but I can’t retrieve the data that I want within Node-RED. I’ve tried lots of combinations of Topic forms, but none work. It may well be that the emonPi isn’t even generating any MQTT data of its own, but I don’t know how to even check this.

Any assistance hugely appreciated.

TIA

[burble61 (Chris Amos)]

@haffle, great to see your progress! I’m rather behind you, but on MQTT I used “MQTT Explorer” (free download for most platforms) to understand the structure of all the topics coming out of the broker - it saved a great deal of frustration trying to figure that out from Node-Red & other MQTT clients.

Also helpful in understanding the format of data items I was publishing…

Regards