Posting Data to Emoncms with MQTT (Python)

Integrations
1

I tried to write my own Python script to test this out. It might be interesting to others and I have some questions at the end. That’s the reason for this step-by-step post.

  1. Introduction
  2. Paho-mqtt
  3. Publish 2 values once
  4. Publish 2 values in a loop
  5. Script to background
  6. Remaining questions

Introduction

I suppose you already have a running emoncms and you have a meter/sensor/… where the values should become available in emoncms. Your steps are:

  1. Check if it works out-of-the-box in emoncms.
  2. If not, check if an interfacer for that meter/sensor/… exists and configure it in emonhub.
  3. If not, post Data to Emoncms yourself. The link explains that you can use ‘Emoncms HTTP Input API‘ or ‘MQTT’. I’ll use the MQTT solution for my situation, but unfortunately the link is very short in explanation. So I tried to write my own Python script to test out.

My situation: I have three daisy chained modbus kWh meters to monitor my household (SDM630), heat pump (SDM120) and solar panels (SDM120). My hardware setup is explained in this topic. An emonhub interfacer exists for one modbus meter per cable/RS485-USB, but not for daisy chained meters yet (see this topic). So I’m looking to write a Python script to publish the data to the MQTT broker of OpenEnergyMonitor. Before diving into details, I explain some core concepts, the way I see them.

  1. Emoncms is an open-source web application or processing, logging and visualising energy, temperature and other environmental data. To be able to process data in emoncms it should first be registered as an input.
  2. EmonHub is a piece of software that can read/subscribe or send/publish data to and from a multitude of services. Eg. a SDS011 air-quality sensor that produces values about PM2.5 and PM10 and where an emonhub interfacer exist to have those values as inputs in emoncms. So some interfacers already exist, but what if you have a situation where such interfacer doesn’t exist yet?
  3. Well, MQTT is used as a way to pass data in the OpenEnergyMonitor (OEM) ecosystem. This ‘Message Queuing Telemetry Transport’ allows you to publish data (sensors, meters, …) to a central server. This central server is called a broker and the software OEM uses is Mosquitto. Every topic that is published automatically appears as an input in emoncms, as emoncms subscribes to these emon-topics. I found ‘MQTT Essensials’ on hivemq.com very interesting to learn more about MQTT.

I will just publish a counter and some random value to inputs at emoncms, to be able to focus on the essentials. The end goal on the input page is:

input-emoncms-mqtt
input-emoncms-mqtt768×123 5.53 KB

Mind you: I’m no specialist in neither emoncms, MQTT or Python. So chances are big that I could have done things better. Please let me know.

Paho-mqtt

My inspiration comes from this script of Steve Cope and uses paho-mqtt as Python client library. To install:

~$ pip install paho-mqtt

Publish 2 values once

The basic steps are:

  1. The settings: server ip, username, password, …
  2. Connecting to the MQTT server broker (localhost, IP, domain name, …)
  3. Publish values to emoncms. The topic must start with emon, but the other parts are free to choose. In my case I use emon/mynode/counter and emon/mynode/random which will create 1 node mynode in emoncms with the 2 inputs counter and random. The value of this input will be 1 for counter and a random number for random.

The script python2emoncms.py:

import paho.mqtt.client as mqtt
import random
import time

# Settings
CLEAN_SESSION=True  #See https://www.hivemq.com/mqtt-essentials/ > part 7
broker="localhost"
username="emonpi"   #See [mqtt]-section of /var/www/emoncms/settings.ini
password="emonpimqtt2016"

# Functions, further on assigned to correct callback of paho (see client1.on_... = ...)
def on_disconnect(client, userdata, flags, rc=0):
    m="def-DisConnected flags "+"result code "+str(rc)
    print(m)

def on_connect(client, userdata, flags, rc):
    print("def-Connected flags ",str(flags),"result code ",str(rc))

def on_message(client, userdata, message):
    print("def-Message received  "  ,str(message.payload.decode("utf-8")))

def on_publish(client, userdata, mid):
    print("def-Message published ", str(mid))

# Creating paho-instance and connect
print("Creating client 1 with clean session set to", CLEAN_SESSION)
client1 = mqtt.Client("Python1",clean_session=CLEAN_SESSION)    #create new instance
client1.username_pw_set(username,password)
client1.on_message=on_message
client1.on_connect=on_connect

print("Connecting to MQTT broker",broker)
client1.on_publish = on_publish
client1.connect(broker)
client1.loop_start()

# Publish values to emoncms
value1 = 1
value2=random.randint(0, 10000)/1000
print(value1, " ; " , value2)
ret=client1.publish("emon/mynode/counter",value1)
ret=client1.publish("emon/mynode/random",value2)

# Disconnect
client1.on_disconnect=on_disconnect
time.sleep(3)
#client1.loop()
client1.disconnect()
client1.loop_stop()

The output:

~$ python3 python2emoncms.py 
Creating client 1 with clean session set to True
Connecting to MQTT broker localhost
1  ;  2.158
def-Connected flags  {'session present': 0} result code  0
def-Message published  1
def-Message published  2
def-DisConnected flags result code 0

Open emoncms > Input and you should see 1 node mynode and the 2 inputs counter and random automatically created, displaying their corresponding values. The input page doesn’t always show all the digits (eg. 2.16) as they are stored (eg. 2.158), but they can be made visible on the dashboard (see this topic). You can reexecute the script: the first value will always be 1, the second value is a random value.

Publish 2 values in a loop

When thinking about sensors or meters you’ll want a loop to take a new value every x seconds. I adapted the script (using this as inspiration) to execute the ‘Publish values to emoncms’ part every 10 seconds:

# Publish values to emoncms
value1 = 0
while True:
    value1 = value1 + 1
    value2=random.randint(0, 10000)/1000
    print(value1, " ; " , value2)
    ret=client1.publish("emon/mynode/counter",value1)
    ret=client1.publish("emon/mynode/random",value2)
    time.sleep(10)

The output (as there is an infinite loop, you’ll have to stop the script with ctrl + c):

~$ python3 python2emoncms.py 
Creating client 1 with clean session set to True
Connecting to MQTT broker localhost
1  ;  2.051
def-Connected flags  {'session present': 0} result code  0
def-Message published  1
def-Message published  2
2  ;  7.439
def-Message published  3
def-Message published  4
3  ;  4.935
def-Message published  5
def-Message published  6

The complete script becomes (where I have disable the on_publish callback to have a more condens output):

import paho.mqtt.client as mqtt
import random
import time

# Settings
CLEAN_SESSION=True  #See https://www.hivemq.com/mqtt-essentials/ > part 7
broker="localhost"
username="emonpi"   #See [mqtt]-section of /var/www/emoncms/settings.ini
password="emonpimqtt2016"

# Functions, further on assigned to correct callback of paho (see client1.on_... = ...)
def on_disconnect(client, userdata, flags, rc=0):
    m="def-DisConnected flags "+"result code "+str(rc)
    print(m)

def on_connect(client, userdata, flags, rc):
    print("def-Connected flags ",str(flags),"result code ",str(rc))

def on_message(client, userdata, message):
    print("def-Message received  "  ,str(message.payload.decode("utf-8")))

def on_publish(client, userdata, mid):
    print("def-Message published ", str(mid))

# Creating paho-instance and connect
print("Creating client 1 with clean session set to", CLEAN_SESSION)
client1 = mqtt.Client("Python1",clean_session=CLEAN_SESSION)    #create new instance
client1.username_pw_set(username,password)
client1.on_message=on_message
client1.on_connect=on_connect

print("Connecting to MQTT broker",broker)
#client1.on_publish = on_publish
client1.connect(broker)
client1.loop_start()

# Publish values to emoncms
value1 = 0
while True:
    value1 = value1 + 1
    value2=random.randint(0, 10000)/1000
    print(value1, " ; " , value2)
    ret=client1.publish("emon/mynode/counter",value1)
    ret=client1.publish("emon/mynode/random",value2)
    time.sleep(10)

# Disconnect
client1.on_disconnect=on_disconnect
time.sleep(3)
#client1.loop()
client1.disconnect()

Script to background

One problem remains: if I close my terminal, the script also stops executing. Luckily I found this topic, so:

~$ nohup python3 python2emoncms.py &

To stop the script for one reason or another:

~$ ps ax | grep python2emoncms.py
 696690 pts/0    Sl     0:00 python3 python2emoncms.py
 696696 pts/0    S+     0:00 grep --color=auto python2emoncms.py
~$ kill -9 696690

Remaining questions

Some questions remain… and I don’t know the answers :frowning: :

  1. CLEAN_SESSION is true or should it be set to false?
  2. What about the QOS levels in MQTT?
  3. I’m not sure if the paho-functions loopstart, loop and loop_stop are necassary and their documentation doesn’t make me any wiser.
  4. Is it a problem that the script will never reach the client1.disconnect? The script will only stop on crash/reboot/kill.
  5. If my server reboots, it doesn’t automatically restart the script. Also, if the script crashes, it will not restart. This Linux service could be an inspiration (but I think it should only start after network and mosquitto is up and running), but manually starting the script is okay for me at the moment.

Of course, the script doesn’t do anything valuable: it should be extended to be able to read sensors, meters, … But I didn’t want to make it too complex and focus on "how can you use Python to publish values to emoncms with MQTT’. If you have any remarks to make it better: shoot :slight_smile: .

3

I edited my first post to have a more step-by-step approach. Hopefully somebody has the time to look at it. @Bill.Thomson, last time you had some valuable feedback on my posts :slight_smile: . Maybe some other are more into (paho-)MQTT than I am. Maybe @TrystanLea?

4

Thanks for the nice words.

Last time, I was able to help because the subject was one I’m familiar with. (Modbus)

Although I do have an EmonPi, I use it only as a test bed.
I haven’t used emonCMS for about 4 years. (I use InfluxDB and Grafana to log and display my data) so I’m not
up to speed on MQTT. I did write a short Python script to get data into my Influx database via a pahoMQTT
client, but I ran it for only a few months. It was for an an energy monitor I stopped using about two years ago.
I’ve not used MQTT since then.

5

You can post the data as a valid JSON key:value pair (so as many values as you like at once).

Use a service file and run the script as a service.

Look at feedwriter as an example of how to do it.

6

I run a Python script as a service to control the heating (via PV divert) of a thermal store and buffer store via MQTT transported data.

Dead easy once you know how; MQTT is great.

The code and service config are in the files ts_* in GitHub - greentangerine/PVOutput

7

Thank you for all the replies. In the meantime I worked on my own script… ATM still “in development” not “in production”. But it is always nice to be able to look “over the wall”.

@Bill.Thomson I have already read about InfluxDB and Grafana. As far as I understand it is more flexible than emoncms, but maybe a bit more harder to learn? To be able to really do something, I decided to focus on emoncms and not on InfluxDB and Grafana.

@borpin, ATM I would like to focus on MQTT instead of JSON, because it might be too much for my brains otherwise :wink: . About feedwriter, I think you mean feedwriter and feedwriter.php? Unfortunately the code is not very commented, so for a newbie like me it will take some time to understand what is going on.

@greentangerine, thanks for sharing. Some more info in README.md would help, but I guess I can focus on ts_mqtt.py and ts_mqtt.service for my situation.

8

Two completely different things. Bit like saying I want to use MQTT and not Python.

MQTT is a transport protocol. JSON is a data structure.

9

Indeed, but my point was more that a simple emon/heatpump/P in MQTT allows me to do what I want to do without the need for a data structure like JSON.

10

Based on ts_mqtt.service from greentangerine, this Linux service and /etc/systemd/system/emoncms_mqtt.service I made my own service file. I had to leave out StandardOutput=file:/var/log/${ID}/${ID}.log and StandardError=file:/var/log/${ID}/${ID}.log from ts_mqtt.service for it to work on my system.

sudo nano -w /etc/systemd/system/sdm_mqtt.service
#see service file below
sudo systemctl enable sdm_mqtt
sudo systemctl start sdm_mqtt.service

                                                         
sudo nano -w /etc/systemd/system/sdm_mqtt.service
#see script below
sudo systemctl enable sdm_mqtt
sudo systemctl start sdm_mqtt.service

                                                         
[Unit]
Description=MQTT control of SDM kWh meters
Wants=emoncms_mqtt.service
After=emoncms_mqtt.service

[Service]
Environment='SCRIPT=SDM2emoncms.py'
Environment='ID=sdm_mqtt'
Environment='USER=m2ts'
ExecStart=/usr/bin/python3 /home/${USER}/${SCRIPT}
Restart=always
RestartSec=60
#StandardOutput=file:/var/log/${ID}/${ID}.log
#StandardError=file:/var/log/${ID}/${ID}.log
SyslogIdentifier=${ID}
ExecStartPre=/bin/mkdir -p /var/log/${ID}
ExecStartPre=/bin/chown ${USER} /var/log/${ID}
ExecStartPre=/bin/touch /var/log/${ID}.log
ExecStartPre=/bin/chmod 666 /var/log/${ID}.log

[Install]
WantedBy=multi-user.target