Community
OpenEnergyMonitor

Community

MQTT - too much subscriptions to local broker

Tags: #<Tag:0x00007fc9b294a560> #<Tag:0x00007fc9b294a308>

I was doing a review in my MQTT configurations in Emoncms, than I found something strange.

Emoncms connects to MQTT Broker to get messages… OK!
But this happens every second. So many times that MQTT Broker have connection conflicts.

1479166130: New client connected from 127.0.0.1 as Emoncms input subscriber (c0, k10, u’emonpi’).
1479166130: New connection from 127.0.0.1 on port 1883.
1479166130: Client Emoncms input subscriber already connected, closing old connection.
1479166130: Client Emoncms input subscriber disconnected.
1479166130: New client connected from 127.0.0.1 as Emoncms input subscriber (c0, k10, u’emonpi’).
1479166131: New connection from 127.0.0.1 on port 1883.
1479166131: Client Emoncms input subscriber already connected, closing old connection.
1479166131: Client Emoncms input subscriber disconnected.
1479166131: New client connected from 127.0.0.1 as Emoncms input subscriber (c0, k10, u’emonpi’).
1479166131: New connection from 127.0.0.1 on port 1883.
1479166131: Client Emoncms input subscriber already connected, closing old connection.
1479166131: Client Emoncms input subscriber disconnected.
1479166131: New client connected from 127.0.0.1 as Emoncms input subscriber (c0, k10, u’emonpi’).
1479166131: New connection from 127.0.0.1 on port 1883.
1479166131: Client Emoncms input subscriber already connected, closing old connection.
1479166131: Client Emoncms input subscriber disconnected.
1479166131: New client connected from 127.0.0.1 as Emoncms input subscriber (c0, k10, u’emonpi’).
1479166131: New connection from 127.0.0.1 on port 1883.
1479166131: Client Emoncms input subscriber already connected, closing old connection.
1479166131: Client Emoncms input subscriber disconnected.
1479166131: New client connected from 127.0.0.1 as Emoncms input subscriber (c0, k10, u’emonpi’).
1479166131: New connection from 127.0.0.1 on port 1883.
1479166131: Client Emoncms input subscriber already connected, closing old connection.
1479166131: Client Emoncms input subscriber disconnected.
1479166131: New client connected from 127.0.0.1 as Emoncms input subscriber (c0, k10, u’emonpi’).
1479166131: New connection from 127.0.0.1 on port 1883.
1479166131: Client Emoncms input subscriber already connected, closing old connection.
1479166131: Client Emoncms input subscriber disconnected.

Who did this integration? Is this usual? This is the correct way to Emoncms works?
Is it not able to connect and keep connected?
*** I am doing auth connection.

Thanx.
Alberto

1 Like

Where did you find that log file?

Are you running emonSD pre built image? If so what version?

On the emonsD pre built image emonHub posts data to MQTT and a php script then suscribes to the /emon MQTT topic and posts to Emoncms.

See

user docs

technical docs:

PHP MQTT script:

This file is MQTT_INPUT .LOG, and Emoncms + MQTT are working perfectly.

1479251104: New connection from 192.168.1.12 on port 1883.
1479251104: New client connected from 192.168.1.12 as mosqpub/22329-openhpi (c1, k60).
1479251104: Client mosqpub/22329-openhpi disconnected.
1479251104: New connection from 192.168.1.12 on port 1883.
1479251104: New client connected from 192.168.1.12 as mosqpub/22330-openhpi (c1, k60).
1479251104: Client mosqpub/22330-openhpi disconnected.

1479251106: New connection from 192.168.1.11 on port 1883.
1479251106: New client connected from 192.168.1.11 as mosqpub/23800-emonpi (c1, k60).

1479251106: Client mosqpub/23800-emonpi disconnected.
1479251106: New connection from 192.168.1.11 on port 1883.
1479251106: New client connected from 192.168.1.11 as mosqpub/23801-emonpi (c1, k60).
1479251106: Client mosqpub/23801-emonpi disconnected.

However, as I sent, MQTT Broker is logging too much connections from Emoncms.
I believe that phpmqtt_input.php is not creating a continuous connection with MQTT Broker.

This is Emonhub log…

2016-11-15 21:14:04,949 INFO MQTT Received… [Node: 12] [Data: 925,289,635,46,9,49,48,12]
2016-11-15 21:14:05,530 INFO MQTT Acknowledged receipt with ‘ok’ from http://192.168.1.11/emoncms
2016-11-15 21:14:05,639 INFO MQTT Received… [Node: 11] [Data: 434,204,230,0,37,53,53,39,72,100,100,0]
2016-11-15 21:14:06,035 INFO MQTT Acknowledged receipt with ‘ok’ from http://192.168.1.11/emoncms

Version:

Server Information
Emoncms Version low-write 9.3 | 2016.01.18
Buffer 52 feed points pending write
Writer Daemon is running with sleep 60s
Server OS Linux 4.4.8+
Host emonpi emonpi (127.0.1.1)
Date 2016-11-15 21:18:20 BRST
Uptime 21:18:20 up 1 day, 1 min, 2 users, load average: 0.61, 0.62, 0.66
HTTP Server Apache/2.2.22 (Debian) HTTP/1.1 CGI/1.1 80
Database Version MySQL 5.5.52-0+deb7u1
Host localhost (127.0.0.1)
Date 2016-11-15 21:18:20 (UTC -02:00‌​)
Stats Uptime: 86504 Threads: 1 Questions: 26103 Slow queries: 0 Opens: 60 Flush tables: 1 Open tables: 53 Queries per second avg: 0.301
Redis Version 2.4.14
Host localhost:6379 (127.0.0.1)
Size 152 keys (621.97K)Flush
Uptime 1 days
MQTT Version n/a
Host localhost:1883 (127.0.0.1)
PHP Version 5.4.45-0+deb7u5 (Zend Version 2.4.0)

Is this a problem? It’s working fine and has been proven to be reliable? Should the connection not get renewed?

It may be a good idea to turn off connection logging to avoid filling up the tmpfa ram log partition too quickly by adding to mosquitto.conf:

connection_messages false
log_type warning

though i do not use emon software as i built my own handler and database collection methods . if you are rapidly firing your MQTT then do not close the connection. leave it open if it local collection … I have my emontx shield sending data every second on average and my pi device that is collecting data - also not using emoncms software. but my own software is collecting data from several devices sending MQTT. and average every 0.2 seconds to the PI… I have not suffered any issues by leaving the local connection open- if I close the connection after each send then it will cause a back up of data as it take several extra moments to open and close connections before/after each send… the only issue i had at first if data stream was interrupted by interference- but now if it interrupted for several times in a row then it it will force a renewal of wifi connection and/or MQTT connection. it being running for a few months now. several power outages pi reboots and device reboots- and I have not lost more then 300 seconds ( about 1500 mqtt sends )of data out of the 36 million mqtt sends and recieves in that same time period

Here the problem is not necessary connections, and not necessary process.
This R-Pi is not for Emoncms only, than to save processing time is important.
Also, this method to connect works, but conflicting with it self is not a good practice.
If I do it for a third-party MQTT broker to catch some data will be like a flood.
I’d like to find where connection is called to minimize this “flood” agains MQTT broker.

This is where the php script connects:

The emonPi OpenHAB, nodeRED and lightwaveRF MQTT also connect to MQTT server. These could be the other connections we can see.

@glyn.hudson It’s not the mqtt_input what is causing the problem. I killed the script but “Emoncms Publisher” keep connecting and disconnecting to my broker. Then i found that you are using another MQTT library to publish emoncms values to a MQTT topic.

Here and here

Interesting, I would have thought that the publish to MQTT would only be connecting if there was an input process configured to post to MQTT in Emoncms. Do you have any input process configured to post to MQTT from within Emoncms?

Yes, i have six inputs sending from emoncms to MQTT.

I read a little about that and if you want to keep the connection, you have to use nodejs and express for that purpose. PHP cannot hold the connection alive, instead it connects to broker, sends the data and then disconnects immediately.

Hi,

I have noticed this on my raspberry today.

It seems to be connecting multiple times every second even when things are not being sent from emoncms to MQTT.

I know this as the only inputs i have setup with publish to MQTT processes configured come in once a minute on the minute. Does the function to connect get called for every input update even if publish to MQTT is not selected?

It’s not proving to be much of a problem except when i want to find something in the log i can be searching through thousands of lines for things that happened a couple of minutes apart.

Matt.