MQTT - too much subscriptions to local broker

all.mac07 · 14 November 2016 23:46

I was doing a review in my MQTT configurations in Emoncms, than I found something strange.

Emoncms connects to MQTT Broker to get messages… OK!
But this happens every second. So many times that MQTT Broker have connection conflicts.

1479166130: New client connected from 127.0.0.1 as Emoncms input subscriber (c0, k10, u’emonpi’).
1479166130: New connection from 127.0.0.1 on port 1883.
1479166130: Client Emoncms input subscriber already connected, closing old connection.
1479166130: Client Emoncms input subscriber disconnected.
1479166130: New client connected from 127.0.0.1 as Emoncms input subscriber (c0, k10, u’emonpi’).
1479166131: New connection from 127.0.0.1 on port 1883.
1479166131: Client Emoncms input subscriber already connected, closing old connection.
1479166131: Client Emoncms input subscriber disconnected.
1479166131: New client connected from 127.0.0.1 as Emoncms input subscriber (c0, k10, u’emonpi’).
1479166131: New connection from 127.0.0.1 on port 1883.
1479166131: Client Emoncms input subscriber already connected, closing old connection.
1479166131: Client Emoncms input subscriber disconnected.
1479166131: New client connected from 127.0.0.1 as Emoncms input subscriber (c0, k10, u’emonpi’).
1479166131: New connection from 127.0.0.1 on port 1883.
1479166131: Client Emoncms input subscriber already connected, closing old connection.
1479166131: Client Emoncms input subscriber disconnected.
1479166131: New client connected from 127.0.0.1 as Emoncms input subscriber (c0, k10, u’emonpi’).
1479166131: New connection from 127.0.0.1 on port 1883.
1479166131: Client Emoncms input subscriber already connected, closing old connection.
1479166131: Client Emoncms input subscriber disconnected.
1479166131: New client connected from 127.0.0.1 as Emoncms input subscriber (c0, k10, u’emonpi’).
1479166131: New connection from 127.0.0.1 on port 1883.
1479166131: Client Emoncms input subscriber already connected, closing old connection.
1479166131: Client Emoncms input subscriber disconnected.
1479166131: New client connected from 127.0.0.1 as Emoncms input subscriber (c0, k10, u’emonpi’).
1479166131: New connection from 127.0.0.1 on port 1883.
1479166131: Client Emoncms input subscriber already connected, closing old connection.
1479166131: Client Emoncms input subscriber disconnected.

Who did this integration? Is this usual? This is the correct way to Emoncms works?
Is it not able to connect and keep connected?
*** I am doing auth connection.

Thanx.
Alberto

glyn.hudson · 15 November 2016 19:36

Where did you find that log file?

Are you running emonSD pre built image? If so what version?

On the emonsD pre built image emonHub posts data to MQTT and a php script then suscribes to the /emon MQTT topic and posts to Emoncms.

See

user docs

technical docs:

github.com

emoncms/emoncms/blob/master/docs/RaspberryPi/MQTT.md

**To install emoncms on a Raspberry Pi see new installation script approach:<br>https://github.com/openenergymonitor/EmonScripts**

---

## Archived: Enabling MQTT
MQTT is a "lightweight" messaging protocol, which enables the publishing of data from Emoncms, as well as subscribing to data from other connected devices. The emonPi uses MQTT to transfer data from emonHub to Emoncms.

### Preparation

Before following this guide, it is essential that emoncms was initially installed by following either the Raspbian Stretch, [Raspbian Jessie](readme.md) or [Raspbian Wheezy](install_Wheezy.md) installation guides, or you have used git to install a working version of Emoncms & emonHub on your Raspberry Pi, as well as a MQTT message broker such as [Mosquitto](http://mosquitto.org/) installed and running.

#### Update Emoncms:

    cd /var/www/emoncms && git pull

### Ensure packages are installed

In addition to Mosquitto MQTT server we will need to install [mosquitto-debian-repository]( http://mosquitto.org/2013/01/mosquitto-debian-repository) and [Mosquitto-PHP library](https://github.com/mgdm/Mosquitto-PHP):

    sudo apt-get install libmosquitto-dev

This file has been truncated. show original

PHP MQTT script:

github.com

emoncms/emoncms/blob/master/scripts/phpmqtt_input.php

<?php

    // TBD: support user target in message schema
    $mqttsettings = array(
        'userid' => 1
    );


    /*
    
    **MQTT input interface script**
    
    SERVICE INSTALL INSTRUCTIONS:
    https://github.com/emoncms/emoncms/blob/master/docs/RaspberryPi/MQTT.md
    
    EXAMPLES:
    
    create an input from emonTx node called power with value 10:
        [basetopic]/emontx/power 10

This file has been truncated. show original

all.mac07 · 15 November 2016 23:23

This file is MQTT_INPUT .LOG, and Emoncms + MQTT are working perfectly.

1479251104: New connection from 192.168.1.12 on port 1883.
1479251104: New client connected from 192.168.1.12 as mosqpub/22329-openhpi (c1, k60).
1479251104: Client mosqpub/22329-openhpi disconnected.
1479251104: New connection from 192.168.1.12 on port 1883.
1479251104: New client connected from 192.168.1.12 as mosqpub/22330-openhpi (c1, k60).
1479251104: Client mosqpub/22330-openhpi disconnected.
…
1479251106: New connection from 192.168.1.11 on port 1883.
1479251106: New client connected from 192.168.1.11 as mosqpub/23800-emonpi (c1, k60).
…
1479251106: Client mosqpub/23800-emonpi disconnected.
1479251106: New connection from 192.168.1.11 on port 1883.
1479251106: New client connected from 192.168.1.11 as mosqpub/23801-emonpi (c1, k60).
1479251106: Client mosqpub/23801-emonpi disconnected.

However, as I sent, MQTT Broker is logging too much connections from Emoncms.
I believe that phpmqtt_input.php is not creating a continuous connection with MQTT Broker.

This is Emonhub log…

2016-11-15 21:14:04,949 INFO MQTT Received… [Node: 12] [Data: 925,289,635,46,9,49,48,12]
2016-11-15 21:14:05,530 INFO MQTT Acknowledged receipt with ‘ok’ from http://192.168.1.11/emoncms
2016-11-15 21:14:05,639 INFO MQTT Received… [Node: 11] [Data: 434,204,230,0,37,53,53,39,72,100,100,0]
2016-11-15 21:14:06,035 INFO MQTT Acknowledged receipt with ‘ok’ from http://192.168.1.11/emoncms

Version:

Server Information
Emoncms Version low-write 9.3 | 2016.01.18
Buffer 52 feed points pending write
Writer Daemon is running with sleep 60s
Server OS Linux 4.4.8+
Host emonpi emonpi (127.0.1.1)
Date 2016-11-15 21:18:20 BRST
Uptime 21:18:20 up 1 day, 1 min, 2 users, load average: 0.61, 0.62, 0.66
HTTP Server Apache/2.2.22 (Debian) HTTP/1.1 CGI/1.1 80
Database Version MySQL 5.5.52-0+deb7u1
Host localhost (127.0.0.1)
Date 2016-11-15 21:18:20 (UTC -02:00‌)
Stats Uptime: 86504 Threads: 1 Questions: 26103 Slow queries: 0 Opens: 60 Flush tables: 1 Open tables: 53 Queries per second avg: 0.301
Redis Version 2.4.14
Host localhost:6379 (127.0.0.1)
Size 152 keys (621.97K)Flush
Uptime 1 days
MQTT Version n/a
Host localhost:1883 (127.0.0.1)
PHP Version 5.4.45-0+deb7u5 (Zend Version 2.4.0)

glyn.hudson · 16 November 2016 19:37

Is this a problem? It’s working fine and has been proven to be reliable? Should the connection not get renewed?

It may be a good idea to turn off connection logging to avoid filling up the tmpfa ram log partition too quickly by adding to mosquitto.conf:

connection_messages false
log_type warning

stephen · 16 November 2016 22:32

though i do not use emon software as i built my own handler and database collection methods . if you are rapidly firing your MQTT then do not close the connection. leave it open if it local collection … I have my emontx shield sending data every second on average and my pi device that is collecting data - also not using emoncms software. but my own software is collecting data from several devices sending MQTT. and average every 0.2 seconds to the PI… I have not suffered any issues by leaving the local connection open- if I close the connection after each send then it will cause a back up of data as it take several extra moments to open and close connections before/after each send… the only issue i had at first if data stream was interrupted by interference- but now if it interrupted for several times in a row then it it will force a renewal of wifi connection and/or MQTT connection. it being running for a few months now. several power outages pi reboots and device reboots- and I have not lost more then 300 seconds ( about 1500 mqtt sends )of data out of the 36 million mqtt sends and recieves in that same time period

all.mac07 · 16 November 2016 22:52

Here the problem is not necessary connections, and not necessary process.
This R-Pi is not for Emoncms only, than to save processing time is important.
Also, this method to connect works, but conflicting with it self is not a good practice.
If I do it for a third-party MQTT broker to catch some data will be like a flood.
I’d like to find where connection is called to minimize this “flood” agains MQTT broker.

glyn.hudson · 16 November 2016 22:56

This is where the php script connects:

github.com

emoncms/emoncms/blob/master/scripts/phpmqtt_input.php#L111

    
      
          require_once "Modules/feed/feed_model.php";
          $feed = new Feed($mysqli,$redis,$settings['feed']);
          
          
require_once "Modules/input/input_model.php";
          $input = new Input($mysqli,$redis,$feed);
          
          
require_once "Modules/process/process_model.php";
          $process = new Process($mysqli,$input,$feed,$user->get_timezone($mqttsettings['userid']));
          
          
$device = false;
          if (file_exists("Modules/device/device_model.php")) {
              require_once "Modules/device/device_model.php";
              $device = new Device($mysqli,$redis);
          }
          /*
              new Mosquitto\Client($id,$cleanSession)
              $id (string) – The client ID. If omitted or null, one will be generated at random.
              $cleanSession (boolean) – Set to true to instruct the broker to clean all messages and subscriptions on disconnect. Must be true if the $id parameter is null.
          */ 
          $mqtt_client = new Mosquitto\Client($settings['mqtt']['client_id'],true);

The emonPi OpenHAB, nodeRED and lightwaveRF MQTT also connect to MQTT server. These could be the other connections we can see.

kenci · 17 November 2016 10:12

@glyn.hudson It’s not the mqtt_input what is causing the problem. I killed the script but “Emoncms Publisher” keep connecting and disconnecting to my broker. Then i found that you are using another MQTT library to publish emoncms values to a MQTT topic.

Here and here

glyn.hudson · 17 November 2016 23:53

Interesting, I would have thought that the publish to MQTT would only be connecting if there was an input process configured to post to MQTT in Emoncms. Do you have any input process configured to post to MQTT from within Emoncms?

kenci · 18 November 2016 07:58

Yes, i have six inputs sending from emoncms to MQTT.

I read a little about that and if you want to keep the connection, you have to use nodejs and express for that purpose. PHP cannot hold the connection alive, instead it connects to broker, sends the data and then disconnects immediately.

spike3312 · 3 February 2018 20:28

Hi,

I have noticed this on my raspberry today.

It seems to be connecting multiple times every second even when things are not being sent from emoncms to MQTT.

I know this as the only inputs i have setup with publish to MQTT processes configured come in once a minute on the minute. Does the function to connect get called for every input update even if publish to MQTT is not selected?

It’s not proving to be much of a problem except when i want to find something in the log i can be searching through thousands of lines for things that happened a couple of minutes apart.

Matt.