Phantom Duplicated MQTT Topics after reboot

glyn.hudson · 2016-05-13T08:16:32.140Z

Interesting, that’s for the invite. Have emonSD build are you running?

kevin:

Do the new inputs have identical names to the old ones? Yes, I’ve seen duplicate topics also appear in “inputs” that are created by the publish to MQTT action, and as they still work in Openhab I expect they have the same name.

Yes, good point. I’m also running openHAB and have noticed this. This proves that the new mqtt topic name is exactly the same. Maybe the name just changed for a second or two which triggered emoncms to created new inputs.

kevin · 2016-05-13T09:33:05.313Z

@glyn.hudson I’m running a home brew image based around your EMONPI image build instructions from the middle of February. To this I added the new MQTT setup from the forum instructions, and also the feed creation and mqtt_input fixes. Its running on a Pi3 with RFM69 without the display script as I need access to all my GPIO’s. It also doesn’t have Node Red or Lightwave.

peter · 2016-05-16T10:31:08.503Z

I just had this duplicate name for the first time - I was updating node-red to publish more topics under the same hierarchy and the original input went n/a and three new ones popped up, one with the same name. Could this be something to do with the way the MQTT subscriber iterates through multiple inputs?

kevin · 2016-05-17T23:16:01.226Z

@glyn.hudson One thing I have discovered, whilst looking for something completely different, is in Openhab.cfg I had

 mqtt:mosquitto.retain=true

set which is odd as I don’t remember ever setting it. I’ve now set

mqtt:mosquitto.retain=false

Will let you know if any duplicates arise with the new setting - as my Openhab is mature it doesn’t get rebooted as often as it used to so it may be a few days before any problems show up.

glyn.hudson · 2016-05-17T23:25:35.670Z

mqtt:mosquitto.retain=true

is set in the default openenergymonitor openhab config

github.com

openenergymonitor/oem_openHab/blob/master/openhab.cfg#L41

    
      
          #
          # Define your MQTT broker connections here for use in the MQTT Binding or MQTT
          # Persistence bundles. Replace <broker> with a id you choose.
          #
          
          
# URL to the MQTT broker, e.g. tcp://localhost:1883 or ssl://localhost:8883
          mqtt:mosquitto.url=tcp://localhost:1883
          mqtt:mosquitto.user=emonpi
          mqtt:mosquitto.pwd=emonpimqtt2016
          mqtt:mosquitto.qos=2
          mqtt:mosquitto.retain=false

How do you think this could be related to the issue we are discussing?

kevin · 2016-05-17T23:39:31.287Z

@glyn.hudson I’m not entirely sure, although I seem to remember someone mentioned retained messages as a possible problem when I reread the thread on the old forum before I started this one
I run a non OEM copy of Openhab, and the value on my system was set, yet on the line above in openhab.cfg it says:

# Optional. True or false. Defines if the broker should retain the messages sent to
# it. Defaults to false.

Suggesting that “true” isn’t a default.
In the V1.8 Runtime I just found in my downloads folder it isn’t set at all, and as I don’t remember setting it, its either a default from the V1.7 days or something I’ve done whilst troubleshooting a long time ago, and definitely before the MQTT changes that lead to the problem appearing.
I can understand how the retained messages setting would be useful, but on a read only file system its likely that it doesn’t work too well unless wherever it retains them has been moved to a writable location. In my case Openhab lives in /home/pi/data/openhab so it can write what it wants, but not knowing about this setting until now everything is setup to not use it.
I’ll leave it at false and report what happens.

Paul · 2016-05-18T08:22:06.323Z

I raised the issue of retained messages here as the likely cause back in March.
Further reading about them can be found in this guide.
@kevin - even though you’ve set it to false, you may need to delete the retained messages which are slushing about in your system. The guide link above explains how, and I flushed mine by setting up a MQTT node in node-red for each of the ‘ghost’ MQTT ID’s, with a zero byte payload and retain flag set.
I changed the ID for each of the ghosts in turn, sending just one message to each, and they have not returned since.

As a test, why not just delete half of them initially, and then see what, if any ghosts reappear.

I described my issue in this post in the node-red forum, and was kindly assisted by Dave Conway-Jones (IBM nodered developer).

Paul

glyn.hudson · 2016-05-18T09:51:11.128Z

Sorry I missed your comment regarding retained messages the first time round.

I have now changed mqtt:mosquitto.retain=false in the default openhab config. This change will get pulled in by the image image since openhab.cfg is symlinked eg.

/etc/openhab/configurations/openhab.cfg -> /home/pi/oem_openHab/openhab.cfg

Unless of course a users has made custom changes to the config. In that case they will need to make the change manually.

kevin · 2016-05-18T10:04:01.921Z

@Paul @glyn.hudson, when I looked into this back then I noted that mosquito.conf referenced a retained messages file on the read only part of the file system. Is it worthwhile amending this and the documentation so it is redirected into somewhere like /tmp that is stores in RAM assuming the permissions are set correctly? That way it can’t retain anything between reboots. (Never quite finished working on this back then due to life getting in the way)

glyn.hudson · 2016-05-18T10:07:50.273Z

Persistence is turned off in Mosquito by default on the current emonSD image. Your correct that persistence_location /var/lib/mosquitto/, is mentioned however since persistence is turned off this location will not be used.

Here is the default mosquitto conf on the image.:

pi@emonpi:~ $ cat /etc/mosquitto/mosquitto.conf 
# Place your local configuration in /etc/mosquitto/conf.d/
#
# A full description of the configuration file is at
# /usr/share/doc/mosquitto/examples/mosquitto.conf.example

pid_file /var/run/mosquitto.pid

persistence false
persistence_location /var/lib/mosquitto/

log_dest file /var/log/mosquitto/mosquitto.log

include_dir /etc/mosquitto/conf.d

allow_anonymous false
password_file /etc/mosquitto/passwd

kevin · 2016-05-18T10:19:36.674Z

@glyn perhaps this is where Openhab tried to store messages when persistence=true was set so after every reboot it got a few messages from the last time the file system was writable

mrandy · 2016-05-27T19:01:25.951Z

I’ve been looking at this issue in my local installation. I think the issue has to do with phpmqtt_input.php and input_model.php in the EmonCMS project. After a reboot, they load a list of node names and input names from Redis (assuming it is enabled), and use that list of channels as a basis for deciding when to create new inputs in the future.

The problem is (at least on my installation) that Redis doesn’t appear to save that info across reboots, and so it feeds an empty list of nodes and channels into EmonCMS.

I switched the code to load the list from MySQL, and suddenly everything is back to normal (almost, I did have to delete all of the extra channels from MySQL).

I’m not sure if this was the right way to fix this, but hopefully it helps someone figure it out. I edited input_model.php around line 126 to always use the MySQL option.

peter · 2016-12-29T14:53:21.568Z

Sorry to wake an old thread but did anything happen with tracking this down in the end?

I recently “lost” my emonpi for a while after a brown out caused it to shutdown and the network switch is was connected to hard failed. Now replaced, plugged the SD card into another RPi and let the long fsck complete before putting it back in place and now I have all duplicated inputs.

Before I manually fix this I am doing a full backup and update, but would love to know if this will likely happen again?

peter · 2016-12-29T16:37:11.044Z

Ah! What I have discovered is simply deleting the “new” phantom entries, the old ones start working again. Phew.

mattjgalloway · 2017-05-19T21:31:36.566Z

I have this a lot, and I just realised that my emonpi I rebooted 2 days ago had a severe case of this and messed up all my logging :-(.

I could be wrong, but my hunch is that Mosquitto PHP client is multi-threaded, and it’s calling our message function multiple times here:

github.com

emoncms/emoncms/blob/master/scripts/phpmqtt_input.php#L157

    
      
                  $log->warn("Not connected, retrying connection");
                  $mqtt_client->setCredentials($settings['mqtt']['user'],$settings['mqtt']['password']);
                  if(isset($settings['mqtt']['capath']) && $settings['mqtt']['capath'] !== null) {
                      $log->warn("mqtt: using ssl");
                      $mqtt_client->setTlsCertificates($settings['mqtt']['capath'],
                                                       $settings['mqtt']['certpath'],
                                                       $settings['mqtt']['keypath'],
                                                       $settings['mqtt']['keypw']);
                  }
                  $mqtt_client->connect($settings['mqtt']['host'], $settings['mqtt']['port'], 5);
                  // moved subscribe to onConnect callback
          
          
    } catch (Exception $e) {
                  $log->error($e);
                  $subscribed = 0;
              }
          }
          
          
// PUBLISH
          // loop through all queued items in redis
          if ($connected && $pub_count>10) {

And then we have a race.

However it wouldn’t account for why the original inputs are not seen as they should be loaded in the get_inputs call.

Or maybe the get_inputs call is returning nothing… That would explain it. Could redis be not alive before this gets called at boot up?

glyn.hudson · 2017-05-20T13:07:05.633Z

mattjgalloway:

Or maybe the get_inputs call is returning nothing… That would explain it. Could redis be not alive before this gets called at boot up?

A while back MQTT input script was changed to use systemd, it should wait for msql and redis to be running:

github.com

emoncms/emoncms/blob/master/scripts/mqtt_input.service#L32


      
          #  See COPYRIGHT.txt and LICENSE.txt.
          #
          #  ---------------------------------------------------------------------
          #  Emoncms - open source energy visualisation
          #  Part of the OpenEnergyMonitor project:
          #  http://openenergymonitor.org
          ###
          
          [Unit]
          Description=Emoncms MQTT Input Script
          After=mosquitto.service mysql.service redis-server.service
          Documentation=https://github.com/emoncms/emoncms/blob/master/docs/RaspberryPi/MQTT.md
          
          [Service]
          Type=idle
          ExecStart=/usr/bin/php /var/www/emoncms/scripts/phpmqtt_input.php
          
          # Uncomment instead of above to use standard log file, else use systemd log
          # Type=forking
          # ExecStart=/bin/sh -c '/usr/bin/php /var/www/emoncms/scripts/phpmqtt_input.php 2>&1 > /var/log/mqtt_input.log &'

However, redis and apache get started in rc.local after the log folders are created in /var/log, this is need since the /var/logis mounted in ram as tmpfs. I can’t see any harm in added a delay into the MQTT input systemd service to ensure all services are up and running properly.

Are you familiar with how to delay a systemd service from starting? It looks like we would need to involve a .timer file

kevin · 2017-05-20T20:55:17.874Z

Why not just change the MQTT script to start from rc.local too?

pb66 · 2017-05-20T21:24:13.863Z

Or better still write a new service and prioritize it to run early in the boot process just to create all these log files before any of the affected services need them, that way the system can work as it was intended rather than being undermined by all the entries in rc.local that could then be totally removed. No delays, no restarting of services, everything would effectively work just as if the logfiles were never lost.

glyn.hudson · 2017-05-20T21:53:08.462Z

Mmm yeah good suggestions, either of those options could work. However it could get tricky trying to upgrade existing emonPi’s. Before we get carried away it would be good to establish if this is indeed the cause of the issue. I’m struggling to recreate this issue.

borpin · 2017-05-22T18:14:41.162Z

I suspect this is a linked topic

MQTT Strangeness Emoncms

@glyn.hudson and @TrystanLea are insistent on everything using the same topics and so that you can target one specific device they must therefore be broken out to individual topics and it is then emoncms that suffers for it. IMO there is no good reason why emonHub couldn’t breakout all the values to non-time sensitive topics (ie current statuses) that can be used locally and ALSO send the same data as a timestamped payload to emoncms also via MQTT if that’s your preference. That way the data is…