MQTT data getting lost between EmonHub and MQTT but is hitting EmonCMS.org?

Emoncms
, ,
1

As a side issue to the PVOutput.org connector I’ve posted about in another thread, I’ve now just about written a NodeRed flow to achieve this (more later after further testing). However, I’ve hit a snag. I’m picking up Power1/Power2 data from my EmonTX via an RFM69Pi on a RaspPi2 running the 3rd May EmonSD image.
The RFM69Pi is pretty solidly picking up the EMonTX data via EmonHub and posting the data to EmonCMS.org.
So far so good.
That same data should turn up via MQTT in NodeRed for further twiddling, but is producing odd results.
So I tried just looping it and storing it straight back back into EmonCMS (but as a separate data stream).

The results are a little worrying…

NodeRed.png1920×1080 201 KB

The Yellow and Blue are the feeds logged from EmonHub, all good. The Green & Red have gone via MQTT thru a Function and back to EmonCMS via the EmonCMS node. In theory they should be identical, but in practice there are missing chunks everywhere.

Anyone had this/noticed this before? Any solutions? Suggestions?

2

How quickly are you posting and how much data? It looks like a bug in mosquitto mqtt server possibly being swamped.

Thanks for reporting

3

Hi Glyn,

It’s the standard 10 second PHPFINA storage and the standard firmware 5 second transmit from the EmonTX.
The RasPi CPU is about 15-20% loaded and most of that is the EmonHub program. Is that usual?

4

That sounds normal, now many feeds?

How long are your feed names? This will cause an overhead for MQTT.

I would guess the issue is with the PHP MQTT library we are using: GitHub - mgdm/Mosquitto-PHP: A wrapper for the Eclipse Mosquitto™ MQTT client library for PHP.

It would be difficult to test, but we could set up a test to log all mosquito MQTT messages to try and work out if the issue is with the mosquito server or the PHP MQTT client.

5

Hi Glyn,

There are about 40 individual feeds / 10 nodes coming in over the air.
Most of these are at 1 min intervals, although a couple are now on 2 min and 1 is just when ‘things change too much’.
Only the EmonTX is every 10 seconds.

The feed names are average. Longest node name is 10 letters (immersion1), longest feedname is 15 letters (immersion_power), but most are 10. The MQTT names are the same.

Can we give the testing a go? As this issue is affecting the PVOutput.Org connector I’ve written in NodeRed, the lost data back from MQTT is causing the accumulated consumption/generation totals to be be way off track.

6

Your use case sounds reasonable, it would expect (or hope!) the system to be able to deal with that sort of data volume.

I posted a reply on another thread of how to use nodered to setup a log of mqtt messages:

I am away on holiday for a couple of weeks tomorrow, I won’t be able to investigate this any further before I leave I’m afraid. Let me know how you get on.

As a work around if its is the PHP MQTT which is causing an issue you could get nodeRED to post directly to local Emoncms or use emonhub to post to local emoncms, create another emoncms interfacer in emonhub.conf:

Place this in the [interfacers] section of emonhub.conf:

[[emoncmslocal]]
    Type = EmonHubEmoncmsHTTPInterfacer
    [[[init_settings]]]
    [[[runtimesettings]]]
        pubchannels = ToRFM12,
        subchannels = ToEmonCMS,
        #url = http://localhost/emoncms
        apikey = xxxxxxxx
        senddata = 1                    # Enable sending data to Emoncms.org
        sendstatus = 1                  # Enable sending WAN IP to Emoncms.org MyIP > https://emoncms.org/myip/list
        sendinterval= 30                # Bulk send interval to Emoncms.org in seconds