Newly created inputs not appearing in emonCMS via MQTT

[thread renamed from “Configuring new emonhub.conf for emon base and two emonTH sensors” to suit the content]

I’ve been monitoring temp and humidity using an emon base (raspberry pi) loaded with an SD image from a couple of years ago, and logging to emoncms successfully for over a year.
I wanted to add a tx (which I don’t yet have) and another th (which I do have) and decided to get the latest SD image as well (Nov 2016).
I’ve managed to set up wifi and set up a local account on the pi, but it’s not seeing any inputs. I can’t work out what my emonhub.conf should look like. Should I comment out the config for node 5 since I’m not using an emonpi? One of the emonth is configured to send as node 20 and I think the other as node 18. Both use 433 RF.
I want to log the inputs/feeds from both emonth modules, both locally and remotely, and have the two sets of api keys.
Thanks for any help.

If you can post your emonhub.conf (apikeys removed) we can check it for you. The unused node definitions shouldn’t pose a problem.

If you are using a RFM2Pi from a couple of years ago, it may not be using the same baud. If it is an RFM69Pi it will be ok at 38400, but if it is the earlier RFM2Pi it will be 9800 baud if fitted with a rfm12 module, a pre-RFM69Pi RFM2Pi with a rfm69cw module can be 38400 or 57600 depending on era.

It’s easy enough to try 9800 and/or 57600 by editing emonhub.conf, failing that can you post some emonhub.log for us to take a look at?

Thanks for the reply Paul.
I changed the baud rate as suggested, to 9600, and one of the emonth started logging on node 19 to emoncms.org.
No inouts seen locally though, and nothing at all from the other emonth. I have definitely had it working, but possibly not at the same time as the first one.
I’ve swapped the batteries between the two emonth, and that makes no difference. However on the one that’s not working the red led is stuck on, rather than going off and then flashing when it transmits.

Here is emonhub.conf:


[hub]
loglevel = DEBUG

[interfacers]
[[RFM2Pi]]
    Type = EmonHubJeeInterfacer
    [[[init_settings]]]
        com_port = /dev/ttyAMA0
        #com_baud = 38400                        # 9600 for old RFM12Pi
        com_baud = 9600                        # 9600 for old RFM12Pi
    [[[runtimesettings]]]
        pubchannels = ToEmonCMS,
        subchannels = ToRFM12,

        group = 210
        frequency = 433
        baseid = 5                              # emonPi / emonBase nodeID
        quiet = true                            # Report incomplete RF packets (no implemented on emonPi)
        calibration = 230V                      # (UK/EU: 230V, US: 110V)
        # interval =  0                         # Interval to transmit time to emonGLCD (seconds)

[[MQTT]]

    Type = EmonHubMqttInterfacer
    [[[init_settings]]]
        mqtt_host = 127.0.0.1
        mqtt_port = 1883
        mqtt_user = emonpi
        mqtt_passwd = emonpimqtt2016

    [[[runtimesettings]]]
        pubchannels = ToRFM12,
        subchannels = ToEmonCMS,

        # emonhub/rx/10/values format
        # Use with emoncms Nodes module
        node_format_enable = 1
        node_format_basetopic = emonhub/

        # emon/emontx/power1 format - use with Emoncms MQTT input
        # http://github.com/emoncms/emoncms/blob/master/docs/RaspberryPi/MQTT.md
        nodevar_format_enable = 1
        nodevar_format_basetopic = emon/

[[emoncmsorg]]
    Type = EmonHubEmoncmsHTTPInterfacer
    [[[init_settings]]]
    [[[runtimesettings]]]
        pubchannels = ToRFM12,
        subchannels = ToEmonCMS,
        url = https://emoncms.org
        apikey = <my write api key for emoncms.org>
        senddata = 1                    # Enable sending data to Emoncms.org
        sendstatus = 1                  # Enable sending WAN IP to Emoncms.org MyIP > https://emoncms.org/myip/list
        sendinterval= 30                # Bulk send interval to Emoncms.org in seconds

[nodes]


[[5]]
    nodename = emonpi
    [[[rx]]]
        names = power1,power2,power1pluspower2,vrms,t1,t2,t3,t4,t5,t6,pulsecount
        datacodes = h, h, h, h, h, h, h, h, h, h, L
        scales = 1,1,1,0.01,0.1,0.1,0.1,0.1,0.1,0.1,1
        units = W,W,W,V,C,C,C,C,C,C,p

[[6]]
    nodename = emontxshield
    [[[rx]]]
       names = power1, power2, power3, power4, vrms
       datacode = h
       scales = 1,1,1,1,0.01
       units =W,W,W,W,V

[[7]]
   nodename = emontx4
   [[[rx]]]
      names = power1, power2, power3, power4, vrms, temp1, temp2, temp3, temp4, temp5, temp6, pulse
      datacodes = h,h,h,h,h,h,h,h,h,h,h,L
      scales = 1,1,1,1,0.01,0.1,0.1, 0.1,0.1,0.1,0.1,1
      units =W,W,W,W,V,C,C,C,C,C,C,p

[[8]]
    nodename = emontx3
    [[[rx]]]
       names = power1, power2, power3, power4, vrms, temp1, temp2, temp3, temp4, temp5, temp6, pulse
       datacodes = h,h,h,h,h,h,h,h,h,h,h,L
       scales = 1,1,1,1,0.01,0.1,0.1, 0.1,0.1,0.1,0.1,1
       units =W,W,W,W,V,C,C,C,C,C,C,p

[[9]]
   nodename = emontx2
   [[[rx]]]
      names = power1, power2, power3, power4, vrms, temp1, temp2, temp3, temp4, temp5, temp6, pulse
      datacode = h
      scales = 1,1,1,1,0.01,0.1,0.1, 0.1,0.1,0.1,0.1,1
      units =W,W,W,W,V,C,C,C,C,C,C,p

[[10]]
    nodename = emontx1
    [[[rx]]]
       names = power1, power2, power3, power4, vrms, temp1, temp2, temp3, temp4, temp5, temp6, pulse
       datacode = h
       scales = 1,1,1,1,0.01,0.1,0.1, 0.1,0.1,0.1,0.1,1
       units =W,W,W,W,V,C,C,C,C,C,C,p

[[11]]
    nodename = 3phase
    [[[rx]]]
       names = powerL1, powerL2, powerL3, power4, Vrms, temp1, temp2, temp3, temp4, temp5, temp6
       datacode = h
       scales = 1,1,1,1,0.01,0.1,0.1,0.1,0.1,0.1,0.1
       units =W,W,W,W,V,C,C,C,C,C,C

[[19]]
   nodename = emonth1
   [[[rx]]]
      names = temperature, external temperature, humidity, battery
      datacode = h
      scales = 0.1,0.1,0.1,0.1
      units = C,C,%,V

[[20]]
   nodename = emonth2
   [[[rx]]]
      names = temperature, external temperature, humidity, battery
      datacode = h
      scales = 0.1,0.1,0.1,0.1
      units = C,C,%,V

[[21]]
   nodename = emonth3
   [[[rx]]]
      names = temperature, external temperature, humidity, battery
      datacode = h
      scales = 0.1,0.1,0.1,0.1
      units = C,C,%,V

[[22]]
   nodename = emonth4
   [[[rx]]]
      names = temperature, external temperature, humidity, battery
      datacode = h
      scales = 0.1,0.1,0.1,0.1
      units = C,C,%,V

[[23]]
    nodename = emonth5
    [[[rx]]]
       names = temperature, external temperature, humidity, battery, pulsecount
       datacodes = h,h,h,h,L
       scales = 0.1,0.1,0.1,0.1,1
       units = C,C,%,V,p

[[24]]
    nodename = emonth6
    [[[rx]]]
       names = temperature, external temperature, humidity, battery, pulsecount
       datacodes = h,h,h,h,L
       scales = 0.1,0.1,0.1,0.1,1
       units = C,C,%,V,p

[[25]]
    nodename = emonth7
    [[[rx]]]
       names = temperature, external temperature, humidity, battery, pulsecount
       datacodes = h,h,h,h,L
       scales = 0.1,0.1,0.1,0.1,1
       units = C,C,%,V,p

[[26]]
    nodename = emonth8
    [[[rx]]]
       names = temperature, external temperature, humidity, battery, pulsecount
       datacodes = h,h,h,h,L
       scales = 0.1,0.1,0.1,0.1,1
       units = C,C,%,V,p

On the emonTH where the LED turns on and remains on:

The LED is turned on at the very start of the setup routine, and should be turned off very close to the end. If it stays on, we can assume that it is not getting to the end of the setup. The likeliest place for it to stop, from experience, is setting up the radio module. If the radio does not respond, the sketch will stop there. Have a very careful look at the soldering around the radio module. It’s possible that you have a bad joint, and moving and working with the unit has given you an intermittent connection.

Thanks Robert. Will get someone to check the soldering out for me.

I’m still not having any luck with local logging of the one working emonTH though.

The emonhub log looks like this, with https://emoncms.org replaced by <emoncms.org> because I’m not allowed to put more than two links yet!

2016-12-18 13:05:21,206 INFO     RFM2Pi     Publishing: emon/18/5 27
2016-12-18 13:05:21,211 INFO     RFM2Pi     Publishing: emon/18/rssi 0
2016-12-18 13:05:21,217 INFO     RFM2Pi     Publishing: emonhub/rx/18/values 1920,4720,0,0,27
2016-12-18 13:05:21,222 INFO     RFM2Pi     Publishing: emonhub/rx/18/rssi 0
2016-12-18 13:05:21,228 DEBUG    RFM2Pi     2728 adding frame to buffer => [1482066321.169473, 18, 1920, 4720, 0, 0, 27]
2016-12-18 13:05:21,231 DEBUG    RFM2Pi     2728 Sent to channel(end)' : ToEmonCMS
2016-12-18 13:05:32,255 INFO     emoncmsorg sending: https://emoncms.org/input/bulk.json?apikey=E-M-O-N-C-M-S-A-P-I-K-E-Y&data=[[1482066321.169473,18,1920,4720,0,0,27]]&sentat=1482066332
2016-12-18 13:05:32,465 DEBUG    emoncmsorg acknowledged receipt with 'ok' from https://emoncms.org
2016-12-18 13:05:32,469 INFO     emoncmsorg sending: https://emoncms.org/myip/set.json?apikey=E-M-O-N-C-M-S-A-P-I-K-E-Y
2016-12-18 13:06:02,308 INFO     emoncmsorg sending: https://emoncms.org/input/bulk.json?apikey=E-M-O-N-C-M-S-A-P-I-K-E-Y&data=[]&sentat=1482066362
2016-12-18 13:06:02,525 DEBUG    emoncmsorg acknowledged receipt with 'ok' from https://emoncms.org
2016-12-18 13:06:02,528 INFO     emoncmsorg sending: https://emoncms.org/myip/set.json?apikey=E-M-O-N-C-M-S-A-P-I-K-E-Y
2016-12-18 13:06:19,168 DEBUG    RFM2Pi     2729 NEW FRAME : 18 138 7 122 18 0 0 0 0 27 0
2016-12-18 13:06:19,178 DEBUG    RFM2Pi     2729 Timestamp : 1482066379.17
2016-12-18 13:06:19,181 DEBUG    RFM2Pi     2729 From Node : 18
2016-12-18 13:06:19,183 DEBUG    RFM2Pi     2729    Values : [1930, 4730, 0, 0, 27]
2016-12-18 13:06:19,194 DEBUG    RFM2Pi     2729 Sent to channel(start)' : ToEmonCMS
2016-12-18 13:06:19,198 INFO     RFM2Pi     Publishing: emon/18/1 1930
2016-12-18 13:06:19,215 INFO     RFM2Pi     Publishing: emon/18/2 4730
2016-12-18 13:06:19,225 INFO     RFM2Pi     Publishing: emon/18/3 0
2016-12-18 13:06:19,236 INFO     RFM2Pi     Publishing: emon/18/4 0
2016-12-18 13:06:19,257 INFO     RFM2Pi     Publishing: emon/18/5 27
2016-12-18 13:06:19,294 INFO     RFM2Pi     Publishing: emon/18/rssi 0
2016-12-18 13:06:19,316 INFO     RFM2Pi     Publishing: emonhub/rx/18/values 1930,4730,0,0,27
2016-12-18 13:06:19,326 INFO     RFM2Pi     Publishing: emonhub/rx/18/rssi 0
2016-12-18 13:06:19,332 DEBUG    RFM2Pi     2729 adding frame to buffer => [1482066379.168057, 18, 1930, 4730, 0, 0, 27]
2016-12-18 13:06:19,335 DEBUG    RFM2Pi     2729 Sent to channel(end)' : ToEmonCMS
2016-12-18 13:06:32,379 INFO     emoncmsorg sending: https://emoncms.org/input/bulk.json?apikey=E-M-O-N-C-M-S-A-P-I-K-E-Y&data=[[1482066379.168057,18,1930,4730,0,0,27]]&sentat=1482066392
2016-12-18 13:06:33,667 DEBUG    emoncmsorg acknowledged receipt with 'ok' from https://emoncms.org
2016-12-18 13:06:33,670 INFO     emoncmsorg sending: https://emoncms.org/myip/set.json?apikey=E-M-O-N-C-M-S-A-P-I-K-E-Y

Sorry, the replacement for the emoncms link doesn’t appear at all.

You should be able to post more links now, your user status has been bumped up a notch.

I have also swapped back all the

<emoncms.org>

references in the log as they were being interpreted as html tags.

Can you expand on

It looks fine in the logs

node 18, 19.3°C, 47.3%, no external temp sensor or pulsecount and 2.7v battery voltage.

What node decoder emonhub.conf entry are you using for node 18? I can’t see an entry in the post above.

The ‘nodename’ should be letters not a number. I don’t think mqtt topics work with numbers e.g the logfile should show posting to mqtt topic like ‘emon/emonth1/temperature’

Wow! That’s interesting and puts a different perspective on things!

Just ran some tests to try this out and found as far as emonHub and Mosquitto are concerned numbers are fine, but any topics with numeric node names or numeric index’s (input names) are ignored by the emoncms MQTT input api.

I currently have an emonPi running the very latest emonSD on the bench, it is also posting to emoncms.org.
I tested this by commenting out the “nodename” line for the emonpi entry in emonhub.conf, without a nodename I could see the data published to emon/5/power1 etc (and the data is posting ok to emoncms.org as node 5) but it is not seen at the local emoncms.

i can however subscribe to the topic from the command line

pi@emonpi(ro):~$ mosquitto_sub -v -u 'emonpi' -P 'emonpimqtt2016' -t 'emon/5/#'
emon/5/power1 0
emon/5/power2 0
emon/5/power1pluspower2 0
emon/5/vrms 0
emon/5/t1 0
emon/5/t2 0
emon/5/t3 0
emon/5/t4 0
emon/5/t5 0
emon/5/t6 0
emon/5/pulsecount 0
emon/5/rssi 0

so this would confirm the MQTT is ok, it’s just not being accepted by emoncms.

I tested this further by commenting out the input names ("names = ") line in the conf

pi@emonpi(ro):~$ mosquitto_sub -v -u 'emonpi' -P 'emonpimqtt2016' -t 'emon/5/#'
emon/5/3 0
emon/5/4 0
emon/5/5 0
emon/5/6 0
emon/5/7 0
emon/5/8 0
emon/5/9 0
emon/5/10 0
emon/5/11 0
emon/5/rssi 0
emon/5/1 0
emon/5/2 0

again it works fine in emonhub and mosquitto, I then tried setting a nodename = 5 which made no noticeable change at either emoncms instance nor mosquitto.

I then returned the nodenames line to “emonpi” expecting the local emoncms to start updating with a second set of “emonpi” inputs named “1” through to “11” but it didn’t, again all ok in mosquitto.

pi@emonpi(ro):~$ mosquitto_sub -v -u 'emonpi' -P 'emonpimqtt2016' -t 'emon/emonpi/#'
emon/emonpi/1 0
emon/emonpi/2 0
emon/emonpi/3 0
emon/emonpi/4 0
emon/emonpi/5 0
emon/emonpi/6 0
emon/emonpi/7 0
emon/emonpi/8 0
emon/emonpi/9 0
emon/emonpi/10 0
emon/emonpi/11 0
emon/emonpi/rssi 0

This confirms the emoncms MQTT input api will not accept numeric node names or input names, or so I thought !!!

With the emonhub.conf back to original, I tried swapping out “power2” to just “2” to test if an explicit numerically named topic would work.

pi@emonpi(ro):~$ mosquitto_sub -v -u 'emonpi' -P 'emonpimqtt2016' -t 'emon/emonpi/#'
emon/emonpi/power1 0
emon/emonpi/2 0
emon/emonpi/power1pluspower2 0
emon/emonpi/vrms 0
emon/emonpi/t1 0
emon/emonpi/t2 0
emon/emonpi/t3 0
emon/emonpi/t4 0
emon/emonpi/t5 0
emon/emonpi/t6 0
emon/emonpi/pulsecount 0
emon/emonpi/rssi 0

I was quite surprised to see an input named “2” was created in the local emoncms (and power2 stopped updating), wondering how that happened I returned that input name back to “power2” in emonhub.conf and edited “power1” to just “1” and was even more surprised to see the previously created input “2” replaced by an input “1” in emoncms, however I do not know if it was the same inputid renamed or it was deleted and replaced with a new inputid.

But as I was confused by how a single “numeric” topic is handled differently to a batch of numerically named topics posted individually, so I tried commenting out the “names =” line again in emonhub.conf and what happened then really threw me. the input “1” (previously input “2”) changed to input “11”, which had not been previously tested individually using a numeric name!!!

<img src=“/uploads/default/original/2X/0/0652b7049b065596a7544b475a4f55cc4ffb1444.png” width="690"height=“270”>

I suspect this last revelation in particular is very much linked to both the “duplicated inputs at reboot” and the “unable to save processlist” issues rather than the MQTT input API’s reluctance to accept numeric node and input names though.

I hope this info will help in debugging the issues mentioned and perhaps the way the emoncms MQTT input API handles node and input names should be improved to accommodate numeric names too?

Sorry for the long post I sort of wrote it as I was doing further tests and had no idea of the full results when I started :slight_smile:

wow, thanks so much for testing @pb66

Very useful to know.

It sounds like the updated numerical node ID is getting into Redis which is why is shows up in the process list but not getting into MYSQL which could be why the feeds stops updating.

Yes, this sounds like the same issue as:

Thanks for sorting out the links Paul.

Adding names to the inputs worked. I now have feeds and am playing around with the graphs. I tried adding another emonTH entry for node 18, but haven’t managed to get that working properly yet, so have commented it out for now. Thanks for all your help.

Good to hear!

@glyn.hudson - I just wanted to add another observation to the findings above. After that little debugging session I had left the emonPi running for other reasons and the names = line was left commented out as per that last test that only showed one numeric input name (2, then 1, then 11) and was ignoring all the others. I have just noticed the other numeric input names are now there, although I had noticed there had been no change a couple of days after my original test.

I suspect a recent reboot has overcome emoncms’s reluctance to accept numeric input names which may indeed point the finger at the role Redis plays.

EDIT - I just tried another reboot to see if they persist a reboot and they do AND my 2nd emonTH has now popped up that was not there when I first wrote this post less than 10mins ago!

It’s a test emonTH and the batteries had died a couple of months ago and I only just got around to replacing them Tuesday morning (48hrs ago) and it isn’t until the emonPi has been rebooted that it has popped up, un-aided, using stock settings. It did appear immediately on my live system though.

EDIT2 - @SianiAnni have you tried rebooting the emonPi since adding the last emonTH? (with the settings uncommented)

To add further to the existing notes. today I set up a gas meter monitor using a hall sensor, I had assigned it a node ID of 30 and the payload currently contains just 2 unsigned longs, pulse count and uptime in milliseconds. Immediately after powering it up I saw the packets arriving at both my own emoncms and emoncms.org, this comes from a forwarding only read-only Pi running (original) emonHub so additional configuration isn’t mandatory. So far so good!

However! when I checked the emonPi, it was receiving ok and forwarding to emoncms.org ok too, but nothing was being seen at the emonpi/emoncms inputs page except for RSSI. Recalling the numeric node id was probably to blame, I set up an entry in emonhub.conf as

[nodes]
    [[30]]
    nodename = GasMon
        [[[rx]]]
            names = GasUnits, Uptime
            datacode = L
            scales = 0.01, 0.001
            units = m3, Secs

This changed the node name from “Node 30” to “GasMon” on the inputs page but did not show any of the inputs, still only RSSI was present. working on the idea of Redis being a factor, I decided to “Flush Redis” from the admin page. At which point all the other inputs ceased to update any longer, the emonPi, emonTx and 2 emonTH’s all stopped updating, I waited sometime and kept refreshing the page with no sign of any updates. Checked emonhub (via emoncms) and could see the data being published ok. So I rebooted (from the admin page) and after the reboot all was well, all my existing nodes were updating and the “GasMon” was properly created and updating.

I have since tried “Flush Redis” again, twice and neither time did it stop the inputs updating. I will if I get a chance create another new node at some point and re-enact what I have done here again, but on the surface it looks like creating a new node (without a pre-existing emonhub.conf entry) not only doesn’t get created on the inputs page, but it also causes a problem for Redis that cannot be Flushed and at the very least requires a reboot to even recognize the new node.

7 posts were split to a new topic: Unable to see emonTx inputs

Did this issue get solved? I’m about to move a system that uses numeric topics and I’d like to not have to redo them all if I can.

no, we have drawn a blank on this one. Any further insight you notice would be useful.

8 posts were split to a new topic: No data from RFM2PI not reaching emoncms

I have sort of lost the thread off this - are you having trouble with the emonhub or emoncms locally?

If it is just a local logging issue (i.e. the data is logging to emoncms.org OK), why not use the http interface to log locally?

@pb66 Paul, if you publish by http instead does it work? If so, it suggests the problem is with the MQTT input process to emoncms. MQTT in general should be fine with numeric topics. Have you tried subscribing to the topics outside of emoncms?

@glyn.hudson What is the thinking behind using MQTT rather than the HTTP interface? Is it to make the data available to other MQTT clients or is there a processing advantage?