Newly created inputs not appearing in emonCMS via MQTT

SianiAnni · 2016-12-18T13:21:08.415Z

I’m still not having any luck with local logging of the one working emonTH though.

The emonhub log looks like this, with https://emoncms.org replaced by <emoncms.org> because I’m not allowed to put more than two links yet!

2016-12-18 13:05:21,206 INFO     RFM2Pi     Publishing: emon/18/5 27
2016-12-18 13:05:21,211 INFO     RFM2Pi     Publishing: emon/18/rssi 0
2016-12-18 13:05:21,217 INFO     RFM2Pi     Publishing: emonhub/rx/18/values 1920,4720,0,0,27
2016-12-18 13:05:21,222 INFO     RFM2Pi     Publishing: emonhub/rx/18/rssi 0
2016-12-18 13:05:21,228 DEBUG    RFM2Pi     2728 adding frame to buffer => [1482066321.169473, 18, 1920, 4720, 0, 0, 27]
2016-12-18 13:05:21,231 DEBUG    RFM2Pi     2728 Sent to channel(end)' : ToEmonCMS
2016-12-18 13:05:32,255 INFO     emoncmsorg sending: https://emoncms.org/input/bulk.json?apikey=E-M-O-N-C-M-S-A-P-I-K-E-Y&data=[[1482066321.169473,18,1920,4720,0,0,27]]&sentat=1482066332
2016-12-18 13:05:32,465 DEBUG    emoncmsorg acknowledged receipt with 'ok' from https://emoncms.org
2016-12-18 13:05:32,469 INFO     emoncmsorg sending: https://emoncms.org/myip/set.json?apikey=E-M-O-N-C-M-S-A-P-I-K-E-Y
2016-12-18 13:06:02,308 INFO     emoncmsorg sending: https://emoncms.org/input/bulk.json?apikey=E-M-O-N-C-M-S-A-P-I-K-E-Y&data=[]&sentat=1482066362
2016-12-18 13:06:02,525 DEBUG    emoncmsorg acknowledged receipt with 'ok' from https://emoncms.org
2016-12-18 13:06:02,528 INFO     emoncmsorg sending: https://emoncms.org/myip/set.json?apikey=E-M-O-N-C-M-S-A-P-I-K-E-Y
2016-12-18 13:06:19,168 DEBUG    RFM2Pi     2729 NEW FRAME : 18 138 7 122 18 0 0 0 0 27 0
2016-12-18 13:06:19,178 DEBUG    RFM2Pi     2729 Timestamp : 1482066379.17
2016-12-18 13:06:19,181 DEBUG    RFM2Pi     2729 From Node : 18
2016-12-18 13:06:19,183 DEBUG    RFM2Pi     2729    Values : [1930, 4730, 0, 0, 27]
2016-12-18 13:06:19,194 DEBUG    RFM2Pi     2729 Sent to channel(start)' : ToEmonCMS
2016-12-18 13:06:19,198 INFO     RFM2Pi     Publishing: emon/18/1 1930
2016-12-18 13:06:19,215 INFO     RFM2Pi     Publishing: emon/18/2 4730
2016-12-18 13:06:19,225 INFO     RFM2Pi     Publishing: emon/18/3 0
2016-12-18 13:06:19,236 INFO     RFM2Pi     Publishing: emon/18/4 0
2016-12-18 13:06:19,257 INFO     RFM2Pi     Publishing: emon/18/5 27
2016-12-18 13:06:19,294 INFO     RFM2Pi     Publishing: emon/18/rssi 0
2016-12-18 13:06:19,316 INFO     RFM2Pi     Publishing: emonhub/rx/18/values 1930,4730,0,0,27
2016-12-18 13:06:19,326 INFO     RFM2Pi     Publishing: emonhub/rx/18/rssi 0
2016-12-18 13:06:19,332 DEBUG    RFM2Pi     2729 adding frame to buffer => [1482066379.168057, 18, 1930, 4730, 0, 0, 27]
2016-12-18 13:06:19,335 DEBUG    RFM2Pi     2729 Sent to channel(end)' : ToEmonCMS
2016-12-18 13:06:32,379 INFO     emoncmsorg sending: https://emoncms.org/input/bulk.json?apikey=E-M-O-N-C-M-S-A-P-I-K-E-Y&data=[[1482066379.168057,18,1930,4730,0,0,27]]&sentat=1482066392
2016-12-18 13:06:33,667 DEBUG    emoncmsorg acknowledged receipt with 'ok' from https://emoncms.org
2016-12-18 13:06:33,670 INFO     emoncmsorg sending: https://emoncms.org/myip/set.json?apikey=E-M-O-N-C-M-S-A-P-I-K-E-Y

SianiAnni · 2016-12-18T13:24:15.353Z

Sorry, the replacement for the emoncms link doesn’t appear at all.

pb66 · 2016-12-18T14:10:59.963Z

You should be able to post more links now, your user status has been bumped up a notch.

I have also swapped back all the

<emoncms.org>

references in the log as they were being interpreted as html tags.

Can you expand on

SianiAnni:

I’m still not having any luck with local logging of the one working emonTH though

It looks fine in the logs

node 18, 19.3°C, 47.3%, no external temp sensor or pulsecount and 2.7v battery voltage.

glyn.hudson · 2016-12-21T10:32:48.616Z

What node decoder emonhub.conf entry are you using for node 18? I can’t see an entry in the post above.

The ‘nodename’ should be letters not a number. I don’t think mqtt topics work with numbers e.g the logfile should show posting to mqtt topic like ‘emon/emonth1/temperature’

pb66 · 2016-12-21T12:32:05.053Z

glyn.hudson:

The ‘nodename’ should be letters not a number. I don’t think mqtt topics work with numbers

Wow! That’s interesting and puts a different perspective on things!

Just ran some tests to try this out and found as far as emonHub and Mosquitto are concerned numbers are fine, but any topics with numeric node names or numeric index’s (input names) are ignored by the emoncms MQTT input api.

I currently have an emonPi running the very latest emonSD on the bench, it is also posting to emoncms.org.
I tested this by commenting out the “nodename” line for the emonpi entry in emonhub.conf, without a nodename I could see the data published to emon/5/power1 etc (and the data is posting ok to emoncms.org as node 5) but it is not seen at the local emoncms.

i can however subscribe to the topic from the command line

pi@emonpi(ro):~$ mosquitto_sub -v -u 'emonpi' -P 'emonpimqtt2016' -t 'emon/5/#'
emon/5/power1 0
emon/5/power2 0
emon/5/power1pluspower2 0
emon/5/vrms 0
emon/5/t1 0
emon/5/t2 0
emon/5/t3 0
emon/5/t4 0
emon/5/t5 0
emon/5/t6 0
emon/5/pulsecount 0
emon/5/rssi 0

so this would confirm the MQTT is ok, it’s just not being accepted by emoncms.

I tested this further by commenting out the input names ("names = ") line in the conf

pi@emonpi(ro):~$ mosquitto_sub -v -u 'emonpi' -P 'emonpimqtt2016' -t 'emon/5/#'
emon/5/3 0
emon/5/4 0
emon/5/5 0
emon/5/6 0
emon/5/7 0
emon/5/8 0
emon/5/9 0
emon/5/10 0
emon/5/11 0
emon/5/rssi 0
emon/5/1 0
emon/5/2 0

again it works fine in emonhub and mosquitto, I then tried setting a nodename = 5 which made no noticeable change at either emoncms instance nor mosquitto.

I then returned the nodenames line to “emonpi” expecting the local emoncms to start updating with a second set of “emonpi” inputs named “1” through to “11” but it didn’t, again all ok in mosquitto.

pi@emonpi(ro):~$ mosquitto_sub -v -u 'emonpi' -P 'emonpimqtt2016' -t 'emon/emonpi/#'
emon/emonpi/1 0
emon/emonpi/2 0
emon/emonpi/3 0
emon/emonpi/4 0
emon/emonpi/5 0
emon/emonpi/6 0
emon/emonpi/7 0
emon/emonpi/8 0
emon/emonpi/9 0
emon/emonpi/10 0
emon/emonpi/11 0
emon/emonpi/rssi 0

This confirms the emoncms MQTT input api will not accept numeric node names or input names, or so I thought !!!

With the emonhub.conf back to original, I tried swapping out “power2” to just “2” to test if an explicit numerically named topic would work.

pi@emonpi(ro):~$ mosquitto_sub -v -u 'emonpi' -P 'emonpimqtt2016' -t 'emon/emonpi/#'
emon/emonpi/power1 0
emon/emonpi/2 0
emon/emonpi/power1pluspower2 0
emon/emonpi/vrms 0
emon/emonpi/t1 0
emon/emonpi/t2 0
emon/emonpi/t3 0
emon/emonpi/t4 0
emon/emonpi/t5 0
emon/emonpi/t6 0
emon/emonpi/pulsecount 0
emon/emonpi/rssi 0

I was quite surprised to see an input named “2” was created in the local emoncms (and power2 stopped updating), wondering how that happened I returned that input name back to “power2” in emonhub.conf and edited “power1” to just “1” and was even more surprised to see the previously created input “2” replaced by an input “1” in emoncms, however I do not know if it was the same inputid renamed or it was deleted and replaced with a new inputid.

But as I was confused by how a single “numeric” topic is handled differently to a batch of numerically named topics posted individually, so I tried commenting out the “names =” line again in emonhub.conf and what happened then really threw me. the input “1” (previously input “2”) changed to input “11”, which had not been previously tested individually using a numeric name!!!

I suspect this last revelation in particular is very much linked to both the “duplicated inputs at reboot” and the “unable to save processlist” issues rather than the MQTT input API’s reluctance to accept numeric node and input names though.

I hope this info will help in debugging the issues mentioned and perhaps the way the emoncms MQTT input API handles node and input names should be improved to accommodate numeric names too?

Sorry for the long post I sort of wrote it as I was doing further tests and had no idea of the full results when I started

glyn.hudson · 2016-12-21T13:10:12.630Z

wow, thanks so much for testing @pb66

pb66:

Just ran some tests to try this out and found as far as emonHub and Mosquitto are concerned numbers are fine, but any topics with numeric node names or numeric index’s (input names) are ignored by the emoncms MQTT input api.

Very useful to know.

pb66:

I was quite surprised to see an input named “2” was created in the local emoncms (and power2 stopped updating), wondering how that happened I returned that input name back to “power2” in emonhub.conf and edited “power1” to just “1” and was even more surprised to see the previously created input “2” replaced by an input “1” in emoncms, however I do not know if it was the same inputid renamed or it was deleted and replaced with a new inputid.

It sounds like the updated numerical node ID is getting into Redis which is why is shows up in the process list but not getting into MYSQL which could be why the feeds stops updating.

Yes, this sounds like the same issue as:

Adding a node with MQTT problem Emoncms

Thanks for the extra debug info @Jon. I’m not surprised that the bug still exists. @TrystanLea couldn’t really figure out the exact cause, other the the data is getting into redis db (hence the input appearing in inputs) but not getting put into the mysql db (hence the error). Adding a delay before connecting to mysql seemed to help but it’s obviously not fixed. It’s strange that after a restart input works OK. Thanks again for your help, we’ll try and revisit this week. It’s an awkward little …

SianiAnni · 2016-12-28T11:03:37.094Z

Thanks for sorting out the links Paul.

SianiAnni · 2016-12-29T10:40:36.928Z

Adding names to the inputs worked. I now have feeds and am playing around with the graphs. I tried adding another emonTH entry for node 18, but haven’t managed to get that working properly yet, so have commented it out for now. Thanks for all your help.

pb66 · 2016-12-29T11:50:02.568Z

SianiAnni:

Adding names to the inputs worked. I now have feeds and am playing around with the graphs.

Good to hear!

@glyn.hudson - I just wanted to add another observation to the findings above. After that little debugging session I had left the emonPi running for other reasons and the names = line was left commented out as per that last test that only showed one numeric input name (2, then 1, then 11) and was ignoring all the others. I have just noticed the other numeric input names are now there, although I had noticed there had been no change a couple of days after my original test.

2016-12-29.png1917×933 43.9 KB

I suspect a recent reboot has overcome emoncms’s reluctance to accept numeric input names which may indeed point the finger at the role Redis plays.

EDIT - I just tried another reboot to see if they persist a reboot and they do AND my 2nd emonTH has now popped up that was not there when I first wrote this post less than 10mins ago!

It’s a test emonTH and the batteries had died a couple of months ago and I only just got around to replacing them Tuesday morning (48hrs ago) and it isn’t until the emonPi has been rebooted that it has popped up, un-aided, using stock settings. It did appear immediately on my live system though.

EDIT2 - @SianiAnni have you tried rebooting the emonPi since adding the last emonTH? (with the settings uncommented)

pb66 · 2017-01-02T17:37:30.448Z

To add further to the existing notes. today I set up a gas meter monitor using a hall sensor, I had assigned it a node ID of 30 and the payload currently contains just 2 unsigned longs, pulse count and uptime in milliseconds. Immediately after powering it up I saw the packets arriving at both my own emoncms and emoncms.org, this comes from a forwarding only read-only Pi running (original) emonHub so additional configuration isn’t mandatory. So far so good!

However! when I checked the emonPi, it was receiving ok and forwarding to emoncms.org ok too, but nothing was being seen at the emonpi/emoncms inputs page except for RSSI. Recalling the numeric node id was probably to blame, I set up an entry in emonhub.conf as

[nodes]
    [[30]]
    nodename = GasMon
        [[[rx]]]
            names = GasUnits, Uptime
            datacode = L
            scales = 0.01, 0.001
            units = m3, Secs

This changed the node name from “Node 30” to “GasMon” on the inputs page but did not show any of the inputs, still only RSSI was present. working on the idea of Redis being a factor, I decided to “Flush Redis” from the admin page. At which point all the other inputs ceased to update any longer, the emonPi, emonTx and 2 emonTH’s all stopped updating, I waited sometime and kept refreshing the page with no sign of any updates. Checked emonhub (via emoncms) and could see the data being published ok. So I rebooted (from the admin page) and after the reboot all was well, all my existing nodes were updating and the “GasMon” was properly created and updating.

I have since tried “Flush Redis” again, twice and neither time did it stop the inputs updating. I will if I get a chance create another new node at some point and re-enact what I have done here again, but on the surface it looks like creating a new node (without a pre-existing emonhub.conf entry) not only doesn’t get created on the inputs page, but it also causes a problem for Redis that cannot be Flushed and at the very least requires a reboot to even recognize the new node.

pb66 · 2017-02-12T12:00:25.079Z

7 posts were split to a new topic: Unable to see emonTx inputs

borpin · 2017-02-11T19:43:49.232Z

glyn.hudson:

It sounds like the updated numerical node ID is getting into Redis which is why is shows up in the process list but not getting into MYSQL which could be why the feeds stops updating.

Yes, this sounds like the same issue as:

Did this issue get solved? I’m about to move a system that uses numeric topics and I’d like to not have to redo them all if I can.

glyn.hudson · 2017-02-12T01:51:54.737Z

no, we have drawn a blank on this one. Any further insight you notice would be useful.

pb66 · 2017-02-16T20:50:27.920Z

8 posts were split to a new topic: No data from RFM2PI not reaching emoncms

borpin · 2017-02-16T13:06:52.619Z

SianiAnni:

Any suggestions as to what else I can try?

I have sort of lost the thread off this - are you having trouble with the emonhub or emoncms locally?

If it is just a local logging issue (i.e. the data is logging to emoncms.org OK), why not use the http interface to log locally?

pb66:

I have since tried “Flush Redis” again, twice and neither time did it stop the inputs updating. I will if I get a chance create another new node at some point and re-enact what I have done here again, but on the surface it looks like creating a new node (without a pre-existing emonhub.conf entry) not only doesn’t get created on the inputs page, but it also causes a problem for Redis that cannot be Flushed and at the very least requires a reboot to even recognize the new node.

@pb66 Paul, if you publish by http instead does it work? If so, it suggests the problem is with the MQTT input process to emoncms. MQTT in general should be fine with numeric topics. Have you tried subscribing to the topics outside of emoncms?

@glyn.hudson What is the thinking behind using MQTT rather than the HTTP interface? Is it to make the data available to other MQTT clients or is there a processing advantage?

glyn.hudson · 2017-02-16T14:15:46.043Z

borpin:

What is the thinking behind using MQTT rather than the HTTP interface? Is it to make the data available to other MQTT clients or is there a processing advantage?

Yes, exactly. Also it’s more lightweight than http so it’s more power and processing efficient for low power connected devices to post via MQTT e.g. ESP8266. On the emonSD MQTT allows us to share data easily between the LCD script, nodeRED, openHAB and Emoncms.

pb66 · 2017-02-16T17:30:12.823Z

borpin:

I have sort of lost the thread off this - are you having trouble with the emonhub or emoncms locally?

He has 2 main issues, firstly data isn’t coming out of the RFM2Pi and secondly there are loads of “unable to connect to MQTT” seen in the emonhub logs whilst trying to debug the lack of RFM data.

borpin:

Paul, if you publish by http instead does it work? If so, it suggests the problem is with the MQTT input process to emoncms. MQTT in general should be fine with numeric topics. Have you tried subscribing to the topics outside of emoncms?

Yes, I have documented all these test steps above and proven there is a problem with the MQTT input of emoncms.

I think Sian’s more recent posts need to be split off to another topic now as this thread has a lot of useful debugging info about that MQTT input issue with emoncms, which is the original issue Sian was posting about…
[Edit - done!. Moved to No data from RFM2PI reaching emoncms)

stuart · 2017-03-16T13:10:14.849Z

Any resolution to this issue? In my recent development work to add new interfacers into emonhub, I also noticed that new data isn’t pushed into the local emonCMS, but is visible in the online emoncms.org site.

All I see is the “rssi” input value.

SianiAnni · 2017-03-16T18:14:36.545Z

Me too.

SianiAnni · 2017-03-16T18:16:35.413Z

I now have my first emonTH_V1 back and inputs on node 19. I fixed the definitions of the inputs in emonhub.conf on the pi, and although I get int temp, ext temp, humidity and battery appearing in emoncms.org, all I see on the pi is rssi.