A question over whether emonCMS is compatible with MQTT

Emoncms
21

No, as I interpret it, the client sends the ping if it knows it hasn’t sent any service messages in the required interval to prevent the broker disconnecting at 150% of the interval.

Or in the words used in the actual MQTT spec

It is the responsibility of the Client to ensure that the interval between Control Packets being sent does not exceed the Keep Alive value. In the absence of sending any other Control Packets, the Client MUST send a PINGREQ Packet

and then

If the Keep Alive value is non-zero and the Server does not receive a Control Packet from the Client within one and a half times the Keep Alive time period, it MUST disconnect the Network Connection to the Client as if the network had failed

Those are the 2 highlighted lines in the “3.1.2.10 Keep Alive” section of this spec page (MQTT Version 3.1.1).

22

Well I got completely the wrong end of that stick! Yes you are completely right, unfortunately I don’t remember seeing any mechanism to do that in these libraries. Perhaps subscribing to one of the $SYS topics would suffice or else create a publish ‘heartbeat’ topic?

23

Aha! Found it!

I think it is in fact part of libmosquitto.

mosquitto_loop_misc() gets called at the very end of the mosquitto_loop() function

(Both these functions lived in lib/mosquitto.c until 11th of April, now they are in lib/loop.c)

It is documented as part of the libraries mosquitto_loop_misc() function but not as part of the libs most commonly used mosquitto_loop() function, nor does it state that the former is included via a call from the latter.

So I now believe you should be seeing both PINGREQ(from the client) and PINGRESP(from the broker) messages, if you are I assume all is well and they are doing there thing, back to the drawing board :slight_smile:

1 Like
24

For my own curiosity I installed tshark to an emonsd image installed to a RPi3 (not an emonpi and no rfm69pi), Here is 3mins traffic on the 1883 port. Then I changed the emoncms keepalive to 60 and ran again.

pi@emonpi(ro):~$ tshark -i any -Y tcp.port==1883
Capturing on 'any'
  2   0.151378          ::1 -> ::1          TCP 90 37804â1883 [PSH, ACK] Seq=1 Ack=1 Win=342 Len=2 TSval=327437 TSecr=326936
  3   0.151607          ::1 -> ::1          TCP 90 1883â37804 [PSH, ACK] Seq=1 Ack=3 Win=342 Len=2 TSval=327437 TSecr=327437
  4   0.151663          ::1 -> ::1          TCP 88 37804â1883 [ACK] Seq=3 Ack=3 Win=342 Len=0 TSval=327437 TSecr=327437
 11   1.507578    127.0.0.1 -> 127.0.0.1    TCP 70 49748â1883 [PSH, ACK] Seq=1 Ack=1 Win=342 Len=2 TSval=327573 TSecr=325172
 12   1.507841    127.0.0.1 -> 127.0.0.1    TCP 70 1883â49748 [PSH, ACK] Seq=1 Ack=3 Win=342 Len=2 TSval=327573 TSecr=327573
 13   1.507904    127.0.0.1 -> 127.0.0.1    TCP 68 49748â1883 [ACK] Seq=3 Ack=3 Win=342 Len=0 TSval=327573 TSecr=327573
 20   5.163670          ::1 -> ::1          TCP 90 37804â1883 [PSH, ACK] Seq=3 Ack=3 Win=342 Len=2 TSval=327938 TSecr=327437
 21   5.163843          ::1 -> ::1          TCP 90 1883â37804 [PSH, ACK] Seq=3 Ack=5 Win=342 Len=2 TSval=327938 TSecr=327938
 22   5.163906          ::1 -> ::1          TCP 88 37804â1883 [ACK] Seq=5 Ack=5 Win=342 Len=0 TSval=327938 TSecr=327938
 35  10.175953          ::1 -> ::1          TCP 90 37804â1883 [PSH, ACK] Seq=5 Ack=5 Win=342 Len=2 TSval=328439 TSecr=327938
 36  10.176182          ::1 -> ::1          TCP 90 1883â37804 [PSH, ACK] Seq=5 Ack=7 Win=342 Len=2 TSval=328439 TSecr=328439
 37  10.176238          ::1 -> ::1          TCP 88 37804â1883 [ACK] Seq=7 Ack=7 Win=342 Len=0 TSval=328439 TSecr=328439
 45  15.188221          ::1 -> ::1          TCP 90 37804â1883 [PSH, ACK] Seq=7 Ack=7 Win=342 Len=2 TSval=328941 TSecr=328439
 46  15.188448          ::1 -> ::1          TCP 90 1883â37804 [PSH, ACK] Seq=7 Ack=9 Win=342 Len=2 TSval=328941 TSecr=328941
 47  15.188505          ::1 -> ::1          TCP 88 37804â1883 [ACK] Seq=9 Ack=9 Win=342 Len=0 TSval=328941 TSecr=328941
 56  20.200491          ::1 -> ::1          TCP 90 37804â1883 [PSH, ACK] Seq=9 Ack=9 Win=342 Len=2 TSval=329442 TSecr=328941
 57  20.200713          ::1 -> ::1          TCP 90 1883â37804 [PSH, ACK] Seq=9 Ack=11 Win=342 Len=2 TSval=329442 TSecr=329442
 58  20.200768          ::1 -> ::1          TCP 88 37804â1883 [ACK] Seq=11 Ack=11 Win=342 Len=0 TSval=329442 TSecr=329442
 64  25.212753          ::1 -> ::1          TCP 90 37804â1883 [PSH, ACK] Seq=11 Ack=11 Win=342 Len=2 TSval=329943 TSecr=329442
 65  25.212986          ::1 -> ::1          TCP 90 1883â37804 [PSH, ACK] Seq=11 Ack=13 Win=342 Len=2 TSval=329943 TSecr=329943
 66  25.213042          ::1 -> ::1          TCP 88 37804â1883 [ACK] Seq=13 Ack=13 Win=342 Len=0 TSval=329943 TSecr=329943
 69  25.510560    127.0.0.1 -> 127.0.0.1    TCP 70 49748â1883 [PSH, ACK] Seq=3 Ack=3 Win=342 Len=2 TSval=329973 TSecr=327573
 70  25.510737    127.0.0.1 -> 127.0.0.1    TCP 70 1883â49748 [PSH, ACK] Seq=3 Ack=5 Win=342 Len=2 TSval=329973 TSecr=329973
 71  25.510801    127.0.0.1 -> 127.0.0.1    TCP 68 49748â1883 [ACK] Seq=5 Ack=5 Win=342 Len=0 TSval=329973 TSecr=329973
 80  30.225045          ::1 -> ::1          TCP 90 37804â1883 [PSH, ACK] Seq=13 Ack=13 Win=342 Len=2 TSval=330444 TSecr=329943
 81  30.225266          ::1 -> ::1          TCP 90 1883â37804 [PSH, ACK] Seq=13 Ack=15 Win=342 Len=2 TSval=330444 TSecr=330444
 82  30.225320          ::1 -> ::1          TCP 88 37804â1883 [ACK] Seq=15 Ack=15 Win=342 Len=0 TSval=330444 TSecr=330444
 88  35.237323          ::1 -> ::1          TCP 90 37804â1883 [PSH, ACK] Seq=15 Ack=15 Win=342 Len=2 TSval=330946 TSecr=330444
 89  35.237555          ::1 -> ::1          TCP 90 1883â37804 [PSH, ACK] Seq=15 Ack=17 Win=342 Len=2 TSval=330946 TSecr=330946
 90  35.237611          ::1 -> ::1          TCP 88 37804â1883 [ACK] Seq=17 Ack=17 Win=342 Len=0 TSval=330946 TSecr=330946
 94  37.676600    127.0.0.1 -> 127.0.0.1    TCP 70 49756â1883 [PSH, ACK] Seq=1 Ack=1 Win=342 Len=2 TSval=331189 TSecr=325189
 95  37.676942    127.0.0.1 -> 127.0.0.1    TCP 70 1883â49756 [PSH, ACK] Seq=1 Ack=3 Win=342 Len=2 TSval=331190 TSecr=331189
 96  37.677074    127.0.0.1 -> 127.0.0.1    TCP 68 49756â1883 [ACK] Seq=3 Ack=3 Win=342 Len=0 TSval=331190 TSecr=331190
107  40.249634          ::1 -> ::1          TCP 90 37804â1883 [PSH, ACK] Seq=17 Ack=17 Win=342 Len=2 TSval=331447 TSecr=330946
108  40.249879          ::1 -> ::1          TCP 90 1883â37804 [PSH, ACK] Seq=17 Ack=19 Win=342 Len=2 TSval=331447 TSecr=331447
109  40.249936          ::1 -> ::1          TCP 88 37804â1883 [ACK] Seq=19 Ack=19 Win=342 Len=0 TSval=331447 TSecr=331447
115  45.261943          ::1 -> ::1          TCP 90 37804â1883 [PSH, ACK] Seq=19 Ack=19 Win=342 Len=2 TSval=331948 TSecr=331447
116  45.262173          ::1 -> ::1          TCP 90 1883â37804 [PSH, ACK] Seq=19 Ack=21 Win=342 Len=2 TSval=331948 TSecr=331948
117  45.262227          ::1 -> ::1          TCP 88 37804â1883 [ACK] Seq=21 Ack=21 Win=342 Len=0 TSval=331948 TSecr=331948
125  49.513432    127.0.0.1 -> 127.0.0.1    TCP 70 49748â1883 [PSH, ACK] Seq=5 Ack=5 Win=342 Len=2 TSval=332373 TSecr=329973
126  49.513628    127.0.0.1 -> 127.0.0.1    TCP 70 1883â49748 [PSH, ACK] Seq=5 Ack=7 Win=342 Len=2 TSval=332373 TSecr=332373
127  49.513694    127.0.0.1 -> 127.0.0.1    TCP 68 49748â1883 [ACK] Seq=7 Ack=7 Win=342 Len=0 TSval=332373 TSecr=332373
131  50.274183          ::1 -> ::1          TCP 90 37804â1883 [PSH, ACK] Seq=21 Ack=21 Win=342 Len=2 TSval=332449 TSecr=331948
132  50.274403          ::1 -> ::1          TCP 90 1883â37804 [PSH, ACK] Seq=21 Ack=23 Win=342 Len=2 TSval=332449 TSecr=332449
133  50.274458          ::1 -> ::1          TCP 88 37804â1883 [ACK] Seq=23 Ack=23 Win=342 Len=0 TSval=332449 TSecr=332449
143  55.285855          ::1 -> ::1          TCP 90 37804â1883 [PSH, ACK] Seq=23 Ack=23 Win=342 Len=2 TSval=332950 TSecr=332449
144  55.286078          ::1 -> ::1          TCP 90 1883â37804 [PSH, ACK] Seq=23 Ack=25 Win=342 Len=2 TSval=332950 TSecr=332950
145  55.286132          ::1 -> ::1          TCP 88 37804â1883 [ACK] Seq=25 Ack=25 Win=342 Len=0 TSval=332950 TSecr=332950
152  60.298124          ::1 -> ::1          TCP 90 37804â1883 [PSH, ACK] Seq=25 Ack=25 Win=342 Len=2 TSval=333452 TSecr=332950
153  60.298342          ::1 -> ::1          TCP 90 1883â37804 [PSH, ACK] Seq=25 Ack=27 Win=342 Len=2 TSval=333452 TSecr=333452
154  60.298398          ::1 -> ::1          TCP 88 37804â1883 [ACK] Seq=27 Ack=27 Win=342 Len=0 TSval=333452 TSecr=333452
166  65.310392          ::1 -> ::1          TCP 90 37804â1883 [PSH, ACK] Seq=27 Ack=27 Win=342 Len=2 TSval=333953 TSecr=333452
167  65.310615          ::1 -> ::1          TCP 90 1883â37804 [PSH, ACK] Seq=27 Ack=29 Win=342 Len=2 TSval=333953 TSecr=333953
168  65.310669          ::1 -> ::1          TCP 88 37804â1883 [ACK] Seq=29 Ack=29 Win=342 Len=0 TSval=333953 TSecr=333953
181  70.322677          ::1 -> ::1          TCP 90 37804â1883 [PSH, ACK] Seq=29 Ack=29 Win=342 Len=2 TSval=334454 TSecr=333953
182  70.322895          ::1 -> ::1          TCP 90 1883â37804 [PSH, ACK] Seq=29 Ack=31 Win=342 Len=2 TSval=334454 TSecr=334454
183  70.322950          ::1 -> ::1          TCP 88 37804â1883 [ACK] Seq=31 Ack=31 Win=342 Len=0 TSval=334454 TSecr=334454
190  73.516288    127.0.0.1 -> 127.0.0.1    TCP 70 49748â1883 [PSH, ACK] Seq=7 Ack=7 Win=342 Len=2 TSval=334773 TSecr=332373
191  73.516541    127.0.0.1 -> 127.0.0.1    TCP 70 1883â49748 [PSH, ACK] Seq=7 Ack=9 Win=342 Len=2 TSval=334773 TSecr=334773
192  73.516600    127.0.0.1 -> 127.0.0.1    TCP 68 49748â1883 [ACK] Seq=9 Ack=9 Win=342 Len=0 TSval=334773 TSecr=334773
196  75.334956          ::1 -> ::1          TCP 90 37804â1883 [PSH, ACK] Seq=31 Ack=31 Win=342 Len=2 TSval=334955 TSecr=334454
197  75.335241          ::1 -> ::1          TCP 90 1883â37804 [PSH, ACK] Seq=31 Ack=33 Win=342 Len=2 TSval=334955 TSecr=334955
198  75.335303          ::1 -> ::1          TCP 88 37804â1883 [ACK] Seq=33 Ack=33 Win=342 Len=0 TSval=334955 TSecr=334955
205  80.347267          ::1 -> ::1          TCP 90 37804â1883 [PSH, ACK] Seq=33 Ack=33 Win=342 Len=2 TSval=335457 TSecr=334955
206  80.347491          ::1 -> ::1          TCP 90 1883â37804 [PSH, ACK] Seq=33 Ack=35 Win=342 Len=2 TSval=335457 TSecr=335457
207  80.347546          ::1 -> ::1          TCP 88 37804â1883 [ACK] Seq=35 Ack=35 Win=342 Len=0 TSval=335457 TSecr=335457
215  85.359548          ::1 -> ::1          TCP 90 37804â1883 [PSH, ACK] Seq=35 Ack=35 Win=342 Len=2 TSval=335958 TSecr=335457
216  85.359868          ::1 -> ::1          TCP 90 1883â37804 [PSH, ACK] Seq=35 Ack=37 Win=342 Len=2 TSval=335958 TSecr=335958
217  85.359924          ::1 -> ::1          TCP 88 37804â1883 [ACK] Seq=37 Ack=37 Win=342 Len=0 TSval=335958 TSecr=335958
224  90.371376          ::1 -> ::1          TCP 90 37804â1883 [PSH, ACK] Seq=37 Ack=37 Win=342 Len=2 TSval=336459 TSecr=335958
225  90.371653          ::1 -> ::1          TCP 90 1883â37804 [PSH, ACK] Seq=37 Ack=39 Win=342 Len=2 TSval=336459 TSecr=336459
226  90.371710          ::1 -> ::1          TCP 88 37804â1883 [ACK] Seq=39 Ack=39 Win=342 Len=0 TSval=336459 TSecr=336459
234  95.383446          ::1 -> ::1          TCP 90 37804â1883 [PSH, ACK] Seq=39 Ack=39 Win=342 Len=2 TSval=336960 TSecr=336459
235  95.383676          ::1 -> ::1          TCP 90 1883â37804 [PSH, ACK] Seq=39 Ack=41 Win=342 Len=2 TSval=336960 TSecr=336960
236  95.383733          ::1 -> ::1          TCP 88 37804â1883 [ACK] Seq=41 Ack=41 Win=342 Len=0 TSval=336960 TSecr=336960
240  97.519342    127.0.0.1 -> 127.0.0.1    TCP 70 49748â1883 [PSH, ACK] Seq=9 Ack=9 Win=342 Len=2 TSval=337174 TSecr=334773
241  97.519599    127.0.0.1 -> 127.0.0.1    TCP 70 1883â49748 [PSH, ACK] Seq=9 Ack=11 Win=342 Len=2 TSval=337174 TSecr=337174
242  97.519658    127.0.0.1 -> 127.0.0.1    TCP 68 49748â1883 [ACK] Seq=11 Ack=11 Win=342 Len=0 TSval=337174 TSecr=337174
243  97.677600    127.0.0.1 -> 127.0.0.1    TCP 70 49756â1883 [PSH, ACK] Seq=3 Ack=3 Win=342 Len=2 TSval=337190 TSecr=331190
244  97.677890    127.0.0.1 -> 127.0.0.1    TCP 70 1883â49756 [PSH, ACK] Seq=3 Ack=5 Win=342 Len=2 TSval=337190 TSecr=337190
245  97.678022    127.0.0.1 -> 127.0.0.1    TCP 68 49756â1883 [ACK] Seq=5 Ack=5 Win=342 Len=0 TSval=337190 TSecr=337190
255 100.395617          ::1 -> ::1          TCP 90 37804â1883 [PSH, ACK] Seq=41 Ack=41 Win=342 Len=2 TSval=337461 TSecr=336960
256 100.395839          ::1 -> ::1          TCP 90 1883â37804 [PSH, ACK] Seq=41 Ack=43 Win=342 Len=2 TSval=337461 TSecr=337461
257 100.395894          ::1 -> ::1          TCP 88 37804â1883 [ACK] Seq=43 Ack=43 Win=342 Len=0 TSval=337461 TSecr=337461
263 105.406947          ::1 -> ::1          TCP 90 37804â1883 [PSH, ACK] Seq=43 Ack=43 Win=342 Len=2 TSval=337963 TSecr=337461
264 105.407176          ::1 -> ::1          TCP 90 1883â37804 [PSH, ACK] Seq=43 Ack=45 Win=342 Len=2 TSval=337963 TSecr=337963
265 105.407232          ::1 -> ::1          TCP 88 37804â1883 [ACK] Seq=45 Ack=45 Win=342 Len=0 TSval=337963 TSecr=337963
310 110.419200          ::1 -> ::1          TCP 90 37804â1883 [PSH, ACK] Seq=45 Ack=45 Win=342 Len=2 TSval=338464 TSecr=337963
311 110.419419          ::1 -> ::1          TCP 90 1883â37804 [PSH, ACK] Seq=45 Ack=47 Win=342 Len=2 TSval=338464 TSecr=338464
312 110.419475          ::1 -> ::1          TCP 88 37804â1883 [ACK] Seq=47 Ack=47 Win=342 Len=0 TSval=338464 TSecr=338464
320 115.431284          ::1 -> ::1          TCP 90 37804â1883 [PSH, ACK] Seq=47 Ack=47 Win=342 Len=2 TSval=338965 TSecr=338464
321 115.431509          ::1 -> ::1          TCP 90 1883â37804 [PSH, ACK] Seq=47 Ack=49 Win=342 Len=2 TSval=338965 TSecr=338965
322 115.431566          ::1 -> ::1          TCP 88 37804â1883 [ACK] Seq=49 Ack=49 Win=342 Len=0 TSval=338965 TSecr=338965
363 120.443459          ::1 -> ::1          TCP 90 37804â1883 [PSH, ACK] Seq=49 Ack=49 Win=342 Len=2 TSval=339466 TSecr=338965
364 120.443676          ::1 -> ::1          TCP 90 1883â37804 [PSH, ACK] Seq=49 Ack=51 Win=342 Len=2 TSval=339466 TSecr=339466
365 120.443731          ::1 -> ::1          TCP 88 37804â1883 [ACK] Seq=51 Ack=51 Win=342 Len=0 TSval=339466 TSecr=339466
371 121.522390    127.0.0.1 -> 127.0.0.1    TCP 70 49748â1883 [PSH, ACK] Seq=11 Ack=11 Win=342 Len=2 TSval=339574 TSecr=337174
372 121.522647    127.0.0.1 -> 127.0.0.1    TCP 70 1883â49748 [PSH, ACK] Seq=11 Ack=13 Win=342 Len=2 TSval=339574 TSecr=339574
373 121.522705    127.0.0.1 -> 127.0.0.1    TCP 68 49748â1883 [ACK] Seq=13 Ack=13 Win=342 Len=0 TSval=339574 TSecr=339574
410 125.455689          ::1 -> ::1          TCP 90 37804â1883 [PSH, ACK] Seq=51 Ack=51 Win=342 Len=2 TSval=339967 TSecr=339466
411 125.455960          ::1 -> ::1          TCP 90 1883â37804 [PSH, ACK] Seq=51 Ack=53 Win=342 Len=2 TSval=339967 TSecr=339967
412 125.456017          ::1 -> ::1          TCP 88 37804â1883 [ACK] Seq=53 Ack=53 Win=342 Len=0 TSval=339967 TSecr=339967
561 130.467994          ::1 -> ::1          TCP 90 37804â1883 [PSH, ACK] Seq=53 Ack=53 Win=342 Len=2 TSval=340469 TSecr=339967
562 130.468212          ::1 -> ::1          TCP 90 1883â37804 [PSH, ACK] Seq=53 Ack=55 Win=342 Len=2 TSval=340469 TSecr=340469
563 130.468267          ::1 -> ::1          TCP 88 37804â1883 [ACK] Seq=55 Ack=55 Win=342 Len=0 TSval=340469 TSecr=340469
598 135.480236          ::1 -> ::1          TCP 90 37804â1883 [PSH, ACK] Seq=55 Ack=55 Win=342 Len=2 TSval=340970 TSecr=340469
599 135.480465          ::1 -> ::1          TCP 90 1883â37804 [PSH, ACK] Seq=55 Ack=57 Win=342 Len=2 TSval=340970 TSecr=340970
600 135.480524          ::1 -> ::1          TCP 88 37804â1883 [ACK] Seq=57 Ack=57 Win=342 Len=0 TSval=340970 TSecr=340970
697 140.490092          ::1 -> ::1          TCP 90 37804â1883 [PSH, ACK] Seq=57 Ack=57 Win=342 Len=2 TSval=341471 TSecr=340970
698 140.490357          ::1 -> ::1          TCP 90 1883â37804 [PSH, ACK] Seq=57 Ack=59 Win=342 Len=2 TSval=341471 TSecr=341471
699 140.490414          ::1 -> ::1          TCP 88 37804â1883 [ACK] Seq=59 Ack=59 Win=342 Len=0 TSval=341471 TSecr=341471
711 145.502450          ::1 -> ::1          TCP 90 37804â1883 [PSH, ACK] Seq=59 Ack=59 Win=342 Len=2 TSval=341972 TSecr=341471
712 145.502683          ::1 -> ::1          TCP 90 1883â37804 [PSH, ACK] Seq=59 Ack=61 Win=342 Len=2 TSval=341972 TSecr=341972
713 145.502741          ::1 -> ::1          TCP 88 37804â1883 [ACK] Seq=61 Ack=61 Win=342 Len=0 TSval=341972 TSecr=341972
714 145.525358    127.0.0.1 -> 127.0.0.1    TCP 70 49748â1883 [PSH, ACK] Seq=13 Ack=13 Win=342 Len=2 TSval=341974 TSecr=339574
715 145.525576    127.0.0.1 -> 127.0.0.1    TCP 70 1883â49748 [PSH, ACK] Seq=13 Ack=15 Win=342 Len=2 TSval=341974 TSecr=341974
716 145.525627    127.0.0.1 -> 127.0.0.1    TCP 68 49748â1883 [ACK] Seq=15 Ack=15 Win=342 Len=0 TSval=341974 TSecr=341974
730 150.514749          ::1 -> ::1          TCP 90 37804â1883 [PSH, ACK] Seq=61 Ack=61 Win=342 Len=2 TSval=342473 TSecr=341972
731 150.514985          ::1 -> ::1          TCP 90 1883â37804 [PSH, ACK] Seq=61 Ack=63 Win=342 Len=2 TSval=342473 TSecr=342473
732 150.515039          ::1 -> ::1          TCP 88 37804â1883 [ACK] Seq=63 Ack=63 Win=342 Len=0 TSval=342473 TSecr=342473
738 155.527026          ::1 -> ::1          TCP 90 37804â1883 [PSH, ACK] Seq=63 Ack=63 Win=342 Len=2 TSval=342975 TSecr=342473
739 155.527261          ::1 -> ::1          TCP 90 1883â37804 [PSH, ACK] Seq=63 Ack=65 Win=342 Len=2 TSval=342975 TSecr=342975
740 155.527319          ::1 -> ::1          TCP 88 37804â1883 [ACK] Seq=65 Ack=65 Win=342 Len=0 TSval=342975 TSecr=342975
744 157.678577    127.0.0.1 -> 127.0.0.1    TCP 70 49756â1883 [PSH, ACK] Seq=5 Ack=5 Win=342 Len=2 TSval=343190 TSecr=337190
745 157.678856    127.0.0.1 -> 127.0.0.1    TCP 70 1883â49756 [PSH, ACK] Seq=5 Ack=7 Win=342 Len=2 TSval=343190 TSecr=343190
746 157.678983    127.0.0.1 -> 127.0.0.1    TCP 68 49756â1883 [ACK] Seq=7 Ack=7 Win=342 Len=0 TSval=343190 TSecr=343190
756 160.539301          ::1 -> ::1          TCP 90 37804â1883 [PSH, ACK] Seq=65 Ack=65 Win=342 Len=2 TSval=343476 TSecr=342975
757 160.539522          ::1 -> ::1          TCP 90 1883â37804 [PSH, ACK] Seq=65 Ack=67 Win=342 Len=2 TSval=343476 TSecr=343476
758 160.539577          ::1 -> ::1          TCP 88 37804â1883 [ACK] Seq=67 Ack=67 Win=342 Len=0 TSval=343476 TSecr=343476
765 165.551565          ::1 -> ::1          TCP 90 37804â1883 [PSH, ACK] Seq=67 Ack=67 Win=342 Len=2 TSval=343977 TSecr=343476
766 165.551798          ::1 -> ::1          TCP 90 1883â37804 [PSH, ACK] Seq=67 Ack=69 Win=342 Len=2 TSval=343977 TSecr=343977
767 165.551855          ::1 -> ::1          TCP 88 37804â1883 [ACK] Seq=69 Ack=69 Win=342 Len=0 TSval=343977 TSecr=343977
774 169.528400    127.0.0.1 -> 127.0.0.1    TCP 70 49748â1883 [PSH, ACK] Seq=15 Ack=15 Win=342 Len=2 TSval=344375 TSecr=341974
775 169.528649    127.0.0.1 -> 127.0.0.1    TCP 70 1883â49748 [PSH, ACK] Seq=15 Ack=17 Win=342 Len=2 TSval=344375 TSecr=344375
776 169.528709    127.0.0.1 -> 127.0.0.1    TCP 68 49748â1883 [ACK] Seq=17 Ack=17 Win=342 Len=0 TSval=344375 TSecr=344375
780 170.563842          ::1 -> ::1          TCP 90 37804â1883 [PSH, ACK] Seq=69 Ack=69 Win=342 Len=2 TSval=344478 TSecr=343977
781 170.564070          ::1 -> ::1          TCP 90 1883â37804 [PSH, ACK] Seq=69 Ack=71 Win=342 Len=2 TSval=344478 TSecr=344478
782 170.564125          ::1 -> ::1          TCP 88 37804â1883 [ACK] Seq=71 Ack=71 Win=342 Len=0 TSval=344478 TSecr=344478
788 175.576167          ::1 -> ::1          TCP 90 37804â1883 [PSH, ACK] Seq=71 Ack=71 Win=342 Len=2 TSval=344979 TSecr=344478
789 175.576396          ::1 -> ::1          TCP 90 1883â37804 [PSH, ACK] Seq=71 Ack=73 Win=342 Len=2 TSval=344979 TSecr=344979
790 175.576452          ::1 -> ::1          TCP 88 37804â1883 [ACK] Seq=73 Ack=73 Win=342 Len=0 TSval=344979 TSecr=344979
797 180.587684          ::1 -> ::1          TCP 90 37804â1883 [PSH, ACK] Seq=73 Ack=73 Win=342 Len=2 TSval=345481 TSecr=344979
798 180.587911          ::1 -> ::1          TCP 90 1883â37804 [PSH, ACK] Seq=73 Ack=75 Win=342 Len=2 TSval=345481 TSecr=345481
799 180.587968          ::1 -> ::1          TCP 88 37804â1883 [ACK] Seq=75 Ack=75 Win=342 Len=0 TSval=345481 TSecr=345481
^C144 packets captured
pi@emonpi(ro):~$ rpi-rw
Filesystem is unlocked - Write access
type ' rpi-ro ' to lock
pi@emonpi(rw):~$ nano /var/www/emoncms/scripts/phpmqtt_input.php
pi@emonpi(rw):~$ sudo service mqtt_input restart

pi@emonpi(rw):~$ tshark -i any -Y tcp.port==1883
Capturing on 'any'
 11  10.889969    127.0.0.1 -> 127.0.0.1    TCP 70 49756â1883 [PSH, ACK] Seq=1 Ack=1 Win=342 Len=2 TSval=355190 TSecr=349190
 12  10.890137    127.0.0.1 -> 127.0.0.1    TCP 70 1883â49756 [PSH, ACK] Seq=1 Ack=3 Win=342 Len=2 TSval=355190 TSecr=355190
 13  10.890272    127.0.0.1 -> 127.0.0.1    TCP 68 49756â1883 [ACK] Seq=3 Ack=3 Win=342 Len=0 TSval=355190 TSecr=355190
 32  22.752908    127.0.0.1 -> 127.0.0.1    TCP 70 49748â1883 [PSH, ACK] Seq=1 Ack=1 Win=342 Len=2 TSval=356376 TSecr=353976
 33  22.753137    127.0.0.1 -> 127.0.0.1    TCP 70 1883â49748 [PSH, ACK] Seq=1 Ack=3 Win=342 Len=2 TSval=356376 TSecr=356376
 34  22.753197    127.0.0.1 -> 127.0.0.1    TCP 68 49748â1883 [ACK] Seq=3 Ack=3 Win=342 Len=0 TSval=356376 TSecr=356376
 61  46.755847    127.0.0.1 -> 127.0.0.1    TCP 70 49748â1883 [PSH, ACK] Seq=3 Ack=3 Win=342 Len=2 TSval=358776 TSecr=356376
 62  46.756092    127.0.0.1 -> 127.0.0.1    TCP 70 1883â49748 [PSH, ACK] Seq=3 Ack=5 Win=342 Len=2 TSval=358776 TSecr=358776
 63  46.756149    127.0.0.1 -> 127.0.0.1    TCP 68 49748â1883 [ACK] Seq=5 Ack=5 Win=342 Len=0 TSval=358776 TSecr=358776
 96  50.935217          ::1 -> ::1          TCP 90 38070â1883 [PSH, ACK] Seq=1 Ack=1 Win=342 Len=2 TSval=359194 TSecr=353179
 97  50.935532          ::1 -> ::1          TCP 90 1883â38070 [PSH, ACK] Seq=1 Ack=3 Win=342 Len=2 TSval=359194 TSecr=359194
 98  50.935595          ::1 -> ::1          TCP 88 38070â1883 [ACK] Seq=3 Ack=3 Win=342 Len=0 TSval=359194 TSecr=359194
742  70.758930    127.0.0.1 -> 127.0.0.1    TCP 70 49748â1883 [PSH, ACK] Seq=5 Ack=5 Win=342 Len=2 TSval=361177 TSecr=358776
743  70.759192    127.0.0.1 -> 127.0.0.1    TCP 70 1883â49748 [PSH, ACK] Seq=5 Ack=7 Win=342 Len=2 TSval=361177 TSecr=361177
744  70.759254    127.0.0.1 -> 127.0.0.1    TCP 68 49748â1883 [ACK] Seq=7 Ack=7 Win=342 Len=0 TSval=361177 TSecr=361177
745  70.890932    127.0.0.1 -> 127.0.0.1    TCP 70 49756â1883 [PSH, ACK] Seq=3 Ack=3 Win=342 Len=2 TSval=361190 TSecr=355190
746  70.891093    127.0.0.1 -> 127.0.0.1    TCP 70 1883â49756 [PSH, ACK] Seq=3 Ack=5 Win=342 Len=2 TSval=361190 TSecr=361190
747  70.891223    127.0.0.1 -> 127.0.0.1    TCP 68 49756â1883 [ACK] Seq=5 Ack=5 Win=342 Len=0 TSval=361190 TSecr=361190
1508  94.761917    127.0.0.1 -> 127.0.0.1    TCP 70 49748â1883 [PSH, ACK] Seq=7 Ack=7 Win=342 Len=2 TSval=363577 TSecr=361177
1509  94.762165    127.0.0.1 -> 127.0.0.1    TCP 70 1883â49748 [PSH, ACK] Seq=7 Ack=9 Win=342 Len=2 TSval=363577 TSecr=363577
1510  94.762224    127.0.0.1 -> 127.0.0.1    TCP 68 49748â1883 [ACK] Seq=9 Ack=9 Win=342 Len=0 TSval=363577 TSecr=363577
1991 110.065193          ::1 -> ::1          TCP 90 38070â1883 [PSH, ACK] Seq=3 Ack=3 Win=342 Len=2 TSval=365107 TSecr=359194
1992 110.065388          ::1 -> ::1          TCP 90 1883â38070 [PSH, ACK] Seq=3 Ack=5 Win=342 Len=2 TSval=365107 TSecr=365107
1993 110.065449          ::1 -> ::1          TCP 88 38070â1883 [ACK] Seq=5 Ack=5 Win=342 Len=0 TSval=365107 TSecr=365107
2273 118.764906    127.0.0.1 -> 127.0.0.1    TCP 70 49748â1883 [PSH, ACK] Seq=9 Ack=9 Win=342 Len=2 TSval=365977 TSecr=363577
2274 118.765149    127.0.0.1 -> 127.0.0.1    TCP 70 1883â49748 [PSH, ACK] Seq=9 Ack=11 Win=342 Len=2 TSval=365977 TSecr=365977
2275 118.765207    127.0.0.1 -> 127.0.0.1    TCP 68 49748â1883 [ACK] Seq=11 Ack=11 Win=342 Len=0 TSval=365977 TSecr=365977
2651 130.891910    127.0.0.1 -> 127.0.0.1    TCP 70 49756â1883 [PSH, ACK] Seq=5 Ack=5 Win=342 Len=2 TSval=367190 TSecr=361190
2652 130.892229    127.0.0.1 -> 127.0.0.1    TCP 70 1883â49756 [PSH, ACK] Seq=5 Ack=7 Win=342 Len=2 TSval=367190 TSecr=367190
2653 130.892323    127.0.0.1 -> 127.0.0.1    TCP 68 49756â1883 [ACK] Seq=7 Ack=7 Win=342 Len=0 TSval=367190 TSecr=367190
3036 142.767909    127.0.0.1 -> 127.0.0.1    TCP 70 49748â1883 [PSH, ACK] Seq=11 Ack=11 Win=342 Len=2 TSval=368378 TSecr=365977
3037 142.768153    127.0.0.1 -> 127.0.0.1    TCP 70 1883â49748 [PSH, ACK] Seq=11 Ack=13 Win=342 Len=2 TSval=368378 TSecr=368378
3038 142.768212    127.0.0.1 -> 127.0.0.1    TCP 68 49748â1883 [ACK] Seq=13 Ack=13 Win=342 Len=0 TSval=368378 TSecr=368378
3797 166.770874    127.0.0.1 -> 127.0.0.1    TCP 70 49748â1883 [PSH, ACK] Seq=13 Ack=13 Win=342 Len=2 TSval=370778 TSecr=368378
3798 166.771123    127.0.0.1 -> 127.0.0.1    TCP 70 1883â49748 [PSH, ACK] Seq=13 Ack=15 Win=342 Len=2 TSval=370778 TSecr=370778
3799 166.771183    127.0.0.1 -> 127.0.0.1    TCP 68 49748â1883 [ACK] Seq=15 Ack=15 Win=342 Len=0 TSval=370778 TSecr=370778
3910 170.196906          ::1 -> ::1          TCP 90 38070â1883 [PSH, ACK] Seq=5 Ack=5 Win=342 Len=2 TSval=371121 TSecr=365107
3911 170.197065          ::1 -> ::1          TCP 90 1883â38070 [PSH, ACK] Seq=5 Ack=7 Win=342 Len=2 TSval=371121 TSecr=371121
3912 170.197125          ::1 -> ::1          TCP 88 38070â1883 [ACK] Seq=7 Ack=7 Win=342 Len=0 TSval=371121 TSecr=371121
^C39 packets captured

The MQTT packet count reduced from 144 to 39, these must all be control packets of some kind as I have no MQTT data being published or subscribed to.

We can see from the time between occurrences that all the PINGREQ and PINGRESP packets are the “::1 → ::1” ones, I have no idea what the “127.0.0.1 → 127.0.0.1” ones are.

Throughout these tests there was absolutely no movement on either the emonhub or emoncms logs, so if those other packets are disconnections and reconnections, or subscriptions or acks of some kind, they are not being shown in the logs despite “debug” and “info” log levels having been set.

1 Like
25

With the command tshark -f "port 1883" being run on the client (emoncms) machine .46, broker on .30 and the keepalive set to 5, I get the following;

24 15.229533722 XXX.XXX.XXX.46 → XXX.XXX.XXX.30 TCP 66 55428 → 1883 [ACK] Seq=17 Ack=438 Win=674 Len=0 TSval=153090669 TSecr=244194
25 20.242282651 XXX.XXX.XXX.46 → XXX.XXX.XXX.30 MQTT 68 Ping Request
26 20.242728828 XXX.XXX.XXX.30 → XXX.XXX.XXX.46 MQTT 68 Ping Response
27 20.242764370 XXX.XXX.XXX.46 → XXX.XXX.XXX.30 TCP 66 55428 → 1883 [ACK] Seq=19 Ack=440 Win=674 Len=0 TSval=153091923 TSecr=245448
28 25.254117711 XXX.XXX.XXX.46 → XXX.XXX.XXX.30 MQTT 68 Ping Request
29 25.254521718 XXX.XXX.XXX.30 → XXX.XXX.XXX.46 MQTT 68 Ping Response
30 25.254557777 XXX.XXX.XXX.46 → XXX.XXX.XXX.30 TCP 66 55428 → 1883 [ACK] Seq=21 Ack=442 Win=674 Len=0 TSval=153093176 TSecr=246701
31 30.267303061 XXX.XXX.XXX.46 → XXX.XXX.XXX.30 MQTT 68 Ping Request
32 30.267683917 XXX.XXX.XXX.30 → XXX.XXX.XXX.46 MQTT 68 Ping Response
33 30.267718094 XXX.XXX.XXX.46 → XXX.XXX.XXX.30 TCP 66 55428 → 1883 [ACK] Seq=23 Ack=444 Win=674 Len=0 TSval=153094429 TSecr=247954
34 30.478143315 XXX.XXX.XXX.30 → XXX.XXX.XXX.46 MQTT 95 Publish Message
35 30.478159558 XXX.XXX.XXX.46 → XXX.XXX.XXX.30 TCP 66 55428 → 1883 [ACK] Seq=23 Ack=473 Win=674 Len=0 TSval=153094482 TSecr=248007
36 30.478227188 XXX.XXX.XXX.30 → XXX.XXX.XXX.46 MQTT 94 Publish Message
37 30.478233594 XXX.XXX.XXX.46 → XXX.XXX.XXX.30 TCP 66 55428 → 1883 [ACK] Seq=23 Ack=501 Win=674 Len=0 TSval=153094482 TSecr=248007
38 30.478274444 XXX.XXX.XXX.30 → XXX.XXX.XXX.46 MQTT 96 Publish Message
39 30.478278331 XXX.XXX.XXX.46 → XXX.XXX.XXX.30 TCP 66 55428 → 1883 [ACK] Seq=23 Ack=531 Win=674 Len=0 TSval=153094482 TSecr=248007
40 30.478341474 XXX.XXX.XXX.30 → XXX.XXX.XXX.46 MQTT 92 Publish Message
41 30.478346637 XXX.XXX.XXX.46 → XXX.XXX.XXX.30 TCP 66 55428 → 1883 [ACK] Seq=23 Ack=557 Win=674 Len=0 TSval=153094482 TSecr=248007
42 30.478385807 XXX.XXX.XXX.30 → XXX.XXX.XXX.46 MQTT 112 Publish Message, Publish Message
43 30.478389160 XXX.XXX.XXX.46 → XXX.XXX.XXX.30 TCP 66 55428 → 1883 [ACK] Seq=23 Ack=603 Win=674 Len=0 TSval=153094482 TSecr=248007
44 30.478465691 XXX.XXX.XXX.30 → XXX.XXX.XXX.46 MQTT 92 Publish Message
45 30.478470834 XXX.XXX.XXX.46 → XXX.XXX.XXX.30 TCP 66 55428 → 1883 [ACK] Seq=23 Ack=629 Win=674 Len=0 TSval=153094482 TSecr=248007
46 30.478519211 XXX.XXX.XXX.30 → XXX.XXX.XXX.46 MQTT 120 Publish Message, Publish Message
47 30.478522975 XXX.XXX.XXX.46 → XXX.XXX.XXX.30 TCP 66 55428 → 1883 [ACK] Seq=23 Ack=683 Win=674 Len=0 TSval=153094482 TSecr=248007
48 30.479802062 XXX.XXX.XXX.30 → XXX.XXX.XXX.46 MQTT 252 Publish Message
49 30.479821050 XXX.XXX.XXX.46 → XXX.XXX.XXX.30 TCP 66 55428 → 1883 [ACK] Seq=23 Ack=869 Win=691 Len=0 TSval=153094482 TSecr=248007
50 30.506759854 XXX.XXX.XXX.46 → XXX.XXX.XXX.30 MQTT 70 Publish Received
51 30.507148899 XXX.XXX.XXX.30 → XXX.XXX.XXX.46 MQTT 70 Publish Release
52 30.509499325 XXX.XXX.XXX.46 → XXX.XXX.XXX.30 MQTT 70 Publish Complete
53 30.553498666 XXX.XXX.XXX.30 → XXX.XXX.XXX.46 TCP 66 1883 → 55428 [ACK] Seq=873 Ack=31 Win=227 Len=0 TSval=248026 TSecr=153094489

You can see the ping request and response every 5 seconds which is apparently initiated by the client. there are a number of publish events, one for each topic. These are published to the broker (by node-red also on .30) to be passed to the client.

With the keepalive removed (so defaulting to 60)

 1 0.000000000 XXX.XXX.XXX.30 → XXX.XXX.XXX.46 MQTT 95 Publish Message
 2 0.044001974 XXX.XXX.XXX.46 → XXX.XXX.XXX.30 TCP 66 59050 → 1883 [ACK] Seq=1 Ack=30 Win=758 Len=0 TSval=153231025 TSecr=384541
 3 0.044187203 XXX.XXX.XXX.30 → XXX.XXX.XXX.46 MQTT 460 Publish Message, Publish Message, Publish Message, Publish Message, Publish Message, Publish Message, Publish Message, Publish Message, Publish Message
 4 0.044193802 XXX.XXX.XXX.46 → XXX.XXX.XXX.30 TCP 66 59050 → 1883 [ACK] Seq=1 Ack=424 Win=775 Len=0 TSval=153231025 TSecr=384553
 5 0.060961345 XXX.XXX.XXX.46 → XXX.XXX.XXX.30 MQTT 70 Publish Received
 6 0.061191797 XXX.XXX.XXX.30 → XXX.XXX.XXX.46 TCP 66 1883 → 59050 [ACK] Seq=424 Ack=5 Win=227 Len=0 TSval=384557 TSecr=153231029
 7 0.061274265 XXX.XXX.XXX.30 → XXX.XXX.XXX.46 MQTT 70 Publish Release
 8 0.063745986 XXX.XXX.XXX.46 → XXX.XXX.XXX.30 MQTT 70 Publish Complete
 9 0.105957101 XXX.XXX.XXX.30 → XXX.XXX.XXX.46 TCP 66 1883 → 59050 [ACK] Seq=428 Ack=9 Win=227 Len=0 TSval=384569 TSecr=153231029
10 30.352396459 XXX.XXX.XXX.30 → XXX.XXX.XXX.46 MQTT 95 Publish Message
11 30.395992481 XXX.XXX.XXX.46 → XXX.XXX.XXX.30 TCP 66 59050 → 1883 [ACK] Seq=9 Ack=457 Win=775 Len=0 TSval=153238613 TSecr=392130
12 30.396385690 XXX.XXX.XXX.30 → XXX.XXX.XXX.46 MQTT 462 Publish Message, Publish Message, Publish Message, Publish Message, Publish Message, Publish Message, Publish Message, Publish Message, Publish Message
13 30.396411673 XXX.XXX.XXX.46 → XXX.XXX.XXX.30 TCP 66 59050 → 1883 [ACK] Seq=9 Ack=853 Win=792 Len=0 TSval=153238613 TSecr=392141
14 30.412855808 XXX.XXX.XXX.46 → XXX.XXX.XXX.30 MQTT 70 Publish Received
15 30.413214560 XXX.XXX.XXX.30 → XXX.XXX.XXX.46 TCP 66 1883 → 59050 [ACK] Seq=853 Ack=13 Win=227 Len=0 TSval=392146 TSecr=153238617
16 30.413282407 XXX.XXX.XXX.30 → XXX.XXX.XXX.46 MQTT 70 Publish Release
17 30.415546171 XXX.XXX.XXX.46 → XXX.XXX.XXX.30 MQTT 70 Publish Complete
18 30.457026066 XXX.XXX.XXX.30 → XXX.XXX.XXX.46 TCP 66 1883 → 59050 [ACK] Seq=857 Ack=17 Win=227 Len=0 TSval=392157 TSecr=153238617

There are no ping requests as there is more data within the keepalive limit.

I had a problem with my secondary router (I think it is dying) so I have not got any data to see if the disconnections have reduced or not. I’ll check in 24Hrs.

[Edit] A note on the columns (tshark -G column-formats gives the detail), they are

  • No.
  • Time
  • Source
  • Destination
  • Protocol
  • Length
  • Info
26

I think it is unlikely the disconnections will reduce as the keepalive pings are working, unless less traffic impacts the load, but that’s unlikely too. But it will be good to know for sure.

I tried the -f "port 1883" instead of -Y tcp.port==1883 and got identical results, but I cannot get any results without specifying -i any or -i lo as the emonsd has both clients and broker on same machine, (omitting a interface defaults to eth0) it would be useful if you could confirm there is no change for you when using -i any option just to find out if the traffic is between client and broker or if it’s just a info broadcast or something similar. It’s difficult for me to tell if those “other packets” belong to broker, client or both as they are both on the same machine.

27

With the -i any option I do not see any change in the output.

1 Like
28

So I found the source of all the other packets yesterday.

By systematically stopping all the services that use MQTT and starting them one by one in turn. I found both the openhab and lwrfd services create control packets on the 1883 port.

It seems the lwrfd mqtt service has a 24s keepalive and openhab has a 60s keepalive, these regular batches of 3 packets are easily spotted when looking at the traffic of just one client at a time, when they were all running at once (lwrfd every 24s, openhab every 60s and mqtt_input every 5 or 60s) and they all have the same IP, it did appear like quite a random pattern.

1 Like
29

@pb66, did you ever get to a resolution of the cleansession question (should we, shouldn’t we) Use a persistent MQTT connection by pb66 · Pull Request #823 · emoncms/emoncms · GitHub?

I have just created a PR MQTT Client ID by borpin · Pull Request #1160 · emoncms/emoncms · GitHub to move the ClientID into the settings. Do you want to add the cleansession parameter as well so it is easily controlled?

30

Off-hand Brian I cannot recall. There were very distinct times when one is used over the other and my proposed solution at the time automatically set the cleansession correctly, I think that went out the window due to the redis stuff for publishing.

The problem with making it definable is that (when working correctly) there is no need to define it manually, but it does expose it to being set incorrectly and shifts the responsibility of getting it right on to the user rather than getting the coding right. Without revisiting the code and the discussions I can’t say with much certainty, but I would probably leave it at just the clientid for now.

1 Like