MQTT Strangeness

borpin · 2017-05-22T13:37:30.862Z

So some strange happenings in MQTT land. I have had MQTT running for a while, recently migrated to a new server locally though I’m not sure that is the reason. Emoncms (and modules) updated to latest stable today and database updates done.

The other thing to check is that I have 2 processes (Emoncms and node-red) reading the same MQTT feed. Is that allowed?

So first oddity, is any new feeds I create keep disappearing from the input page, then reappearing when new data is posted (every 120s). No pattern, sometimes it is a new value for a different node. Any reason why this should happen?

Secondly, the input does not always capture all the MQTT topics. You can see the same feed from the console and Emoncms. The last item from the console is missing. Next time it may appear again.

mqttemoncmspywws.PNG256×599 4.19 KB

pb66 · 2017-05-22T14:52:30.256Z

borpin:

The other thing to check is that I have 2 processes (Emoncms and node-red) reading the same MQTT feed. Is that allowed?

Yes that is the intended key strength of MQTT.

borpin:

So first oddity, is any new feeds I create keep disappearing from the input page, then reappearing when new data is posted (every 120s). No pattern, sometimes it is a new value for a different node. Any reason why this should happen?

There are some known “gremlins” with the emoncms MQTT inputs so this isn’t as odd as you would think,

borpin:

Secondly, the input does not always capture all the MQTT topics. You can see the same feed from the console and Emoncms. The last item from the console is missing. Next time it may appear again.

I think this is linked to the first point, just more successful than other posts, but none the less, it’s not as it is supposed to work and perhaps this new info might help towards debugging the issue.

Have you tried setting the log level to “INFO” in emoncms/settings.php and checking emoncms.log to see if there are any clues when data is incomplete?

You can open another ssh terminal window and use tail -f /var/log/emoncms/emoncms.log for a live output of the incoming data.

borpin · 2017-05-22T16:37:47.821Z

Changed the log level and had a look at the logs. Unfortunately it seems there is no log information coming out of the input part only, the Feed part of the system. I can see the MQTT messages arriving at the broker, but there are no corresponding log entries as the input page on Emoncms updates.

borpin · 2017-05-22T16:42:55.738Z

On a related subject, should I be able to have 2 separate MQTT feeds, going to the same node but with different keys?

So, MQTT sender sends emon/CO2/Hall and a second device sends to the same broker emon/CO2/Bedroom.

Does Emoncms assume that all keys in a node are updated at the same time? Could that explain the loss of the final key if it took long enough to process so the time was different?

borpin · 2017-05-22T16:54:12.422Z

So the more I look at this the stranger it gets but I think it is a timeout / timing issue.

Firstly note that the update time of the first key and the last key differs by a second. They are all published at the same time (and is true for both these different nodes, published by completely different physical machines to the same broker).

Now look at these two. This is the same update cycle for the same node;

mqttemoncmspywws2.PNG532×651 13.2 KB

mqttemoncmspywws3.PNG531×614 12.7 KB

In the second screenshot, the last key has vanished!
{edit}
Just checked the Server Load and it seems fine.

pb66 · 2017-05-22T17:22:42.320Z

borpin:

So, MQTT sender sends emon/CO2/Hall and a second device sends to the same broker emon/CO2/Bedroom.

Yes that should be fine, it is how MQTT is designed to wok, but I cannot vouch for whether emoncms is ok with it, I can see no reason why it should object.

borpin:

Does Emoncms assume that all keys in a node are updated at the same time?

No, or at least it shouldn’t, the “leaves” as it were on each “branch” of any topic can come from different locations and at different update intervals.[quote=“borpin, post:4, topic:4082”]
Could that explain the loss of the final key if it took long enough to process so the time was different?
[/quote]

It could be the reason you see differences in the update time and thus, as part of the whole “now you see me, now you don’t” issue with the emoncms mqtt inputs. The inputs should never disappear until deleted from the inputs page by you, even if the sender of emon/CO2/Hall went offline the input should always remain visible, although the “Updated” time would increase until it reached “n/a” even though the emon/CO2/Bedroom device continues to publish and that input remains green and a shorter last “Updated” time.

borpin:

They are all published at the same time

Actually no! Unlike the RFM type payload, each of these inputs are totally independent and they are posted consecutively one after the other by the source device. So it is perfectly reasonable to see a time difference “mid batch” as although they will be published in very rapid succesion, emoncms input only deals in whole seconds so something arriving at xxxxxx0.999 secs unixtime will appear to be a whole second earlier than one arriving at xxxxxx1.0 secs unixtime.

borpin:

In the second screenshot, the last key has vanished!

That shouldn’t happen, and is a key part in the emoncms mqtt inputs issues.

borpin · 2017-05-22T17:40:14.357Z

pb66:

Actually no! Unlike the RFM type payload, each of these inputs are totally independent and they are posted consecutively one after the other by the source device.

Yes I realised that but as I see the mosquitto_sub process that I am running alongside this, prints all the topics in a fraction of a second, it seems to me EmonCMS is taking an inordinate time to process the input and that I suspect is part of the issue. I also meant that I am sending them as a mosquitto (or node red) multi-message format rather than one at a time in the sending code itself.

pb66:

No, or at least it shouldn’t

I agree but that is an assumption.

pb66:

although the “Updated” time would increase until it reached “n/a”

And that is what I see for older inputs created previously (by the old MQTT interface).

I have always thought the MQTT data structure, of a single value per topic was flawed and should have been a JSON input with a timestamp so multiple data items came as a single package (actually more efficient as well).

I think I may well process this through node-red and push it to Emoncms using the HTTP API.

pb66 · 2017-05-22T18:06:59.371Z

borpin:

I have always thought the MQTT data structure, of a single value per topic was flawed and should have been a JSON input with a timestamp so multiple data items came as a single package (actually more efficient as well).

@glyn.hudson and @TrystanLea are insistent on everything using the same topics and so that you can target one specific device they must therefore be broken out to individual topics and it is then emoncms that suffers for it. IMO there is no good reason why emonHub couldn’t breakout all the values to non-time sensitive topics (ie current statuses) that can be used locally and ALSO send the same data as a timestamped payload to emoncms also via MQTT if that’s your preference. That way the data is available in either format, if you insist on getting individual topics at emoncms you can. But those of us wanting correctly timestamped data that arrives in a fixed order for accurate processing would no longer be excluded from using MQTT with emoncms.

borpin:

I think I may well process this through node-red and push it to Emoncms using the HTTP API.

I think that may be more reliable, but it seems a bit like writing an email and then printing it off and faxing it to the recipient!!!

borpin · 2017-05-22T18:13:03.010Z

pb66:

I think that may be more reliable, but it seems a bit like writing an email and then printing it off and faxing it to the recipient!!!

LOL

borpin · 2017-05-22T18:45:34.766Z

Sadly, I’m sitting here watching this…

What actually happens is that the pywws node is the second to last node. It updates with a lower frequency than the CO2 nodes. When the pywws node updates, the final key is present but any CO2 node is removed. When a CO2 node data comes in it seems to wipe out the last key of the pywws node.

What is more interesting is that when a different CO2 value comes in, it wipes out the existing CO2 node.

It looks to me like I am at the end of the index or something. I do have a Node 30, and then there are 3 below that

Is there a limit on number of nodes? In the distant past I do remember something about that. I am going for a reboot as that seems to have solved MQTT problems for others.

{edit}
Seems a reboot is the answer. Everything working as expected and the CO2 Node is accepting 2 inputs from 2 different devices (different MQTT Topics).

Looks to me like a range index is getting corrupted somewhere in the input process.

TrystanLea · 2017-05-25T11:34:27.402Z

Im not necessarily against the idea of having a emoncms mqtt topic that accepts json format including a timestamp. I think its one suggestion I proposed early on From what I understood the downside is that it makes subscribing to topics and using topics outside of emoncms, e.g by other applications such as nodered and openhab harder as they would then need to decode the json sent from say emonhub.

Im sure a modification could be made to emoncms/scripts/phpmqtt_input.php to accept both approaches if there was the desire to have this option. Happy to accept a pull request that would support both.

@borpin this sounds like the ongoing issue, for some reason I thought we where further with it?

community.openenergymonitor.org/t/newly-created-inputs-not-appearing-in-emoncms-via-mqtt/2515/40
community.openenergymonitor.org/t/emonbase-no-inputs-problem-with-mqtt/1698

Are you able to replicate the issue above reliably? that would be a great place to start with getting to the bottom of it?

TrystanLea · 2017-05-25T11:37:53.259Z

The nodeid limit that is usually applied to inputs coming in via HTTP route is not applied on the MQTT route. Testing here I seem to be able to use numeric nodeid’s and input names without an issue which @pb66 highlighted as being an issue, but then I have only just started the mqtt service script.

pb66 · 2017-05-25T13:05:14.697Z

TrystanLea:

Testing here I seem to be able to use numeric nodeid’s and input names without an issue which @pb66 highlighted as being an issue,

It was actually Glyn that highlighted it to me,

Newly created inputs not appearing in emonCMS via MQTT

The ‘nodename’ should be letters not a number. I don’t think mqtt topics work with numbers e.g the logfile should show posting to mqtt topic like ‘emon/emonth1/temperature’

I just confirmed that the issue lay with the mqtt_input.php and not a Mosquitto or emonHub issue. At the time I was able to recreate the issue at will and documented it all in that same thread. While I was able to get numeric node names working on occasion, the mqtt_input.php behaved quite erratically and this was more so with numeric nodenames.

TrystanLea:

but then I have only just started the mqtt service script

That may be a vital clue to gain insight to the issue. I did not try restarting (Mosquitto or) the emoncms mqtt_input service, I only tried a reboot after flushing Redis didn’t work. Rebooting has worked every time and I have recommended a reboot many times here on the forum when new inputs do not appear (even those with text names) and in most instances this has worked.

Perhaps we can try just restarting the service rather than rebooting to get us closer to understanding the cause

TrystanLea:

From what I understood the downside is that it makes subscribing to topics and using topics outside of emoncms, e.g by other applications such as nodered and openhab harder as they would then need to decode the json sent from say emonhub.

That is pretty much the crux of it, but if emonhub is able to publish a payload similar to the current http bulk upload api but via MQTT purely for the benefit of emoncms and also publish each and every value in it’s own topic for anything and everything (local?) to be able to use then everyone would be happy.

I’m less concerned about whether it’s JSON formatted (I’m not opposed, I’m just not closed off to options), what is important is the inclusion of timestamps, the (optional?) ability to bulk upload following network outages or server downtime.and a predictable order of input processing so that cross-input processes can be relied upon.

A “bulk over MQTT” approach would also pave the way to a multi-user mqtt solution as an apikey could also be included in the payload.

Another benefit of a “tandem publish” by emonhub is that the “one topic per value” status’s need not be published to the same broker as the “bulk over MQTT” for example if emoncms.org adopts MQTT but users still want MQTT around the house.

[edit] In addition to the above “tandem publish” I would also like to add an emonhub interfacer that uses the current “fetch” api to get the current values of several feeds in one go and publish them on individual topics based on base topic and names defined in emohub.conf, this would make ALL emoncms feeds (incl virtual) available for publishing on the local broker (regardless of emoncms location) without having to use the “Publish to MQTT” process which slows down the input processing (if too many values published) and it would do it without opening and closing the MQTT connection for each value published as the current “Publish to MQTT” does.

To move towards the option of a full MQTT (no http) emoncms we could then look to using similar queries to the http api, expanding the capability of MQTT implemented in emoncms, for example executing the above “fetch” api via MQTT by publishing the query to the “emoncms rx only” topic and the response is published by emoncms to the “emoncms tx only” topic that emonhub is subscribed to so it can publish the individual topics as above.

By making it possible to delegate local stuff including “per value topics” to emonHub and creating a “super highway” between it and emoncms, in a user choice of MQTT or HTTP, users get the best of both worlds, lots of local “status” data readily available on individual topics AND a better emoncms, less loss of data (buffered bulk upload), more accurate data (timestamped data) and faster processing as there is no need for Publish to MQTT’s to be in the input processing. Not to mention a undoubtedly more reliable MQTT experience as emoncms doesn’t need to subscribe to so many topics and/or process each value as a completely separate input that happens to share the same node id as other inputs rather than processing them as a group.

TrystanLea · 2017-05-25T20:31:10.375Z

Thanks @pb66, ok lets aim to support a /input/post equivalent in the phpmqtt_input script as a starting point. I will get back to you properly on your other points soon

borpin · 2017-05-25T20:51:56.880Z

I can’t reproduce it, sorry.

On a timestamp, as Emoncms can publish MQTT, any data needed as a single item can be published this way. Equally, as it needs a specific topic to bring the data in, the data structure has in effect been created specifically for Emoncms anyway.

I have wondered why the topic to be subscribed to, could not be configured on a topic/key basis. i.e. you could subscribe to any MQTT topic.

pb66 · 2017-05-26T08:29:27.880Z

TrystanLea:

I will get back to you properly on your other points soon

Great it would be good to hear your thoughts.

TrystanLea:

ok lets aim to support a /input/post equivalent in the phpmqtt_input script as a starting point.

Maybe, what exactly do you have in mind?

jimmy · 2017-12-22T22:02:55.376Z

pb66:

That may be a vital clue to gain insight to the issue. I did not try restarting (Mosquitto or) the emoncms mqtt_input service, I only tried a reboot after flushing Redis didn’t work. Rebooting has worked every time and I have recommended a reboot many times here on the forum when new inputs do not appear (even those with text names) and in most instances this has worked.

Perhaps we can try just restarting the service rather than rebooting to get us closer to understanding the cause

Hi sorry I’m late to the party here, but now using something purely MQTT and I was finding that when posting to a topic such as emon/heating/boiler this would create the node heating, and then the key boiler as expected

Then if I posted to emon/heating/pump the key would be changed from boiler to pump.

restarting the service by running sudo service mqtt_input did get this working again.

This is on emoncms 9.8.25 if it helps with the debugging strangeness…

glyn.hudson · 2017-12-22T23:44:29.383Z

This issue with MQTT should fixed in 9.8.27, see commit:

github.com/emoncms/emoncms

mysqli ping to keep connection open

committed 01:53PM - 21 Dec 17 UTC

glynhudson

+19 -6

~~9.8.27 is currently in master branch, it will be merged into stable in the next few days.~~

Update: 9.8.27 has now been merged into stable and is available via standard emonPi update.

GitHub

Release V9.8.27 | 2017.12.21 · emoncms/emoncms

Add group module support @cagabi #730 French language localization @thib66 Fix MQTT > Emoncms input process error and input creation issues #752 #541 @TrystanLea c21302a

borpin · 2017-12-28T12:46:44.140Z

glyn.hudson:

Update: 9.8.27 has now been merged into stable and is available via standard emonPi update.

Looks like this has solved my MQTT issues. An announcement perhaps?

TrystanLea:

Im not necessarily against the idea of having a emoncms mqtt topic that accepts json format including a timestamp.

Could this be looked at again please?