MQTT Strangeness

So some strange happenings in MQTT land. I have had MQTT running for a while, recently migrated to a new server locally though I’m not sure that is the reason. Emoncms (and modules) updated to latest stable today and database updates done.

The other thing to check is that I have 2 processes (Emoncms and node-red) reading the same MQTT feed. Is that allowed?

So first oddity, is any new feeds I create keep disappearing from the input page, then reappearing when new data is posted (every 120s). No pattern, sometimes it is a new value for a different node. Any reason why this should happen?

Secondly, the input does not always capture all the MQTT topics. You can see the same feed from the console and Emoncms. The last item from the console is missing. Next time it may appear again.

Yes that is the intended key strength of MQTT.

There are some known “gremlins” with the emoncms MQTT inputs so this isn’t as odd as you would think,

I think this is linked to the first point, just more successful than other posts, but none the less, it’s not as it is supposed to work and perhaps this new info might help towards debugging the issue.

Have you tried setting the log level to “INFO” in emoncms/settings.php and checking emoncms.log to see if there are any clues when data is incomplete?

You can open another ssh terminal window and use tail -f /var/log/emoncms/emoncms.log for a live output of the incoming data.

Changed the log level and had a look at the logs. Unfortunately it seems there is no log information coming out of the input part only, the Feed part of the system. I can see the MQTT messages arriving at the broker, but there are no corresponding log entries as the input page on Emoncms updates.

On a related subject, should I be able to have 2 separate MQTT feeds, going to the same node but with different keys?

So, MQTT sender sends emon/CO2/Hall and a second device sends to the same broker emon/CO2/Bedroom.

Does Emoncms assume that all keys in a node are updated at the same time? Could that explain the loss of the final key if it took long enough to process so the time was different?

So the more I look at this the stranger it gets but I think it is a timeout / timing issue.

Firstly note that the update time of the first key and the last key differs by a second. They are all published at the same time (and is true for both these different nodes, published by completely different physical machines to the same broker).

Now look at these two. This is the same update cycle for the same node;

In the second screenshot, the last key has vanished!
{edit}
Just checked the Server Load and it seems fine.

Yes that should be fine, it is how MQTT is designed to wok, but I cannot vouch for whether emoncms is ok with it, I can see no reason why it should object.

No, or at least it shouldn’t, the “leaves” as it were on each “branch” of any topic can come from different locations and at different update intervals.[quote=“borpin, post:4, topic:4082”]
Could that explain the loss of the final key if it took long enough to process so the time was different?
[/quote]

It could be the reason you see differences in the update time and thus, as part of the whole “now you see me, now you don’t” issue with the emoncms mqtt inputs. The inputs should never disappear until deleted from the inputs page by you, even if the sender of emon/CO2/Hall went offline the input should always remain visible, although the “Updated” time would increase until it reached “n/a” even though the emon/CO2/Bedroom device continues to publish and that input remains green and a shorter last “Updated” time.

Actually no! Unlike the RFM type payload, each of these inputs are totally independent and they are posted consecutively one after the other by the source device. So it is perfectly reasonable to see a time difference “mid batch” as although they will be published in very rapid succesion, emoncms input only deals in whole seconds so something arriving at xxxxxx0.999 secs unixtime will appear to be a whole second earlier than one arriving at xxxxxx1.0 secs unixtime.

That shouldn’t happen, and is a key part in the emoncms mqtt inputs issues.

Yes I realised that but as I see the mosquitto_sub process that I am running alongside this, prints all the topics in a fraction of a second, it seems to me EmonCMS is taking an inordinate time to process the input and that I suspect is part of the issue. I also meant that I am sending them as a mosquitto (or node red) multi-message format rather than one at a time in the sending code itself.

I agree but that is an assumption.

And that is what I see for older inputs created previously (by the old MQTT interface).

I have always thought the MQTT data structure, of a single value per topic was flawed and should have been a JSON input with a timestamp so multiple data items came as a single package (actually more efficient as well).

I think I may well process this through node-red and push it to Emoncms using the HTTP API.

@glyn.hudson and @TrystanLea are insistent on everything using the same topics and so that you can target one specific device they must therefore be broken out to individual topics and it is then emoncms that suffers for it. IMO there is no good reason why emonHub couldn’t breakout all the values to non-time sensitive topics (ie current statuses) that can be used locally and ALSO send the same data as a timestamped payload to emoncms also via MQTT if that’s your preference. That way the data is available in either format, if you insist on getting individual topics at emoncms you can. But those of us wanting correctly timestamped data that arrives in a fixed order for accurate processing would no longer be excluded from using MQTT with emoncms.

I think that may be more reliable, but it seems a bit like writing an email and then printing it off and faxing it to the recipient!!!

1 Like

LOL :joy:

Sadly, I’m sitting here watching this…

What actually happens is that the pywws node is the second to last node. It updates with a lower frequency than the CO2 nodes. When the pywws node updates, the final key is present but any CO2 node is removed. When a CO2 node data comes in it seems to wipe out the last key of the pywws node.

What is more interesting is that when a different CO2 value comes in, it wipes out the existing CO2 node.

It looks to me like I am at the end of the index or something. I do have a Node 30, and then there are 3 below that

Is there a limit on number of nodes? In the distant past I do remember something about that. I am going for a reboot as that seems to have solved MQTT problems for others.

{edit}
Seems a reboot is the answer. Everything working as expected and the CO2 Node is accepting 2 inputs from 2 different devices (different MQTT Topics).

Looks to me like a range index is getting corrupted somewhere in the input process.

Im not necessarily against the idea of having a emoncms mqtt topic that accepts json format including a timestamp. I think its one suggestion I proposed early on :slight_smile: From what I understood the downside is that it makes subscribing to topics and using topics outside of emoncms, e.g by other applications such as nodered and openhab harder as they would then need to decode the json sent from say emonhub.

Im sure a modification could be made to emoncms/scripts/phpmqtt_input.php to accept both approaches if there was the desire to have this option. Happy to accept a pull request that would support both.

@borpin this sounds like the ongoing issue, for some reason I thought we where further with it?

Are you able to replicate the issue above reliably? that would be a great place to start with getting to the bottom of it?

The nodeid limit that is usually applied to inputs coming in via HTTP route is not applied on the MQTT route. Testing here I seem to be able to use numeric nodeid’s and input names without an issue which @pb66 highlighted as being an issue, but then I have only just started the mqtt service script.

It was actually Glyn that highlighted it to me,

I just confirmed that the issue lay with the mqtt_input.php and not a Mosquitto or emonHub issue. At the time I was able to recreate the issue at will and documented it all in that same thread. While I was able to get numeric node names working on occasion, the mqtt_input.php behaved quite erratically and this was more so with numeric nodenames.

That may be a vital clue to gain insight to the issue. I did not try restarting (Mosquitto or) the emoncms mqtt_input service, I only tried a reboot after flushing Redis didn’t work. Rebooting has worked every time and I have recommended a reboot many times here on the forum when new inputs do not appear (even those with text names) and in most instances this has worked.

Perhaps we can try just restarting the service rather than rebooting to get us closer to understanding the cause

That is pretty much the crux of it, but if emonhub is able to publish a payload similar to the current http bulk upload api but via MQTT purely for the benefit of emoncms and also publish each and every value in it’s own topic for anything and everything (local?) to be able to use then everyone would be happy.

I’m less concerned about whether it’s JSON formatted (I’m not opposed, I’m just not closed off to options), what is important is the inclusion of timestamps, the (optional?) ability to bulk upload following network outages or server downtime.and a predictable order of input processing so that cross-input processes can be relied upon.

A “bulk over MQTT” approach would also pave the way to a multi-user mqtt solution as an apikey could also be included in the payload.

Another benefit of a “tandem publish” by emonhub is that the “one topic per value” status’s need not be published to the same broker as the “bulk over MQTT” for example if emoncms.org adopts MQTT but users still want MQTT around the house.

[edit] In addition to the above “tandem publish” I would also like to add an emonhub interfacer that uses the current “fetch” api to get the current values of several feeds in one go and publish them on individual topics based on base topic and names defined in emohub.conf, this would make ALL emoncms feeds (incl virtual) available for publishing on the local broker (regardless of emoncms location) without having to use the “Publish to MQTT” process which slows down the input processing (if too many values published) and it would do it without opening and closing the MQTT connection for each value published as the current “Publish to MQTT” does.

To move towards the option of a full MQTT (no http) emoncms we could then look to using similar queries to the http api, expanding the capability of MQTT implemented in emoncms, for example executing the above “fetch” api via MQTT by publishing the query to the “emoncms rx only” topic and the response is published by emoncms to the “emoncms tx only” topic that emonhub is subscribed to so it can publish the individual topics as above.

By making it possible to delegate local stuff including “per value topics” to emonHub and creating a “super highway” between it and emoncms, in a user choice of MQTT or HTTP, users get the best of both worlds, lots of local “status” data readily available on individual topics AND a better emoncms, less loss of data (buffered bulk upload), more accurate data (timestamped data) and faster processing as there is no need for Publish to MQTT’s to be in the input processing. Not to mention a undoubtedly more reliable MQTT experience as emoncms doesn’t need to subscribe to so many topics and/or process each value as a completely separate input that happens to share the same node id as other inputs rather than processing them as a group.

Thanks @pb66, ok lets aim to support a /input/post equivalent in the phpmqtt_input script as a starting point. I will get back to you properly on your other points soon

I can’t reproduce it, sorry.

On a timestamp, as Emoncms can publish MQTT, any data needed as a single item can be published this way. Equally, as it needs a specific topic to bring the data in, the data structure has in effect been created specifically for Emoncms anyway.

I have wondered why the topic to be subscribed to, could not be configured on a topic/key basis. i.e. you could subscribe to any MQTT topic.

Great it would be good to hear your thoughts.

Maybe, what exactly do you have in mind?

Hi sorry I’m late to the party here, but now using something purely MQTT and I was finding that when posting to a topic such as emon/heating/boiler this would create the node heating, and then the key boiler as expected

Then if I posted to emon/heating/pump the key would be changed from boiler to pump.

restarting the service by running sudo service mqtt_input did get this working again.

This is on emoncms 9.8.25 if it helps with the debugging strangeness…

This issue with MQTT should fixed in 9.8.27, see commit:

9.8.27 is currently in master branch, it will be merged into stable in the next few days.

Update: 9.8.27 has now been merged into stable and is available via standard emonPi update.

Looks like this has solved my MQTT issues. An announcement perhaps? :smile:

Could this be looked at again please?

1 Like