I haven’t suggested otherwise. I’m suggesting you explore the only useful info you currently have right now and that is that some nodes are affected and some are not, that doesn’t suggest a randomness. If you where able to get the same node to be held in RAM or not, depending on a change of QoS setting, topic length, payload content or update frequency etc, you have reduced the field considerably.
I’m suggesting you shake the tree and see what falls out, If you try nothing and nothing changes, you have learnt nothing and made no progress. If you do try something, even if it doesn’t change anything, you’ve learn’t something and that’s progress, even if it is just ruling something out.
Have you checked the logs or set something up to confirm that everything is ok going into emoncms? Are there any missing datapoints in the feeds? Are the posts stuck in RAM processed or not?
I previously asked for data on all the nodes, update intervals, topics and payloads.
Have you tried subscribing to the emoncms topic from the command line (using QoS 2) to confirm there is nothing funky going on with the data going into emoncms?
Have you tried stopping any of the mqtt nodes to see if it reduces the issues?
Have you tried setting up a test node with a simple payload of a incrementing packet id or timestamp so that you can see if data is getting lost along the route? Does that test node appear in the memory dump?
You suggest this fault is present all the time, starts at boot and grows. This should give you loads of opportunity as you can try something and know almost immediately if it’s made a difference, you are not at the mercy of the fault occurring periodically at an unknown interval.
I can find no mention of MQTT in the IoTaWatt repo and there are several posts on the IoTaWatt forum in March confirming it wasn’t available at that time, so I think mentioning IoTaWatt here maybe a red herring as it is most likely posting via http and irrelevant to this discussion.
To my knowledge no one else is experiencing this, so whilst I agree this is a bug, I am also fairly convinced it is apparent on this system due to a certain way it’s configured or because of the data that’s being passed, the largest part of diagnosing this issue is to identify why it is happening on this setup and perhaps only to certain nodes. That can only be done by looking closely at the finer details of this setup and playing with the settings and/or data content to establish what is failing and why, then a fix can be administered so that the current setup does work. I’m in no way suggesting the issue is due to an incorrect user setting.
just as an observation, it looks like every one of the lines in your “filtered data” above has a space or special character in it. What are we looking at there? is it all topic+value or are some of the payloads multi value? Why are there so many colons? is the data corrupt as well as truncated? what does a good topic+payload look like? I would expect to see braces if there are colons, but I can’t find any example code right now.
The bulk of the lines come from “Solax 2” which contains a space, whilst that is valid MQTT it is bad practice as spaces cause issues else where, perhaps emoncms mqtt input is struggling with the spaces? It tends to everywhere else in emoncms.
Don’t use spaces in a topic
A space is the natural enemy of each programmer, they often make it much harder to read and debug topics, when things are not going the way, they should be. So similar to the first one, only because something is allowed doesn’t mean it should be used. UTF-8 knows many different white space types, it’s pretty obvious that such uncommon characters should be avoided.
(from MQTT Topics & Best Practices - MQTT Essentials: Part 5)