I just started tracking two new items using MQTT to publish each item (Key). And found something very odd! Every time a new value is published to those two Keys, two new lines are added to the Inputs web page. I think its created 50-60 new items.
Not sure where to start looking for this one. Could this be related to any recent changes??
EDIT: This is with the current emonPi Update. And I did a sudo reboot.
EDIT2: This may be the issue or it could be me trying to delete the duplicate info:
pi@emonpi(ro):~$ sudo journalctl -f -u mqtt_input -o short
-- Logs begin at Tue 2017-01-10 21:05:48 CST. --
Jan 10 21:21:26 emonpi mqtt_input[1703]: PHP Notice: Undefined index: processList in /var/www/emoncms/Modules/input/input_model.php on line 147
Jan 10 21:21:26 emonpi mqtt_input[1703]: Notice: Undefined index: processList in /var/www/emoncms/Modules/input/input_model.php on line 147
Jan 10 21:21:35 emonpi mqtt_input[1703]: PHP Notice: Undefined index: nodeid in /var/www/emoncms/Modules/input/input_model.php on line 145
Jan 10 21:21:35 emonpi mqtt_input[1703]: Notice: Undefined index: nodeid in /var/www/emoncms/Modules/input/input_model.php on line 145
Jan 10 21:21:35 emonpi mqtt_input[1703]: PHP Notice: Undefined index: name in /var/www/emoncms/Modules/input/input_model.php on line 147
Jan 10 21:21:35 emonpi mqtt_input[1703]: Notice: Undefined index: name in /var/www/emoncms/Modules/input/input_model.php on line 147
Jan 10 21:21:35 emonpi mqtt_input[1703]: PHP Notice: Undefined index: id in /var/www/emoncms/Modules/input/input_model.php on line 147
Jan 10 21:21:35 emonpi mqtt_input[1703]: Notice: Undefined index: id in /var/www/emoncms/Modules/input/input_model.php on line 147
Jan 10 21:21:35 emonpi mqtt_input[1703]: PHP Notice: Undefined index: processList in /var/www/emoncms/Modules/input/input_model.php on line 147
Jan 10 21:21:35 emonpi mqtt_input[1703]: Notice: Undefined index: processList in /var/www/emoncms/Modules/input/input_model.php on line 147
^Cpi@emonpi(ro):~$
and each time MQTT (or something related) saw the bad character it created a new Key. Removing the single quote from the Key name fixed it instantly.
I’ll dig through MQTT docs tomorrow to determine what are invalid and valid characters for MQTT topic names. I thought just # and + were not acceptable as part of a name.
Best I can tell the single quote should be a valid character for an MQTT name. And most any ASCII character should also be valid except for # and +. Spaces are also OK but are discouraged.
Hope no-one minds me breaking this out to another thread, it has nothing to do with the use of systemd for the mqtt_input daemon.
It looks like it is ignoring certain symbols when it saves the input but still used the “with excluded symbols” copy to search for a pre-existing input, and not finding it as it was previously saved in a “without certain symbols” form and creating another copy(s).
@Jon I think you are right this does have more in common with not being able to use numeric topic levels (ie node ids) too.
These restrictions are not in MQTT itself, but in the way it’s implemented in emoncms.
Even if it is decided that emoncms will have it’s own convention, it needs fixing so that error such as you describe here does not occur, I think it would be better to fix the input code if possible.
From what I read " / # + $ " should not be used in topic naming and spaces should be avoided, personally I think both single and double quotes should be avoided too if possible, but I do not see that documented anywhere.
what is the filename of the input code? (I’d like to take a look)
For MQTT Invalid Characters: I’ve looked in the docs below and a few others. I haven’t found any reference single or double quotes. They seem to be valid. “$” is to be avoided as it is reserved for internal statistics of the MQTT broker. And then I found this:
When topic-based wildcards are not wild
The wildcard characters ‘+’ and ‘#’ have no special meaning when they are mixed with other characters (including themselves) in a topic level.
This means that topics that contain ‘+’ or ‘#’ together with other characters in a topic level can be published.
For example, consider the following two topics:
level0/level1/+/level4/#
level0/level1/#+/level4/level#
In the first example, the characters ‘+’ and ‘#’ are treated as wildcards and are therefore not valid in a topic string that is to be published to but are valid in a subscription.
In the second example, the characters ‘+’ and ‘#’ are not treated as wildcards and therefore the topic string can be both published and subscribed to.
This is all very confusion. I wish I could find an official “valid” or “invalid” character list! I’d hate to think one needs to be created for emon/OEM.
No, as I said this was a personal recommendation. Over the years I have all too often found many errors have found to be caused by use of unexpected chars, spaces and quotes being the worst offenders in my opinion. I have learnt that keeping it simple is so much easier, we have seen issues in the past with certain characters in feed names in emoncms, wifi psk’s in the wifi module and also the user login names on emoncms etc.
Not all characters can be fully used in emoncms. For example, if you start adding commas, curly braces, colons, equal signs and quote marks to input and feed names, what happens to the JSON output from api calls to list inputs and feeds? That’s just one “other convention”. There is php, javascript, html, ajax, sql etc to consider too as those input and feed names will need processing, displaying and storing too.
I’m not suggesting it cannot be done, but to make an emoncms capable of utilizing all chars across the board would probably be a mean task. It may be easier for both the devs and the users to have clearly defined restrictions, that are then coded into emoncms so that error handling can be improved and notify the user “Oi! that’s an illegal character!” rather than silently doing nothing, something quite unexpected or falling over completely.
