Adding a node with MQTT problem

[mastrogippo]

I’m trying to build a custom node with an ESP8266. I don’t want to publish data with the URL+KEY method, so I’m trying to write (and read) data via MQTT.
The message I am posting is “emon/EVSE/power1” with an int power value.
I added this in the configuration file under [nodes]

[[42]]
    nodename = EVSE
    [[[rx]]]
        names = power1
        datacodes = h
        scales = 1
        units = W

I can see the new node in the Inputs tab, but when I try to add a “Log to feed” I get this error as soon as I click save:

“ERROR: could not save processlist. undefined.”

I then see the feed in the Feeds page, but the value is always NaN and doesn’t update.
Are there more steps I should have taken to add a new node?

My software version is low-write 9.7.2 | 2016.07.04, self hosted on a Raspi.

[mastrogippo]

Screenshot:

[glyn.hudson]

No emonhub node decoder config is needed since you are posting directly to MQTT. The emonhub node decoder is only needed to decode data received via RFM69CW / RFM12B, emonhub the posts the resulting (decoded) data to MQTT. See flow chart:

The error you are seeing replicates a known Emoncms issue: Issue logging input to feed: "Error could not save process list Undefined" · Issue #541 · emoncms/emoncms · GitHub. This is not fault of yours, it should work. I am not sure what’s causing this error. I have asked @TrystanLea to investigate.

Update: Ah, just noticed in your post that your self-hosted on a RasPi. Are you runnin our pre built emonSD SD card image? If so what version. If not, are you running MQTT Input script:

emoncms/emoncms/blob/master/docs/RaspberryPi/MQTT.md#run-emoncms-phpmqtt_input-script

You can find out by trying to restart it:

$ sudo service mqtt_input restart

[mastrogippo]

Thank you for your prompt answer! I’m running the prebuilt emonSD-03May16.
The service restarts without errors.
Is there any way I can help speed up the bugfix, like giving access to my setup or donating some bounty? (I purchased the official kit that I could have easily diy just to support the developers ). I’d love to fix it myself but I’m a hardware guy…

[Jon (Jon Murphy)]

I am still experiencing the same ERROR: Could not save processlist. undefined

Just curious how the debugging or fix was going for this error? Is there additional information I can provide?

 

Continuing the discussion from Publish to MQTT Issue:

Continuing the discussion from Cannot create Feed from MQTT input:

EDIT: added link to github issue.

Issue logging input to feed: "Error could not save process list Undefined"

[Jon (Jon Murphy)]

@glyn.hudson & @TrystanLea - Still getting this same error. I tried the sudo service mqtt_input restart but it did not work. What other info can I provide to help get this fixed?!?

[glyn.hudson]

Sorry for the delay looking into this.We have been struggling to re-create this issue reliably.

I have posted on the github issue. It would be great if you could try and test:

Issue logging input to feed: "Error could not save process list Undefined"

[Jon (Jon Murphy)]

I’ll take a look at the github & test.

---

For me this is repeatable. These are my steps:

Create a new MQTT topic. This could be:
emon/existing_device/new_topic (e.g., emon/Nest/target_temperature_high_c)
-or-
emon/new_device/new_topic (e.g., emon/Device/Item)

 
Click Inject and the new item will appear in the Inputs list:

 

One of the hints things are going bad is the new item will disappear in a minute.

Note: Adding new data to the mqtt item will cause emon/Device/Item to reappear.
 

Add the Feed (within the one minute):

 

Ugh!

 

Test node-red flow:
[{“id”:“2a9a2dbe.d0eed2”,“type”:“mqtt out”,“z”:“ab9c389a.5a7d8”,“name”:"",“topic”:“emon/Device/Item”,“qos”:“2”,“retain”:"",“broker”:“19211dbb.e6dee2”,“x”:590,“y”:160,“wires”:[]},{“id”:“40d39cfe.835bb4”,“type”:“inject”,“z”:“ab9c389a.5a7d8”,“name”:"",“topic”:"",“payload”:"",“payloadType”:“date”,“repeat”:"",“crontab”:"",“once”:false,“x”:260,“y”:180,“wires”:[[“2a9a2dbe.d0eed2”]]},{“id”:“19211dbb.e6dee2”,“type”:“mqtt-broker”,“z”:"",“broker”:“localhost”,“port”:“1883”,“clientid”:"",“usetls”:false,“verifyservercert”:true,“compatmode”:true,“keepalive”:“15”,“cleansession”:true,“willTopic”:"",“willQos”:“0”,“willRetain”:“false”,“willPayload”:"",“birthTopic”:"",“birthQos”:“0”,“birthRetain”:“false”,“birthPayload”:""}]

[Jon (Jon Murphy)]

Glyn - the github fix did not work. I am still seeing the same error. For me the disappearing input seems to be the big indicator that something is not right.

@Jon

One of the hints things are going bad is the new item will disappear in a minute.

Note: Adding new data to the mqtt item will cause emon/Device/Item to reappear.

 

EDIT: The info will disappear from the Inputs page when using this command also:

mosquitto_pub -u 'emonpi' -P 'emonpimqtt2016' -t 'emon/Device2/Item2' -m '9999'

 

EDIT2: When I grab the sudo service mqtt_input status I get this exited message:

Active: active (exited) since Tue 2016-12-13 17:09:32 CST; 6 days ago

And that corresponds with the emonPi uptime since: 2016-12-13 17:08:53.
 

pi@emonpi(rw):~$ sudo service mqtt_input status
● mqtt_input.service - LSB: Starts phpmqtt_input agent daemon at boot time
   Loaded: loaded (/etc/init.d/mqtt_input)
   Active: active (exited) since Tue 2016-12-13 17:09:32 CST; 6 days ago

Warning: Journal has been rotated since unit was started. Log output is incomplete or unavailable.

pi@emonpi(rw):~$ sudo service mqtt_input restart

pi@emonpi(rw):~$ sudo service mqtt_input status
● mqtt_input.service - LSB: Starts phpmqtt_input agent daemon at boot time
   Loaded: loaded (/etc/init.d/mqtt_input)
   Active: active (running) since Mon 2016-12-19 20:06:57 CST; 2s ago
  Process: 32484 ExecStop=/etc/init.d/mqtt_input stop (code=exited, status=0/SUCCESS)
  Process: 32490 ExecStart=/etc/init.d/mqtt_input start (code=exited, status=0/SUCCESS)
   CGroup: /system.slice/mqtt_input.service
           └─32494 /usr/bin/php /var/www/emoncms/scripts/phpmqtt_input.php

Dec 19 20:06:57 emonpi mqtt_input[32490]: Starting Daemon for the emoncms MQTT script...t.
Dec 19 20:06:57 emonpi systemd[1]: Started LSB: Starts phpmqtt_input agent daemon at...me.
Hint: Some lines were ellipsized, use -l to show in full.
pi@emonpi(rw):~$ 

pi@emonpi(ro):~$ uptime -s
2016-12-13 17:08:53
pi@emonpi(ro):~$ 

 
Then I ran sudo service mqtt_input restart. Now any new data into MQTT works as it should. So where do errors from mqtt_input end up?

[Jon (Jon Murphy)]

and the exited time corresponds with this error on Line 34:

1481670572: Socket error on client mosq/FrFNBb;BP\]MYF;pv<, disconnecting.

mosquitto.log.txt (10.7 KB)

[Jon (Jon Murphy)]

@glyn.hudson - I ran out of knowledge and don’t know where to go from here. I am not sure what it means when the mqtt_input service is active (exited)

Active: active (exited) since Tue 2016-12-13 17:09:32 CST; 6 days ago

Is there any info I can provide you to help troubleshoot?

[Bill.Thomson]

@Jon

when the mqtt_input service is active (exited)

Jon,

Maybe this will be of some help:

What does status "active (exited)" mean for a systemd service?

[glyn.hudson]

Thanks for the extra debug info @Jon. I’m not surprised that the bug still exists. @TrystanLea couldn’t really figure out the exact cause, other the the data is getting into redis db (hence the input appearing in inputs) but not getting put into the mysql db (hence the error). Adding a delay before connecting to mysql seemed to help but it’s obviously not fixed. It’s strange that after a restart input works OK.

Thanks again for your help, we’ll try and revisit this week. It’s an awkward little bug! Possibly an issue with underlying progress.

[Jon (Jon Murphy)]

it is not related to the Active: active (exited). This time the status was Active: active (running) and my inputs were disappearing. The sudo service mqtt_input restart allowed my input to work as it should.
 

pi@emonpi(rw):~$ sudo service mqtt_input status
● mqtt_input.service - LSB: Starts phpmqtt_input agent daemon at boot time
   Loaded: loaded (/etc/init.d/mqtt_input)
   Active: active (running) since Mon 2016-12-19 20:06:57 CST; 17h ago
  Process: 32484 ExecStop=/etc/init.d/mqtt_input stop (code=exited, status=0/SUCCESS)
  Process: 32490 ExecStart=/etc/init.d/mqtt_input start (code=exited, status=0/SUCCESS)
   CGroup: /system.slice/mqtt_input.service
           └─32494 /usr/bin/php /var/www/emoncms/scripts/phpmqtt_input.php

Dec 19 20:06:57 emonpi mqtt_input[32490]: Starting Daemon for the emoncms MQTT script: mqtt_input.
Dec 19 20:06:57 emonpi systemd[1]: Started LSB: Starts phpmqtt_input agent daemon at boot time.

pi@emonpi(rw):~$ sudo service mqtt_input restart

[glyn.hudson]

Hi @Jon, happy new year! Hope you have had a good Christmas break.

I am continuing to try and take steps to debug and fix this issue. I have made a change to the mqtt_input script to move to running it as a systemd service rather than init.d. I have posted this on another thread, please could you run emonPi update to pull in the changes and continue to monitor. Sorry it’s taking so long to try and fix this issue.

Mqtt_input script run as systemd instead of initd

The mqtt_input service runs the phpmqtt_input script which subscribes to MQTT topics and posts data to local Emoncms. It’s used on the emonSD which is running on emonPi / emonBase. Following on from mqtt_input script debugging, see threads: https://community.openenergymonitor.org/t/adding-a-node-with-mqtt-problem/1396 https://community.openenergymonitor.org/t/cannot-create-feed-from-mqtt-input/1150 https://community.openenergymonitor.org/t/publish-to-mqtt-issue/929 and GitHub issue: I h…

[Jon (Jon Murphy)]

Excellent! Thank you!

[greenthegarden (Philip Cutler)]

I have also just run into this issue. I had successfully set up both CT sources (power1 and power2) to log to feed, so they both showed up as expected in the feeds view. However, I inadvertently deleted (via Emoncms) power2. Although the input was recreated, when I attempt to ‘log to feed’ (feed last value:NaN) is displayed. I have rebooted but the problem still exists. I have not attempted to change the running of the mqtt_input script, but can give it a go if you think it may help. Thanks for any assistance.

[glyn.hudson]

Try running an emonpi update (admin > update) in local emoncms

[greenthegarden (Philip Cutler)]

Thanks Glyn for the quick response. I thought I was up-to-date. After running an update and reboot all seems to be good. Thanks for all the work put into Emoncms.

[margusv (Margus Veer)]

I have to problem with MQTT and node-red:

I have done soma solution before emonpi, with raspberry Pi and have some info in external MySQL from where I get some average, MIN and MAX temp. Let’s say I push info with MQTT to emon/emonpi/t8 and second info to emon/emonpi/t7, however various sources of information will go to both output first info is let’s sai 25 and second info 59 and whe look to inpit list then in one moment shows emonpi t7 with 25 and next moment t8 with 59, but not in same time