Adding a node with MQTT problem

mastrogippo · 2016-08-29T17:03:50.218Z

Screenshot:

glyn.hudson · 2016-08-29T17:30:19.718Z

No emonhub node decoder config is needed since you are posting directly to MQTT. The emonhub node decoder is only needed to decode data received via RFM69CW / RFM12B, emonhub the posts the resulting (decoded) data to MQTT. See flow chart:

emonPi_System_Diagram.png1031×511 83.5 KB

The error you are seeing replicates a known Emoncms issue: Issue logging input to feed: "Error could not save process list Undefined" · Issue #541 · emoncms/emoncms · GitHub. This is not fault of yours, it should work. I am not sure what’s causing this error. I have asked @TrystanLea to investigate.

Update: Ah, just noticed in your post that your self-hosted on a RasPi. Are you runnin our pre built emonSD SD card image? If so what version. If not, are you running MQTT Input script:

github.com

emoncms/emoncms/blob/master/docs/RaspberryPi/MQTT.md#run-emoncms-phpmqtt_input-script

**To install emoncms on a Raspberry Pi see new installation script approach:<br>https://github.com/openenergymonitor/EmonScripts**

---

## Archived: Enabling MQTT
MQTT is a "lightweight" messaging protocol, which enables the publishing of data from Emoncms, as well as subscribing to data from other connected devices. The emonPi uses MQTT to transfer data from emonHub to Emoncms.

### Preparation

Before following this guide, it is essential that emoncms was initially installed by following either the Raspbian Stretch, [Raspbian Jessie](readme.md) or [Raspbian Wheezy](install_Wheezy.md) installation guides, or you have used git to install a working version of Emoncms & emonHub on your Raspberry Pi, as well as a MQTT message broker such as [Mosquitto](http://mosquitto.org/) installed and running.

#### Update Emoncms:

    cd /var/www/emoncms && git pull

### Ensure packages are installed

In addition to Mosquitto MQTT server we will need to install [mosquitto-debian-repository]( http://mosquitto.org/2013/01/mosquitto-debian-repository) and [Mosquitto-PHP library](https://github.com/mgdm/Mosquitto-PHP):

    sudo apt-get install libmosquitto-dev

This file has been truncated. show original

You can find out by trying to restart it:

$ sudo service mqtt_input restart

mastrogippo · 2016-08-30T10:34:24.638Z

Thank you for your prompt answer! I’m running the prebuilt emonSD-03May16.
The service restarts without errors.
Is there any way I can help speed up the bugfix, like giving access to my setup or donating some bounty? (I purchased the official kit that I could have easily diy just to support the developers ). I’d love to fix it myself but I’m a hardware guy…

Jon · 2016-09-21T20:24:52.307Z

I am still experiencing the same ERROR: Could not save processlist. undefined

Screen Shot 2016-09-21 at 3.17.32 PM.png2608×1238 265 KB

Just curious how the debugging or fix was going for this error? Is there additional information I can provide?

Continuing the discussion from Publish to MQTT Issue:

Continuing the discussion from Cannot create Feed from MQTT input:

EDIT: added link to github issue.

github.com/emoncms/emoncms

Issue logging input to feed: "Error could not save process list Undefined"

opened 01:01PM - 23 Jun 16 UTC

closed 12:06PM - 14 Jun 18 UTC

glynhudson

bug

Trying to log an input from an MQTT test input to Emoncms (local) V9.7.1 results… in "Error could not save process list Undefined" when I try and save the process list setup after logging to a feed. Logging an emonPi feed (also coming from MQTT) works fine. I think the issue is `contextid=0` and object id is 0, see console output. Any idea what could be causing this? Test input seems to be updating correctly. ![screenshot 2016-06-23 at 13 56 53](https://cloud.githubusercontent.com/assets/758844/16303795/752e4132-394a-11e6-9f8f-13c694da07cc.png) ![screenshot 2016-06-23 at 13 56 20](https://cloud.githubusercontent.com/assets/758844/16303796/75522e8a-394a-11e6-9af2-f26caa82aa34.png)

Jon · 2016-12-06T16:24:43.123Z

@glyn.hudson & @TrystanLea - Still getting this same error. I tried the sudo service mqtt_input restart but it did not work. What other info can I provide to help get this fixed?!?

Screen Shot 2016-12-06 at 10.18.49 AM.png2680×1852 423 KB

glyn.hudson · 2016-12-06T17:36:51.257Z

Sorry for the delay looking into this.We have been struggling to re-create this issue reliably.

I have posted on the github issue. It would be great if you could try and test:

github.com/emoncms/emoncms

Issue logging input to feed: "Error could not save process list Undefined"

opened 01:01PM - 23 Jun 16 UTC

closed 12:06PM - 14 Jun 18 UTC

glynhudson

bug

Trying to log an input from an MQTT test input to Emoncms (local) V9.7.1 results… in "Error could not save process list Undefined" when I try and save the process list setup after logging to a feed. Logging an emonPi feed (also coming from MQTT) works fine. I think the issue is `contextid=0` and object id is 0, see console output. Any idea what could be causing this? Test input seems to be updating correctly. ![screenshot 2016-06-23 at 13 56 53](https://cloud.githubusercontent.com/assets/758844/16303795/752e4132-394a-11e6-9f8f-13c694da07cc.png) ![screenshot 2016-06-23 at 13 56 20](https://cloud.githubusercontent.com/assets/758844/16303796/75522e8a-394a-11e6-9af2-f26caa82aa34.png)

Jon · 2016-12-06T19:35:33.673Z

I’ll take a look at the github & test.

For me this is repeatable. These are my steps:

Create a new MQTT topic. This could be:
emon/existing_device/new_topic (e.g., emon/Nest/target_temperature_high_c)
-or-
emon/new_device/new_topic (e.g., emon/Device/Item)

Screen Shot 2016-12-06 at 12.29.22 PM.png2676×1282 270 KB

Click Inject and the new item will appear in the Inputs list:

Screen Shot 2016-12-06 at 12.29.00 PM.png2676×1282 274 KB

One of the hints things are going bad is the new item will disappear in a minute.

Screen Shot 2016-12-06 at 12.29.35 PM.png2676×1282 283 KB

Note: Adding new data to the mqtt item will cause emon/Device/Item to reappear.

Add the Feed (within the one minute):

Screen Shot 2016-12-06 at 12.15.15 PM.png2722×1848 451 KB

Ugh!

Screen Shot 2016-12-06 at 12.15.20 PM.png2722×1848 434 KB

Test node-red flow:
[{“id”:“2a9a2dbe.d0eed2”,“type”:“mqtt out”,“z”:“ab9c389a.5a7d8”,“name”:"",“topic”:“emon/Device/Item”,“qos”:“2”,“retain”:"",“broker”:“19211dbb.e6dee2”,“x”:590,“y”:160,“wires”:[]},{“id”:“40d39cfe.835bb4”,“type”:“inject”,“z”:“ab9c389a.5a7d8”,“name”:"",“topic”:"",“payload”:"",“payloadType”:“date”,“repeat”:"",“crontab”:"",“once”:false,“x”:260,“y”:180,“wires”:[[“2a9a2dbe.d0eed2”]]},{“id”:“19211dbb.e6dee2”,“type”:“mqtt-broker”,“z”:"",“broker”:“localhost”,“port”:“1883”,“clientid”:"",“usetls”:false,“verifyservercert”:true,“compatmode”:true,“keepalive”:“15”,“cleansession”:true,“willTopic”:"",“willQos”:“0”,“willRetain”:“false”,“willPayload”:"",“birthTopic”:"",“birthQos”:“0”,“birthRetain”:“false”,“birthPayload”:""}]

Jon · 2016-12-20T01:20:14.383Z

Glyn - the github fix did not work. I am still seeing the same error. For me the disappearing input seems to be the big indicator that something is not right.

Jon:

One of the hints things are going bad is the new item will disappear in a minute.

Screen Shot 2016-12-06 at 12.29.35 PM.png2676×1282 283 KB

Note: Adding new data to the mqtt item will cause emon/Device/Item to reappear.

EDIT: The info will disappear from the Inputs page when using this command also:

mosquitto_pub -u 'emonpi' -P 'emonpimqtt2016' -t 'emon/Device2/Item2' -m '9999'

EDIT2: When I grab the sudo service mqtt_input status I get this exited message:

Active: active (exited) since Tue 2016-12-13 17:09:32 CST; 6 days ago

And that corresponds with the emonPi uptime since: 2016-12-13 17:08:53.

pi@emonpi(rw):~$ sudo service mqtt_input status
● mqtt_input.service - LSB: Starts phpmqtt_input agent daemon at boot time
   Loaded: loaded (/etc/init.d/mqtt_input)
   Active: active (exited) since Tue 2016-12-13 17:09:32 CST; 6 days ago

Warning: Journal has been rotated since unit was started. Log output is incomplete or unavailable.

pi@emonpi(rw):~$ sudo service mqtt_input restart

pi@emonpi(rw):~$ sudo service mqtt_input status
● mqtt_input.service - LSB: Starts phpmqtt_input agent daemon at boot time
   Loaded: loaded (/etc/init.d/mqtt_input)
   Active: active (running) since Mon 2016-12-19 20:06:57 CST; 2s ago
  Process: 32484 ExecStop=/etc/init.d/mqtt_input stop (code=exited, status=0/SUCCESS)
  Process: 32490 ExecStart=/etc/init.d/mqtt_input start (code=exited, status=0/SUCCESS)
   CGroup: /system.slice/mqtt_input.service
           └─32494 /usr/bin/php /var/www/emoncms/scripts/phpmqtt_input.php

Dec 19 20:06:57 emonpi mqtt_input[32490]: Starting Daemon for the emoncms MQTT script...t.
Dec 19 20:06:57 emonpi systemd[1]: Started LSB: Starts phpmqtt_input agent daemon at...me.
Hint: Some lines were ellipsized, use -l to show in full.
pi@emonpi(rw):~$ 

pi@emonpi(ro):~$ uptime -s
2016-12-13 17:08:53
pi@emonpi(ro):~$

Then I ran sudo service mqtt_input restart. Now any new data into MQTT works as it should. So where do errors from mqtt_input end up?

Jon · 2016-12-20T02:40:37.439Z

and the exited time corresponds with this error on Line 34:

1481670572: Socket error on client mosq/FrFNBb;BP\]MYF;pv<, disconnecting.

mosquitto.log.txt (10.7 KB)

Jon · 2016-12-20T17:11:10.256Z

@glyn.hudson - I ran out of knowledge and don’t know where to go from here. I am not sure what it means when the mqtt_input service is active (exited)

Active: active (exited) since Tue 2016-12-13 17:09:32 CST; 6 days ago

Is there any info I can provide you to help troubleshoot?

Bill.Thomson · 2016-12-20T17:59:38.864Z

Jon:

when the mqtt_input service is active (exited)

Jon,

Maybe this will be of some help:

unix.stackexchange.com

What does status "active (exited)" mean for a systemd service?

linux, systemd, services, sysvinit

asked by Highly Irregular on 10:15PM - 09 Nov 15 UTC

glyn.hudson · 2016-12-20T18:11:01.299Z

Thanks for the extra debug info @Jon. I’m not surprised that the bug still exists. @TrystanLea couldn’t really figure out the exact cause, other the the data is getting into redis db (hence the input appearing in inputs) but not getting put into the mysql db (hence the error). Adding a delay before connecting to mysql seemed to help but it’s obviously not fixed. It’s strange that after a restart input works OK.

Thanks again for your help, we’ll try and revisit this week. It’s an awkward little bug! Possibly an issue with underlying progress.

Jon · 2016-12-20T20:34:45.958Z

it is not related to the Active: active (exited). This time the status was Active: active (running) and my inputs were disappearing. The sudo service mqtt_input restart allowed my input to work as it should.

pi@emonpi(rw):~$ sudo service mqtt_input status
● mqtt_input.service - LSB: Starts phpmqtt_input agent daemon at boot time
   Loaded: loaded (/etc/init.d/mqtt_input)
   Active: active (running) since Mon 2016-12-19 20:06:57 CST; 17h ago
  Process: 32484 ExecStop=/etc/init.d/mqtt_input stop (code=exited, status=0/SUCCESS)
  Process: 32490 ExecStart=/etc/init.d/mqtt_input start (code=exited, status=0/SUCCESS)
   CGroup: /system.slice/mqtt_input.service
           └─32494 /usr/bin/php /var/www/emoncms/scripts/phpmqtt_input.php

Dec 19 20:06:57 emonpi mqtt_input[32490]: Starting Daemon for the emoncms MQTT script: mqtt_input.
Dec 19 20:06:57 emonpi systemd[1]: Started LSB: Starts phpmqtt_input agent daemon at boot time.

pi@emonpi(rw):~$ sudo service mqtt_input restart

glyn.hudson · 2017-01-05T10:51:44.658Z

Hi @Jon, happy new year! Hope you have had a good Christmas break.

I am continuing to try and take steps to debug and fix this issue. I have made a change to the mqtt_input script to move to running it as a systemd service rather than init.d. I have posted this on another thread, please could you run emonPi update to pull in the changes and continue to monitor. Sorry it’s taking so long to try and fix this issue.

Mqtt_input script run as systemd instead of initd Emoncms

The mqtt_input service runs the phpmqtt_input script which subscribes to MQTT topics and posts data to local Emoncms. It’s used on the emonSD which is running on emonPi / emonBase. Following on from mqtt_input script debugging, see threads: https://community.openenergymonitor.org/t/adding-a-node-with-mqtt-problem/1396 https://community.openenergymonitor.org/t/cannot-create-feed-from-mqtt-input/1150 https://community.openenergymonitor.org/t/publish-to-mqtt-issue/929 and GitHub issue: I h…

Jon · 2017-01-05T21:35:03.273Z

Excellent! Thank you!

greenthegarden · 2017-01-15T12:08:57.235Z

I have also just run into this issue. I had successfully set up both CT sources (power1 and power2) to log to feed, so they both showed up as expected in the feeds view. However, I inadvertently deleted (via Emoncms) power2. Although the input was recreated, when I attempt to ‘log to feed’ (feed last value:NaN) is displayed. I have rebooted but the problem still exists. I have not attempted to change the running of the mqtt_input script, but can give it a go if you think it may help. Thanks for any assistance.

glyn.hudson · 2017-01-15T13:47:49.585Z

Try running an emonpi update (admin > update) in local emoncms

greenthegarden · 2017-01-16T04:04:32.904Z

Thanks Glyn for the quick response. I thought I was up-to-date. After running an update and reboot all seems to be good. Thanks for all the work put into Emoncms.

margusv · 2017-01-16T04:46:53.445Z

I have to problem with MQTT and node-red:

I have done soma solution before emonpi, with raspberry Pi and have some info in external MySQL from where I get some average, MIN and MAX temp. Let’s say I push info with MQTT to emon/emonpi/t8 and second info to emon/emonpi/t7, however various sources of information will go to both output first info is let’s sai 25 and second info 59 and whe look to inpit list then in one moment shows emonpi t7 with 25 and next moment t8 with 59, but not in same time

margusv · 2017-01-16T05:15:09.154Z

After update to last low-write 9.7.9 | 2017.01.05, all works like charm