Emonpi low-write 9.8.25 - inputs stopped working

Pukka · 12 December 2017 19:16

Hi all,

I have been running the later version for the emonpi low-write 9.8.25, some of the inputs stopped working today

2017-12-04 09:08:24.215|WARN|phpmqtt_input.php|Starting MQTT Input script
2017-12-04 09:08:25.480|WARN|phpmqtt_input.php|Subscribing to: emon/#
2017-12-04 09:08:25.481|WARN|phpmqtt_input.php|Not connected, retrying connection
2017-12-04 09:08:25.482|WARN|phpmqtt_input.php|Connecting to MQTT server: Connection Accepted.: code: 0
2017-12-04 09:08:25.482|WARN|phpmqtt_input.php|Subscribed to topic: emon/#
2017-12-04 09:08:28.510|WARN|phpmqtt_input.php|Starting MQTT Input script
2017-12-04 09:08:29.644|WARN|phpmqtt_input.php|Subscribing to: emon/#
2017-12-04 09:08:29.644|WARN|phpmqtt_input.php|Not connected, retrying connection
2017-12-04 09:08:29.645|WARN|phpmqtt_input.php|Connecting to MQTT server: Connection Accepted.: code: 0
2017-12-04 09:08:29.645|WARN|phpmqtt_input.php|Subscribed to topic: emon/#
2017-12-10 08:51:06.452|ERROR|phpmqtt_input.php|exception 'Mosquitto\Exception' with message 'The client is not currently connected.' in /var/www/emoncms/scripts/phpmqtt_input.php:112
Stack trace:
#0 /var/www/emoncms/scripts/phpmqtt_input.php(112): Mosquitto\Client->loop()
#1 {main}
2017-12-10 08:51:07.257|WARN|phpmqtt_input.php|Not connected, retrying connection
2017-12-12 07:09:49.215|ERROR|phpmqtt_input.php|exception 'Mosquitto\Exception' in /var/www/emoncms/scripts/phpmqtt_input.php:112
Stack trace:
#0 /var/www/emoncms/scripts/phpmqtt_input.php(112): Mosquitto\Client->loop()
#1 {main}
2017-12-12 07:09:51.433|WARN|phpmqtt_input.php|Not connected, retrying connection
2017-12-12 07:19:14.784|ERROR|phpmqtt_input.php|exception 'RedisException' with message 'read error on connection' in /var/www/emoncms/Modules/process/process_processlist.php:657
Stack trace:
#0 /var/www/emoncms/Modules/process/process_processlist.php(657): Redis->exists('process:whaccum...')
#1 [internal function]: Process_ProcessList->wh_accumulator('17', 1513062619, 2487, NULL, Object(Process))
#2 /var/www/emoncms/Modules/process/process_model.php(60): call_user_func_array(Array, Array)
#3 /var/www/emoncms/Modules/process/process_model.php(103): Process->__call('wh_accumulator', Array)
#4 /var/www/emoncms/Modules/process/process_model.php(103): Process->wh_accumulator('17', 1513062619, 2487, NULL)
#5 /var/www/emoncms/scripts/phpmqtt_input.php(270): Process->input(1513062619, '2487', '1:16,2:1,34:17')
#6 [internal function]: message(Object(Mosquitto\Message))
#7 /var/www/emoncms/scripts/phpmqtt_input.php(112): Mosquitto\Client->loop()
#8 {main}

[attached is the log file emoncms(1).txt (2.4 KB)]

I logged in via SSH

sudo systemctl restart mqtt_input
did not make any difference

but
sudo service emonhub restart
kick it back in to life. I did try restarting from the web interface

Paul

[Title was ambiguous - expanded to clarify. Moderator (RW)]

pb66 · 12 December 2017 20:43

Hi Paul, Can you tells us about the inputs that stopped. Would I be right in thinking they were all RFM networked? (eg emonTH emonTx etc)

Can you supply the emonhub.log if it reoccurs?

Thanks for supplying the emoncms.log, whilst I do not think any of those errors are the cause of this issue, I’m sure the info will be useful.

Pukka · 13 December 2017 09:03

Hi Paul

Where do i find the emoncms.log (it would be great if they was a button on the web interface that would zip up all of the log files and allow you to download them)

The interfaces that stopped Emonpi, emonth5, BMWi3, 2 x node-red (modbus) 1x node-red (weather).
Interfaces still working were openevse and IotaWatt

pb66 · 13 December 2017 09:30

You have already shown us the emoncms.log, which is accessible/downloadable via the emoncms=>admin page, the emonhub.log is accessible and downloadable via the emoncms=>emonhub page (on an emonSD).

I assume you are running the latest emonSD on and emonPi? You say emoncms is the latest, have you updated via the “update emonpi” button in emoncms=>admin?

At this point I have no idea how restarting emonHub could impact the “2 x node-red (modbus) 1x node-red (weather).” inputs directly, I assume they are posting direct to emoncms and not through emonHub, are they posting via MQTT or HTTP?

Pukka · 13 December 2017 09:38

I will keep my eye on it and grab the log files next time they is a problem

The node-red is posting via MQTT the same as the inbuilt weather flow

The update was installed via the “update emonpi” on admin, also click the database update&check

Pukka · 24 December 2017 17:55

It has crashed again.

Screenshot and log files attached.

emoncms(2).txt (7.5 KB)
emonhub.txt (773.8 KB)

sudo systemctl status mqtt_input -n50 -l
● mqtt_input.service - Emoncms MQTT Input Script
Loaded: loaded (/etc/systemd/system/mqtt_input.service; enabled)
Active: active (running) since Tue 2017-12-19 21:31:16 GMT; 4 days ago
Docs: https://github.com/emoncms/emoncms/blob/master/docs/RaspberryPi/MQTT.md
Main PID: 3275 (php)
CGroup: /system.slice/mqtt_input.service
└─3275 /usr/bin/php /var/www/emoncms/scripts/phpmqtt_input.php

Dec 24 01:44:26 emonpi mqtt_input[3275]: PHP Fatal error: Call to a member function fetch_object() on boolean in /var/www/emoncms/Modules/input/input_model.php on line 605
Dec 24 01:44:26 emonpi mqtt_input[3275]: Fatal error: Call to a member function fetch_object() on boolean in /var/www/emoncms/Modules/input/input_model.php on line 605
Warning: Journal has been rotated since unit was started. Log output is incomplete or unavailable.

I will leave it in a broken state for a few hours if you would like a remote session to have a look, just send me a PM

Paul

glyn.hudson · 24 December 2017 23:41

Try and update to 9.8.27:

This release includes a significant fix to mqtt input script. The issue was the MYSQL connection was timing out after the mqtt input script had been working for a while. This has now been fixed.

Pukka · 25 December 2017 22:20

I have upgraded and will report back in a few days.

Pukka · 29 December 2017 16:51

just been to look and it looks like the MQTT service stopped 2 days ago

Pukka · 20 January 2018 19:46

I am still having a problem with the a service stopping or crashing, i have upgraded to the latest version low-write 9.8.27

If i look under administration it reports “Daemon is not running, start it at ~/scripts/feedwriter”

emoncms log looks like

2018-01-20 19:17:01.688|ERROR|phpmqtt_input.php|exception ‘ErrorException’ with message ‘Invalid argument supplied for foreach()’ in /var/www/emoncms/Modules/input/input_model.php:197
Stack trace:
#0 /var/www/emoncms/Modules/input/input_model.php(197): exceptions_error_handler(2, ‘Invalid argumen…’, ‘/var/www/emoncm…’, 197, Array)
#1 /var/www/emoncms/Modules/input/input_model.php(182): Input->redis_get_inputs(1)
#2 /var/www/emoncms/scripts/phpmqtt_input.php(220): Input->get_inputs(1)
#3 [internal function]: message(Object(Mosquitto\Message))
#4 /var/www/emoncms/scripts/phpmqtt_input.php(116): Mosquitto\Client->loop()
#5 {main}
2018-01-20 19:17:01.690|ERROR|phpmqtt_input.php|exception ‘ErrorException’ with message ‘Invalid argument supplied for foreach()’ in /var/www/emoncms/Modules/input/input_model.php:200
Stack trace:
#0 /var/www/emoncms/Modules/input/input_model.php(200): exceptions_error_handler(2, ‘Invalid argumen…’, ‘/var/www/emoncm…’, 200, Array)
#1 /var/www/emoncms/Modules/input/input_model.php(182): Input->redis_get_inputs(1)
#2 /var/www/emoncms/scripts/phpmqtt_input.php(220): Input->get_inputs(1)
#3 [internal function]: message(Object(Mosquitto\Message))
#4 /var/www/emoncms/scripts/phpmqtt_input.php(116): Mosquitto\Client->loop()
#5 {main}
2018-01-20 19:17:01.702|ERROR|phpmqtt_input.php|exception ‘ErrorException’ with message ‘Invalid argument supplied for foreach()’ in /var/www/emoncms/Modules/input/input_model.php:197
Stack trace:
#0 /var/www/emoncms/Modules/input/input_model.php(197): exceptions_error_handler(2, ‘Invalid argumen…’, ‘/var/www/emoncm…’, 197, Array)
#1 /var/www/emoncms/Modules/input/input_model.php(182): Input->redis_get_inputs(1)
#2 /var/www/emoncms/scripts/phpmqtt_input.php(220): Input->get_inputs(1)
#3 [internal function]: message(Object(Mosquitto\Message))
#4 /var/www/emoncms/scripts/phpmqtt_input.php(116): Mosquitto\Client->loop()
#5 {main}

Anything to try?

glyn.hudson · 27 January 2018 12:47

It looks like the feedwriter scrip has stopped this can be restarted with

sudo service feedwriter restart

This question is why is stopped in the first place. This error is interesting, I’ve not seen this before:

What version emonSD are you running?

What are you running on the emonPi? What inputs are you posting? Anything non standard?

If you update to the latest version V9.8.28 (via emonPi update) you will be able to copy all the info admin info page. Please paste the full output here:

See the copy to clipboard button

Pukka · 27 January 2018 15:10

Hi Glyn,
I am running 2.8.27
I did try running “sudo service feedwriter restart” and the buffer counted down but the data was still missing.

Input wise i have 2 node-red running modbus every 5 seconds, BMW input, OpenEVSE, EmonPi, Iotawatt, weather.

I have clicked the emonpi update button but it has not download the latest version

Paul

glyn.hudson · 27 January 2018 21:02

That’s strange, emonPi update should update to 9.8.28. Please post your emonpiupdate log

Pukka · 27 January 2018 21:53

Log File Attached emonpiupdate.txt (6.9 KB)

glyn.hudson · 28 January 2018 21:41

Thanks. From the logfile is seems that default.emonpi.settings.php and settings.php have been user modified. The change to default.emonpi.settings.php is causing emonPi update to not be able to pull in any new changes.

If’s possible to keep your custom settings in settings.php and just restore default.emonpi.settings.php to it’s default state. This file is is only used to store the default settings for reference, its not actually used by the application. You can reset default.emonpi.settings.php back to it’s default state by running

rpi-rw
cd  /var/www/emoncms
git checkout default.emonpi.settings.php
git pull

The final git pull command should now pull in the latest changes and update you to 9.8.28. This won’t necessarily fix the issue you have reported but should enable us to debug on a common platform. Once updated to 9.8.28 please use the new copy to clipboard function to copy your Emoncms system info to clipboard and paste into the forum. It should look something like this:

Pukka · 29 January 2018 08:54

Hi Glyn,
Thats fixed it

Server Information


Emoncms	Version	low-write 9.8.28 : 2018.01.27
	Modules	Administration : App v1.1.0 : Backup v1.1.2 : EmonHub Config v1.0.0 : Dashboard v1.1.1 : EventProcesses : Feed : Graph v1.2.0 : Input : postprocess : CoreProcess : Schedule : setup : Time : User : Visualisation : WiFi v1.0.0
	Buffer	loading…
	Writer	Daemon is running with sleep 60s
Server	OS	Linux 4.9.35-v7+
	Host	emonpi emonpi (127.0.1.1)
	Date	2018-01-29 08:54:05 UTC
	Uptime	08:54:05 up 3 days, 6 min, 1 user, load average: 0.57, 0.62, 0.44
HTTP	Server	Apache/2.4.10 (Raspbian) HTTP/1.1 CGI/1.1 80
MySQL	Version	5.5.57-0+deb8u1
	Host	localhost (127.0.0.1)
	Date	2018-01-29 08:54:04 (UTC 00:00‌)
	Stats	Uptime: 259601 Threads: 2 Questions: 320456 Slow queries: 0 Opens: 68 Flush tables: 1 Open tables: 61 Queries per second avg: 1.234
Redis	Version	2.8.17
	Host	localhost:6379 (127.0.0.1)
	Size	518 keys (616.68K)
	Uptime	3 days
MQTT	Version	1.4.14
	Host	localhost:1883 (127.0.0.1)
Pi	CPU Temp	53.69°C
	Release	emonSD-07Nov16
	File-system	Set root file-system temporarily to read-write, (default read-only)
Memory	RAM	Used: 87.39% Total: 970.93 MB Used: 848.46 MB Free: 122.46 MB
Disk	Mount	Stats
	/	Used: 65.37% Total: 3.33 GB Used: 2.18 GB Free: 1015.86 MB
	/boot	Used: 36.32% Total: 59.95 MB Used: 21.77 MB Free: 38.17 MB
	/home/pi/data	Used: 5.75% Total: 24.98 GB Used: 1.44 GB Free: 22.26 GB
PHP	Version	5.6.30-0+deb8u1 (Zend Version 2.6.0)
	Modules	apache2handler : bcmath : bz2 : calendar : Core v5.6.30-0+deb8u1 : ctype : curl : date v5.6.30-0+deb8u1 : dba : dio v0.0.4RC4 : dom v20031129 : ereg : exif v1.4 : fileinfo v1.0.5 : filter v0.11.0 : ftp : gettext : hash v1.0 : iconv : json v1.3.6 : libxml : mbstring : mcrypt : mhash : mosquitto v0.3.0 : mysql v1.0 : mysqli v0.1 : openssl : pcre : PDO v1.0.4dev : pdo_mysql v1.0.2 : Phar v2.0.2 : posix : readline v5.6.30-0+deb8u1 : redis v2.2.7 : Reflection : session : shmop : SimpleXML v0.1 : soap : sockets : SPL v0.2 : standard v5.6.30-0+deb8u1 : sysvmsg : sysvsem : sysvshm : tokenizer v0.1 : wddx : xml : xmlreader v0.1 : xmlwriter v0.1 : Zend OPcache v7.0.6-devFE : zip v1.12.5 : zlib v2.0 :

Client Information












HTTP
Browser
Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:56.0) Gecko/20100101 Firefox/56.0


Screen
Resolution
1920 x 1080


Window
Size
1903 x 948

Bernie_Carter · 6 February 2018 00:48

It looks like I’m having the same issue. Trystan recommended I update to 9.8.28 as Glyn suggested. So, let’s run this and see what happens.

Paul, Are you having a failure between 2 and 6 days the same as I am?

Also, Can you send me a few excerpts of your raw input?

Thanks
Bernie Carter

Pukka · 7 February 2018 14:33

Hi @Bernie_Carter

Yes that sounds correct. It looks like it’s something to do with the MQTT /nodered service, as anything that is posting to that stops. But other services like openevse and iotawatt till post ok.

I am away at the moment so I am unable to post any log files

Paul

Bernie_Carter · 7 February 2018 15:43

@Pukka

That is possible, I will review my logs again and see, but I didn’t notice anything out of line the first run through. Just a reference to a broken pipe.

I’ve had a ton of issues with nodered right from the start. In fact this is my 3 try at keeping it going. The first two ran into all kinds of programming issues, and had to be wiped to clear all.

It’s also possible that your issue is unrelated to mine, they just have similar outcomes.

Bernie Carter

Pukka · 22 February 2018 15:52

It has just crashed again, does anyone want to login and have a look at it? I will leave it broken over night.

Paul


HTTP	Browser	Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:56.0) Gecko/20100101 Firefox/56.0
Screen	Resolution	1920 x 1080
Window	Size	1903 x 948