Emonpi low-write 9.8.25 - inputs stopped working

I am still having a problem with the a service stopping or crashing, i have upgraded to the latest version low-write 9.8.27

If i look under administration it reports “Daemon is not running, start it at ~/scripts/feedwriter”

emoncms log looks like

2018-01-20 19:17:01.688|ERROR|phpmqtt_input.php|exception ‘ErrorException’ with message ‘Invalid argument supplied for foreach()’ in /var/www/emoncms/Modules/input/input_model.php:197
Stack trace:
#0 /var/www/emoncms/Modules/input/input_model.php(197): exceptions_error_handler(2, ‘Invalid argumen…’, ‘/var/www/emoncm…’, 197, Array)
#1 /var/www/emoncms/Modules/input/input_model.php(182): Input->redis_get_inputs(1)
#2 /var/www/emoncms/scripts/phpmqtt_input.php(220): Input->get_inputs(1)
#3 [internal function]: message(Object(Mosquitto\Message))
#4 /var/www/emoncms/scripts/phpmqtt_input.php(116): Mosquitto\Client->loop()
#5 {main}
2018-01-20 19:17:01.690|ERROR|phpmqtt_input.php|exception ‘ErrorException’ with message ‘Invalid argument supplied for foreach()’ in /var/www/emoncms/Modules/input/input_model.php:200
Stack trace:
#0 /var/www/emoncms/Modules/input/input_model.php(200): exceptions_error_handler(2, ‘Invalid argumen…’, ‘/var/www/emoncm…’, 200, Array)
#1 /var/www/emoncms/Modules/input/input_model.php(182): Input->redis_get_inputs(1)
#2 /var/www/emoncms/scripts/phpmqtt_input.php(220): Input->get_inputs(1)
#3 [internal function]: message(Object(Mosquitto\Message))
#4 /var/www/emoncms/scripts/phpmqtt_input.php(116): Mosquitto\Client->loop()
#5 {main}
2018-01-20 19:17:01.702|ERROR|phpmqtt_input.php|exception ‘ErrorException’ with message ‘Invalid argument supplied for foreach()’ in /var/www/emoncms/Modules/input/input_model.php:197
Stack trace:
#0 /var/www/emoncms/Modules/input/input_model.php(197): exceptions_error_handler(2, ‘Invalid argumen…’, ‘/var/www/emoncm…’, 197, Array)
#1 /var/www/emoncms/Modules/input/input_model.php(182): Input->redis_get_inputs(1)
#2 /var/www/emoncms/scripts/phpmqtt_input.php(220): Input->get_inputs(1)
#3 [internal function]: message(Object(Mosquitto\Message))
#4 /var/www/emoncms/scripts/phpmqtt_input.php(116): Mosquitto\Client->loop()
#5 {main}

Anything to try?

It looks like the feedwriter scrip has stopped this can be restarted with

sudo service feedwriter restart

This question is why is stopped in the first place. This error is interesting, I’ve not seen this before:

What version emonSD are you running?

What are you running on the emonPi? What inputs are you posting? Anything non standard?

If you update to the latest version V9.8.28 (via emonPi update) you will be able to copy all the info admin info page. Please paste the full output here:

See the copy to clipboard button

Hi Glyn,
I am running 2.8.27
I did try running “sudo service feedwriter restart” and the buffer counted down but the data was still missing.

Input wise i have 2 node-red running modbus every 5 seconds, BMW input, OpenEVSE, EmonPi, Iotawatt, weather.

I have clicked the emonpi update button but it has not download the latest version

Paul

That’s strange, emonPi update should update to 9.8.28. Please post your emonpiupdate log

Log File Attached emonpiupdate.txt (6.9 KB)

Thanks. From the logfile is seems that default.emonpi.settings.php and settings.php have been user modified. The change to default.emonpi.settings.php is causing emonPi update to not be able to pull in any new changes.

If’s possible to keep your custom settings in settings.php and just restore default.emonpi.settings.php to it’s default state. This file is is only used to store the default settings for reference, its not actually used by the application. You can reset default.emonpi.settings.php back to it’s default state by running

rpi-rw
cd  /var/www/emoncms
git checkout default.emonpi.settings.php
git pull

The final git pull command should now pull in the latest changes and update you to 9.8.28. This won’t necessarily fix the issue you have reported but should enable us to debug on a common platform. Once updated to 9.8.28 please use the new copy to clipboard function to copy your Emoncms system info to clipboard and paste into the forum. It should look something like this:

Hi Glyn,
Thats fixed it :smiley:

Server Information
Emoncms Version low-write 9.8.28 : 2018.01.27
Modules Administration : App v1.1.0 : Backup v1.1.2 : EmonHub Config v1.0.0 : Dashboard v1.1.1 : EventProcesses : Feed : Graph v1.2.0 : Input : postprocess : CoreProcess : Schedule : setup : Time : User : Visualisation : WiFi v1.0.0
Buffer loading…
Writer Daemon is running with sleep 60s
Server OS Linux 4.9.35-v7+
Host emonpi emonpi (127.0.1.1)
Date 2018-01-29 08:54:05 UTC
Uptime 08:54:05 up 3 days, 6 min, 1 user, load average: 0.57, 0.62, 0.44
HTTP Server Apache/2.4.10 (Raspbian) HTTP/1.1 CGI/1.1 80
MySQL Version 5.5.57-0+deb8u1
Host localhost (127.0.0.1)
Date 2018-01-29 08:54:04 (UTC 00:00‌​)
Stats Uptime: 259601 Threads: 2 Questions: 320456 Slow queries: 0 Opens: 68 Flush tables: 1 Open tables: 61 Queries per second avg: 1.234
Redis Version 2.8.17
Host localhost:6379 (127.0.0.1)
Size 518 keys (616.68K)
Uptime 3 days
MQTT Version 1.4.14
Host localhost:1883 (127.0.0.1)
Pi CPU Temp 53.69°C
Release emonSD-07Nov16
File-system Set root file-system temporarily to read-write, (default read-only)
Memory RAM Used: 87.39% Total: 970.93 MB Used: 848.46 MB Free: 122.46 MB
Disk Mount Stats
/ Used: 65.37% Total: 3.33 GB Used: 2.18 GB Free: 1015.86 MB
/boot Used: 36.32% Total: 59.95 MB Used: 21.77 MB Free: 38.17 MB
/home/pi/data Used: 5.75% Total: 24.98 GB Used: 1.44 GB Free: 22.26 GB
PHP Version 5.6.30-0+deb8u1 (Zend Version 2.6.0)
Modules apache2handler : bcmath : bz2 : calendar : Core v5.6.30-0+deb8u1 : ctype : curl : date v5.6.30-0+deb8u1 : dba : dio v0.0.4RC4 : dom v20031129 : ereg : exif v1.4 : fileinfo v1.0.5 : filter v0.11.0 : ftp : gettext : hash v1.0 : iconv : json v1.3.6 : libxml : mbstring : mcrypt : mhash : mosquitto v0.3.0 : mysql v1.0 : mysqli v0.1 : openssl : pcre : PDO v1.0.4dev : pdo_mysql v1.0.2 : Phar v2.0.2 : posix : readline v5.6.30-0+deb8u1 : redis v2.2.7 : Reflection : session : shmop : SimpleXML v0.1 : soap : sockets : SPL v0.2 : standard v5.6.30-0+deb8u1 : sysvmsg : sysvsem : sysvshm : tokenizer v0.1 : wddx : xml : xmlreader v0.1 : xmlwriter v0.1 : Zend OPcache v7.0.6-devFE : zip v1.12.5 : zlib v2.0 :

Client Information
HTTP Browser Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:56.0) Gecko/20100101 Firefox/56.0
Screen Resolution 1920 x 1080
Window Size 1903 x 948

1 Like

It looks like I’m having the same issue. Trystan recommended I update to 9.8.28 as Glyn suggested. So, let’s run this and see what happens.

Paul, Are you having a failure between 2 and 6 days the same as I am?

Also, Can you send me a few excerpts of your raw input?

Thanks
Bernie Carter

Hi @Bernie_Carter

Yes that sounds correct. It looks like it’s something to do with the MQTT /nodered service, as anything that is posting to that stops. But other services like openevse and iotawatt till post ok.

I am away at the moment so I am unable to post any log files

Paul

@Pukka

That is possible, I will review my logs again and see, but I didn’t notice anything out of line the first run through. Just a reference to a broken pipe.

I’ve had a ton of issues with nodered right from the start. In fact this is my 3 try at keeping it going. The first two ran into all kinds of programming issues, and had to be wiped to clear all.

It’s also possible that your issue is unrelated to mine, they just have similar outcomes.

Bernie Carter

It has just crashed again, does anyone want to login and have a look at it? I will leave it broken over night.

Paul

@Bernie_Carter @TrystanLea @glyn.hudson @pb66

Are you any further along or have you found anything else out? mine has crashed again and I am almost at the point of a full reinstall and restore of the database or even just create a job to reboot the pi every 2 days.

When it loses connection it dumps about 2 days worth of data. even if you start the feed writer after it crashed, the number in cache counts down but the data is still lost.

Anyone else have any ideas how to fix or even better do we need a watchdog service.

If anyone wants access to it when its crashed let me know and i will setup an ssh session to the pi

Paul

@Pukka

I ruled out NodeRed, as mine was completely down after a failed upgrade,
and MQTT still crashed.

I will look into the situation further this weekend and see if anything
else pops up.

It sounds odd that you are losing data pre-crash. Mine just doesn’t log
any new data.

If you want to set me up with an SSH, ill certainly take a look at it and
compare, just PM me the details.

Bernie

@Pukka As I suggested above, the best way to monitor this is to install MONIT and use the ‘Timestamp test’ to look for when the feed directories are accessed / written to.

You could get it to run a small script to output the status of the services to a file.

You can also monitor the feedwriter service and try a restart.

Looking a bit deeper at the log you posted, it looks to me like a MQTT mosquitto feed that is causing the problem. Which feeds are MQTT feeds? Have you tried disabling any of these feeds?

@glyn.hudson - Paul says he is using the ‘low-write’ version. The phpmqtt_input.php file in the low-write branch does not have 220 lines! I’m not really sure how this all hangs together but could this be an issue?

These things can be a real pain to work out…

@borpin

I’m running the exact same version of low-write and my phpmqtt_input has 297 lines.
Line 220 is exactly as shown in the error log
$dbinputs = $input->get_inputs($userid);

Is there a different version?

@borpin - The low-write btanch is a bit of a relic, last updated in Apr '16, “low-write” in now a “mode” rather than a separate branch or version.

The line cited in the log is still current in the “stable” branch,

For that line to trip up I would expect there is no userid, no user with that userid or no inputs for that user. Since the userid is hard coded to "1! and almost all emoncms instances have a user “1” setup, I doubt it could be either of those so perhaps it couldn’t find any inputs for that user, redis flushed or mysql offline or something along those lines?

It would seem that log file is 6weeks old, can you be sure the problem is the same ? You are now running emoncms v9.8.28 and have updated the emonPi since then, so some fresh logs would be useful.

The best course of action is always to fix the cause rather than hide it behind a watchdog and restart.

By all means set up a watchdog to tell you when it misbehaves, and grab the logfiles and info when it does fail, but if you start setting up auto restarts etc you may never find the issue. You might not of even had this issue if prior issues were properly diagnosed and resolved rather than resorting to workarounds.

Can you provide new logs for the time leading up to and beyond a fail (emonhub and emoncms)?

Also check the status of the mqtt_input, emonhub and feedwriter services and post the info?

1 Like

Hi All

Thanks for your help so far

I have updated the PI with the latest os updates and node-red. if/when it crashes in the next few days i will post the log files here. i will also try and download the log files everyday.

Paul

Exactly, that’s the direction I’m looking as well.

Mine shows up as a broken pipe, Pauls shows up with a connection error. Whether these 2 incidents are related remains to be seen, but it appears to me then there’s an issue with the MQTT Broker while running in this configuration.

I’m now running an update on mine as well so that I’m inline with pauls setup, and I will again see what happens.

I cannot really comment on the issue you are seeing as you have not posted any info or logs

I’ve not seen any signs of the Mosquitto broker failing, it does seem the issues are all common to MQTT, but that’s where the most traffic is on an emonPi/SD, plus all the “tricky” stuff, eg bespoke node-red flows and the more exotic emonhub interfacers.

Just looking at the log again…

exception is raised on 116:

Examining this bit of code (and excuse my lack of PHP knowledge) I’d have expected an exception on 116 to be caught by the try catch. If this is in a namesapce (don’t really understand that) the ‘Exception’ should have a leading \ (apparently).

The systemd service is set to restart when failed. Next time it crashes can you check the status of the mqtt_input.service and the mosquitto.service?