It looks like the feedwriter scrip has stopped this can be restarted with
sudo service feedwriter restart
This question is why is stopped in the first place. This error is interesting, I’ve not seen this before:
What version emonSD are you running?
What are you running on the emonPi? What inputs are you posting? Anything non standard?
If you update to the latest version V9.8.28 (via emonPi update) you will be able to copy all the info admin info page. Please paste the full output here:
Thanks. From the logfile is seems that default.emonpi.settings.php and settings.php have been user modified. The change to default.emonpi.settings.php is causing emonPi update to not be able to pull in any new changes.
If’s possible to keep your custom settings in settings.php and just restore default.emonpi.settings.php to it’s default state. This file is is only used to store the default settings for reference, its not actually used by the application. You can reset default.emonpi.settings.php back to it’s default state by running
rpi-rw
cd /var/www/emoncms
git checkout default.emonpi.settings.php
git pull
The final git pull command should now pull in the latest changes and update you to 9.8.28. This won’t necessarily fix the issue you have reported but should enable us to debug on a common platform. Once updated to 9.8.28 please use the new copy to clipboard function to copy your Emoncms system info to clipboard and paste into the forum. It should look something like this:
Yes that sounds correct. It looks like it’s something to do with the MQTT /nodered service, as anything that is posting to that stops. But other services like openevse and iotawatt till post ok.
I am away at the moment so I am unable to post any log files
That is possible, I will review my logs again and see, but I didn’t notice anything out of line the first run through. Just a reference to a broken pipe.
I’ve had a ton of issues with nodered right from the start. In fact this is my 3 try at keeping it going. The first two ran into all kinds of programming issues, and had to be wiped to clear all.
It’s also possible that your issue is unrelated to mine, they just have similar outcomes.
Are you any further along or have you found anything else out? mine has crashed again and I am almost at the point of a full reinstall and restore of the database or even just create a job to reboot the pi every 2 days.
When it loses connection it dumps about 2 days worth of data. even if you start the feed writer after it crashed, the number in cache counts down but the data is still lost.
Anyone else have any ideas how to fix or even better do we need a watchdog service.
If anyone wants access to it when its crashed let me know and i will setup an ssh session to the pi
@Pukka As I suggested above, the best way to monitor this is to install MONIT and use the ‘Timestamp test’ to look for when the feed directories are accessed / written to.
You could get it to run a small script to output the status of the services to a file.
You can also monitor the feedwriter service and try a restart.
Looking a bit deeper at the log you posted, it looks to me like a MQTT mosquitto feed that is causing the problem. Which feeds are MQTT feeds? Have you tried disabling any of these feeds?
@glyn.hudson - Paul says he is using the ‘low-write’ version. The phpmqtt_input.php file in the low-write branch does not have 220 lines! I’m not really sure how this all hangs together but could this be an issue?
I’m running the exact same version of low-write and my phpmqtt_input has 297 lines.
Line 220 is exactly as shown in the error log
$dbinputs = $input->get_inputs($userid);
@borpin - The low-write btanch is a bit of a relic, last updated in Apr '16, “low-write” in now a “mode” rather than a separate branch or version.
The line cited in the log is still current in the “stable” branch,
For that line to trip up I would expect there is no userid, no user with that userid or no inputs for that user. Since the userid is hard coded to "1! and almost all emoncms instances have a user “1” setup, I doubt it could be either of those so perhaps it couldn’t find any inputs for that user, redis flushed or mysql offline or something along those lines?
It would seem that log file is 6weeks old, can you be sure the problem is the same ? You are now running emoncms v9.8.28 and have updated the emonPi since then, so some fresh logs would be useful.
The best course of action is always to fix the cause rather than hide it behind a watchdog and restart.
By all means set up a watchdog to tell you when it misbehaves, and grab the logfiles and info when it does fail, but if you start setting up auto restarts etc you may never find the issue. You might not of even had this issue if prior issues were properly diagnosed and resolved rather than resorting to workarounds.
Can you provide new logs for the time leading up to and beyond a fail (emonhub and emoncms)?
Also check the status of the mqtt_input, emonhub and feedwriter services and post the info?
I have updated the PI with the latest os updates and node-red. if/when it crashes in the next few days i will post the log files here. i will also try and download the log files everyday.
Exactly, that’s the direction I’m looking as well.
Mine shows up as a broken pipe, Pauls shows up with a connection error. Whether these 2 incidents are related remains to be seen, but it appears to me then there’s an issue with the MQTT Broker while running in this configuration.
I’m now running an update on mine as well so that I’m inline with pauls setup, and I will again see what happens.
I cannot really comment on the issue you are seeing as you have not posted any info or logs
I’ve not seen any signs of the Mosquitto broker failing, it does seem the issues are all common to MQTT, but that’s where the most traffic is on an emonPi/SD, plus all the “tricky” stuff, eg bespoke node-red flows and the more exotic emonhub interfacers.
Examining this bit of code (and excuse my lack of PHP knowledge) I’d have expected an exception on 116 to be caught by the try catch. If this is in a namesapce (don’t really understand that) the ‘Exception’ should have a leading \ (apparently).
The systemd service is set to restart when failed. Next time it crashes can you check the status of the mqtt_input.service and the mosquitto.service?