Recently, I came across multiple posts in which users are willing to troubleshoot. When troubleshooting the first time myself, I also asked the question which commands and log files are interesting to look at.
Initially conceived as a very rudimentary quick and dirty fix script, it evolved into something more refined. Or so I hope 
I tried to keep the skill level required to use the script as low as possible. That being said, you should at least be able to:
- be able to log in on your emonpi using ssh (or putty)
- be able to download the script and put it on your emonpi (using scp, winscp, …)
- be able to retrieve a file from your emonpi (using scp, winscp, …) and upload it on this forum.
#!/bin/bash
# get read-only API key from http://EM.ON.CMS.IP/emoncms/feed/api
# put read-only API key below
APIKEY=abc123_PUT_YOUR_API_KEY_HERE_abc123
# get feed id from http://EM.ON.CMS.IP/emoncms/feed/list
# put feed id below
feed_temp=15
feed_pwr1=1
feed_pwr2=17
WATCHDOG_LIMIT=300 # 5 minutes
###########################################
### NO CHANGES REQUIRED BELOW THIS LINE ###
###########################################
LOGDIR=/home/pi/watchlog
TRIGGER=/tmp/watchdog-triggered
###
### save logfiles and relevant debug information
###
function watchdog_log {
now=$(date '+%s')
timestamp=$(date +%Y%m%d-%H%M%S%z -d@$now)
# create trigger file to indicate watchdog has been triggered
touch $TRIGGER
rpi-rw >/dev/null
# make sure the LOGDIR exists
mkdir -p $LOGDIR/$timestamp/
# write collected data
file=$LOGDIR/$timestamp/data.txt
date +%s > $file
echo $output_temp >> $file
echo $output_pwr1 >> $file
echo $output_pwr2 >> $file
# make a copy of log files
cp -prv /var/log/syslog $LOGDIR/$timestamp/
cp -prv /var/log/emoncms.log $LOGDIR/$timestamp/
# save output of multiple commands
ps -AHf > $LOGDIR/$timestamp/cmd-ps.txt
service --status-all 2>&1 > $LOGDIR/$timestamp/cmd-service.txt
top -bn1 > $LOGDIR/$timestamp/cmd-top.txt
# create a tar archive for easy upload to https://community.openenergymonitor.org forum
TARFILE=$LOGDIR/watchdog-$timestamp.tar.gz
tar -czvf $TARFILE $LOGDIR/$timestamp
ls -l $LOGDIR/$timestamp/
echo
echo "You can now upload $TARFILE to the forum"
echo "Alternatively, upload specific files from $LOGDIR/$timestamp/"
rpi-ro >/dev/null
}
###
### probe EmonCMS API and, if needed, collect logs
###
function watchdog_probe {
###
### COLLECT data from EmonCMS
###
output_temp=`wget -qO- "http://127.0.0.1/emoncms/feed/timevalue.json?id=${feed_temp}&apikey=$APIKEY"`
output_pwr1=`wget -qO- "http://127.0.0.1/emoncms/feed/timevalue.json?id=${feed_pwr1}&apikey=$APIKEY"`
output_pwr2=`wget -qO- "http://127.0.0.1/emoncms/feed/timevalue.json?id=${feed_pwr2}&apikey=$APIKEY"`
###
### EVALUATE the measurements and take action if needed
###
now=`date +%s`
# get time of feed
time_temp=$(echo $output_temp | sed 's/.*"time":\([^,]*\).*/\1/')
time_pwr1=$(echo $output_pwr1 | sed 's/.*"time":\([^,]*\).*/\1/')
time_pwr2=$(echo $output_pwr2 | sed 's/.*"time":\([^,]*\).*/\1/')
# calculate difference between last measurement and current timestamp
diff_temp=$(($now-$time_temp))
diff_pwr1=$(($now-$time_pwr1))
diff_pwr2=$(($now-$time_pwr2))
# $WATCHDOG_LIMIT stayalive timer exceeded?
if [ $diff_temp -gt $WATCHDOG_LIMIT ] || [ $diff_pwr1 -gt $WATCHDOG_LIMIT ] || [ $diff_pwr2 -gt $WATCHDOG_LIMIT ]; then
# if the trigger file already exists, don't trigger the watchdog anymore
if [ -f $TRIGGER ]; then
echo "Watchdog detected anomaly but it was triggered before. Not collecting logs again." | logger -it watchdog
exit 0
fi
echo "Watchdog detected anomaly and was not triggered before. Collecting logs." | logger -it watchdog
watchdog_log
#sudo reboot
exit 0
fi
}
###
### check if CRON calls the watchdog
###
function watchdog_status {
crontab -l | grep watchdog
}
###
### install in CRON (requires rpi-rw, run as sudo)
###
function watchdog_install {
crontab -l | grep watchdog
if [ $? -ne 0 ]; then
echo '*/1 * * * * /home/pi/watchdog.sh' | tee -a /var/spool/cron/crontabs/pi
crontab -l | grep watchdog
fi
}
###
### uninstall from CRON (requires rpi-rw, run as sudo)
###
function watchdog_uninstall {
awk '! /.*watchdog.sh$/' /var/spool/cron/crontabs/pi > /var/spool/cron/crontabs/pi
}
#grep ^PPid /proc/$$/status | sed 's/.*:[ \t]*\([0-9]*\)/\1/'
#PPID=`ps -p $$ -o ppid=`
PCOM=`ps -p $PPID -o comm=`
if [ "$PCOM" == "bash" ]; then
echo 'called from bash. Just collecting logs.'
watchdog_log
else
#echo 'called from ['$PPID'] '$PCOM > /tmp/watchdog
#ps -AHf > /tmp/watchps
watchdog_probe
fi
Download the script watchdog.txt (3.8 KB)
Login via SSH on your EmonPI and make the file system writeable
pi@emonpi(ro):~$ rpi-rw
Filesystem is unlocked - Write access
type ' rpi-ro ' to lock
pi@emonpi(rw):~$
Open it in an editor of your choice (notepad, …) and adjust the APIKEY, feed_temp, feed_pwr1 and feed_pwr2 to your installation.
Copy it to the home directory of your EmonPI /home/pi/
Rename it
mv watchdog.txt watchdog.sh
Make it executable
chmod a+x watchdog.sh
Set the filesystem back to read-only.
rpi-ro
Execute the script
pi@emonpi(ro):~$ ~/watchdog.sh
called from bash. Just collecting logs.
‘/var/log/syslog’ -> ‘/home/pi/watchlog/20180425-164546+0000/syslog’
‘/var/log/emoncms.log’ -> ‘/home/pi/watchlog/20180425-164546+0000/emoncms.log’
[ ? ] ntp-backup
tar: Removing leading `/' from member names
/home/pi/watchlog/20180425-164546+0000/
/home/pi/watchlog/20180425-164546+0000/cmd-ps.txt
/home/pi/watchlog/20180425-164546+0000/cmd-service.txt
/home/pi/watchlog/20180425-164546+0000/emoncms.log
/home/pi/watchlog/20180425-164546+0000/syslog
/home/pi/watchlog/20180425-164546+0000/cmd-top.txt
/home/pi/watchlog/20180425-164546+0000/data.txt
total 480
-rw-r--r-- 1 pi pi 12213 Apr 25 16:45 cmd-ps.txt
-rw-r--r-- 1 pi pi 1135 Apr 25 16:45 cmd-service.txt
-rw-r--r-- 1 pi pi 12206 Apr 25 16:45 cmd-top.txt
-rw-r--r-- 1 pi pi 14 Apr 25 16:45 data.txt
-rw-rw-rw- 1 pi pi 1057 Apr 15 05:19 emoncms.log
-rw-r----- 1 pi adm 451251 Apr 25 16:45 syslog
You can now upload /home/pi/watchlog/watchdog-20180425-164546+0000.tar.gz to the forum
Alternatively, upload specific files from /home/pi/watchlog/20180425-164546+0000/
pi@emonpi(ro):~$