Collecting Local EmonCMS debug information

Recently, I came across multiple posts in which users are willing to troubleshoot. When troubleshooting the first time myself, I also asked the question which commands and log files are interesting to look at.

Initially conceived as a very rudimentary quick and dirty fix script, it evolved into something more refined. Or so I hope :slight_smile:

I tried to keep the skill level required to use the script as low as possible. That being said, you should at least be able to:

  1. be able to log in on your emonpi using ssh (or putty)
  2. be able to download the script and put it on your emonpi (using scp, winscp, …)
  3. be able to retrieve a file from your emonpi (using scp, winscp, …) and upload it on this forum.
#!/bin/bash

# get read-only API key from http://EM.ON.CMS.IP/emoncms/feed/api
# put read-only API key below 
APIKEY=abc123_PUT_YOUR_API_KEY_HERE_abc123

# get feed id from http://EM.ON.CMS.IP/emoncms/feed/list
# put feed id below
feed_temp=15
feed_pwr1=1
feed_pwr2=17

WATCHDOG_LIMIT=300 # 5 minutes

###########################################
### NO CHANGES REQUIRED BELOW THIS LINE ###
###########################################
LOGDIR=/home/pi/watchlog
TRIGGER=/tmp/watchdog-triggered

###
### save logfiles and relevant debug information
###
function watchdog_log {
	now=$(date '+%s')
	timestamp=$(date +%Y%m%d-%H%M%S%z [email protected]$now)

	# create trigger file to indicate watchdog has been triggered
	touch $TRIGGER

	rpi-rw >/dev/null

	# make sure the LOGDIR exists
	mkdir -p $LOGDIR/$timestamp/
	
	# write collected data
	file=$LOGDIR/$timestamp/data.txt
	date +%s > $file
	echo $output_temp >> $file
	echo $output_pwr1 >> $file
	echo $output_pwr2 >> $file

	# make a copy of log files
	cp -prv /var/log/syslog $LOGDIR/$timestamp/
	cp -prv /var/log/emoncms.log $LOGDIR/$timestamp/

	# save output of multiple commands
	ps -AHf > $LOGDIR/$timestamp/cmd-ps.txt
	service --status-all 2>&1 > $LOGDIR/$timestamp/cmd-service.txt
	top -bn1 > $LOGDIR/$timestamp/cmd-top.txt
	
	# create a tar archive for easy upload to https://community.openenergymonitor.org forum
	TARFILE=$LOGDIR/watchdog-$timestamp.tar.gz
	tar -czvf $TARFILE $LOGDIR/$timestamp
	
	ls -l $LOGDIR/$timestamp/
	echo
	echo "You can now upload $TARFILE to the forum"
	echo "Alternatively, upload specific files from $LOGDIR/$timestamp/"

	rpi-ro >/dev/null
}

###
### probe EmonCMS API and, if needed, collect logs
###
function watchdog_probe {
	###
	### COLLECT data from EmonCMS
	###
	output_temp=`wget -qO- "http://127.0.0.1/emoncms/feed/timevalue.json?id=${feed_temp}&apikey=$APIKEY"`
	output_pwr1=`wget -qO- "http://127.0.0.1/emoncms/feed/timevalue.json?id=${feed_pwr1}&apikey=$APIKEY"`
	output_pwr2=`wget -qO- "http://127.0.0.1/emoncms/feed/timevalue.json?id=${feed_pwr2}&apikey=$APIKEY"`


	###
	### EVALUATE the measurements and take action if needed
	###
	now=`date +%s`

	# get time of feed
	time_temp=$(echo $output_temp | sed 's/.*"time":\([^,]*\).*/\1/')
	time_pwr1=$(echo $output_pwr1 | sed 's/.*"time":\([^,]*\).*/\1/')
	time_pwr2=$(echo $output_pwr2 | sed 's/.*"time":\([^,]*\).*/\1/')

	# calculate difference between last measurement and current timestamp
	diff_temp=$(($now-$time_temp))
	diff_pwr1=$(($now-$time_pwr1))
	diff_pwr2=$(($now-$time_pwr2))

	# $WATCHDOG_LIMIT stayalive timer exceeded?
	if [ $diff_temp -gt $WATCHDOG_LIMIT ] || [ $diff_pwr1 -gt $WATCHDOG_LIMIT ] || [ $diff_pwr2 -gt $WATCHDOG_LIMIT ]; then
		# if the trigger file already exists, don't trigger the watchdog anymore
		if [ -f $TRIGGER ]; then
			echo "Watchdog detected anomaly but it was triggered before. Not collecting logs again." | logger -it watchdog
			exit 0
		fi
		echo "Watchdog detected anomaly and was not triggered before. Collecting logs." | logger -it watchdog
		watchdog_log
		#sudo reboot
		exit 0
	fi
}

###
### check if CRON calls the watchdog
###
function watchdog_status {
	crontab -l | grep watchdog
}
###
### install in CRON (requires rpi-rw, run as sudo)
###
function watchdog_install {
	crontab -l | grep watchdog
	if [ $? -ne 0 ]; then
		echo '*/1 * * * * /home/pi/watchdog.sh' | tee -a /var/spool/cron/crontabs/pi
		crontab -l | grep watchdog
	fi
}
###
### uninstall from CRON (requires rpi-rw, run as sudo)
###
function watchdog_uninstall {
	awk '! /.*watchdog.sh$/' /var/spool/cron/crontabs/pi > /var/spool/cron/crontabs/pi
}

#grep ^PPid /proc/$$/status | sed 's/.*:[ \t]*\([0-9]*\)/\1/'
#PPID=`ps -p $$ -o ppid=`
PCOM=`ps -p $PPID -o comm=`
if [ "$PCOM" == "bash" ]; then
	echo 'called from bash. Just collecting logs.'
	watchdog_log
else
	#echo 'called from ['$PPID'] '$PCOM > /tmp/watchdog
	#ps -AHf > /tmp/watchps
	watchdog_probe
fi

Download the script watchdog.txt (3.8 KB)
Login via SSH on your EmonPI and make the file system writeable

[email protected](ro):~$ rpi-rw 
Filesystem is unlocked - Write access
type ' rpi-ro ' to lock
[email protected](rw):~$ 

Open it in an editor of your choice (notepad, …) and adjust the APIKEY, feed_temp, feed_pwr1 and feed_pwr2 to your installation.

Copy it to the home directory of your EmonPI /home/pi/

Rename it

mv watchdog.txt watchdog.sh

Make it executable

chmod a+x watchdog.sh

Set the filesystem back to read-only.

rpi-ro

Execute the script

[email protected](ro):~$ ~/watchdog.sh 
called from bash. Just collecting logs.
‘/var/log/syslog’ -> ‘/home/pi/watchlog/20180425-164546+0000/syslog’
‘/var/log/emoncms.log’ -> ‘/home/pi/watchlog/20180425-164546+0000/emoncms.log’
[ ? ]  ntp-backup
tar: Removing leading `/' from member names
/home/pi/watchlog/20180425-164546+0000/
/home/pi/watchlog/20180425-164546+0000/cmd-ps.txt
/home/pi/watchlog/20180425-164546+0000/cmd-service.txt
/home/pi/watchlog/20180425-164546+0000/emoncms.log
/home/pi/watchlog/20180425-164546+0000/syslog
/home/pi/watchlog/20180425-164546+0000/cmd-top.txt
/home/pi/watchlog/20180425-164546+0000/data.txt
total 480
-rw-r--r-- 1 pi pi   12213 Apr 25 16:45 cmd-ps.txt
-rw-r--r-- 1 pi pi    1135 Apr 25 16:45 cmd-service.txt
-rw-r--r-- 1 pi pi   12206 Apr 25 16:45 cmd-top.txt
-rw-r--r-- 1 pi pi      14 Apr 25 16:45 data.txt
-rw-rw-rw- 1 pi pi    1057 Apr 15 05:19 emoncms.log
-rw-r----- 1 pi adm 451251 Apr 25 16:45 syslog

You can now upload /home/pi/watchlog/watchdog-20180425-164546+0000.tar.gz to the forum
Alternatively, upload specific files from /home/pi/watchlog/20180425-164546+0000/

[email protected](ro):~$
2 Likes

One “gotcha” to be aware of when using Notepad (Windows version).

Notepad terminates each line with a Carriage Return and a Line Feed.
Linux wants to see only a Line Feed. The CR / LF combo can cause weird things to happen when
Linux / Unix tries to read a file with CR/ LF terminated text lines.

One way to avoid that is to use an editor that lets you choose the line terminator character(s).
For Windows, Notepad++ is one of many available with that feature.