emonPi update oddness: no update

Jon · 18 June 2018 22:09

Hello! I just tried to run an emonPi update and things didn’t work as hoped. After click the emonPi Update (on Setup > Administration webpage) the log from 4 days ago went flying by in the Update log window. After waiting a few minutes I did a reboot.

The emonpi update seems to have happen before the emonPi reboot. And since there are LOTS of errors in the emonpiupdate.log (enclosed below) I tried to do another emonpi update. But the same issue (the log from an hour ago went flying by in the Update log window).

I’m hoping if I can get the emonPi update to run then the emonpiupdate.log errors might disappear…

48%20PM%20copy%202

2018-06-18 at 16.14.42 - emonpiupdate.log.txt (45.3 KB)

EDIT: added server info

Server Information












Emoncms
Version
low-write 9.8.30 : 2018.05.08



Modules
Administration : App v1.1.0 : Backup v1.1.2 : EmonHub Config v1.0.0 : Dashboard v1.3.0 : EventProcesses : Feed : Graph v1.2.0 : Input : postprocess : CoreProcess : Schedule : setup : sync : Time : User : Visualisation : WiFi v1.1.1



Buffer
loading…



Writer
Daemon is running with sleep 60s


Server
OS
Linux 4.9.35-v7+



Host
emonpi emonpi (127.0.1.1)



Date
2018-06-18 17:12:47 CDT



Uptime
17:12:47 up 59 min,  0 users,  load average: 0.20, 0.30, 0.43


HTTP
Server
Apache/2.4.10 (Raspbian) HTTP/1.1 CGI/1.1 80


MySQL
Version
5.5.60-0+deb8u1



Host
localhost (127.0.0.1)



Date
2018-06-18 17:12:47 (UTC -05:00‌)



Stats
Uptime: 3535  Threads: 3  Questions: 10306  Slow queries: 0  Opens: 59  Flush tables: 1  Open tables: 52  Queries per second avg: 2.915


Redis
Version
2.8.17



Host
localhost:6379 (127.0.0.1)



Size
635 keys  (618.53K)



Uptime
0 days


MQTT
Version




Host
localhost:1883 (127.0.0.1)


Pi
CPU Temp
36.86°C



Release
emonSD-07Nov16



File-system
Set root file-system temporarily to read-write, (default read-only)


Memory
RAM
Used: 30.38% Total: 970.93 MB Used: 294.93 MB Free: 676.01 MB


Disk
Mount
Stats



/
Used: 77.08% Total: 3.33 GB Used: 2.57 GB Free: 616.38 MB



/boot
Used: 36.35% Total: 59.95 MB Used: 21.79 MB Free: 38.16 MB



/home/pi/data
Used: 34.23% Total: 3.75 GB Used: 1.29 GB Free: 2.28 GB



/mnt/usb
Used: 29.20% Total: 57.81 GB Used: 16.88 GB Free: 37.99 GB


PHP
Version
5.6.33-0+deb8u1 (Zend Version 2.6.0)



Modules
apache2handler : bcmath : bz2 : calendar : Core v5.6.33-0+deb8u1 : ctype : curl : date v5.6.33-0+deb8u1 : dba : dio v0.0.4RC4 : dom v20031129 : ereg : exif v1.4 : fileinfo v1.0.5 : filter v0.11.0 : ftp : gettext : hash v1.0 : iconv : json v1.3.6 : libxml : mbstring : mcrypt : mhash : mosquitto v0.3.0 : mysql v1.0 : mysqli v0.1 : openssl : pcre : PDO v1.0.4dev : pdo_mysql v1.0.2 : Phar v2.0.2 : posix : readline v5.6.33-0+deb8u1 : redis v2.2.7 : Reflection : session : shmop : SimpleXML v0.1 : soap : sockets : SPL v0.2 : standard v5.6.33-0+deb8u1 : sysvmsg : sysvsem : sysvshm : tokenizer v0.1 : wddx : xml : xmlreader v0.1 : xmlwriter v0.1 : Zend OPcache v7.0.6-devFE : zip v1.12.5 : zlib v2.0 :








Client Information











HTTP
Browser
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/11.1.1 Safari/605.1.15


Screen
Resolution
2560 x 1440


Window
Size
1296 x 1240








EDIT 2: I ran this to force an update.



Errors not reported in emonpiupdate.log file

rpi-rw && rm /home/pi/data/emonpiupdate.log && /home/pi/emonpi/firstbootupdate



I still have lots of errors related to emonpi2c
rm: cannot remove ‘hardware/emonpi/emonpi2c/emonpi2c_V1.2/emonPi2C_V1.2_BOM.ods’: Permission
denied

And I cannot get the emopi Update to run via the webpage.  ugh!

Jon · 18 June 2018 23:52

to correct these errors:

rm: cannot remove ‘hardware/emonpi/emonpi2c/emonpi2c_V1.2/emonPi2C_V1.2_BOM.ods’: Permission denied
rm: cannot remove ‘hardware/emonpi/emonpi2c/emonpi2c_V1.2/emonPi2C_V1.2_GERBERS/emonPi2C_V1.2.GTO’: Permission denied
rm: cannot remove ‘hardware/emonpi/emonpi2c/emonpi2c_V1.2/emonPi2C_V1.2_GERBERS/emonPi2C_V1.2.TXT’: Permission denied
rm: cannot remove ‘hardware/emonpi/emonpi2c/emonpi2c_V1.2/emonPi2C_V1.2_GERBERS/emonPi2C_V1.2.GBS’: Permission denied
rm: cannot remove ‘hardware/emonpi/emonpi2c/emonpi2c_V1.2/emonPi2C_V1.2_GERBERS/emonPi2C_V1.2.GTL’: Permission denied
rm: cannot remove ‘hardware/emonpi/emonpi2c/emonpi2c_V1.2/emonPi2C_V1.2_GERBERS/emonPi2C_V1.2.gpi’: Permission denied
rm: cannot remove ‘hardware/emonpi/emonpi2c/emonpi2c_V1.2/emonPi2C_V1.2_GERBERS/emonPi2C_V1.2.GBO’: Permission denied
rm: cannot remove ‘hardware/emonpi/emonpi2c/emonpi2c_V1.2/emonPi2C_V1.2_GERBERS/emonPi2C_V1.2.dri’: Permission denied
rm: cannot remove ‘hardware/emonpi/emonpi2c/emonpi2c_V1.2/emonPi2C_V1.2_GERBERS/emonPi2C_V1.2.GTS’: Permission denied
rm: cannot remove ‘hardware/emonpi/emonpi2c/emonpi2c_V1.2/emonPi2C_V1.2_GERBERS/emonPi2C_V1.2.GBL’: Permission denied
rm: cannot remove ‘hardware/emonpi/emonpi2c/emonpi2c_V1.2/emonPi2C_V1.2_GERBERS/emonPi2C_V1.2.GTP’: Permission denied
rm: cannot remove ‘hardware/emonpi/emonpi2c/emonpi2c_V1.2/emonPi2C_V1.2.brd’: Permission denied
rm: cannot remove ‘hardware/emonpi/emonpi2c/emonpi2c_V1.2/emonPi2C_V1.2.sch’: Permission denied
rm: cannot remove ‘hardware/emonpi/emonpi2c/emonpi2c_V1.1/emonPi2C_V1.1_Gerbers.zip’: Permission denied
rm: cannot remove ‘hardware/emonpi/emonpi2c/emonpi2c_V1.1/emonPi2C_V1.1.sch’: Permission denied
rm: cannot remove ‘hardware/emonpi/emonpi2c/emonpi2c_V1.1/Eagle_7.3_gerber_export.cam.txt’: Permission denied
rm: cannot remove ‘hardware/emonpi/emonpi2c/emonpi2c_V1.1/emonPi2C_V1.1.brd’: Permission denied
rm: cannot remove ‘hardware/emonpi/emonpi2c/emonpi2c_dev/emonpi2c.sch’: Permission denied
rm: cannot remove ‘hardware/emonpi/emonpi2c/emonpi2c_dev/emonpi2c.brd’: Permission denied

this line needs to have root permissions:

github.com

openenergymonitor/emonpi/blob/master/service-runner-update.sh#L50

    
      
                  if [ "$image_version" == "$image_name" ]; then
                      echo "emonSD base image check passed...continue update"
                      valid=1
                  fi
              done <<< "$save_to_update"
          
          
    if [ $valid == 0 ]; then
                  echo "ERROR: emonSD base image old or undefined...update will not continue"
                  echo "See latest verson: https://github.com/openenergymonitor/emonpi/wiki/emonSD-pre-built-SD-card-Download-&-Change-Log"
                  echo "Stopping update"
                  exit 0
              fi
          fi
          
          
# -----------------------------------------------------------------
          
          
# Pull in latest emonpi repo before then running updated update scripts
          echo "git pull $usrdir/emonpi"
          cd $usrdir/emonpi
          sudo rm -rf hardware/emonpi/emonpi2c/
          git branch

pb66 · 19 June 2018 10:20

Hi @Jon, looking at your log (among other things) it looks like your file system was in read-only mode for the larger latter portion and therefore many commands failed and those that didn’t will not have stuck.

The update script doesn’t call rpi-rw very often so once it’s gone RO it’s unlikely to recover, and even if it did, then there could be missing bit go un-noticed or partial updates that cause issues. When I wrote an install/update script fora RO emonSD image just before the launch of the emonPi, I wrapped each command in a function that checked the FS status and called rpi-rw (if needed) at every step but that approach is not used in the emonSD so this sort of thing can happen at anytime.

The FS can be mounted RO at anytime due to a user command (rpi-ro) in another console/window or indeed another script of some description running in the background that has the same ability as the update script to lock the FS when it’s done. The way I did it, the status of the FS immediately before the wrapped command was called determined if the FS got locked or not on completion, so there was no chance of one process/script locking down the FS while another was still in progress.

There is also the “auto-lock” function that runs hourly and mounts the FS read-only, if you are really unlucky you could call the update seconds before that cron task runs and locks the FS again before the update completes, in this instance though I think the node-red update was just so long that it eventually timed out, that has been reported before here on the forum.

One way to “dodge” the node-red update (for test or to split the updates into smaller chunks) might be to use “update emonbase” rather than “update emonpi” as I previously noticed only emonpi users get the node-red update, it’s not part of the emonBase update routine. So you could run the update twice (once emonBase and then emonPi) to get the full update.

Or you could look at the /etc/crontab file and determine when your cron-hourly is run and work around that knowing the FS will re lock each hour at that time (eg 17mins past the hour in the following example) or you could temporarily disable the cron job.

# /etc/crontab: system-wide crontab
# Unlike any other crontab you don't have to run the `crontab'
# command to install the new version when you edit this file
# and files in /etc/cron.d. These files also have username fields,
# that none of the other crontabs do.

SHELL=/bin/sh
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin

# m h dom mon dow user  command
17 *    * * *   root    cd / && run-parts --report /etc/cron.hourly
25 6    * * *   root    test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily )
47 6    * * 7   root    test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.weekly )
52 6    1 * *   root    test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.monthly )
#

You could also keep an eye on the /var/log/service-runner.log file (or something close to that name) for errors as not all the update commands output goes to the update log.

[edit]

…and if you have not successfully updated since 7th June then there is the added possibility that some but not all of the changes made to switch the service-runner to using a redis queue may have stuck but not others if it bombed out part way though, potentially leaving you with a broken service-runner/updater. I only say this based on your above comment, I have not looked closely at how it’s done to know if it’s definitely the case.

Jon · 20 July 2018 02:45

sorry for the late reply! my bad! I got the first set of errors fixed with the sudo command.

github.com

openenergymonitor/emonpi/blob/master/service-runner-update.sh#L50


      
                  if [ "$image_version" == "$image_name" ]; then
                      echo "emonSD base image check passed...continue update"
                      valid=1
                  fi
              done <<< "$save_to_update"
          
              if [ $valid == 0 ]; then
                  echo "ERROR: emonSD base image old or undefined...update will not continue"
                  echo "See latest verson: https://github.com/openenergymonitor/emonpi/wiki/emonSD-pre-built-SD-card-Download-&-Change-Log"
                  echo "Stopping update"
                  exit 0
              fi
          fi
          
          # -----------------------------------------------------------------
          
          # Pull in latest emonpi repo before then running updated update scripts
          echo "git pull $usrdir/emonpi"
          cd $usrdir/emonpi
          sudo rm -rf hardware/emonpi/emonpi2c/
          git branch

The second set of errors was as you said. my emonPi went into the read-only mode part way thru the emonPi update. I got two of those types of errors in a row (odd coincidence?).

I looked at the service-runner and all looked like a correct copy of the service runner on GitHub. But I do have a recent problem with redis.

I still haven’t figured out why I cannot get the emopi Update to run via the webpage. Too many odd issues! No worries, I’ll figure it out someday!


Emoncms	Version	low-write 9.8.30 : 2018.05.08
	Modules	Administration : App v1.1.0 : Backup v1.1.2 : EmonHub Config v1.0.0 : Dashboard v1.3.0 : EventProcesses : Feed : Graph v1.2.0 : Input : postprocess : CoreProcess : Schedule : setup : sync : Time : User : Visualisation : WiFi v1.1.1
	Buffer	loading…
	Writer	Daemon is running with sleep 60s
Server	OS	Linux 4.9.35-v7+
	Host	emonpi emonpi (127.0.1.1)
	Date	2018-06-18 17:12:47 CDT
	Uptime	17:12:47 up 59 min, 0 users, load average: 0.20, 0.30, 0.43
HTTP	Server	Apache/2.4.10 (Raspbian) HTTP/1.1 CGI/1.1 80
MySQL	Version	5.5.60-0+deb8u1
	Host	localhost (127.0.0.1)
	Date	2018-06-18 17:12:47 (UTC -05:00‌)
	Stats	Uptime: 3535 Threads: 3 Questions: 10306 Slow queries: 0 Opens: 59 Flush tables: 1 Open tables: 52 Queries per second avg: 2.915
Redis	Version	2.8.17
	Host	localhost:6379 (127.0.0.1)
	Size	635 keys (618.53K)
	Uptime	0 days
MQTT	Version
	Host	localhost:1883 (127.0.0.1)
Pi	CPU Temp	36.86°C
	Release	emonSD-07Nov16
	File-system	Set root file-system temporarily to read-write, (default read-only)
Memory	RAM	Used: 30.38% Total: 970.93 MB Used: 294.93 MB Free: 676.01 MB
Disk	Mount	Stats
	/	Used: 77.08% Total: 3.33 GB Used: 2.57 GB Free: 616.38 MB
	/boot	Used: 36.35% Total: 59.95 MB Used: 21.79 MB Free: 38.16 MB
	/home/pi/data	Used: 34.23% Total: 3.75 GB Used: 1.29 GB Free: 2.28 GB
	/mnt/usb	Used: 29.20% Total: 57.81 GB Used: 16.88 GB Free: 37.99 GB
PHP	Version	5.6.33-0+deb8u1 (Zend Version 2.6.0)
	Modules	apache2handler : bcmath : bz2 : calendar : Core v5.6.33-0+deb8u1 : ctype : curl : date v5.6.33-0+deb8u1 : dba : dio v0.0.4RC4 : dom v20031129 : ereg : exif v1.4 : fileinfo v1.0.5 : filter v0.11.0 : ftp : gettext : hash v1.0 : iconv : json v1.3.6 : libxml : mbstring : mcrypt : mhash : mosquitto v0.3.0 : mysql v1.0 : mysqli v0.1 : openssl : pcre : PDO v1.0.4dev : pdo_mysql v1.0.2 : Phar v2.0.2 : posix : readline v5.6.33-0+deb8u1 : redis v2.2.7 : Reflection : session : shmop : SimpleXML v0.1 : soap : sockets : SPL v0.2 : standard v5.6.33-0+deb8u1 : sysvmsg : sysvsem : sysvshm : tokenizer v0.1 : wddx : xml : xmlreader v0.1 : xmlwriter v0.1 : Zend OPcache v7.0.6-devFE : zip v1.12.5 : zlib v2.0 :


HTTP	Browser	Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/11.1.1 Safari/605.1.15
Screen	Resolution	2560 x 1440
Window	Size	1296 x 1240