Update failing - confused about the "safe-update" list

emonSD
1

Relatively new user of emonPi bought Dec-19 with pre-installed emonSD-17Oct19 image.

I went to my Admin tab yesterday and checked for updates. The Update Log window came back :-

Starting update via service-runner-update.sh (v3.0) >
- emonSD version: emonSD-17Oct19
ERROR: emonSD base image old or undefined...update will not continue
See latest verson: https://github.com/openenergymonitor/emonpi/wiki/emonSD-pre-built-SD-card-Download-&-Change-Log
Stopping update

I was worried about a fried SD card, but all seemed functional overnight. Came back and looked at again this afternoon, same message. Had a look at the service-runner-update.sh script, and see it goes to find the list of safe update versions from github with a curl line, which gets the response

curl -s https://raw.githubusercontent.com/openenergymonitor/e/master/safe-update
emonSD-30Oct18
emonSD-26Oct17
emonSD-07Nov16

So emonSD-17Oct19 is not (at least not yesterday or today) on the happy list of safe updates. I’ve searched for this a little in this community forum, but not found anything recent.

Have I lost the plot on driving search engines, or is there really an issue here with last October’s SD build. If so what is the resolution time frame?

Also do I need to do anything to assist my recovery? I’m minded to get a spare 16GB SD card or two to give myself a chance to recover, and certainly would prefer not lose my hard built dataset.

2

The chances are there was a problem with the update getting the file from GitHUb. sadly the check does not distinguish between not getting the file with not matching a good build.

Just try again.

3

Not sure if it is correct, or which repo is the right one, but using a browser and navigating to
https://raw.githubusercontent.com/openenergymonitor/emonpi/master/safe-update
returns

emonSD-30Oct18
emonSD-26Oct17
emonSD-07Nov16

Looking at the emonpi GitHub repo (https://github.com/openenergymonitor/emonpi), that file was last updated in May last year, so it is very unlikely to include an Oct19 entry!

4

The new location for the safe update list on the latest image is EmonScripts/safe-update at master · openenergymonitor/EmonScripts · GitHub

This is checked from this point EmonScripts/service-runner-update.sh at master · openenergymonitor/EmonScripts · GitHub in the update scripts.

What happens @Simonsse if you try again? was it an intermittent network fault?

5

Thanks for clarifying @TrystanLea.

According to the first post, he tried twice on two consecutive days, and the contents of the safe update list received during those attempts matched the version in the emonpi repo.

Unfortunately, the curl command he posted doesn’t actually indicate which Repo his update script tried to use, the URL posted indicates the repo is openenergymonitor/e, which doesn’t exist.

Is it possible that his image is somehow still pointing to the emonpi repo?

6

@TrystanLea I have seen this before. If there is an issue pulling the file, the check that is done does not distinguish between a blank file returned and the text not matching.

GitHub has had intermittent tissues recently.

Would it be worth putting the URL used to do the check into the log so it’s possible to see if it is right or not?

7

Gentlemen, thank you for your attention to my noob query - much appreciated as I have simply been enjoying a lazy Sunday in the sunshine.

@borpin @TrystanLea I tried again and I get the same error - but only after a new error precedes it :-

Starting update via service-runner-update.sh (v3.0) >
- emonSD version: emonSD-17Oct19
/opt/openenergymonitor/EmonScripts/update/service-runner-update.sh: line 45: cannot create temp file for here-document: No space left on device
ERROR: emonSD base image old or undefined...update will not continue
See latest verson: https://github.com/openenergymonitor/emonpi/wiki/emonSD-pre-built-SD-card-Download-&-Change-Log
Stopping update

I don’t understand which device is complaining about lack of space, my server & client information are :-

Server Information

Server Information

Services

Emoncms

Server

  • OS :- Linux 4.19.75-v7+
  • Host :- emonpi | emonpi | (10.5.1.215)
  • Date :- 2020-04-26 18:53:57 BST
  • Uptime :- 18:53:57 up 1 day, 23:07, 0 users, load average: 0.69, 0.55, 0.28

Memory

  • RAM :- Used: 26.79%
    • Total :- 975.62 MB
    • Used :- 261.4 MB
    • Free :- 714.22 MB
  • Swap :- Used: 0.25%
    • Total :- 100 MB
    • Used :- 256 KB
    • Free :- 99.75 MB
      Write Load Period

Disk

  • / :- Used: 52.16%
    • Total :- 3.92 GB
    • Used :- 2.05 GB
    • Free :- 1.68 GB
    • Write Load :- 6.11 KB/s (1 days 1 hours 42 mins)
  • /var/opt/emoncms :- Used: 0.39%
    • Total :- 9.98 GB
    • Used :- 40.03 MB
    • Free :- 9.43 GB
    • Write Load :- 66.61 B/s (1 days 1 hours 42 mins)
  • /boot :- Used: 20.55%
    • Total :- 252.05 MB
    • Used :- 51.79 MB
    • Free :- 200.26 MB
    • Write Load :- 0 B/s (1 days 1 hours 42 mins)
  • /var/log :- Used: 67.84%
    • Total :- 50 MB
    • Used :- 33.92 MB
    • Free :- 16.08 MB
    • Write Load :- n/a

HTTP

  • Server :- Apache/2.4.38 (Raspbian) HTTP/1.1 CGI/1.1 80

MySQL

  • Version :- 5.5.5-10.3.17-MariaDB-0+deb10u1
  • Host :- localhost:6379 (127.0.0.1)
  • Date :- 2020-04-26 18:53:57 (UTC 01:00‌​)
  • Stats :- Uptime: 174068 Threads: 10 Questions: 12610 Slow queries: 0 Opens: 46 Flush tables: 1 Open tables: 40 Queries per second avg: 0.072

Redis

  • Version :-
    • Redis Server :- 5.0.3
    • PHP Redis :- 5.0.2
  • Host :- localhost:6379
  • Size :- 218 keys (853.93K)
  • Uptime :- 2 days

MQTT Server

  • Version :- Mosquitto 1.5.7
  • Host :- localhost:1883 (127.0.0.1)

PHP

  • Version :- 7.3.9-1~deb10u1 (Zend Version 3.3.9)
  • Modules :- apache2handlercalendar Core ctype curl date dom v20031129exif fileinfo filter ftp gd gettext hash iconv json v1.7.0libxml mbstring mosquitto v0.4.0mysqli mysqlnd vmysqlnd 5.0.12-dev - 20150407 - $Id: 7cc7cc96e675f6d72e5cf0f267f48e167c2abb23 $openssl pcre PDO pdo_mysql Phar posix readline redis v5.0.2Reflection session shmop SimpleXML sockets sodium SPL standard sysvmsg sysvsem sysvshm tokenizer wddx xml xmlreader xmlwriter xsl Zend OPcache zlib

Pi

  • Model :- Raspberry Pi 3 Model B Rev 1.2 - 1GB (Sony UK)

  • Serial num. :- ECC50CE4

  • Temperature :- 49.93°C - 50.5°C

  • emonpiRelease :- emonSD-17Oct19

  • File-system :- read-write

Client Information

Client Information

HTTP

  • Browser :- Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.113 Safari/537.36
  • Language :- en-GB,en;q=0.9

Window

  • Size :- 1204 x 1283

Screen

  • Resolution :- 2560 x 1440

Nothing immediately jumps out as being out of space.

The contents of my /opt/openenergymonitor/EmonScripts/update/service-runner.sh script are :-

#!/bin/bash
DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"
cd $DIR
source load_config.sh

if [ -z "$2" ]; then
    type="all"
    firmware=$1
else
    type=$1
    firmware=$2
fi

# Clear log update file
cat /dev/null > /var/log/emoncms/emonupdate.log

echo "Starting update via service-runner-update.sh (v3.0) >"

# -----------------------------------------------------------------

# Check emonSD version
image_version=$(ls /boot | grep emonSD)

if [ "$image_version" = "" ]; then
    echo "- Could not find emonSD version file"
    emonSD_pi_env=0
else
    echo "- emonSD version: $image_version"
    valid=0

    save_to_update=$(curl -s https://raw.githubusercontent.com/openenergymonitor/EmonScripts/master/safe-update)
    while read -r image_name; do
        if [ "$image_version" == "$image_name" ]; then
            echo "emonSD base image check passed...continue update"
            valid=1
        fi
    done <<< "$save_to_update"

    if [ $valid == 0 ]; then
        echo "ERROR: emonSD base image old or undefined...update will not continue"
        echo "See latest verson: https://github.com/openenergymonitor/emonpi/wiki/emonSD-pre-built-SD-card-Download-&-Change-Log"
        echo "Stopping update"
        exit 0
    fi
fi

# -----------------------------------------------------------------

# Pull in latest EmonScripts repo before then running updated update scripts
echo "git pull $openenergymonitor_dir/EmonScripts"
cd $openenergymonitor_dir/EmonScripts
git branch
git status
git pull

# Run update in main update script
$openenergymonitor_dir/EmonScripts/update/main.sh $type $firmware

Still confused … (But sun still just about shining)

8

First things does

curl -s https://raw.githubusercontent.com/openenergymonitor/EmonScripts/master/safe-update

Work from the command line?

Could you post the output from

df -h

reboot run, df -h again and then try the update again please.

9

So the command line curl (on an SSH session) works fine

pi@emonpi:~ $ curl -s https://raw.githubusercontent.com/openenergymonitor/EmonScripts/master/safe-update
emonSD-02Oct19
emonSD-17Oct19
pi@emonpi:~

The first df -h

pi@emonpi:~ $ df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/root       4.0G  2.1G  1.7G  55% /
devtmpfs        484M     0  484M   0% /dev
tmpfs           488M     0  488M   0% /dev/shm
tmpfs           488M   19M  470M   4% /run
tmpfs           5.0M  4.0K  5.0M   1% /run/lock
tmpfs           488M     0  488M   0% /sys/fs/cgroup
tmpfs           1.0M     0  1.0M   0% /var/lib/php/sessions
tmpfs            30M   30M     0 100% /tmp
tmpfs           1.0M     0  1.0M   0% /var/tmp
/dev/mmcblk0p3   10G   41M  9.5G   1% /var/opt/emoncms
/dev/mmcblk0p1  253M   52M  201M  21% /boot
log2ram          50M   37M   14M  74% /var/log
tmpfs            98M     0   98M   0% /run/user/1000
pi@emonpi:~ $

Reboot [see below], and second df -h

pi@emonpi:~ $ df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/root       4.0G  2.1G  1.7G  55% /
devtmpfs        484M     0  484M   0% /dev
tmpfs           488M     0  488M   0% /dev/shm
tmpfs           488M  6.5M  482M   2% /run
tmpfs           5.0M  4.0K  5.0M   1% /run/lock
tmpfs           488M     0  488M   0% /sys/fs/cgroup
tmpfs           1.0M     0  1.0M   0% /var/lib/php/sessions
tmpfs            30M     0   30M   0% /tmp
tmpfs           1.0M     0  1.0M   0% /var/tmp
/dev/mmcblk0p3   10G   41M  9.5G   1% /var/opt/emoncms
/dev/mmcblk0p1  253M   52M  201M  21% /boot
log2ram          50M   38M   13M  75% /var/log
tmpfs            98M     0   98M   0% /run/user/1000
pi@emonpi:~ $

Now about this rebooting thing.

Maybe related to my other issues, could be nothing but I am find rebooting troublesome. Today when trying the reboot sandwich in between a pair of df -h I struggled. I first asked the emonPi to reboot from the button at the bottom of Setup / Admin from a browser login to it’s IP address.

Certainly yes the separate PuTTY session in to port 22 died immediately, and after 3 minutes (measured by a timer on my phone) I tried PuTTY again - timed out. Went to the emonPi (inconveniently located but aren’t all utility meters), no LCD backlight and no response to the button.

Removed 5V power, waited around 2 minutes minutes, repowered. Went through splash screen, sensor detection, and gets to Raspberry Pi booting … message. After another 5 minutes no progress on LCD, backlight and booting message still there (that message needs to be animated so you can tell when it sticks). Removed 5V power and AC voltage sensor power, left for 15 minutes.

Reapplied AC voltage then 5V power. Went through normal boot sequence to WiFi Yes xx% & DHCP’d IP address. Happy, or so I think.

I extract the second df -h and write it here. Then try browsing to the usual web page at the Pi’s IP. Very strange result - rather than a login landing page I get

Fatal error : Uncaught Error: Call to a member function fetch_array() on bool in /var/www/emoncms/core.php:43 Stack trace: #0 /var/www/emoncms/index.php(82): db_check(Object(mysqli), ‘emoncms’) #1 {main} thrown in /var/www/emoncms/core.php on line 43

Tried on two different browsers (edge & chrome) and to the bare IP or at IP:80. All the same result.

So is my SDcard gradually disintegrating? Do I have a hardware issue? What is the issue with not powering down gracefully?

ConfusionLevel++

10

Apologies, now I look I see that the /tmp mount was full prior to the reboot, hence maybe the original second failure to update.

But I’m definitely in a worse place now. How do I get my Pi back?

11

This might be related /tmp being full. One alternative is from the command line sudo reboot now.

Should be OK.

SSH in and reboot from the command line. May take a few (wait 10) minutes before you can reconnect to the Pi via Putty (tip - right click on banner and select restart session). Hard reboot when there are updates going on can be a very bad thing! Patience is a necessity.

1 Like
12

The /tmp file being full is interesting, it may be caused by the apache2 sessions folder filling up, do you have any other OEM hardware running EmonESP by any chance?

13

@borpin Now completely borked. I did sudo reboot now from PuTTY and that was the last my network has seen of the Pi. LCD got as far as Raspberry Pi rebooting… Where it has sat for some hours.

About to download the image and burn a new card.

@TrystanLea I was running the vanilla image as supplied from the shop, with the only addition being an install of node-red last weekend. That’s obviously the prime suspect, but I think I was having issues before anyways.

Today has been a bad day for technology in my day job as well.

Happiness–

14

That sounds like the best course of action at this point. Sorry that this has failed.
If you want to try and recover any data on the image you could try Update & Upgrade — OpenEnergyMonitor 0.0.1 documentation you may need to use fsck to recover the card see Import / Backup / Restore / Update — OpenEnergyMonitor 0.0.1 documentation

15

Ah, well, that could easily explain it. I’d run NR on a separate Pi.

16

Today is a different day.

Downloaded image and flashed to virgin 16GB SDcard using etcher. Installed in emonPi, powered up and walked away for a few hours. Did initial set up on WiFi - will come back to wired Ethernet issue later. Ran Full Update, but I think it had already done so, as nothing was pulled in.

Enabled SSH from button by LCD. Put old SDcard into USB reader and plugged into emonPi. Tried Setup | Backup | Import USB - this was the output.

=== USB Emoncms import start ===
2020-04-29-12:59:27
Backup module version:
    "version"      : "2.2.2"
EUID: 1000
Reading /opt/emoncms/modules/backup/config.cfg....
Location of data databases: /var/opt/emoncms
Location of emonhub.conf: /etc/emonhub
Location of Emoncms: /var/www/emoncms

USB drive not found

Assumed SDcard corrupted. From SSH session ran

pi@emonpi:~ $ sudo fsck.ext4 /dev/sda2
e2fsck 1.44.5 (15-Dec-2018)
rootfs contains a file system with errors, check forced.
Pass 1: Checking inodes, blocks, and sizes
Inode 98182 is in use, but has dtime set.  Fix<y>? yes
Inodes that were part of a corrupted orphan linked list found.  Fix<y>? yes
Inode 98184 was part of the orphaned inode list.  FIXED.
Inode 98182 has a bad extended attribute block 32768.  Clear<y>? yes
Inode 98182, i_size is 576460752303423488, should be 0.  Fix<y>? yes
Pass 2: Checking directory structure
Directory inode 42052, block #0, offset 520: directory corrupted
Salvage<y>? yes
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Unattached zero-length inode 8197.  Clear<y>? yes
Inode 98182 (...) has invalid mode (01000).
Clear<y>? yes
Inode 98184 (...) has invalid mode (00).
Clear<y>? yes
Pass 5: Checking group summary information
Block bitmap differences:  -(155136--155378)
Fix<y>? yes
Free inodes count wrong for group #12 (8050, counted=8048).
Fix ('a' enables 'yes' to all) <y>? yes
Free inodes count wrong (190683, counted=190681).
Fix ('a' enables 'yes' to all) <y>? yes

rootfs: ***** FILE SYSTEM WAS MODIFIED *****
rootfs: 74903/265584 files (0.1% non-contiguous), 559467/1053440 blocks

And then

pi@emonpi:~ $ sudo fsck.ext2 /dev/sda3
e2fsck 1.44.5 (15-Dec-2018)
/dev/sda3: clean, 40/665088 files, 210847/10634240 blocks
pi@emonpi:~ $

Tried Import USB again, same message. Unplugged USB card reader, plugged into different USB port and tried again - still nothing found.

No linux guru so not understanding that Import USB can’t see the card in the reader, but fsck has seen and fixed some errors. Were the errors that fsck fixed found on the SDcard, or some other storage on the Pi motherboard? Are /dev/sda2 and /dev/sda3 on the SDcard or somewhere else?

What is my next step to attempt to restore my feeds and data? Painful and disappointing if I can’t but not disastrous.

Now to the Wired Ethernet. Before Corruption (BC) I had tried and failed to move the emonPi onto my wired network. I used a Powerline plug pair to present 10/100Mb LAN at the utility meter. A normal laptop happily DHCP’d there to my 10.5.1.0/24 home network and connected to all network resources. The emonPi did not, it came up on the LCD as 169.254.204.239, and still does now After Corruption (AC).

This makes me think that either (a) there is be some other stored state in the emonPi - like an MMC, Flash or EEPROM chip that is corrupted, or (b) there’s a fault in my emonPi’s Ethernet port.

Can anyone comment.

17

PS Port lights on the emonPi and Powerline plug showed Link Up and Active.

PPS I had also directly linked a laptop to the emonPi using a straight wired RJ45 - good link LEDs at both ends, and the laptop assigned itself a compatible 169.254.0.0/16 address. I could not web browse to the emonPi’s IP address, with or without a :80 suffix, or SSH to its port 22.

18

I see 2 different IP addresses (when I interrogate the network) - one for wired (eth0) one for Wi-Fi (wlan0).

try ip a for the IP addresses.

19

The 2 commands have been successful when run after a hard power down due to a power cut.

Sometimes the card is just too badly corrupted.

20

Sorry about this @Simonsse I think this is an issue with the import script, I’ve been trying to scan for likely USB card reader names and dont seem to be able to catch all the variations, see the lines here backup/usb-import.sh at master · emoncms/backup · GitHub

Could you run:

find /dev/disk/by-id/