Backup module usb-import.sh script (beta)

TrystanLea · 2019-12-17T18:51:49.755Z

One advantage of this usb import approach is that it can in certain cases be used to recover a corrupted SD card were a previous backup archive was not made - or is quite old. If the SD card mounts ok the script may run without trouble, but if the damage is greater, running the file system consistency check utility (fsck) maybe needed before doing an import.

Both Glyn and I had our home systems go down following the same power cut in our village last weekend. While my system booted there were issues with emonhub and redis resulting from SD card corruption. I was able to recover using the importer tool without running fsck. Glyn’s on the other hand would not boot initially - but after running fsck we were able to mount the old SD card and run the import script.

Here are the steps to do this:

1. Insert SD card in SD card reader and plug in to the emonPi/emonbase/raspberrypi running a fresh copy of the emonSD image.
2. Attempt to recover the SD card partitions e.g:

sudo fsck.ext4 /dev/sda2 (root OS partition)
sudo fsck.ext2 /dev/sda3 (data partition)

3. Once complete, run the importer tool as above.

borpin · 2019-12-17T19:09:06.622Z

TrystanLea:

Both Glyn and I had our home systems go down following the same power cut

And you’ve now both got spike protection…

borpin · 2019-12-17T19:09:43.111Z

TrystanLea:

Here are the steps to do this:

Worth adding a page on the docs for this I’d suggest.

borpin · 2019-12-17T19:13:16.284Z

Actually, @TrystanLea a comprehensive guide on upgrading an SDCard would be fantastic. I cannot remember what is out there right now.

My current advice would be

You are running an older setup. My advice is to upgrade to the lastest version. emonSD-17Oct19 release

To do this;

get a new SD Card (you can always put the old one back in if there is an issue)
download & flash a new EmonSD image emonSD pre built SD card Download & Change Log · openenergymonitor/emonpi Wiki · GitHub
take a backup of the existing system. You may need to clone GitHub - emoncms/backup: emoncms backup module and do a manual export.
install the new SD card & power up
wait, wait, wait, make a cup of coffee, wait, wait, wait… (lots of updates etc)
import the backup

TrystanLea · 2020-01-11T19:53:22.528Z

I’ve added the USB import method now to the backup module guide and to a new guide on updating and upgrading emonpi/emonbase systems.

Backup Guide:
Import / Backup - Guide | OpenEnergyMonitor

Update and Upgrade
Update & Upgrade - Guide | OpenEnergyMonitor

Thanks @borpin for your start on this, you will see I’ve made use of what you wrote here: Restore Data From SD Card (USB) · openenergymonitor/emonpi Wiki · GitHub

borpin · 2020-01-12T14:46:00.951Z

@TrystanLea great. Could you add in a bookmark for the Upgrade section so a direct link can be created to it please?

Couple of things.

“Place the old SD card in a SD” I think it should be ‘an’ so “Place the old SD card in an SD”

“Create a temporary user on the new system if you have not done so already.”

This instruction could be confusing as to whether you mean a user on the OS or on the EmonCMS system. Perhaps

“From the EmonCMS login page, click register and create a temporary user. Once the import is complete the original user details will be used.”

and add

“5. Once the import is complete, log out and back into the EmonCMS page with the original user details”.

TrystanLea · 2020-01-12T19:22:08.417Z

Great will add those in, thanks @borpin

TrystanLea · 2020-01-13T12:32:20.128Z

Thanks @borpin I’ve made the changes you suggested. Apparently bookmarks work to any heading, you just need to type the full heading in lower case and with dashes:

Update & Upgrade - Guide | OpenEnergyMonitor

borpin · 2020-01-13T12:59:46.784Z

For some headings the link appears when you mouse over as in

Pukka · 2020-01-18T17:12:45.158Z

I could not get the USB import to work so went to a ssh session and found my USB device was found as

image1019×69 3.47 KB

I edited the backup/usb-import.sh at master · emoncms/backup · GitHub
usb=$(ls $disk_id | grep ‘usb-Generic_Mass-Storage’) to usb=$(ls $disk_id | grep ‘usb-Generic’)

Pukka · 2020-01-18T17:15:58.555Z

also they should be a little more checking, if you click import from USB without the usb the web interface dies with
’ Can’t connect to database, please verify credentials/configuration in settings.php
Error message: Unknown database ‘emoncms’’

TrystanLea · 2020-01-18T17:30:17.664Z

Thanks @Pukka, that’s useful to know, I did wonder how many varients on the grep search term for the usb-Generic… there might be. I’ve created a github issue to keep track of this usb-Generic vs usb-Generic_Mass-Storage detection · Issue #52 · emoncms/backup · GitHub.

Bill.Thomson · 2020-01-18T22:46:24.238Z

Pukka:

if you click import from USB without the usb the web interface dies with

Yet another example of why new “features” need to be “wrung out” before getting released.

TrystanLea · 2020-01-19T09:49:11.810Z

Pukka:

also they should be a little more checking, if you click import from USB without the usb the web interface dies with
’ Can’t connect to database, please verify credentials/configuration in settings.php
Error message: Unknown database ‘emoncms’ ’

I was sure I’d tested this one, its now fixed, thanks!

PeteF · 2020-02-11T14:15:31.491Z

I have just updated to emonSD-17Oct19 and get “USB drive not found”. The USB drive is:

Screenshot 2020-02-11 at 14.02.551450×326 184 KB

PeteF · 2020-02-11T14:28:41.112Z

OK I just read the code and I think I see that the master hasn’t been updated yet, pending Trystan’s GitHub issue investigation - completely understand why. I assume Paul created his own branch to get it working. I’m not a GitHub user so I might need some help to create my own mod

borpin · 2020-02-11T16:11:35.001Z

PeteF:

I might need some help to create my own mod

Pukka:

I edited the https://github.com/emoncms/backup/blob/master/usb-import.sh
usb=$(ls $disk_id | grep ‘usb-Generic_Mass-Storage’) to usb=$(ls $disk_id | grep ‘usb-Generic’)

If you just edit the file line 34 in the folder /opt/emoncms/modules/backup/

github.com

emoncms/backup/blob/d474916321f9705edadb4230ac8c18dbfc2450ac/usb-import.sh#L34


      
          emoncms_mqtt=$(systemctl show emoncms_mqtt | grep LoadState | cut -d"=" -f2)
          
          echo
          
          disk=false
          # Scan through disks to find 'usb-Generic_Mass-Storage'
          for diskname in 'sda' 'sdb' 'sdc'
            do
            disk_id=$(find /dev/disk/by-id/ -lname "*$diskname")
            if [ $disk_id ]; then
                usb=$(ls $disk_id | grep 'usb-Generic_Mass-Storage')
                if [ $usb ]; then
                    echo "Found: $disk_id at /dev/$diskname"
                    disk="$diskname"
                fi
                usb=$(ls $disk_id | grep 'usb-Mass_Storage_Device')
                if [ $usb ]; then
                    echo "Found: $disk_id at /dev/$diskname"
                    disk="$diskname"
                fi
            fi

to read

usb=$(ls $disk_id | grep ‘usb-Generic’)

it should work.

Just note, you will need to undo this before you are able to update the backup module again.

PeteF · 2020-02-11T16:18:19.074Z

Thank you Brian! After an initial attempt using Git, I went ahead and did exactly what you suggest about half an hour ago, and the import worked a treat

However, I wouldn’t have known to undo it for future updates to work though - so double-thanks for that. I made a copy of the file first though

It would be nice to know how to have done it properly though, via Git…
I created my own fork, edited it and committed it back to my private repo. Now though, how would I apply the update on the EmonPi in order to test it? And then how would I revert back to the master Emoncms release afterwards?

borpin · 2020-02-11T17:06:20.658Z

PeteF:

how would I apply the update on the EmonPi in order to test it? And then how would I revert back to the master Emoncms release afterwards?

If you want to play be my guest

First an explanation of terms

upstream is the original emoncms repo you forked.
origin is your fork of the upstream
local is the repo on your machine

However because the original clone was done direct from the upstream, it calls itself origin in this case (git doesn’t make anything easy - trust me). We won’t bother creating the link though - it doesn’t matter here. It does matter (a bit) if you are doing development work.

First part right, create a fork. Ideally then create your own branch on that fork - if you subsequently want to create a PR (Pull Request) to submit something you have done to the main emoncms repo, this makes it much easier.

To use your repo, you first need to add it to your local machine in the same directory the original is in.

git remote add myrepo https://github.com/YOUR_NAME/backup.git

The ‘myrepo’ can be any name but not one of the 3 above (they have special meaning) git remote -v should look similar to this.

$ git remote -v
myrepo  https://github.com/borpin/backup.git (fetch)
myrepo  https://github.com/borpin/backup.git (push)
origin  https://github.com/emoncms/backup.git (fetch)
origin  https://github.com/emoncms/backup.git (push)

Then the commands

git fetch myrepo
git checkout myrepo/BRANCH_NAME

You will get a message about detatched head - don’t worry about that.

git status

tells you the branch you are on and you are now using your code.

to go back simply do

git checkout origin/master

or

git checkout origin/stable

depending what branch you were on before you started messing about.

HTH

PeteF · 2020-02-11T17:25:41.054Z

Many thanks for taking the time and trouble to write that, Brian. I will master it, but I may need to go and have a little lie down for a bit first