Backup module usb-import.sh script (beta)

emonSD
1

After reading through @mjseeley’s thread on migrating from an old 2015 image to the latest (LCD Screen says "Updating - Do Not Unplug" (updating system from V8.5)) it got me thinking again about an idea for migrating that would avoid the need to update software or run a script on the original image.

The idea is to take the original SD card and mount it as a ‘USB drive’ on to the Pi running the latest image, then run an import script on the new image to copy the files across directly.

2019-10-21
2019-10-211280×601 301 KB

I’ve got a preliminary version of this to work here and tested migration from emonSD-30Oct18 and emonSD-17Jun2015 images. I’ve uploaded the script to the master branch of the backup module for testing: GitHub - emoncms/backup: emoncms backup module, see: usb-import.sh.

Inputs and feeds imported fine on the emonSD-17Jun2015 image, a basic dashboard with one widget needed clicking on the widget and re-configuring. The import tool copied the old emonhub.conf which was not compatible with the new image so I had to restore with a new default copy from emonhub/emonpi.default.emonhub.conf at emon-pi · openenergymonitor/emonhub · GitHub

How to use the script

1. SSH into the Pi running the new image
2. Place the old SD card in a SD card reader and plug into any of the USB ports on the Pi running the new image.
3. Navigate to directory /opt/emoncms/modules/backup

cd /opt/emoncms/modules/backup

4. Run the import script:

./usb-import.sh

5. Logout and log back into your emoncms account with your original credentials.

Note: The script assumes that the USB reader and old SD card is mounted on /dev/sda which appears to be consistent over 2 different systems and 3 migration tests that I’ve performed so far.

Next steps: I’d like to make this script runnable from a button click in the emoncms UI to remove the need to use SSH.

Can anyone see any obvious issues with this approach that I may be missing?

2 Likes
2

I think it is a good idea and most useful to those without the backup module installed, but…

  1. Not everyone will have a reader so the backup file method should not be abandoned.
  2. Some installations are not on SD Cards - might be on a VM or a HDD.

I also think being able to import (either file or mounted filesystem) at the ‘register’ point of a new install would make migration much easier as well.

All progress :grin:.

1 Like
3

This script is now available via the emoncms backup UI (currently on the master branch for testing):

image
image1242×929 67.2 KB

1 Like
4

Excellent. What I would say is that it would be really useful if this was an option on first setup rather than having to register a user, then upload the data.

An option at the register stage to import data would be even better :grinning:.

Will it pick up in-built usb card readers on laptops?

5

I agree that would be nice.

At the moment it looks for ‘usb-Generic_Mass-Storage’ in the disk name, so probably not.

6

Stupid question actually, as designed, it need to be plugged into the actual Pi.

7

I was thinking you might want to use it, to import an SD card to a local copy of emoncms running on your laptop… I think it should work, if you use a USB reader.

8

No I was thinking of accessing the web interface - totally daft suggestion.

9

One advantage of this usb import approach is that it can in certain cases be used to recover a corrupted SD card were a previous backup archive was not made - or is quite old. If the SD card mounts ok the script may run without trouble, but if the damage is greater, running the file system consistency check utility (fsck) maybe needed before doing an import.

Both Glyn and I had our home systems go down following the same power cut in our village last weekend. While my system booted there were issues with emonhub and redis resulting from SD card corruption. I was able to recover using the importer tool without running fsck. Glyn’s on the other hand would not boot initially - but after running fsck we were able to mount the old SD card and run the import script.

Here are the steps to do this:

1. Insert SD card in SD card reader and plug in to the emonPi/emonbase/raspberrypi running a fresh copy of the emonSD image.
2. Attempt to recover the SD card partitions e.g:

sudo fsck.ext4 /dev/sda2 (root OS partition)
sudo fsck.ext2 /dev/sda3 (data partition)

3. Once complete, run the importer tool as above.

1 Like
10

And you’ve now both got spike protection… :grinning:

1 Like
11

Worth adding a page on the docs for this I’d suggest.

1 Like
12

Actually, @TrystanLea a comprehensive guide on upgrading an SDCard would be fantastic. I cannot remember what is out there right now.

My current advice would be

You are running an older setup. My advice is to upgrade to the lastest version. emonSD-17Oct19 release

To do this;

1 Like
13

I’ve added the USB import method now to the backup module guide and to a new guide on updating and upgrading emonpi/emonbase systems.

Backup Guide:
Import / Backup - Guide | OpenEnergyMonitor

Update and Upgrade
Update & Upgrade - Guide | OpenEnergyMonitor

Thanks @borpin for your start on this, you will see I’ve made use of what you wrote here: Restore Data From SD Card (USB) · openenergymonitor/emonpi Wiki · GitHub

1 Like
14

@TrystanLea great. Could you add in a bookmark for the Upgrade section so a direct link can be created to it please?

Couple of things.

“Place the old SD card in a SD” I think it should be ‘an’ so “Place the old SD card in an SD”

“Create a temporary user on the new system if you have not done so already.”

This instruction could be confusing as to whether you mean a user on the OS or on the EmonCMS system. Perhaps

“From the EmonCMS login page, click register and create a temporary user. Once the import is complete the original user details will be used.”

and add

“5. Once the import is complete, log out and back into the EmonCMS page with the original user details”.

1 Like
15

Great will add those in, thanks @borpin

1 Like
16

Thanks @borpin I’ve made the changes you suggested. Apparently bookmarks work to any heading, you just need to type the full heading in lower case and with dashes:

Update & Upgrade - Guide | OpenEnergyMonitor

17

For some headings the link appears when you mouse over as in

image

18

I could not get the USB import to work so went to a ssh session and found my USB device was found as

image
image1019×69 3.47 KB

I edited the backup/usb-import.sh at master · emoncms/backup · GitHub
usb=$(ls $disk_id | grep ‘usb-Generic_Mass-Storage’) to usb=$(ls $disk_id | grep ‘usb-Generic’)

1 Like
19

also they should be a little more checking, if you click import from USB without the usb the web interface dies with
’ Can’t connect to database, please verify credentials/configuration in settings.php
Error message: Unknown database ‘emoncms’’

20

Thanks @Pukka, that’s useful to know, I did wonder how many varients on the grep search term for the usb-Generic… there might be. I’ve created a github issue to keep track of this usb-Generic vs usb-Generic_Mass-Storage detection · Issue #52 · emoncms/backup · GitHub.

1 Like