Community
OpenEnergyMonitor

Community

A simple method to run the linux partitions of the emonSD image from an attached USB HDD/SSD

Tags: #<Tag:0x00007f8dee1cc4a0> #<Tag:0x00007f8dee1cc388>

Having experienced a number of SD card corruptions to my emonBase on a Pi2B owing to random power cuts, I decided to rebuild a new system using the latest emonSD-17Oct19.img image on a USB HDD connected to the high speed USB3 port of a Pi4.

Whilst the expert Linux user will find this post simplistic and overly long, the intended audience is the novice to emonBase. The post veers towards the ‘how to’ in the hope that the user can assimulate new knowledge largely risk free without fear of wrecking their system in the process of manipulating partitions and file systems. EmonPi/eMonBase is an elegant system developed by experts over many years but it is undeniably complex. An easy recovery path is important if a new user is not to hesitate over the execution of a command fearing the consequences of a mistake.

The aim of this post is to describe a simple procedure to create a robust system running entirely from a HDD drive apart from the essentially run time read-only files on the /boot partition of the SD card which are easily edited on a Windows/Apple machine if needs be. The new system should also be immune against any number of other USB SD/HDD/SSDs attached to the Pi at boot.

The method is a distillation of ideas taken from other OpenEnergyMonitor Community, Raspberry Pi and Linux fora for which I can take no credit. However, in drawing these ideas together, I feel this post may benefit others, who, like me, have no deep understanding of Linux programming, per se, but just want to get a reliable and robust system working.

Some background. My first post on the use of an HDD drive was made in Jun 2016 in archived post:
Moving the emonSD Linux partitions to a USB HDD
in which I described my experience of using the helpful script created by Paul Reed:
https://github.com/emoncms/emoncms/blob/stable/docs/RaspberryPi/USB_HDD.md

However, this script only moves the root partion to the HDD leaving the /var/www/emoncms data partition on the SD card. A later post expressed the desirability of also moving the data partition and included a set of build notes by John Banks for moving this partition:

This post describes an alternative approach.

A complete descriptive outline of the steps is given below. My annotated build notes showing the linux commands are given in the attached file: Linux Steps.doc (99.5 KB) .

A). Burn the SD card image file onto BOTH the new SD card and HDD/SSD drive using your program of choice (balenaEtcher seems to be in wide use but I prefer ApplePi-Baker v2.2.3 at Tweaking4All.com on MacOS which is now available as a 64bit app and will also backup the modified image).

B). Connect the hardware together and power up the Pi4 logging in as user: pi and password: emonpi2016.

C). Use the raspi-config program to activate ssh if wanting to run without monitor and keyboard from another computer. In this case establish either a LAN or WiFi IP connection to the host.

D). Make a copy of the /boot/cmdline.txt file to /boot/cmdline.sd to enable easy switching back back and forth between the SD card and HDD.

E). Add 2 additional lines to the /boot/config.txt to increase the USB boot wait timeout and increase the available USB drive current to 1200mA to cater for slower, power hungry drives.

F). Install the gdisk partition tool to the root file system on the SD card.

G). Using the gdisk tool convert the HDD from a MBR to EFI GPT partitioning scheme. gdisk leaves a protected MBR sector that ensures fdisk will not trash the scheme if used later. Delete the HDD /boot partition and create a new GPT partition in its place. Randomise the UUID and PARTUUID values of all HDD partitions. This is necessary as the partitions on the HDD and SD card have identical values as yet, since they were created from the same image file. The partition PARTUUID value is essentially a concept introduced with GPT and cannot be changed using the default fdisk partitioning tool. The PTUUID value is not the same and cannot be used to uniquely identify the partition in the /boot/cmdline.txt file. (Comments in Paul Reed’s script also hint at these differences).

H). At this stage it is worth using the simple /dev/sda[n] descriptors to test that it is possible to boot from the HDD and run emoncms. Edit the /etc/fstab table on the /dev/sda2 HDD partition and edit the /boot/cmdline.txt file to point to the /dev/sda2 partition.

I). Reboot and if all is well the system should now boot from the HDD (It is necessary to run raspi-config again to enable ssh, install `gdisk and setup the LAN and WiFi IP addresses on the HDD).

J). If this test is passed, copy the old /boot/cmdline.sd to /boot/cmdline.txt and reboot to the SD card. I prefer not to make partitioning and file system changes on a live boot partition in case of typos and to avoid kernel problems.

K). Use gdisk to expand the HDD /var/www/emoncms data partition to the desired size. I expanded this partition to around 330GB on my 500GB HDD in order to leave room for an optional 100GB 4th partition on which to store my emoncms backups.

L). Expand the emoncms file system to fill the newly sized data partition.

M). OPTIONAL. Create a 4th partition on the HDD and initialise a new file system to hold old emoncms backups.

N). Edit the /etc/fstab file on the /dev/sda2 HDD partition to use the new HDD UUID values of the HDD partitions generated earlier.

O). Finally, edit the /boot cmdline.txt again to use the PARTUUID value of /dev/sda2 HDD partition and reboot.

This may seem like a complex procedure but each step is straightforward AND it has the added advantage of an easy recovery if a mistake is made; the SD card root and data partitions are left untouched and the HDD can always be flashed with the SD card image again for a fresh attempt.

If a bare bones installation using only /dev/sda[n] partition descriptors is required, the steps involving gdisk can be omitted with fdisk used to expand the partition size in step G). However, this will not guarantee that the system will boot if additional drives are attached.

The use of the backup and restore scripts on the SD image before and after implementing the above steps should allow for this procedure to move existing installations containing valuable data to the HDD, although I have not done this.

Whilst these procedures clearly work and, in the barebones case, are particularly simple, I would encourage the acknowledged linux experts in the Community to correct any misplaced explanations and transcription errors that I have made to prevent the novice being led astray.

1 Like

Thanks for sharing @mhamer