A simple method to run the linux partitions of the emonSD image from an attached USB HDD/SSD

mhamer · 2020-01-17T12:10:12.470Z

Having experienced a number of SD card corruptions to my emonBase on a Pi2B owing to random power cuts, I decided to rebuild a new system using the latest emonSD-17Oct19.img image on a USB HDD connected to the high speed USB3 port of a Pi4.

Whilst the expert Linux user will find this post simplistic and overly long, the intended audience is the novice to emonBase. The post veers towards the ‘how to’ in the hope that the user can assimulate new knowledge largely risk free without fear of wrecking their system in the process of manipulating partitions and file systems. EmonPi/eMonBase is an elegant system developed by experts over many years but it is undeniably complex. An easy recovery path is important if a new user is not to hesitate over the execution of a command fearing the consequences of a mistake.

The aim of this post is to describe a simple procedure to create a robust system running entirely from a HDD drive apart from the essentially run time read-only files on the /boot partition of the SD card which are easily edited on a Windows/Apple machine if needs be. The new system should also be immune against any number of other USB SD/HDD/SSDs attached to the Pi at boot.

The method is a distillation of ideas taken from other OpenEnergyMonitor Community, Raspberry Pi and Linux fora for which I can take no credit. However, in drawing these ideas together, I feel this post may benefit others, who, like me, have no deep understanding of Linux programming, per se, but just want to get a reliable and robust system working.

Some background. My first post on the use of an HDD drive was made in Jun 2016 in archived post:
Moving the emonSD Linux partitions to a USB HDD
in which I described my experience of using the helpful script created by Paul Reed:
https://github.com/emoncms/emoncms/blob/stable/docs/RaspberryPi/USB_HDD.md

However, this script only moves the root partion to the HDD leaving the /var/www/emoncms data partition on the SD card. A later post expressed the desirability of also moving the data partition and included a set of build notes by John Banks for moving this partition:

EmonSD save data to an external HDD/SDD instead of SD emonSD

Hi, is there a way to change the location of EmonPI SQL database? I found some info in this thread that suggests to use the script /var/www/emoncms/scripts/usb_hdd/usb_hdd, unfortunally seems that this script just moves root partition to an external hdd and database is still stored in the SD card. Any suggestions? Thanks

This post describes an alternative approach.

A complete descriptive outline of the steps is given below. My annotated build notes showing the linux commands are given in the attached file: Linux Steps.doc (99.5 KB) .

A). Burn the SD card image file onto BOTH the new SD card and HDD/SSD drive using your program of choice (balenaEtcher seems to be in wide use but I prefer ApplePi-Baker v2.2.3 at Tweaking4All.com on MacOS which is now available as a 64bit app and will also backup the modified image).

B). Connect the hardware together and power up the Pi4 logging in as user: pi and password: emonpi2016.

C). Use the raspi-config program to activate ssh if wanting to run without monitor and keyboard from another computer. In this case establish either a LAN or WiFi IP connection to the host.

D). Make a copy of the /boot/cmdline.txt file to /boot/cmdline.sd to enable easy switching back back and forth between the SD card and HDD.

E). Add 2 additional lines to the /boot/config.txt to increase the USB boot wait timeout and increase the available USB drive current to 1200mA to cater for slower, power hungry drives.

F). Install the gdisk partition tool to the root file system on the SD card.

G). Using the gdisk tool convert the HDD from a MBR to EFI GPT partitioning scheme. gdisk leaves a protected MBR sector that ensures fdisk will not trash the scheme if used later. Delete the HDD /boot partition and create a new GPT partition in its place. Randomise the UUID and PARTUUID values of all HDD partitions. This is necessary as the partitions on the HDD and SD card have identical values as yet, since they were created from the same image file. The partition PARTUUID value is essentially a concept introduced with GPT and cannot be changed using the default fdisk partitioning tool. The PTUUID value is not the same and cannot be used to uniquely identify the partition in the /boot/cmdline.txt file. (Comments in Paul Reed’s script also hint at these differences).

H). At this stage it is worth using the simple /dev/sda[n] descriptors to test that it is possible to boot from the HDD and run emoncms. Edit the /etc/fstab table on the /dev/sda2 HDD partition and edit the /boot/cmdline.txt file to point to the /dev/sda2 partition.

I). Reboot and if all is well the system should now boot from the HDD (It is necessary to run raspi-config again to enable ssh, install `gdisk and setup the LAN and WiFi IP addresses on the HDD).

J). If this test is passed, copy the old /boot/cmdline.sd to /boot/cmdline.txt and reboot to the SD card. I prefer not to make partitioning and file system changes on a live boot partition in case of typos and to avoid kernel problems.

K). Use gdisk to expand the HDD /var/www/emoncms data partition to the desired size. I expanded this partition to around 330GB on my 500GB HDD in order to leave room for an optional 100GB 4th partition on which to store my emoncms backups.

L). Expand the emoncms file system to fill the newly sized data partition.

M). OPTIONAL. Create a 4th partition on the HDD and initialise a new file system to hold old emoncms backups.

N). Edit the /etc/fstab file on the /dev/sda2 HDD partition to use the new HDD UUID values of the HDD partitions generated earlier.

O). Finally, edit the /boot cmdline.txt again to use the PARTUUID value of /dev/sda2 HDD partition and reboot.

This may seem like a complex procedure but each step is straightforward AND it has the added advantage of an easy recovery if a mistake is made; the SD card root and data partitions are left untouched and the HDD can always be flashed with the SD card image again for a fresh attempt.

If a bare bones installation using only /dev/sda[n] partition descriptors is required, the steps involving gdisk can be omitted with fdisk used to expand the partition size in step G). However, this will not guarantee that the system will boot if additional drives are attached.

The use of the backup and restore scripts on the SD image before and after implementing the above steps should allow for this procedure to move existing installations containing valuable data to the HDD, although I have not done this.

Whilst these procedures clearly work and, in the barebones case, are particularly simple, I would encourage the acknowledged linux experts in the Community to correct any misplaced explanations and transcription errors that I have made to prevent the novice being led astray.

TrystanLea · 2020-01-18T18:02:59.192Z

Thanks for sharing @mhamer

jpalo · 2020-08-05T14:42:46.882Z

Very nice, was able to switch to SSD on RPi using the EXCELLENT “Linux Steps.doc” document. Thank you so much @mhamer!

One small correction to step J): sudo cp /boot/config.sd /boot/config.txt should be sudo cp /boot/cmdline.sd /boot/cmdline.txt

AndyMc · 2021-03-01T20:53:59.499Z

Thank you Malcolm for your procedure on transferring emonSD image to an SSD, in my case a Samsung 250. As usual I am having a few hiccups along the way. I have got the system booting off of the SSD but have not got the data partition extended to take advantage of the larger size drive. I am still operating off of a 10 gig data partition. I was going to live with this limitation because i have not been able to make the gdisk do my bidding and extend the partition, but…

The system will only boot off of the SSD if the SD card is installed.
I tried to change this in the raspi-config but the boot order in the menu does not work correctly and there are errors.

I also tried to alter the boot preference in the boot loader eeprom but this image of Emon will not allow me to edit the eeprom config.
I am using the emonSD-24Jul20.img

I had success booting from the SSD with the newest Raspi OS but limited joy with EmonSD image.

I would appreciate any insight you might have.

Best Regards: Andrew McIvor

borpin · 2021-03-02T16:53:50.716Z

I think the optimal solution is to install the base OS move to SSD if required and then run EmonScripts to install emoncms.

chrissg6eui · 2021-03-02T22:28:08.708Z

Looks like I missed this topic, I have posted my method in this topic

Which RPi to use, Argon One case, M.2 SSD emonSD

Ok done a bit more digging and the booting from SSD vs SD card turned out to be a simple edit of the fstab in the SSD /etc directory. here are the changes I made. (I’m not sure if there is a way to format this). *#/dev/mmcblk0p1 /boot vfat defaults 0 2* *#/dev/mmcblk0p2 / ext4 defaults,noatime,nodiratime 0 1* *#/dev/mmcblk0p3 /var/opt/emoncms ext2 defaults,noatime,nodiratime 0 2* *#* */dev/sda1 /boot vfat defaults 0…

AndyMc · 2021-03-02T22:36:46.106Z

borpin:

EmonScripts to install emoncms.

Hello Brian
Following your recommendations, I have achieved an EmonCms that boots and operates from an SSD. Now i just have to slip it under the emonpi and see if it picks up all the emontx’s i have scattered about. Do you have any recommendations about calibrating the CTs or will transferring the values from the Emonhub config be sufficient?
Best Regards and Thanks: Andrew McIvor

Robert.Wall · 2021-03-03T00:26:18.971Z

AndyMc:

Do you have any recommendations about calibrating the CTs or will transferring the values from the Emonhub config be sufficient?

Calibration depends on the hardware, so if that is the same, and hasn’t got mixed up and moved around, the calibration won’t change.

AndyMc · 2021-03-03T02:48:52.168Z

The only thing that will change is the raspi3+ going to a raspi4.
Thanks Andy

Robert.Wall · 2021-03-03T10:42:37.102Z

There’s no analogue processing inside the Pi, so obviously calibration won’t be affected, provided the maths in emonHub and emonCMS remain the same, i.e. you don’t change the scales in emonHub nor any multiplications or additions in emonCMS.

AndyMc · 2021-03-05T06:39:16.988Z

The new raspi4 combined with the new ssd has brought a new level of performance to emoncms. Emoncms picked up all the emontx’s deployed and the inputs page listed them all in order (first time for that) and only the ones deployed and operating are listed.
I will need to wire up another courtesy plug in the panel to run the USB hub for the ssd and that will make the wiring a little disorganized with lots of wallwarts for 9V ac adapters and 5V usb plugs as well as CT’s running in thru walls. Could you make one 9V AC adapter serve for more than one emontx?

The raspi 3 used to pause and crunch away from time to time but now, with the raspi 4, I have not been able to cause that to happen. If the ssd proves to be more reliable than the SD card then the upgrade will be worth while.

Thanks for your help: Andy

Robert.Wall · 2021-03-05T13:21:19.677Z

AndyMc:

Could you make one 9V AC adapter serve for more than one emontx?

Yes - but only if they also have a 5 V USB supply. And you’re content to measure only the voltage of one leg.

The problem is, if you take too much current from the a.c. adapter, you dent the voltage you’re trying to measure.

If you have a split single-phase N.American supply, I’d recommend one emonTx per leg, each with its own a.c. adapter, but they can share a 5 V USB supply. If you need more than two, then have a 5 V USB supply for all the emonTx’s (you’ll need a lot of emonTx’s to reach the 1 A limit of the adapter) but one a.c. adapter for each leg. And of course open the link to prevent the emonTx powering itself from the a.c. side.

You need to maintain polarity when connecting the barrel plugs - all the outers need to be connected together and all the inners together - otherwise you’ll get a short via the d.c. side!

Bill.Thomson · 2021-03-05T17:34:08.891Z

AndyMc:

If the ssd proves to be more reliable than the SD card then the upgrade will be worth while.

I think you’ll find it’ll last many times longer than a µSD card.

Ref:
https://techreport.com/review/27909/the-ssd-endurance-experiment-theyre-all-dead/