Community
OpenEnergyMonitor

Community

Increase emonSD pre-built SD card to 8GB min

Tags: #<Tag:0x00007f6e13df7a18>

Currently emonSD-03May16 (just!) fits on a 4GB SD card. This only leaves 40Mb free in the data partition for data logging and only 280Mb free in the root Linux partition.

All SD cards we ship via the shop including pre-installed in all emonPi and emonBase include an 8GB SD card with the data partition expanded to fill the card.

I’m working on the next release and considering just moving to an 8GB min SD card size. I think given the low cost of SD cards (in bulk 8GB cards were cheaper than 4GB!) I would not recommend anyone uses a 4GB card anymore. Using a 4GB card will just create issues with running out of space in the near future.
.
Do you think this is a reasonable move?

The extra space will create space for future dist updates and allow platformIO (for onboard firmware compiling and uploading) and HomeAssistant to be installed and configured.

2 Likes

Smart move. Having a only 40mb for space leaves very little room for data storage. I know as for me if I were setup a raspberry pi, I would like to set it up, and not worry about running out of space within a month. And have to worry about moving the data etc. Given the cost of the SD cards, has fallen dramatically in the recent years, worth the move. Having 4gb of left over space for data will be enough to trend for a long time.

I have my raspberry pi 3 coming in today, and will be using a 32gb card.

1 Like

Yes - 8GB min SD card size sounds good to me. HomeAssistant is on my “to-do” list so having that installed sounds great.

Can the root Linux partition be increased (at the expense of data)? I seem to run out of space in the root area fairly quick.

Yes, given a 8GB card we can afford to have a larger root partition as 4GB of data is a hell of a lot of emoncms logging! Given an 8GB card I wonder how much we should allocate to the root?

This is where I am at today after starting with a new emonSD-03May16 image on Sep 23 and then adding in some other things.

[email protected]:/ $ df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/root       3.4G  2.3G  895M  73% /
devtmpfs        483M     0  483M   0% /dev
tmpfs           487M     0  487M   0% /dev/shm
tmpfs           487M   50M  438M  11% /run
tmpfs           5.0M  4.0K  5.0M   1% /run/lock
tmpfs           487M     0  487M   0% /sys/fs/cgroup
tmpfs           1.0M  4.0K 1020K   1% /var/lib/dhcpcd5
tmpfs            40M  6.6M   34M  17% /var/lib/openhab
tmpfs           1.0M     0  1.0M   0% /var/lib/dhcp
/dev/mmcblk0p1   60M   20M   40M  34% /boot
tmpfs            50M  9.5M   41M  19% /var/log
tmpfs            30M   52K   30M   1% /tmp
/dev/mmcblk0p3  3.8G  499M  3.1G  14% /home/pi/data
[email protected]:/ $ 

I’d be happy with +1 GB for root (and -1GB for data).

What does a emonSD-03May16 image on a blank 8GB SD look like?

or maybe the better question: what does the new emonSD-Oct2016 image (with platformIO and HomeAssistant) on a blank 8GB SD look like?

If moving to a 8GB, why not use the full Jessie OS, instead of the cutdown version?

It would future proof the installation, especially as additional future packages may need the wider dependencies, and give users the full flexibility of the OS.

Apart from savings on disk space (which would no longer be an issue), the other benefits of using the cutdown OS are anecdotal, which are easily outweighed by the benefits of using the full Jessie OS.

Paul
@glyn.hudson - I’ll bet that you were anticipating this comment!!

Hahah yes, I did expect that comment! I’m sure you will also be expecting my reply :wink:

IMHO it’s unnecessary to run a full desktop OS on a headless box.

As for future proofness, Jessie lite is officially released and supported by the Raspbian team and can access any Jessie package from the repositories.

I’m not planning on doing a full image re-build this time, just an interim release pulling in all the latest security updates, fixes and possible adding a couple of new features.

If a user wishes Jessie Lite can be ‘upgraded’ to full Jessie by installing the following:

adwaita-icon-theme alacarte alsa-base aspell aspell-en blt bluej claws-mail
claws-mail-i18n coinor-libcbc3 coinor-libcgl1 coinor-libclp1
coinor-libcoinmp1:armhf coinor-libcoinutils3 coinor-libosi1 cryptsetup-bin
cups-bsd cups-client cups-common dbus-x11 dconf-gsettings-backend:armhf
dconf-service debian-reference-common debian-reference-en desktop-base
desktop-file-utils dh-python dictionaries-common dillo eject emacsen-common
epiphany-browser epiphany-browser-data esound-common fontconfig
fontconfig-config fonts-dejavu fonts-dejavu-core fonts-dejavu-extra
fonts-freefont-ttf fonts-opensymbol fonts-roboto fonts-sil-gentium-basic
freepats fuse galculator gconf-service gconf2 gconf2-common gdebi-core
gettext-base giblib1:armhf gir1.2-atk-1.0 gir1.2-freedesktop:armhf
gir1.2-gdkpixbuf-2.0 gir1.2-glib-2.0:armhf gir1.2-gmenu-3.0
gir1.2-gtk-3.0:armhf gir1.2-pango-1.0:armhf git git-core git-man gksu
glib-networking:armhf glib-networking-common glib-networking-services
gnome-desktop3-data gnome-

from https://www.raspberrypi.org/forums/viewtopic.php?t=145665

1 Like

PlatformIO is installed but not Home Assistant yet. Just re-partitioned and this is how things look like on an 8GB SD. Plenty of space :slight_smile:

[email protected]:~ $ df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/root       4.9G  2.9G  1.8G  63% /
devtmpfs        483M     0  483M   0% /dev
tmpfs           487M     0  487M   0% /dev/shm
tmpfs           487M  6.6M  480M   2% /run
tmpfs           5.0M  4.0K  5.0M   1% /run/lock
tmpfs           487M     0  487M   0% /sys/fs/cgroup
tmpfs            40M  6.6M   34M  17% /var/lib/openhab
tmpfs           1.0M  4.0K 1020K   1% /var/lib/dhcpcd5
tmpfs           1.0M     0  1.0M   0% /var/lib/dhcp
tmpfs            50M  800K   50M   2% /var/log
tmpfs            30M   40K   30M   1% /tmp
/dev/mmcblk0p1   60M   21M   40M  35% /boot
/dev/mmcblk0p3  1.9G  171M  1.7G  10% /home/pi/data

@glyn.hudson - make sure the pre-release built is clean;

“apt-get clean all” - for example this cleaned up ~700MB of apt cache on my system.
I’m currently looking at the rest of the file system to see what else can be cleaned up :slight_smile:

/var/swap exists in the image - despite swap being off, so thats another ~100MB to clean up.

Nice find,

Does the just clean up remove unused tempory package files. Is there no detrimental effects? Why is clean no run automatically as part of the linux package instal?

I’ve added it onto my next image ‘to do’ list:

This may help answer your question:

1 Like

Packages are NOT removed, only cached information (and therefore cached copies of installed applications).
Since the installed base isn’t likely to change - I would encourage you to “apt-get clean all” to remove the crap.

IF you want to remove installed but un-used applications, that should be "apt-get autoremove --purge"
However your system is already clean;

[email protected](rw):~$ sudo apt-get autoremove --purge
Reading package lists... Done
Building dependency tree        
Reading state information... Done
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.

Another nice addition - you might notice my prompt that tells me when the filesystem is “ro” or “rw” - add the following code to the end of /etc/bash.bashrc

set_bash_prompt(){
    fs_mode=$(mount | sed -n -e "s/^\/dev\/mmcblk0p2 on \/ .*(\(r[w|o]\).*/\1/p")

    if [ $(id -u) -eq 0 ];
      then
        PS1='\[\033[01;31m\]\[email protected]\h${fs_mode:+($fs_mode)}\[\033[00m\]:\[\033[01;34m\]\W\[\033[00m\]\$ '
      else
        PS1='\[\033[01;32m\]\[email protected]\h${fs_mode:+($fs_mode)}\[\033[00m\]:\[\033[01;34m\]\W\[\033[00m\]\$ '
    fi
}

# setup fancy prompt
PROMPT_COMMAND=set_bash_prompt

This also turns the whole prompt RED if you are root :slight_smile:
Hope those help.

2 Likes

Post clean-up scores…

[email protected](ro):~$ df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/root       3.4G  1.5G  1.8G  46% /
1 Like

Wow, that’s really useful! I’ve lost count of the number of times I edit a file with nano but then unable to save changes since I forgot to toggle RW. Nice work, I’ll make sure that get’s into the next release :thumbsup:

I guess it might also be possible to add it to existing systems via emonpi update…

1 Like

Of course it is.

As for the filesystem sizing, you could (now that we have a proven way to clean up) actually reduce your root volume to 3G leaving another 400MB for logging data.

Not sure what you use for closing the SD card image, but the way I do it is to use a modified version of this;
https://raw.githubusercontent.com/billw2/rpi-clone/master/rpi-clone

My modifications include creating specifically sized partitions (rather than filling the target SD card) and writing zeros (from /dev/zero) in all of the free space, so that when I then pull the image back, truncate it and compress it - I get a SERIOUS compression advantage (because compressed zeros take up zero space…)

Hope those tips help.

Nice, thanks for the tips. That script looks useful. I have been using dd after removing the SD card from the Pi and Gparted if partitions need editing.

So I use that to clone the working system onto a new SD card, mine is modified to not copy things like .bash_history etc, so I have some added excludes (you can find these easily enough)

        --exclude '/var/log' \
        --exclude '/home/pi/.ssh' \
        --exclude '/home/pi/.bash_history' \
        --exclude '/root/.bash_history' \

Then at the end - before the target is unmounted;

mkdir -p $CLONE/var/log
#
# Clean the disk (full the partition with zeros to help with later compression)
#
echo ""
echo "*** Writing zero's to the remaining disk space, this will take a few mins..."
echo ""

dd if=/dev/zero of=$CLONE/large.file bs=1M count=2048
sleep 5

rm -rf $CLONE/large.file

Once the image write has completed, I then use DD to pull the whole image back, and then use truncate to rip the image down to only the used space on the SD Card;
Read this: https://evilshit.wordpress.com/2014/02/07/how-to-trim-disk-images-to-partition-size/

Now you have your collapsed SD Card image, you can zip that as normal and it should get significantly reduced in size.

I should have added this part too, this is where i set the size of the target root filesystem;

# Borrowed from do_expand_rootfs in raspi-config
expand_rootfs()
        {
        # Get the starting offset of the root partition
        # (with Jessie's parted, now need to strip trailing 's' from PART_START)
        PART_START=$(parted /dev/mmcblk0 -ms unit s p  \
                        | grep "^2" | cut -f 2 -d: | cut -f 1 -d s)
        [ "$PART_START" ] || return 1
        # Return value will likely be error for fdisk as it fails to reload the
        # partition table because the root fs is mounted
        fdisk /dev/$DST_DISK > /dev/null <<EOF
p
d
2
n
p
2
$PART_START
+1780M
p
w
q
EOF
        }

You will need to modify this to set the root partition size AND to create the /home/pi/data partition too

That’s triggered something I’ve wanted to ask for ages, but never got around to.

Is there a good reason why there are two separate scripts, one for “ro” and one for “rw”, instead of one that takes “-ro” and “-rw” as parameters?

Then, the command becomes syntactically the same as everything else that takes options.

Adding this to the /etc/bash.bashrc will remove the need for the files altogether :slight_smile:

# Aliases to control re-mounting
alias rpi-ro='sudo mount -o remount,ro / ; sudo mount -o remount,ro /boot'
alias rpi-rw='sudo mount -o remount,rw / ; sudo mount -o remount,rw /boot'

I should point out that if you have other scripts that rely on running ‘rpi-ro’ and or ‘rpi-rw’ - using aliases is probably not a good idea…