emonSD-17Oct19 Image partition sizes

Just got to this, thanks work!
Sorry for the simple question, but with previous image and a 32GB card i have a nice big partition in pi/home to install home assistant. This time around with latest image, I can’t (don’t need to?) run emonSDexpand command as not found, and I no longer have the full card to put data on - missing 16GB. What have I missed?

From fdisk command (snipped /ramX disks):

Disk /dev/mmcblk0: 29.7 GiB, 31914983424 bytes, 62333952 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x6c586e13

Device         Boot   Start      End  Sectors  Size Id Type
/dev/mmcblk0p1         8192   532479   524288  256M  c W95 FAT32 (LBA)
/dev/mmcblk0p2       532480  8959999  8427520    4G 83 Linux
/dev/mmcblk0p3      8960000 30228479 21268480 10.1G 83 Linux

I’m going to split this question out. [and edited as my sizes were wrong]

The image is optimised for emoncms data going onto an Ext2 partition (to improve life of card). It is also an image designed for a 16Gb card.

sudo parted -l

gives more information.

Yes you can create a partition in the spare space but I’m not exactly sure how. Partly depends what you intend to add to this Pi.

If you use the scripts to install, you can amend the sizes of the resulting partitions so rootfs will be bigger.

I used the Raspbian script for expanding a new card and just added a new partition to it. Currently the start of the Ext2 partition is roughly 4 GB from the start of the card, so following the Pi preparation instruction on a 32GB card would result in a 28GB Ext2 partition. This can be used for other things but it might make more sense if it was 8-10GB from the end - I doubt emoncms data will exceed that - equally it could be an option when installing from the script.

Sadly I can’t remember why I ended up with 8960000 as the figure. I think it needed to be divisible by 1024 (Ext2 block size) so you could start again from scratch (Raspbian Lite image) and at the point where you copy the init_size script, edit the figure to be 26,880,000 as the start figure. I’m sure the rootfs will just fill the space.

@TrystanLea, it might make more sense if it was the Ext2 partition that was a fixed size from the end rather than the beginning.

Just looking at my card (an image I burned), the size of the Ext2 partition is 21268480 sectors ~10.1GB. I’ll use that.

That line I think would therefore be Card_Size - 10GB - 1 so

TARGET_END=$((ROOT_DEV_SIZE -  21268480 - 1))

@Tockley - you could try this if you like…

One other thought, there probably should be a check on card size during this process. You could run Emoncms on either a 4GB or 8GB card in reality so perhaps the init script should take account of that (future dev). Equally the documentation should state a minimum 16GB card for this image and the Scripted install (PR done).

1 Like

I’ll add that I’m not convinced of my numbers and if someone with more knowledge of sectors/blocks etc wants to chip in, feel free.

Thanks for the detail @borpin! I’ve ignored the skipped space for now, and gone with a docker & HA install in the new emoncms partition… 10GB is a fair amount of space still obviously. :wink:

All running ok, zero issues with the emonPi reflash, install, connection, etc. As always, ran all the latest pi and raspbian update gets (update, upgrade, dist) and no issues seen. Backup restore worked flawlessly and I was back up and running my solar graphing super quickly. Maybe I’m also well practiced!

1 Like

I love HA!

I have had a chance to try this out and the sizes above are not correct. This added to the init_resize.sh script will work.

#EmonSD Fix end of Rootfs
if [ $ROOT_DEV_SIZE -gt 16000000 ]; then
  #16GB card or above, assign last 10GB for data 
  TARGET_END=$(($ROOT_DEV_SIZE - 20972544 - 1 ))
elif [ $ROOT_DEV_SIZE -gt 8000000 ]; then
  #8GB card, assign last 4GB for data 
  TARGET_END=$(($ROOT_DEV_SIZE - 7341056 - 1 ))
fi

The original sizes I proposed did not align optimally.

For future ref, the key figure is 2048. parted wants the start of a partition to start on a multiple of 2048. Because of the calculation, and the fact the whole SD card is not a multiple of 2048, this caused a problem as the rootfs did not finish on a 2048 boundary.

The Ext2 partition is not a multiple of 2048 (it doesn’t need to be).

I am, however, not convinced this is robust for all SD cards - it assumes that the whole card is not a multiple of 2048 - I hate assumptions!

Can anyone suggest a way in bash to check the alignment figure?

1 Like

Stupid question I am sure, but here goes!
I am using previous SD Oct18, I can update to latest but what advantage would I get from this? Everything running fine at the moment.
Thanks

No such thing as a stupid question IMHO.

There are 2 levels of update. Going into the Admin page and clicking ‘update’ will give you the most recent stable release of Emoncms and the installed Modules.

The new image has introduced some changes (such as folder structure), improvements (such as log file management) and is based on the most recent Raspbian release of ‘Buster’. The value of this update, however, is up to yourself.

It is reasonably simple to move to the updated image. You can use the ‘Backup’ feature to export all your settings etc which can be imported then to the new image.

Just thought it was worth posting a quick update on this release. I haven’t done anything about the partition sizes, but everything working beautifully. Good, solid release guys - ticking along very nicely, well done yet again! :grinning:

2 Likes

Just thought i would post a link here also

can/dev/mmcblk0p3 not be mounted to var/opt and not var/opt/emoncms

The whole point of the partiton is for the data regularly written. The problem are the limits set for PHP file upload IIRC.

I do think the rootfs is too small in relation to the data partition, just can’t find the time to get the setup right.

I thought the upgrade would take about 2 hours, I have spent most of the day trying to get this to work. I feel like the 17Oct19 needs to back to beta and have the partition table sorted, why mess about with 16gb when 32/64gb cards are the norm.

I think we could put the backup upload in var/opt/emoncms/backup which would not need a partition table modification - but should give us what we need.

1 Like

I would disagree with that. 16GB is plenty big enough unless your data capture is unusual.

Yup i am unusual.

Server Information

Server Information

Services

  • emonhub :- Active Running
  • emoncms_mqtt :- Active Running
  • feedwriter :- Active Running - sleep 60s 82 feed points pending write
  • service-runner :- Active Running
  • emonPiLCD :- Active Running
  • redis-server :- Active Running
  • mosquitto :- Active Running

Emoncms

Server

  • OS :- Linux 4.14.71-v7+
  • Host :- emonpi | emonpi | (192.168.1.215)
  • Date :- 2020-01-12 21:27:29 UTC
  • Uptime :- 21:27:29 up 3:07, 0 users, load average: 0.54, 0.59, 0.52

Memory

  • RAM :- Used: 20.42%
    • Total :- 976.74 MB
    • Used :- 199.43 MB
    • Free :- 777.3 MB
  • Swap :- Used: 0.00%
    • Total :- 100 MB
    • Used :- 0 B
    • Free :- 100 MB
      Write Load Period

Disk

  • / :- Used: 42.97%
    • Total :- 3.81 GB
    • Used :- 1.64 GB
    • Free :- 2 GB
    • Write Load :- 449.03 KB/s (2 mins)
  • /boot :- Used: 51.69%
    • Total :- 42.52 MB
    • Used :- 21.98 MB
    • Free :- 20.54 MB
    • Write Load :- 0 B/s (2 mins)
  • /home/pi/data :- Used: 32.49%
    • Total :- 54 GB
    • Used :- 17.54 GB
    • Free :- 33.71 GB
    • Write Load :- 2.94 KB/s (2 mins)

HTTP

  • Server :- Apache/2.4.25 (Raspbian) HTTP/1.1 CGI/1.1 80

MySQL

  • Version :- 5.5.5-10.1.23-MariaDB-9+deb9u1
  • Host :- localhost:6379 (127.0.0.1)
  • Date :- 2020-01-12 21:27:29 (UTC 00:00‌​)
  • Stats :- Uptime: 11230 Threads: 3 Questions: 124993 Slow queries: 0 Opens: 31 Flush tables: 1 Open tables: 25 Queries per second avg: 11.130

Redis

  • Version :-
    • Redis Server :- 3.2.6
    • PHP Redis :- 4.1.1
  • Host :- localhost:6379
  • Size :- 649 keys (928.09K)
  • Uptime :- 0 days

MQTT Server

  • Version :- Mosquitto 1.4.10
  • Host :- localhost:1883 (127.0.0.1)

PHP

  • Version :- 7.0.30-0+deb9u1 (Zend Version 3.0.0)
  • Modules :- apache2handler | calendar v7.0.30-0+deb9u1 | Core v7.0.30-0+deb9u1 | ctype v7.0.30-0+deb9u1 | curl v7.0.30-0+deb9u1 | date v7.0.30-0+deb9u1 | dom v20031129 | exif v7.0.30-0+deb9u1 | fileinfo v1.0.5 | filter v7.0.30-0+deb9u1 | ftp v7.0.30-0+deb9u1 | gd v7.0.30-0+deb9u1 | gettext v7.0.30-0+deb9u1 | hash v1.0 | iconv v7.0.30-0+deb9u1 | igbinary v2.0.1 | json v1.4.0 | libxml v7.0.30-0+deb9u1 | mbstring v7.0.30-0+deb9u1 | mcrypt v7.0.30-0+deb9u1 | mosquitto v0.4.0 | mysqli v7.0.30-0+deb9u1 | mysqlnd vmysqlnd 5.0.12-dev - 20150407 - $Id: b5c5906d452ec590732a93b051f3827e02749b83 $ | openssl v7.0.30-0+deb9u1 | pcre v7.0.30-0+deb9u1 | PDO v7.0.30-0+deb9u1 | pdo_mysql v7.0.30-0+deb9u1 | Phar v2.0.2 | posix v7.0.30-0+deb9u1 | readline v7.0.30-0+deb9u1 | redis v4.1.1 | Reflection v7.0.30-0+deb9u1 | session v7.0.30-0+deb9u1 | shmop v7.0.30-0+deb9u1 | SimpleXML v7.0.30-0+deb9u1 | sockets v7.0.30-0+deb9u1 | SPL v7.0.30-0+deb9u1 | standard v7.0.30-0+deb9u1 | sysvmsg v7.0.30-0+deb9u1 | sysvsem v7.0.30-0+deb9u1 | sysvshm v7.0.30-0+deb9u1 | tokenizer v7.0.30-0+deb9u1 | wddx v7.0.30-0+deb9u1 | xml v7.0.30-0+deb9u1 | xmlreader v7.0.30-0+deb9u1 | xmlwriter v7.0.30-0+deb9u1 | xsl v7.0.30-0+deb9u1 | Zend OPcache v7.0.30-0+deb9u1 | zlib v7.0.30-0+deb9u1

Pi

  • Model :- Raspberry Pi 3 Model B Rev 1.2 - 1GB (Sony UK)

  • Serial num. :- 43DDB2EE

  • Temperature :- 63.91°C - 63.4°C

  • emonpiRelease :- emonSD-30Oct18

  • File-system :- read-write

Client Information

Client Information

HTTP

  • Browser :- Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:72.0) Gecko/20100101 Firefox/72.0
  • Language :- en-US,en;q=0.5

Window

  • Size :- 1903 x 966

Screen

  • Resolution :- 1920 x 1080

Fair enough, but that is exactly why for most users 16GB is fine and there is a mechanism to expand to larger SD Cards. Moving the folder the backup uses is a better solution than changing the mount point.

@borpin

Expanding the 16GB image to fill a 32GB SDHC
This may/may not help but what I’ve done is …

Do   sudo parted
Then do    print
This will show #3 as the Ext2 data partition which is to be expanded.
Then do   resizepart    and respond with   3
Respond   Yes  to the question about the partition already being mounted 
At the prompt   End? (15.5GB)?  … enter the new ending point in MB …
I used    31600
Verify the change has been made by doing    print
Finish by doing    quit
Finally expand the Ext2 file system to use its newly enlarged partition …
Do   sudo resize2fs /dev/mmcblk0p3
Verify all is OK by doing     df -hT

A few days ago, Amazon(UK) had SanDisk Ultra Class 10 32 GB SDHC’s that were a few pennies less expensive than the 16 GB version.

1 Like