emonSD-17Oct19 Image partition sizes

Tockley · 2019-10-26T05:14:01.213Z

Just got to this, thanks work!
Sorry for the simple question, but with previous image and a 32GB card i have a nice big partition in pi/home to install home assistant. This time around with latest image, I can’t (don’t need to?) run emonSDexpand command as not found, and I no longer have the full card to put data on - missing 16GB. What have I missed?

image1265×466 24.8 KB

From fdisk command (snipped /ramX disks):

Disk /dev/mmcblk0: 29.7 GiB, 31914983424 bytes, 62333952 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x6c586e13

Device         Boot   Start      End  Sectors  Size Id Type
/dev/mmcblk0p1         8192   532479   524288  256M  c W95 FAT32 (LBA)
/dev/mmcblk0p2       532480  8959999  8427520    4G 83 Linux
/dev/mmcblk0p3      8960000 30228479 21268480 10.1G 83 Linux

borpin · 2019-10-26T11:32:17.031Z

I’m going to split this question out. [and edited as my sizes were wrong]

The image is optimised for emoncms data going onto an Ext2 partition (to improve life of card). It is also an image designed for a 16Gb card.

sudo parted -l

gives more information.

Yes you can create a partition in the spare space but I’m not exactly sure how. Partly depends what you intend to add to this Pi.

If you use the scripts to install, you can amend the sizes of the resulting partitions so rootfs will be bigger.

I used the Raspbian script for expanding a new card and just added a new partition to it. Currently the start of the Ext2 partition is roughly 4 GB from the start of the card, so following the Pi preparation instruction on a 32GB card would result in a 28GB Ext2 partition. This can be used for other things but it might make more sense if it was 8-10GB from the end - I doubt emoncms data will exceed that - equally it could be an option when installing from the script.

github.com

openenergymonitor/EmonScripts/blob/79dc389c8427180c5049d247c883d7f5a4ae28cf/install/init_resize.sh#L66-L66

    
      
          TARGET_END=$((8960000 - 1))

Sadly I can’t remember why I ended up with 8960000 as the figure. I think it needed to be divisible by 1024 (Ext2 block size) so you could start again from scratch (Raspbian Lite image) and at the point where you copy the init_size script, edit the figure to be 26,880,000 as the start figure. I’m sure the rootfs will just fill the space.

@TrystanLea, it might make more sense if it was the Ext2 partition that was a fixed size from the end rather than the beginning.

Just looking at my card (an image I burned), the size of the Ext2 partition is 21268480 sectors ~10.1GB. I’ll use that.

That line I think would therefore be Card_Size - 10GB - 1 so

TARGET_END=$((ROOT_DEV_SIZE -  21268480 - 1))

@Tockley - you could try this if you like…

One other thought, there probably should be a check on card size during this process. You could run Emoncms on either a 4GB or 8GB card in reality so perhaps the init script should take account of that (future dev). Equally the documentation should state a minimum 16GB card for this image and the Scripted install (PR done).

borpin · 2019-10-26T12:10:39.469Z

I’ll add that I’m not convinced of my numbers and if someone with more knowledge of sectors/blocks etc wants to chip in, feel free.

Tockley · 2019-10-26T23:33:47.582Z

Thanks for the detail @borpin! I’ve ignored the skipped space for now, and gone with a docker & HA install in the new emoncms partition… 10GB is a fair amount of space still obviously.

All running ok, zero issues with the emonPi reflash, install, connection, etc. As always, ran all the latest pi and raspbian update gets (update, upgrade, dist) and no issues seen. Backup restore worked flawlessly and I was back up and running my solar graphing super quickly. Maybe I’m also well practiced!

borpin · 2019-10-27T09:01:24.319Z

Tockley:

HA

I love HA!

borpin · 2019-10-28T10:10:41.351Z

I have had a chance to try this out and the sizes above are not correct. This added to the init_resize.sh script will work.

#EmonSD Fix end of Rootfs
if [ $ROOT_DEV_SIZE -gt 16000000 ]; then
  #16GB card or above, assign last 10GB for data 
  TARGET_END=$(($ROOT_DEV_SIZE - 20972544 - 1 ))
elif [ $ROOT_DEV_SIZE -gt 8000000 ]; then
  #8GB card, assign last 4GB for data 
  TARGET_END=$(($ROOT_DEV_SIZE - 7341056 - 1 ))
fi

The original sizes I proposed did not align optimally.

For future ref, the key figure is 2048. parted wants the start of a partition to start on a multiple of 2048. Because of the calculation, and the fact the whole SD card is not a multiple of 2048, this caused a problem as the rootfs did not finish on a 2048 boundary.

The Ext2 partition is not a multiple of 2048 (it doesn’t need to be).

I am, however, not convinced this is robust for all SD cards - it assumes that the whole card is not a multiple of 2048 - I hate assumptions!

Can anyone suggest a way in bash to check the alignment figure?

github.com/openenergymonitor/EmonScripts

Review partition sizes

opened 10:10AM - 28 Oct 19 UTC

borpin

enhancement

re PR #66 (documentation) re Issue #58 https://community.openenergymonitor.…org/t/emonsd-17oct19-image-partition-sizes/12282/ The current script/image fixes the size of the `rootfs` and only works for 16GB cards. Suggest the script is amended to fix the Ext2 partition to the end of the card at a max size. ```shell #EmonSD Fix end of Rootfs if [ $ROOT_DEV_SIZE -gt 16000000 ]; then #16GB card or above, assign last 10GB for data TARGET_END=$(($ROOT_DEV_SIZE - 20972544 - 1 )) elif [ $ROOT_DEV_SIZE -gt 8000000 ]; then #8GB card, assign last 4GB for data TARGET_END=$(($ROOT_DEV_SIZE - 7341056 - 1 )) fi ``` More discussion on community thread. I'm not certain this is a robust solution.

FatJuan · 2019-10-31T09:48:02.730Z

Stupid question I am sure, but here goes!
I am using previous SD Oct18, I can update to latest but what advantage would I get from this? Everything running fine at the moment.
Thanks

borpin · 2019-10-31T12:02:40.547Z

FatJuan:

Stupid question I am sure

No such thing as a stupid question IMHO.

FatJuan:

I am using previous SD Oct18, I can update to latest but what advantage would I get from this?

There are 2 levels of update. Going into the Admin page and clicking ‘update’ will give you the most recent stable release of Emoncms and the installed Modules.

The new image has introduced some changes (such as folder structure), improvements (such as log file management) and is based on the most recent Raspbian release of ‘Buster’. The value of this update, however, is up to yourself.

It is reasonably simple to move to the updated image. You can use the ‘Backup’ feature to export all your settings etc which can be imported then to the new image.

Tockley · 2019-11-05T10:39:17.728Z

Just thought it was worth posting a quick update on this release. I haven’t done anything about the partition sizes, but everything working beautifully. Good, solid release guys - ticking along very nicely, well done yet again!

Pukka · 2020-01-12T16:47:42.095Z

Just thought i would post a link here also

Backup and restore to lastest EmonPI Image Emoncms

I have taken a backup its 505mb I loaded the image to a 16gb card and tried a restore got an error back about no disk Cannot write: No space left on device. I found a 64gb card and ran sudo /opt/emoncms/modules/usefulscripts/sdpart/./sdpart_imagefile . waited 30 min the partition /var/opt/emoncms now shows 50.55gb free. but the restore still fails. it looks like the restore uses /opt/openenergymonitor/data/upload I dont know how to fix this. server info below
Server Information
<a name=…

can/dev/mmcblk0p3 not be mounted to var/opt and not var/opt/emoncms

borpin · 2020-01-12T16:52:30.870Z

Pukka:

can/dev/mmcblk0p3 not be mounted to var/opt and not var/opt/emoncms

The whole point of the partiton is for the data regularly written. The problem are the limits set for PHP file upload IIRC.

I do think the rootfs is too small in relation to the data partition, just can’t find the time to get the setup right.

Pukka · 2020-01-12T17:28:25.005Z

I thought the upgrade would take about 2 hours, I have spent most of the day trying to get this to work. I feel like the 17Oct19 needs to back to beta and have the partition table sorted, why mess about with 16gb when 32/64gb cards are the norm.

TrystanLea · 2020-01-12T19:14:40.929Z

I think we could put the backup upload in var/opt/emoncms/backup which would not need a partition table modification - but should give us what we need.

borpin · 2020-01-12T19:36:10.921Z

Pukka:

why mess about with 16gb when 32/64gb cards are the norm.

I would disagree with that. 16GB is plenty big enough unless your data capture is unusual.

Pukka · 2020-01-12T21:21:57.900Z

Yup i am unusual.

image1670×640 20.7 KB

Server Information

Services

emonhub :- Active Running
emoncms_mqtt :- Active Running
feedwriter :- Active Running - sleep 60s 82 feed points pending write
service-runner :- Active Running
emonPiLCD :- Active Running
redis-server :- Active Running
mosquitto :- Active Running

Emoncms

Version :- low-write 10.1.13
Modules :- Administration | App v2.0.9 | Backup v2.2.0 | EmonHub Config v2.0.4 | Dashboard v2.0.5 | Device v2.0.3 | EventProcesses | Feed | Graph v2.0.7 | Input | Postprocess v2.1.2 | CoreProcess | Schedule | Network Setup v1.0.0 | sync | Time | User | Visualisation | WiFi v2.0.2
Git :-
- URL :- GitHub - emoncms/emoncms: Web-app for processing, logging and visualising energy, temperature and other environmental data
- Branch :- * stable
- Describe :- 10.1.11-7-g90f39158

Server

OS :- Linux 4.14.71-v7+
Host :- emonpi | emonpi | (192.168.1.215)
Date :- 2020-01-12 21:27:29 UTC
Uptime :- 21:27:29 up 3:07, 0 users, load average: 0.54, 0.59, 0.52

Memory

RAM :- Used: 20.42%
- Total :- 976.74 MB
- Used :- 199.43 MB
- Free :- 777.3 MB
Swap :- Used: 0.00%
- Total :- 100 MB
- Used :- 0 B
- Free :- 100 MB
  Write Load Period

Disk

/ :- Used: 42.97%
- Total :- 3.81 GB
- Used :- 1.64 GB
- Free :- 2 GB
- Write Load :- 449.03 KB/s (2 mins)
/boot :- Used: 51.69%
- Total :- 42.52 MB
- Used :- 21.98 MB
- Free :- 20.54 MB
- Write Load :- 0 B/s (2 mins)
/home/pi/data :- Used: 32.49%
- Total :- 54 GB
- Used :- 17.54 GB
- Free :- 33.71 GB
- Write Load :- 2.94 KB/s (2 mins)

HTTP

Server :- Apache/2.4.25 (Raspbian) HTTP/1.1 CGI/1.1 80

MySQL

Version :- 5.5.5-10.1.23-MariaDB-9+deb9u1
Host :- localhost:6379 (127.0.0.1)
Date :- 2020-01-12 21:27:29 (UTC 00:00‌)
Stats :- Uptime: 11230 Threads: 3 Questions: 124993 Slow queries: 0 Opens: 31 Flush tables: 1 Open tables: 25 Queries per second avg: 11.130

Redis

Version :-
- Redis Server :- 3.2.6
- PHP Redis :- 4.1.1
Host :- localhost:6379
Size :- 649 keys (928.09K)
Uptime :- 0 days

MQTT Server

Version :- Mosquitto 1.4.10
Host :- localhost:1883 (127.0.0.1)

PHP

Version :- 7.0.30-0+deb9u1 (Zend Version 3.0.0)
Modules :- apache2handler | calendar v7.0.30-0+deb9u1 | Core v7.0.30-0+deb9u1 | ctype v7.0.30-0+deb9u1 | curl v7.0.30-0+deb9u1 | date v7.0.30-0+deb9u1 | dom v20031129 | exif v7.0.30-0+deb9u1 | fileinfo v1.0.5 | filter v7.0.30-0+deb9u1 | ftp v7.0.30-0+deb9u1 | gd v7.0.30-0+deb9u1 | gettext v7.0.30-0+deb9u1 | hash v1.0 | iconv v7.0.30-0+deb9u1 | igbinary v2.0.1 | json v1.4.0 | libxml v7.0.30-0+deb9u1 | mbstring v7.0.30-0+deb9u1 | mcrypt v7.0.30-0+deb9u1 | mosquitto v0.4.0 | mysqli v7.0.30-0+deb9u1 | mysqlnd vmysqlnd 5.0.12-dev - 20150407 - $Id: b5c5906d452ec590732a93b051f3827e02749b83 $ | openssl v7.0.30-0+deb9u1 | pcre v7.0.30-0+deb9u1 | PDO v7.0.30-0+deb9u1 | pdo_mysql v7.0.30-0+deb9u1 | Phar v2.0.2 | posix v7.0.30-0+deb9u1 | readline v7.0.30-0+deb9u1 | redis v4.1.1 | Reflection v7.0.30-0+deb9u1 | session v7.0.30-0+deb9u1 | shmop v7.0.30-0+deb9u1 | SimpleXML v7.0.30-0+deb9u1 | sockets v7.0.30-0+deb9u1 | SPL v7.0.30-0+deb9u1 | standard v7.0.30-0+deb9u1 | sysvmsg v7.0.30-0+deb9u1 | sysvsem v7.0.30-0+deb9u1 | sysvshm v7.0.30-0+deb9u1 | tokenizer v7.0.30-0+deb9u1 | wddx v7.0.30-0+deb9u1 | xml v7.0.30-0+deb9u1 | xmlreader v7.0.30-0+deb9u1 | xmlwriter v7.0.30-0+deb9u1 | xsl v7.0.30-0+deb9u1 | Zend OPcache v7.0.30-0+deb9u1 | zlib v7.0.30-0+deb9u1

Pi

Model :- Raspberry Pi 3 Model B Rev 1.2 - 1GB (Sony UK)
Serial num. :- 43DDB2EE
Temperature :- 63.91°C - 63.4°C
emonpiRelease :- emonSD-30Oct18
File-system :- read-write

Client Information

HTTP

Browser :- Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:72.0) Gecko/20100101 Firefox/72.0
Language :- en-US,en;q=0.5

Window

Size :- 1903 x 966

Screen

Resolution :- 1920 x 1080

borpin · 2020-01-12T21:33:55.593Z

Pukka:

Yup i am unusual.

Fair enough, but that is exactly why for most users 16GB is fine and there is a mechanism to expand to larger SD Cards. Moving the folder the backup uses is a better solution than changing the mount point.

johnbanks · 2020-01-13T22:32:02.882Z

@borpin

Expanding the 16GB image to fill a 32GB SDHC
This may/may not help but what I’ve done is …

Do   sudo parted
Then do    print
This will show #3 as the Ext2 data partition which is to be expanded.
Then do   resizepart    and respond with   3
Respond   Yes  to the question about the partition already being mounted 
At the prompt   End? (15.5GB)?  … enter the new ending point in MB …
I used    31600
Verify the change has been made by doing    print
Finish by doing    quit
Finally expand the Ext2 file system to use its newly enlarged partition …
Do   sudo resize2fs /dev/mmcblk0p3
Verify all is OK by doing     df -hT

A few days ago, Amazon(UK) had SanDisk Ultra Class 10 32 GB SDHC’s that were a few pennies less expensive than the 16 GB version.