HOWTO: TrueNAS Scale: from 32gb USB to booting from mirrored larger enterprise SSD's, create add'l partition/data pool from remaining free space

[NovaSaltado]

The inspiration for this post, which was theoretical and helped guide me towards a working solution: https://www.truenas.com/community/t...sds-with-boot-and-data-pools-for-scale.89428/

FLUFF
My previous setup was an R720XD running ESXI 7.0.3 from mirrored SDCards. ESXI VM's would run from a mirrored SSD's (using onboard RAID card) and the HBA passed through to TrueNAS Core with the HDD's.

My new server is an R730XD. Which cannot easily connect the main backplane to an HBA and connect the flexbay with the onboard raid card.

Original requirements based on my R720xd:

• Run hypervisor (esxi/proxmox) from usb
• Setup 2x enterprise SSD's for VM's including TrueNAS Scale
• passthrough HDD's to TrueNAS and setup the main zpool

/FLUFF


Adjust requirements:

note: ESXI is no longer recommended to run from USB​

note: Proxmox is not recommended to run from USB/Flash​

note: When passing through disks in Proxmox, TrueNAS cannot interact with SMART to view data or run tests (this is bad)​

• TrueNAS Scale on baremetal, installed on enterprise mirrored SSD's *only using roughly 40GB)
• Remaining SSD space for VM's
• TrueNAS Scale interacting with SMART

Process Overview:

1. Install TrueNAS Scale on 32GB SanDisk Ultra Fit (sector size 512)
2. boot into TrueNAS Scale on the USB drive
3. mirror USB partition table to enterprise SSD's (sector size 4096)
4. Setup 4th partition on SSD's from remaining space (OPTIONAL: if you don't care to use your remaining free space)

Code:

# determine the drives you want to work on ie /dev/<drive>
# you could also remove your HDD's to shorten the list
lsblk -l

# set vars
# omit the /dev/ as the script below accounts for that.
bootdisk=sds
ssd1=sdl
ssd2=sdm
# optional: 4th partition
partitionpool=vm-pool

# we're dumping the current bootdrive partition data
# to a file, in the event you want to change sectors e.g. 512->4096
sfdisk --dump /dev/${bootdisk} > original.boot.disk.part.dump

###########################
# OPTIONAL if going to 4k sector size and/or want to use remaining free space as an additional pool
# USB sectors sizee 512 -> enterprise SSD sector size 4096
#
# 1. Dump current ssd partition table to grab last-lba for SSD
#
# sfdisk --dump /dev/${ssd1} > ssd.reference.dump
# cat ssd.reference.dump
#
#
# 2. edit the working dump with the SSD last-lba
#    divide all sector start/size by 4, rounding up to whole numbers
#    make sure the 'start' + 'size' does not exceed 'start' of the next partition
#
# nano original.boot.disk.part.dump
#
# 3. save
#
# /OPTIONAL
###########################
# edit sectors if you'd like, otherwise, move on
sfdisk --force /dev/${ssd1} < original.boot.disk.part.dump
sfdisk --force /dev/${ssd2} < original.boot.disk.part.dump

# attaching the 3rd partition to the boot-pool
zpool attach boot-pool ${bootdisk}3 ${ssd1}3
zpool attach boot-pool ${bootdisk}3 ${ssd2}3

# copy files from bootdisk,1 to drive1,1 & drive2,1
dd if=/dev/${bootdisk}1 of=/dev/${ssd1}1 status=progress
dd if=/dev/${bootdisk}1 of=/dev/${ssd2}1 status=progress

# copy files from bootdisk,1 to drive1,2 & drive2,2
# these 2 lines may be unnecessary, as I still had to copy efi files further down
dd if=/dev/${bootdisk}2 of=/dev/${ssd1}2 status=progress
dd if=/dev/${bootdisk}2 of=/dev/${ssd2}2 status=progress

# prep dirs needed for copying EFI partition
mkdir -p /mnt/${bootdisk}2/
mkdir -p /mnt/${ssd1}2/
mkdir -p /mnt/${ssd2}2/

# grub can have issues with installing here
# set the filesystem to fat32 for drive1,2 & drive2,2
mkfs.vfat /dev/${ssd1}2
mkfs.vfat /dev/${ssd2}2

# mount necessary dirs
mount /dev/${bootdisk}2 /mnt/${bootdisk}2/
mount /dev/${ssd1}2 /mnt/${ssd1}2/
mount /dev/${ssd2}2 /mnt/${ssd2}2/

# copy contents
cp -r /mnt/${bootdisk}2/ /mnt/{ssd1}2/
cp -r /mnt/${bootdisk}2/ /mnt/{ssd2}2/

# confirm resilvering is complete before detaching USB from boot-pool
watch -n 5 -d zpool status boot-pool

# offline and remove original bootdisk from the pool
zpool offline boot-pool ${bootdisk}3
zpool detach boot-pool ${bootdisk}3

###########################
# OPTIONAL: create a new SSD partitions and mirrored pool#
# 1. create partition on SSD1
# fdisk /dev/${ssd1}
# <input> n <press enter>
# <press enter to input defaults and create 4th partition>
# <input> w <press enter>
#
# 2. create partition on SSD2
# fdisk /dev/${ssd2}
# <input> n <press enter>
# <press enter to input defaults and create 4th partition>
# <input> w <press enter>
#
# 3. create mirrored pool from 4th partition and export to see in TrueNAS CLI/GUI
# zpool create ${partitionpool} /dev/${ssd1}4 /dev/${ssd2}4
# zpool export ${partitionpool}
#
# /OPTIONAL
###########################

 

[NovaSaltado]

For whatever reason, while entering in code blocks syntax highlighting works, but doesn't on the post itself.

Here's a pastebin that's a bit easier on the eyes.

HOWTO: TrueNAS Scale: from 32gb USB to booting from mirrored larger enterprise SSD's, create add'l - Pastebin.com

Pastebin.com is the number one paste tool since 2002. Pastebin is a website where you can store text online for a set period of time.

pastebin.com

 

[JkktBkkt]

Thank you for the work done here. There are a few things I've noticed while following this:
1. The pastebin has some of the code and commentaries cut, i.e. setting variables
2. We're changing sector size from 512 bytes to 4 kilobytes (which 8 times bigger), but in instructions you state divide all sector start/size by 4, rounding up to whole numbers - perhaps an oversight?
And, perhaps obviously so, but after the zpool export in the end the pool with/for vms is not accessible requiring to be re-imported to be attached (perhaps automatically done so during reboot?)
Thanks again :)

 

[NovaSaltado]

> 1. The pastebin has some of the code and commentaries cut, i.e. setting variables
Whoops, thanks for catching that, fixed.

> 2. We're changing sector size from 512 bytes to 4 kilobytes (which 8 times bigger), but in instructions you state divide all sector start/size by 4, rounding up to whole numbers - perhaps an oversight?

I haven't messed with manual partitioning in a long time and while the "math doesn't add up" it's for good reason, at least to me. I didn't want to run into space constraints on any partitions. And while you're not wrong about "8x bigger", when going from 512>4k, the sector count decreases (Use this LBA Calculator). Input "1000000" and switch between 512/4k and see the difference in capacity.

 

[JkktBkkt]

> And while you're not wrong about "8x bigger", when going from 512>4k, the sector count decreases
By rounding?
> Input "1000000" and switch between 512/4k and see the difference in capacity.
Indeed, using the site's calc: 1000000 sectors in 512 bytes each is 488.28125 MB. That * 8 = 3906.25 MB. Switch to 4K and we see.. 3906.25 MB. All good. That's why we can round up, to have exactly same or more space. There's also possibility to add more in case we're doing it via creating a mirror of boot pool temporarily and expect boot pool to have very tight space constraints and a lot of small files at the same time. Though I'm not much of an experienced user with truenas to see how the boot pool might grow, still used a 64gb usb drive at first steps instead in my case, just to keep a bit more storage free and because it was first laying around.

 

[NovaSaltado]

> by rounding
The rounding is in the event you adjust your sectors and end up with a .5, which would be invalid.