NovaSaltado
Cadet
- Joined
- Feb 3, 2018
- Messages
- 7
The inspiration for this post, which was theoretical and helped guide me towards a working solution: https://www.truenas.com/community/t...sds-with-boot-and-data-pools-for-scale.89428/
FLUFF
My previous setup was an R720XD running ESXI 7.0.3 from mirrored SDCards. ESXI VM's would run from a mirrored SSD's (using onboard RAID card) and the HBA passed through to TrueNAS Core with the HDD's.
My new server is an R730XD. Which cannot easily connect the main backplane to an HBA and connect the flexbay with the onboard raid card.
Original requirements based on my R720xd:
Adjust requirements:
Process Overview:
FLUFF
My previous setup was an R720XD running ESXI 7.0.3 from mirrored SDCards. ESXI VM's would run from a mirrored SSD's (using onboard RAID card) and the HBA passed through to TrueNAS Core with the HDD's.
My new server is an R730XD. Which cannot easily connect the main backplane to an HBA and connect the flexbay with the onboard raid card.
Run hypervisor (esxi/proxmox) from usbSetup 2x enterprise SSD's for VM's including TrueNAS Scalepassthrough HDD's to TrueNAS and setup the main zpool
Adjust requirements:
note: ESXI is no longer recommended to run from USB
note: Proxmox is not recommended to run from USB/Flash
note: When passing through disks in Proxmox, TrueNAS cannot interact with SMART to view data or run tests (this is bad)
- TrueNAS Scale on baremetal, installed on enterprise mirrored SSD's *only using roughly 40GB)
- Remaining SSD space for VM's
- TrueNAS Scale interacting with SMART
Process Overview:
- Install TrueNAS Scale on 32GB SanDisk Ultra Fit (sector size 512)
- boot into TrueNAS Scale on the USB drive
- mirror USB partition table to enterprise SSD's (sector size 4096)
- Setup 4th partition on SSD's from remaining space (OPTIONAL: if you don't care to use your remaining free space)
Code:
# determine the drives you want to work on ie /dev/<drive> # you could also remove your HDD's to shorten the list lsblk -l # set vars # omit the /dev/ as the script below accounts for that. bootdisk=sds ssd1=sdl ssd2=sdm # optional: 4th partition partitionpool=vm-pool # we're dumping the current bootdrive partition data # to a file, in the event you want to change sectors e.g. 512->4096 sfdisk --dump /dev/${bootdisk} > original.boot.disk.part.dump ########################### # OPTIONAL if going to 4k sector size and/or want to use remaining free space as an additional pool # USB sectors sizee 512 -> enterprise SSD sector size 4096 # # 1. Dump current ssd partition table to grab last-lba for SSD # # sfdisk --dump /dev/${ssd1} > ssd.reference.dump # cat ssd.reference.dump # # # 2. edit the working dump with the SSD last-lba # divide all sector start/size by 4, rounding up to whole numbers # make sure the 'start' + 'size' does not exceed 'start' of the next partition # # nano original.boot.disk.part.dump # # 3. save # # /OPTIONAL ########################### # edit sectors if you'd like, otherwise, move on sfdisk --force /dev/${ssd1} < original.boot.disk.part.dump sfdisk --force /dev/${ssd2} < original.boot.disk.part.dump # attaching the 3rd partition to the boot-pool zpool attach boot-pool ${bootdisk}3 ${ssd1}3 zpool attach boot-pool ${bootdisk}3 ${ssd2}3 # copy files from bootdisk,1 to drive1,1 & drive2,1 dd if=/dev/${bootdisk}1 of=/dev/${ssd1}1 status=progress dd if=/dev/${bootdisk}1 of=/dev/${ssd2}1 status=progress # copy files from bootdisk,1 to drive1,2 & drive2,2 # these 2 lines may be unnecessary, as I still had to copy efi files further down dd if=/dev/${bootdisk}2 of=/dev/${ssd1}2 status=progress dd if=/dev/${bootdisk}2 of=/dev/${ssd2}2 status=progress # prep dirs needed for copying EFI partition mkdir -p /mnt/${bootdisk}2/ mkdir -p /mnt/${ssd1}2/ mkdir -p /mnt/${ssd2}2/ # grub can have issues with installing here # set the filesystem to fat32 for drive1,2 & drive2,2 mkfs.vfat /dev/${ssd1}2 mkfs.vfat /dev/${ssd2}2 # mount necessary dirs mount /dev/${bootdisk}2 /mnt/${bootdisk}2/ mount /dev/${ssd1}2 /mnt/${ssd1}2/ mount /dev/${ssd2}2 /mnt/${ssd2}2/ # copy contents cp -r /mnt/${bootdisk}2/ /mnt/{ssd1}2/ cp -r /mnt/${bootdisk}2/ /mnt/{ssd2}2/ # confirm resilvering is complete before detaching USB from boot-pool watch -n 5 -d zpool status boot-pool # offline and remove original bootdisk from the pool zpool offline boot-pool ${bootdisk}3 zpool detach boot-pool ${bootdisk}3 ########################### # OPTIONAL: create a new SSD partitions and mirrored pool# # 1. create partition on SSD1 # fdisk /dev/${ssd1} # <input> n <press enter> # <press enter to input defaults and create 4th partition> # <input> w <press enter> # # 2. create partition on SSD2 # fdisk /dev/${ssd2} # <input> n <press enter> # <press enter to input defaults and create 4th partition> # <input> w <press enter> # # 3. create mirrored pool from 4th partition and export to see in TrueNAS CLI/GUI # zpool create ${partitionpool} /dev/${ssd1}4 /dev/${ssd2}4 # zpool export ${partitionpool} # # /OPTIONAL ###########################