HOWTO: Split SSD with boot-pool to create partition for data (no USB install, easy config migration)

[MaxForTrueNas]

Hello, while there already exist HOWTOs about "shrinking" boot-pool, e.g.
HOWTO: Using a pair of large SSDs with boot and data pools for SCALE
HOWTO: TrueNAS Scale: from 32gb USB to booting from mirrored larger enterprise SSD's, create add'l partition/data pool from remaining free space

Credit:
While writing this guide, I found a reddit thread reporting the identical steps.
Since I didn't find it before spending two days figuring out installation process, and this guide is sufficiently complete, I'm posting this anyway.

I encountered an issue with the 1st step in both of them - namely installing TrueNAS on USB didn't work with any of my hardware.

After having a look around, I found awesome truenas-install script, where the disks are partitioned.
It was extremely easy to adjust the script to not occupy whole ssd with boot-pool.

The method should work without any issue for any number of SSDs (verified with one disk).
The method is likely able to preserve your configuration (verified on a VM).
THIS METHOD IS LIKELY TO BREAK UPDATES, BACK-UP YOUR DATA BEFORE RUNNING AN UPDATE !!!

UNSUPPORTED DEPLOYMENT AHEAD, DEPENDS ON TRUENAS INSTALLER WORKING THE SAME WAY AS IT DOES NOW, PROCEED WITH CAUTION
But to be honest, I see no reason why this shouldn't work fine. Just mirror more SSDs if you're worried, and backup things once in a while.

Steps​

Main assumption - you have graphical access to the system where is/are the disk/s for installation.

This method works even on top of actually working system (verified on a VM with _single_ virtual drive, YMMV).
You are of course encouraged to export your zpools, back-up your data and proceed with caution.

Installation​

• Use installer, boot with installer
• Select "shell" from the installer menu
• Edit `/sbin/truenas-install` (use vi)

Code:

# This is currently line 410 in the truenas-install
# This is part of create_partitions function

    # Create boot pool
    if ! sgdisk -n3:0:<put your desired size here, e.g. +50G> -t3:BF01 /dev/${_disk}; then
        return 1
    fi

• Note BF01 - this is marking disk as ZFS; Refer to sgdisk man for details
• Once done editing, run `truenas-install` and proceed as usual
  • You can select multiple disks if you'd like to have duplication, but I didn't verify if this will work
• You can choose to migrate your data or install anew
• I replaced the boot environment, unsure what exactly that (or create new one) does

New partition creation​

After you installed and booted TrueNAS, you're free to allocate free disk space (pun intended) however you want.

I used
```
gdisk /dev/sXX

n <for new part>
<default start - after last part>
<default size - till the end>
BF01 <to mark disk as ZFS>
w <write>
```

```
zpool create <pool name> /dev/sxxN
zpool export <pool name>
```

Now you can import the pool from the UI and do whatever you want with it.
I recommend only using it for something you don't care about, like k8s containers and VMs you don't mind having to re-create.

 

[papasan]

just registered to say that this method didn't work for me on a 22.12.3.3 install. system came up with the full SSD in boot-pool.

 

[sretalla]

system came up with the full SSD in boot-pool.

Seems not so plausible if you followed the instructions and created the partitions as instructed... interesting to see if anyone else who followed the instructions gets the same.

Did you put the size like this?:

+50G

meaning:
if ! sgdisk -n3:0:+50G> -t3:BF01 /dev/${_disk}; then

 

[papasan]

I used +32G, but yes. might be because of the 16GB swap part creation, not sure.

 

[papasan]

can't edit posts? anyways, according to this the nomenclature is "+32GiB". haven't tested as the 'install on smaller and copy partitions to SSD' method seems to be working for me.

 

[kiler129]

This does work on the newest TNS. When using the 16GB swap partition, it will be created properly as well. Example with s 240GB SSD:

Code:

# gdisk /dev/sda
(...)

Command (? for help): p
Disk /dev/sda: 468862128 sectors, 223.6 GiB
Model: *****
Sector size (logical/physical): 512/512 bytes
Disk identifier (GUID): *****
Partition table holds up to 128 entries
Main partition table begins at sector 2 and ends at sector 33
First usable sector is 34, last usable sector is 468862094
Partitions will be aligned on 2048-sector boundaries
Total free space is 367148141 sectors (175.1 GiB)

Number  Start (sector)    End (sector)  Size       Code  Name
   1            4096            6143   1024.0 KiB  EF02
   2            6144         1054719   512.0 MiB   EF00
   3        34609152       101718015   32.0 GiB    BF01
   4         1054720        34609151   16.0 GiB    8200

Command (? for help):

Adding a new partition works flawlessly as well:

Code:

Command (? for help): n
Partition number (5-128, default 5):
First sector (34-468862094, default = 101718016) or {+-}size{KMGTP}:
Last sector (101718016-468862094, default = 468862094) or {+-}size{KMGTP}:
Current type is 8300 (Linux filesystem)
Hex code or GUID (L to show codes, Enter = 8300): BF01
Changed type of partition to 'Solaris /usr & Mac ZFS'

Command (? for help): p
Disk /dev/sda: 468862128 sectors, 223.6 GiB
Model: ******
Sector size (logical/physical): 512/512 bytes
Disk identifier (GUID): ******
Partition table holds up to 128 entries
Main partition table begins at sector 2 and ends at sector 33
First usable sector is 34, last usable sector is 468862094
Partitions will be aligned on 2048-sector boundaries
Total free space is 4062 sectors (2.0 MiB)

Number  Start (sector)    End (sector)  Size       Code  Name
   1            4096            6143   1024.0 KiB  EF02
   2            6144         1054719   512.0 MiB   EF00
   3        34609152       101718015   32.0 GiB    BF01
   4         1054720        34609151   16.0 GiB    8200
   5       101718016       468862094   175.1 GiB   BF01  Solaris /usr & Mac ZFS

Command (? for help): w

And it can be confirmed as well:

Code:

# fdisk -l /dev/sda
Disk /dev/sda: 223.57 GiB, 240057409536 bytes, 468862128 sectors
Disk model: ******
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: ******

Device         Start       End   Sectors   Size Type
/dev/sda1       4096      6143      2048     1M BIOS boot
/dev/sda2       6144   1054719   1048576   512M EFI System
/dev/sda3   34609152 101718015  67108864    32G Solaris /usr & Apple ZFS
/dev/sda4    1054720  34609151  33554432    16G Linux swap
/dev/sda5  101718016 468862094 367144079 175.1G Solaris /usr & Apple ZFS

Partition table entries are not in disk order.

 

[lucaxxaa]

I try this operation, on TrueNAS-SCALE-23.10.1.3, running on virtual machine on virtualbox .
I try more option, +30G or +30GiB, but not work for me

 

[lucaxxaa]

...now is right, i have created partition, but i am not able to create pool, in that partition.
It is /dev/sda5,

For create pool, is necessary more that One hard disk, or partition ?

admin@truenas[~]$ sudo fdisk -l /dev/sda Disk /dev/sda: 80 GiB, 85899345920 bytes, 167772160 sectors Disk model: VBOX HARDDISK Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disklabel type: gpt Disk identifier: ACFCB4E4-DD61-4BBF-B7F7-2E116F4FE9F6 Device Start End Sectors Size Type /dev/sda1 4096 6143 2048 1M BIOS boot /dev/sda2 6144 1054719 1048576 512M EFI System /dev/sda3 34609152 101718015 67108864 32G Solaris /usr & Apple ZFS /dev/sda4 1054720 34609151 33554432 16G Linux swap /dev/sda5 101718016 167770111 66052096 31.5G Solaris /usr & Apple ZFS Partition table entries are not in disk order.

 

[Ericloewe]

...now is right, i have created partition, but i am not able to create pool, in that partition.

What's error, exactly?

 

[lucaxxaa]

What's error, exactly?

I reboot virtual machine, and run this commands, without error, with sudo

but i can't see it, in graphical interface

zpool create pippo /dev/sda5 zpool export pippo

 

[lucaxxaa]

I reboot virtual machine, and run this commands, without error, with sudo

but i can't see it, in graphical interface

zpool create pippo /dev/sda5 zpool export pippo .

i wrong procedure. Now i find "import" command, on graphical interface, and it work
Excuseme, but i am newbee

 

[Ericloewe]

Yeah, Import [a ZFS pool] vs Import [data from other filesystems]. Not great, but nobody's been able to come up with a better naming scheme over the past decade+.

 

[TCWL]

I did this and it seems to work fine, except the GUI doesn't list any data VDEVs:

When I open "Manage Devices", it shows the entire disk, not the partition:

Does ZFS or TrueNAS not understand partitions as vdevs or what gives?

 

[TCWL]

It seems ZFS understands partitions just fine.

I know this isn't officially supported use-case but I'd like to learn ZFS properly and this behavior seems odd to me, maybe I am doing something wrong.

 

[sretalla]

maybe I am doing something wrong.

Setting up TrueNAS in an unsupported way will bring unexpected results... Since the developers don't expect to see the scenario you present... Because they don't give you a means to do it, they don't account for it when writing the GUI.

The expectation is that TrueNAS will only use entire disks as pool members, so that's how the GUI is built.

Either play the game as expected or put up with the abnormal things that result.

 

[TCWL]

Setting up TrueNAS in an unsupported way will bring unexpected results... Since the developers don't expect to see the scenario you present... Because they don't give you a means to do it, they don't account for it when writing the GUI.

The expectation is that TrueNAS will only use entire disks as pool members, so that's how the GUI is built.

Either play the game as expected or put up with the abnormal things that result.

I see that's happening. However, I am intrigued why the GUI doesn't read the vdevs directly from zpool status command, it would be much more robust approach. As of now, there seems some kind of disparity and duplication of information (GUI seems to store the pool configuration separately from zpool command).

 

[sretalla]

When you use the GUI and add a disk/create a pool the supported way, TrueNAS creates 2 partitions on the disk, using a 2 GB partition 1 as swap and the rest of the capacity in partition 2 as the pool member... It shows that as the whole disk in use as a pool member.

It's not only TrueNAS, but if you look at the OpenZFS docs, you'll see that it's recommend to only give entire disks to ZFS pools.

 

[Patrick M. Hausen]

When you use the UUID of the partition instead of the raw device name to create your "aux" pool, the UI will probably pick up and display the state.

Never use device names for data pools, especially when you perform replace operations on the command line - which is perfectly fine - make sure to use UUIDs only.

 

[kiriak]

I see absolutely no reason for anyone to split disks with the boot pool beyond the scientific or hobbyist interest to go beyond the limits or to explore new things.
Even if one is in need to use all the available connectors for the data etc. pools he can use available usb port(s) for the boot pool.

 

[TCWL]

Never use device names for data pools, especially when you perform replace operations on the command line - which is perfectly fine - make sure to use UUIDs only.

Probably a good advice, but doesn't TrueNAS use device names as well?

I see absolutely no reason for anyone to split disks with the boot pool beyond the scientific or hobbyist interest to go beyond the limits or to explore new things.
Even if one is in need to use all the available connectors for the data etc. pools he can use available usb port(s) for the boot pool.

There are only so many available pcie lanes on a motherboard, putting non-critical data beside boot pool saves space (inside the case).

I like my computer being built inside the case, no hanging wires or dongles.