Add NVMe to Boot Pool

[dakotta]

I installed TrueNAS [edit - TrueNAS-12.0-U5] Core] a few months ago using a 120 GiB USB drive as the boot pool. I am finally getting around to modifying that by adding a 232 GiB NVMe-to-USB stick as a mirror.

1. Does the new disk need to have a GPT partition table on it before adding it to the pool?
2. Does the new disk need to be formatted with a file system before it can be attached? Or is the file system created during the install?

Initially, TrueNAS would not recognize the disk, so I partitioned it with cgdisk (on Linux). After that, TrueNAS recognized the disk.

I "attched" it using the GUI, with:
System ==> Boot ==> Actions ==> Boot Pool Status
Right-click on existing boot pool disk ==> Attch ==> Select disk (da5, 232.89 GiB) ==> Use all disk space ==> Submit

This resulted in:
"Error: [EFAULT] concurrent.futures.process._RemoteTraceback: """ Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/middlewared/plugins/zfs.py", line 214, in extend i['target'].attach(newvdev) File "libzfs.pyx", line 391, in libzfs.ZFS.__exit__ File "/usr/local/lib/python3.9/site-packages/middlewared/plugins/zfs.py", line 214, in extend i['target'].attach(newvdev) File "libzfs.pyx", line 2030, in libzfs.ZFSVdev.attach libzfs.ZFSException: can only attach to mirrors and top-level disks During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.9/concurrent/futures/process.py", line 243, in _process_worker r = call_item.fn(*call_item.args, **call_item.kwargs) File "/usr/local/lib/python3.9/site-packages/middlewared/worker.py", line 94, in main_worker res = MIDDLEWARE._run(*call_args) File "/usr/local/lib/python3.9/site-packages/middlewared/worker.py", line 45, in _run return self._call(name, serviceobj, methodobj, args, job=job) File "/usr/local/lib/python3.9/site-packages/middlewared/worker.py", line 39, in _call return methodobj(*params) File "/usr/local/lib/python3.9/site-packages/middlewared/worker.py", line 39, in _call return methodobj(*params) File "/usr/local/lib/python3.9/site-packages/middlewared/schema.py", line 977, in nf return f(*args, **kwargs) File "/usr/local/lib/python3.9/site-packages/middlewared/plugins/zfs.py", line 217, in extend raise CallError(str(e), e.code) middlewared.service_exception.CallError: [EZFS_BADTARGET] can only attach to mirrors and top-level disks """ The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/middlewared/job.py", line 367, in run await self.future File "/usr/local/lib/python3.9/site-packages/middlewared/job.py", line 398, in __run_body rv = await self.middleware._call_worker(self.method_name, *self.args, job={'id': self.id}) File "/usr/local/lib/python3.9/site-packages/middlewared/main.py", line 1219, in _call_worker return await self.run_in_proc(main_worker, name, args, job) File "/usr/local/lib/python3.9/site-packages/middlewared/main.py", line 1146, in run_in_proc return await self.run_in_executor(self.__procpool, method, *args, **kwargs) File "/usr/local/lib/python3.9/site-packages/middlewared/main.py", line 1120, in run_in_executor return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs)) middlewared.service_exception.CallError: [EZFS_BADTARGET] can only attach to mirrors and top-level disks"

Any suggsetions?

Cheers,

TrueNAS-12.0-U5

Kingston 250GB NV1 M.2 2280 NVMe Internal SSD (boot drive)

https://www.bhphotovideo.com/c/product/1676932-REG/kingston_snvs_250g_250gb_nv1_m_2_2280.html

Sabrent NVMe PCIe M.2 to USB 3.1 Gen 2 USB Type-C SSD Enclosure (Gray) [boot drive enclosure]

https://www.bhphotovideo.com/c/product/1448870-REG/sabrent_ec_nvme_usb_3_1_aluminum_enclosure.html

 

[dakotta]

I wiped the partition structure (on Linux) with...
# sgdisk --zap-all /dev/sdc

Then I tried to attach again (like it my first post). That failed, with the same error message.

Looking at the disk in Linux, I see

Code:

             FSTYPE        FSVER    LABEL
/dev/sdc1    vfat          FAT16                  260M    EFI System
/dev/sdc2    zfs_member    5000      boot-pool    119G    FreeBSD         ZFS

So, I guess the answers are:
1. You do not need to create a partition table first.
2. You do not need to format the disks with a file system.

Still don't know why it failed though.

Guess I'll just buy a dozen cheap USB disks and swap them out as they fail. :(

 

[pschatz100]

I had difficulty following your post. What is your question? You have a working TrueNAS system, right?

Why are you running commands in Linux to reformat the disk before using it in TrueNAS? If you want to prepare a disk for TrueNAS, you would be better off doing this from the TrueNAS GUI. You might want to check the documentation for instructions about wiping a disk - and then go from there.

 

[dakotta]

Really? I thought the questions were pretty straight-forward.

1. Does the new disk need to have a GPT partition table on it before adding it to the pool?
2. Does the new disk need to be formatted with a file system before it can be attached? Or is the file system created during the install?

As to why I was using not using the GUI, I thought I made that clear also: "TrueNAS would not recognize the disk." Meaning, that the disk did not appear on any GUI screen or menu option. I could not find any TrueNAS documentation on creating partition tables, so I figured the partition table was platform independent.

It was only after creating a GPT partition table, that the TrueNAS operating system recognized the disk. At that point, I tried to use the TrueNAS GUI (as suggested) to format the disk, but that operation failed. Or rather, the "formatting" seemed to succeed [file systems were installed], but the "attachment" did not.

From the TrueNAS GUI:
System ==> Boot ==> Actions ==> Boot Pool Status
Right-click on existing boot pool disk ==> Attach ==> Select disk (da5, 232.89 GiB) ==> Use all disk space ==> Submit]

So, yes, I have a working TrueNAS system with a single USB thumb drive as the boot pool.

I believe that additional steps are necessary to make the NVMe-to-USB disk work as a boot pool and that those instructions are not included in the documentation.

As a result, I intend to purchase a dozen cheap USB drives and swap them out as necessary. Not optimal, but I believe it will work.

Cheers,

 

[jgreco]

Does the new disk need to have a GPT partition table on it before adding it to the pool?

No. Disks are expected to be blank.

Does the new disk need to be formatted with a file system before it can be attached? Or is the file system created during the install?

Again, disks are expected to be blank. No filesystems are created when you attach a new disk. The new disk(s) become available as part of the pool, but are only a component of the larger storage system.

 

[dakotta]

Thank you!

Cheers,