Can't add mirror drive to boot pool: "can only attach to mirrors and top-level disks"

[garyo]

I just reinstalled the latest TrueNAS Core (TrueNAS-13.0-U5.3). My boot drive is a USB thumb drive (I know... better drive will be here soon.) Because of that I thought I'd attach a second one as a mirror. I tried using the GUI to attach it in System > Boot, but I get this error:

Error: [EFAULT] concurrent.futures.process._RemoteTraceback: """ Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/middlewared/plugins/zfs.py", line 220, in extend i['target'].attach(newvdev) File "libzfs.pyx", line 402, in libzfs.ZFS.__exit__ File "/usr/local/lib/python3.9/site-packages/middlewared/plugins/zfs.py", line 220, in extend i['target'].attach(newvdev) File "libzfs.pyx", line 2117, in libzfs.ZFSVdev.attach libzfs.ZFSException: can only attach to mirrors and top-level disks During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.9/concurrent/futures/process.py", line 246, in _process_worker r = call_item.fn(*call_item.args, **call_item.kwargs) File "/usr/local/lib/python3.9/site-packages/middlewared/worker.py", line 111, in main_worker res = MIDDLEWARE._run(*call_args) File "/usr/local/lib/python3.9/site-packages/middlewared/worker.py", line 45, in _run return self._call(name, serviceobj, methodobj, args, job=job) File "/usr/local/lib/python3.9/site-packages/middlewared/worker.py", line 39, in _call return methodobj(*params) File "/usr/local/lib/python3.9/site-packages/middlewared/worker.py", line 39, in _call return methodobj(*params) File "/usr/local/lib/python3.9/site-packages/middlewared/schema.py", line 985, in nf return f(*args, **kwargs) File "/usr/local/lib/python3.9/site-packages/middlewared/plugins/zfs.py", line 223, in extend raise CallError(str(e), e.code) middlewared.service_exception.CallError: [EZFS_BADTARGET] can only attach to mirrors and top-level disks """ The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/middlewared/job.py", line 355, in run await self.future File "/usr/local/lib/python3.9/site-packages/middlewared/job.py", line 386, in __run_body rv = await self.middleware._call_worker(self.method_name, *self.args, job={'id': self.id}) File "/usr/local/lib/python3.9/site-packages/middlewared/main.py", line 1250, in _call_worker return await self.run_in_proc(main_worker, name, args, job) File "/usr/local/lib/python3.9/site-packages/middlewared/main.py", line 1169, in run_in_proc return await self.run_in_executor(self.__procpool, method, *args, **kwargs) File "/usr/local/lib/python3.9/site-packages/middlewared/main.py", line 1152, in run_in_executor return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs)) middlewared.service_exception.CallError: [EZFS_BADTARGET] can only attach to mirrors and top-level disks

I've seen on some forums that this error can come from other root causes, so here's my `gpart show` for the two drives:

Code:

=>      40  60125104  da0  GPT  (29G)
        40      1024    1  freebsd-boot  (512K)
      1064  60096512    2  freebsd-zfs  (29G)
  60097576     27568       - free -  (13M)
=>      40  60088240  da1  GPT  (29G)
        40      1024    1  freebsd-boot  (512K)
      1064  60087216    2  freebsd-zfs  (29G)

 

[garyo]

Doing `zpool attach boot-pool da0p2 da1p2` also gives the same error.

 

[Arwen]

I just reinstalled the latest TrueNAS Core (TrueNAS-13.0-U5.3). My boot drive is a USB thumb drive (I know... better drive will be here soon.) Because of that I thought I'd attach a second one as a mirror. I tried using the GUI to attach it in System > Boot, but I get this error:

...

I've seen on some forums that this error can come from other root causes, so here's my `gpart show` for the two drives:

Code:

=>      40  60125104  da0  GPT  (29G)
        40      1024    1  freebsd-boot  (512K)
      1064  60096512    2  freebsd-zfs  (29G)
  60097576     27568       - free -  (13M)
=>      40  60088240  da1  GPT  (29G)
        40      1024    1  freebsd-boot  (512K)
      1064  60087216    2  freebsd-zfs  (29G)

It is not possible to add a sub-mirror that has fewer blocks that the source. Your "da0p2" has 9296 more blocks than the "da1p2".

60096512 - 60087216 = 9296

This is one of the annoying problems with storage today. Devices sold as one size being slightly different block count than a similar product from even the same vendor. Let alone different vendor.


One way around this, is to backup your configuration, (preferably multiple times, and with any encryption keys...). Then, re-install Core on to the smaller of the 2 USB drives. And last, use the larger drive as the mirror. The installation might even allow you to select both as a mirror, but make sure you select the smaller of the 2 first.

 

[garyo]

Aha, that makes sense! Terrible error message though. :(