SOLVED Boot pool degraded, resilver won't work, exception thrown

[angelus249]

Hi,

I run TrueNAS-13.0-U5.2, specs in signature.

The boot pool is mirrored on 2 USB drives. Sunday night one drive failed, so I ordered a new one. Same size, but it's actually even a bit bigger (old 28.65GiB, new 28.67GiB). Shouldn't matter.

Now tried to replace the faulted one with the new drive, but (probably the same) error as seen below was thrown. So then I first detached the faulted drive, effectively running the boot pool on a single USB drive. Then tried attaching the new drive. I can select it, click Submit and after a few seconds the following error is thrown (repeatedly). Reboots didn't solve the issue. I can't add a the new drive.

Error: [EFAULT] concurrent.futures.process._RemoteTraceback: """ Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/middlewared/plugins/zfs.py", line 220, in extend i['target'].attach(newvdev) File "libzfs.pyx", line 402, in libzfs.ZFS.__exit__ File "/usr/local/lib/python3.9/site-packages/middlewared/plugins/zfs.py", line 220, in extend i['target'].attach(newvdev) File "libzfs.pyx", line 2117, in libzfs.ZFSVdev.attach libzfs.ZFSException: can only attach to mirrors and top-level disks During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.9/concurrent/futures/process.py", line 246, in _process_worker r = call_item.fn(*call_item.args, **call_item.kwargs) File "/usr/local/lib/python3.9/site-packages/middlewared/worker.py", line 111, in main_worker res = MIDDLEWARE._run(*call_args) File "/usr/local/lib/python3.9/site-packages/middlewared/worker.py", line 45, in _run return self._call(name, serviceobj, methodobj, args, job=job) File "/usr/local/lib/python3.9/site-packages/middlewared/worker.py", line 39, in _call return methodobj(*params) File "/usr/local/lib/python3.9/site-packages/middlewared/worker.py", line 39, in _call return methodobj(*params) File "/usr/local/lib/python3.9/site-packages/middlewared/schema.py", line 985, in nf return f(*args, **kwargs) File "/usr/local/lib/python3.9/site-packages/middlewared/plugins/zfs.py", line 223, in extend raise CallError(str(e), e.code) middlewared.service_exception.CallError: [EZFS_BADTARGET] can only attach to mirrors and top-level disks """ The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/middlewared/job.py", line 355, in run await self.future File "/usr/local/lib/python3.9/site-packages/middlewared/job.py", line 386, in __run_body rv = await self.middleware._call_worker(self.method_name, *self.args, job={'id': self.id}) File "/usr/local/lib/python3.9/site-packages/middlewared/main.py", line 1250, in _call_worker return await self.run_in_proc(main_worker, name, args, job) File "/usr/local/lib/python3.9/site-packages/middlewared/main.py", line 1169, in run_in_proc return await self.run_in_executor(self.__procpool, method, *args, **kwargs) File "/usr/local/lib/python3.9/site-packages/middlewared/main.py", line 1152, in run_in_executor return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs)) middlewared.service_exception.CallError: [EZFS_BADTARGET] can only attach to mirrors and top-level disks

Somewhere in the middle it says "[EZFS_BADTARGET] can only attach to mirrors and top-level disks" - well I'm somewhat sure I'm not doing it wrong. Have replaced disks on boot and storage before....

Does anyone know how to solve the problem?

Cheers

 

[angelus249]

Ok, ofc after I posted I found the right threads :) Some digging brought this old thread to light

Can't Mirror - Error: "can only attach to mirrors and top-level disks"

Context: I'm on Truenas 12.0 I've been a longtime user of using a good USB drive as my boot device. I'm trying to move off that to an SSD. I was intending to mirror that boot pool over to the SSD but keep running into issues. I keep trying to set up a mirror of my freenas-boot but it keeps...

www.truenas.com

Which led to this bug report

[NAS-108568] - iXsystems TrueNAS Jira

ixsystems.atlassian.net

Essentially, the error message is still showing bogus/unrefined error messages.

The culprit was the block size - for some reason.
Although it shouldn't differ, as I've ever only used the same size and type of USB drives, the option "-o ashift=9" worked, setting the block size to 512KB (2^9) when attaching the drive.

Code:

zpool attach -o ashift=9 boot-pool /dev/da16p2 /dev/da17p2

This via CLI then solved the issue. Added and resilvered without issues.

Cheers

 

[samarium]

Good that it works.

FYI ashift is not blocksize, similar but distinct.
And 2^9 is 512 bytes, not 512kB. The other common shift is 12 is 4096 bytes or 4kB.
Might trip you up in the future, or someone else using this as a reference.

 

[Philip Robar]

It's really disappointing that iXsystems, a company that supposedly makes storage "Appliances", takes a will not fix attitude to problems like this that show up so repeatedly in this and other forums.