Couldn't replace mirrored disk, so I detached disk and now can't extend Vdev

[punish_your_machine]

Hello,

TrueNAS Core 13.0-U3.1 running 12 discs that are mirrored (24 disks total), and I wanted to replace one 3TB SAS disk in a mirror with a larger 14TB SAS disk. I have done this several times before on previous versions of TrueNAS (like maybe version 12 or earlier). However, it didn't work today. I wish I would have copied the error message when trying to replace, but I didn't. It seemed to say something along the lines of waiting to detach. Anyway, I detached the offline disk, thinking I could just extend the Vdev and create the mirror again. But extending the Vdev isn't working either. I don't understand where I went wrong, or perhaps I found a bug? Any help would be appreciated. Here is the error I am getting while trying to extend the Vdev:

Error: Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/middlewared/job.py", line 355, in run await self.future File "/usr/local/lib/python3.9/site-packages/middlewared/job.py", line 391, in __run_body rv = await self.method(*([self] + args)) File "/usr/local/lib/python3.9/site-packages/middlewared/schema.py", line 975, in nf return await f(*args, **kwargs) File "/usr/local/lib/python3.9/site-packages/middlewared/plugins/pool_/attach_disk.py", line 86, in attach await job.wrap(extend_job) File "/usr/local/lib/python3.9/site-packages/middlewared/job.py", line 484, in wrap raise CallError(subjob.exception) middlewared.service_exception.CallError: [EFAULT] concurrent.futures.process._RemoteTraceback: """ Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/middlewared/plugins/zfs.py", line 220, in extend i['target'].attach(newvdev) File "libzfs.pyx", line 392, in libzfs.ZFS.__exit__ File "/usr/local/lib/python3.9/site-packages/middlewared/plugins/zfs.py", line 220, in extend i['target'].attach(newvdev) File "libzfs.pyx", line 2093, in libzfs.ZFSVdev.attach libzfs.ZFSException: can only attach to mirrors and top-level disks During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.9/concurrent/futures/process.py", line 246, in _process_worker r = call_item.fn(*call_item.args, **call_item.kwargs) File "/usr/local/lib/python3.9/site-packages/middlewared/worker.py", line 111, in main_worker res = MIDDLEWARE._run(*call_args) File "/usr/local/lib/python3.9/site-packages/middlewared/worker.py", line 45, in _run return self._call(name, serviceobj, methodobj, args, job=job) File "/usr/local/lib/python3.9/site-packages/middlewared/worker.py", line 39, in _call return methodobj(*params) File "/usr/local/lib/python3.9/site-packages/middlewared/worker.py", line 39, in _call return methodobj(*params) File "/usr/local/lib/python3.9/site-packages/middlewared/schema.py", line 979, in nf return f(*args, **kwargs) File "/usr/local/lib/python3.9/site-packages/middlewared/plugins/zfs.py", line 223, in extend raise CallError(str(e), e.code) middlewared.service_exception.CallError: [EZFS_BADTARGET] can only attach to mirrors and top-level disks """ The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/middlewared/job.py", line 355, in run await self.future File "/usr/local/lib/python3.9/site-packages/middlewared/job.py", line 386, in __run_body rv = await self.middleware._call_worker(self.method_name, *self.args, job={'id': self.id}) File "/usr/local/lib/python3.9/site-packages/middlewared/main.py", line 1249, in _call_worker return await self.run_in_proc(main_worker, name, args, job) File "/usr/local/lib/python3.9/site-packages/middlewared/main.py", line 1168, in run_in_proc return await self.run_in_executor(self.__procpool, method, *args, **kwargs) File "/usr/local/lib/python3.9/site-packages/middlewared/main.py", line 1151, in run_in_executor return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs)) middlewared.service_exception.CallError: [EZFS_BADTARGET] can only attach to mirrors and top-level disks

 

[punish_your_machine]

Hello, I've been poking around in forums and I believe the issue might have to do with the 3TB disk having a sector size of 512, and the 14TB disk having a sector size of 512 but a stripe size of 4096. The 3TB disk has ashift = 9, and the 14TB disk has ashift = 12. From what I've read, the ashift probably needs to be the same for both disks. Alexander Motin commented on this here:

https://ixsystems.atlassian.net/browse/NAS-112093

Alexander Motin
April 28, 2022 at 1:36 PM
Edited


I was able to reproduce the problem. It happens for pools created before TrueNAS 12. Those pools do not have ashift property set, since TrueNAS 11 didn't have one and used loader tunable for ashift specification. As result on 12/13/SCALE such pools have neither the tunable nor property to set ashift and so on new vdev addition default to disk's physical sector size.