Unable to replace failed disks in Raidz-2 pool

[notrox]

I had a old file server with FreeBSD and ZFS which I took the drives from and put into a new chassis with a fresh installation of TrueNas 12.0-U1. I've imported the pool and have attempted to replace one of the drives that is "Offline" but I get a Error replacing Disk - already in replacing/spare config; wait for completion or use 'zpool detach'

Note: I tried to replace the UNAVAILABLE drive the same way and received the same error.

I've attached the following "More info" from the error.

This is
Error: concurrent.futures.process._RemoteTraceback:
"""
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/middlewared/plugins/zfs.py", line 277, in replace
target.replace(newvdev)
File "libzfs.pyx", line 391, in libzfs.ZFS.__exit__
File "/usr/local/lib/python3.8/site-packages/middlewared/plugins/zfs.py", line 277, in replace
target.replace(newvdev)
File "libzfs.pyx", line 2060, in libzfs.ZFSVdev.replace
libzfs.ZFSException: already in replacing/spare config; wait for completion or use 'zpool detach'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/local/lib/python3.8/concurrent/futures/process.py", line 239, in _process_worker
r = call_item.fn(*call_item.args, **call_item.kwargs)
File "/usr/local/lib/python3.8/site-packages/middlewared/worker.py", line 91, in main_worker
res = MIDDLEWARE._run(*call_args)
File "/usr/local/lib/python3.8/site-packages/middlewared/worker.py", line 45, in _run
return self._call(name, serviceobj, methodobj, args, job=job)
File "/usr/local/lib/python3.8/site-packages/middlewared/worker.py", line 39, in _call
return methodobj(*params)
File "/usr/local/lib/python3.8/site-packages/middlewared/worker.py", line 39, in _call
return methodobj(*params)
File "/usr/local/lib/python3.8/site-packages/middlewared/schema.py", line 977, in nf
return f(*args, **kwargs)
File "/usr/local/lib/python3.8/site-packages/middlewared/plugins/zfs.py", line 279, in replace
raise CallError(str(e), e.code)
middlewared.service_exception.CallError: [EZFS_BADTARGET] already in replacing/spare config; wait for completion or use 'zpool detach'
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/middlewared/job.py", line 361, in run
await self.future
File "/usr/local/lib/python3.8/site-packages/middlewared/job.py", line 397, in __run_body
rv = await self.method(*([self] + args))
File "/usr/local/lib/python3.8/site-packages/middlewared/schema.py", line 973, in nf
return await f(*args, **kwargs)
File "/usr/local/lib/python3.8/site-packages/middlewared/plugins/pool_/replace_disk.py", line 122, in replace
raise e
File "/usr/local/lib/python3.8/site-packages/middlewared/plugins/pool_/replace_disk.py", line 102, in replace
await self.middleware.call(
File "/usr/local/lib/python3.8/site-packages/middlewared/main.py", line 1238, in call
return await self._call(
File "/usr/local/lib/python3.8/site-packages/middlewared/main.py", line 1203, in _call
return await self._call_worker(name, *prepared_call.args)
File "/usr/local/lib/python3.8/site-packages/middlewared/main.py", line 1209, in _call_worker
return await self.run_in_proc(main_worker, name, args, job)
File "/usr/local/lib/python3.8/site-packages/middlewared/main.py", line 1136, in run_in_proc
return await self.run_in_executor(self.__procpool, method, *args, **kwargs)
File "/usr/local/lib/python3.8/site-packages/middlewared/main.py", line 1110, in run_in_executor
return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs))
middlewared.service_exception.CallError: [EZFS_BADTARGET] already in replacing/spare config; wait for completion or use 'zpool detach'

pool: raid0
state: DEGRADED
status: One or more devices could not be opened. Sufficient replicas exist for
the pool to continue functioning in a degraded state.
action: Attach the missing device and online it using 'zpool online'.
see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-2Q
scan: resilvered 26.4M in 00:59:42 with 0 errors on Sun Jan 10 19:02:21 2021
config:

NAME STATE READ WRITE CKSUM
raid0 DEGRADED 0 0 0
raidz2-0 DEGRADED 0 0 0
label/disk0 DEGRADED 0 0 0 too many errors
12654245418753434265 UNAVAIL 0 0 0 was /dev/da13s1
label/disk2 DEGRADED 0 0 0 too many errors
label/disk3 DEGRADED 0 0 0 too many errors
da9s1 DEGRADED 0 0 0 too many errors
13081339411024284783 OFFLINE 0 0 0 was /dev/label/disk5
label/disk6 DEGRADED 0 0 0 too many errors
da1s1 DEGRADED 0 0 0 too many errors
label/disk8 DEGRADED 0 0 0 too many errors
label/disk9 DEGRADED 0 0 0 too many errors

errors: No known data errors

Any help would be greatly appreciated. Thanks everyone

 

[sretalla]

It looks like your controller has had a bad reaction to something... maybe a reboot will improve your situation.

 

[notrox]

I have tried rebooting a few times along with scrubbing the pool and it doesn't seem to make a difference. Is there anything else I can try?