dra6onfire
Dabbler
- Joined
- Jan 19, 2016
- Messages
- 10
I am currently running TrueNAS core 12.0-u2.1. My Setup has 3 pools and a boot volume. 1 of my pools is a mirrored vdev of 2 small SSDs used for virtualization (we'll call this pool 1 to keep things straight). The other two are larger with each pool containing one raidz1 vdev with 4 spinning disks (we will call these pool 2 and pool 3).
I am currently going through and replacing each drive in the pools with larger drives to increase storage. I started with pool 1. I offlined one drive from the zpool status UI menu, shutdown the server, replaced the offline disk, started the server, and executed the replace operation from the zpool status UI menu. As it always has in the past, it went flawlessly and once the resilvering was completed, I repeated the same for the other disk in pool 1. I moved on to pool 2. Same process on the first disk but when I attempted the replace operation I got a gnarly error that seems to be reporting a disk not found somehow (full trace below). I replaced the original disk and tried replacing another in pool 2 with the exact same result. As a sanity check, I put the original disks for pool 2 back and attempted pool 3. Pool 3, other than disk size, is configured in the same raidz1 config as pool 2 and was provisioned at the same time as pool 2. Pool 3 was able to replace all drives and resilver with absolutely no issues just like pool 1. I went back and re-attempted pool 2 again after completing pool 3 and got the same error.
I have no idea how to proceed. I tried attempting the same operations from command line and I see exactly the same error. The disks that failed to be used as replacements in pool 2 worked perfectly for pool 3. Ideally I don't want to have to copy all the data to another storage location, rebuild a new pool with the new disks and then move all the data back. Any help would be fantastic.
I am currently going through and replacing each drive in the pools with larger drives to increase storage. I started with pool 1. I offlined one drive from the zpool status UI menu, shutdown the server, replaced the offline disk, started the server, and executed the replace operation from the zpool status UI menu. As it always has in the past, it went flawlessly and once the resilvering was completed, I repeated the same for the other disk in pool 1. I moved on to pool 2. Same process on the first disk but when I attempted the replace operation I got a gnarly error that seems to be reporting a disk not found somehow (full trace below). I replaced the original disk and tried replacing another in pool 2 with the exact same result. As a sanity check, I put the original disks for pool 2 back and attempted pool 3. Pool 3, other than disk size, is configured in the same raidz1 config as pool 2 and was provisioned at the same time as pool 2. Pool 3 was able to replace all drives and resilver with absolutely no issues just like pool 1. I went back and re-attempted pool 2 again after completing pool 3 and got the same error.
Code:
[2021/02/28 14:14:49] (ERROR) middlewared.job.run():379 - Job <bound method accepts.<locals>.wrap.<locals>.nf of <middlewared.plugins.pool_.replace_disk.PoolService object at 0x81ce9e130>> failed concurrent.futures.process._RemoteTraceback: """ Traceback (most recent call last): File "/usr/local/lib/python3.8/site-packages/middlewared/plugins/zfs.py", line 277, in replace target.replace(newvdev) File "libzfs.pyx", line 391, in libzfs.ZFS.__exit__ File "/usr/local/lib/python3.8/site-packages/middlewared/plugins/zfs.py", line 277, in replace target.replace(newvdev) File "libzfs.pyx", line 2060, in libzfs.ZFSVdev.replace libzfs.ZFSException: already in replacing/spare config; wait for completion or use 'zpool detach' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.8/concurrent/futures/process.py", line 239, in _process_worker r = call_item.fn(*call_item.args, **call_item.kwargs) File "/usr/local/lib/python3.8/site-packages/middlewared/worker.py", line 91, in main_worker res = MIDDLEWARE._run(*call_args) File "/usr/local/lib/python3.8/site-packages/middlewared/worker.py", line 45, in _run return self._call(name, serviceobj, methodobj, args, job=job) File "/usr/local/lib/python3.8/site-packages/middlewared/worker.py", line 39, in _call return methodobj(*params) File "/usr/local/lib/python3.8/site-packages/middlewared/worker.py", line 39, in _call return methodobj(*params) File "/usr/local/lib/python3.8/site-packages/middlewared/schema.py", line 977, in nf return f(*args, **kwargs) File "/usr/local/lib/python3.8/site-packages/middlewared/plugins/zfs.py", line 279, in replace raise CallError(str(e), e.code) middlewared.service_exception.CallError: [EZFS_BADTARGET] already in replacing/spare config; wait for completion or use 'zpool detach' """ The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.8/site-packages/middlewared/job.py", line 367, in run await self.future File "/usr/local/lib/python3.8/site-packages/middlewared/job.py", line 403, in __run_body rv = await self.method(*([self] + args)) File "/usr/local/lib/python3.8/site-packages/middlewared/schema.py", line 973, in nf return await f(*args, **kwargs) File "/usr/local/lib/python3.8/site-packages/middlewared/plugins/pool_/replace_disk.py", line 122, in replace raise e File "/usr/local/lib/python3.8/site-packages/middlewared/plugins/pool_/replace_disk.py", line 102, in replace await self.middleware.call( File "/usr/local/lib/python3.8/site-packages/middlewared/main.py", line 1238, in call return await self._call( File "/usr/local/lib/python3.8/site-packages/middlewared/main.py", line 1203, in _call return await self._call_worker(name, *prepared_call.args) File "/usr/local/lib/python3.8/site-packages/middlewared/main.py", line 1209, in _call_worker return await self.run_in_proc(main_worker, name, args, job) File "/usr/local/lib/python3.8/site-packages/middlewared/main.py", line 1136, in run_in_proc return await self.run_in_executor(self.__procpool, method, *args, **kwargs) File "/usr/local/lib/python3.8/site-packages/middlewared/main.py", line 1110, in run_in_executor return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs)) middlewared.service_exception.CallError: [EZFS_BADTARGET] already in replacing/spare config; wait for completion or use 'zpool detach'
I have no idea how to proceed. I tried attempting the same operations from command line and I see exactly the same error. The disks that failed to be used as replacements in pool 2 worked perfectly for pool 3. Ideally I don't want to have to copy all the data to another storage location, rebuild a new pool with the new disks and then move all the data back. Any help would be fantastic.