SOLVED How to remove top-level vdev from mirrored pool

[internat_user]

System:
TrueNAS-12.0-U8

I was doing some cleanup / maintenance on my TrueNAS system and noticed there is a single vdev in my mirrored pool. There are no known errors on the pool or on the drive. When attempting to extend the vdev to a mirror I receive an error that the disk cannot be found although I can see the disk under 'Disks' and both devices under 'Multipaths'. I am unable to remove the disk from the pool.

Multipath/disk49p2 is the disk in questions. Ideally I want to extend this so that it is a mirror and not a single vdev.

I have tried replacing the vdev and retrying the process but I get the same error. I have attempted to wipe the disks before adding them as well.

Is there a way to extend this vdev via cli or somehow transfer the data off of the vdev and drop it?

I am new, if any additional information is required please let me know, happy to provide. I have attached several screenshots as well.

 

[NugentS]

You won't be able to remove the disk from the pool, its a top level vdev. If you did remove the vdev, you lose the pool

Please follow the forum rules, about posting hardware details

 

[danb35]

You won't be able to remove the disk from the pool, its a top level vdev.

Yes, you can; this has been possible since 11.2. But better would be to add a mirror to that disk. I'm not sure how multipath changes things, though.

 

[NugentS]

Really? Because its mirrors?
I thought you could remove special vdevs etc as mirrors, not the actual data disks - but could have the wrong end of the stick.

I agree - much better to add a mirror - which is the bit that isn't working according to the OP

 

[danb35]

You can remove top-level vdevs if, and only if, all vdevs in the pool are either single disks or mirrors (so not if they're any flavor of RAIDZ). It's even supported in the GUI now, though the GUI is somewhat more restrictive than ZFS itself is. But multipathing is something I haven't dealt with and don't feel comfortable making recommendations on. It might be worth filing a bug--IMO at least, the GUI should never give a stack trace like that during normal operation--but perhaps someone more knowledgeable with this scenario can chime in.

 

[internat_user]

Thank you for the replies and detail, it helps me understand the issues a bit more. Unfortunately I cannot offline the disk from the gui as that throws an error as well.

I would like to either safley remove the disk or add it to a mirror with it currently being a single point of failure I fear if the disk is lost then I will lost the pool.

 

[internat_user]

When I do attempt to offline the vdev I receive the error [EZFS_NOREPLICAS] no valid replicas.

Code:

Error: concurrent.futures.process._RemoteTraceback:
"""
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/middlewared/plugins/zfs.py", line 226, in __zfs_vdev_operation
    op(target, *args)
  File "libzfs.pyx", line 391, in libzfs.ZFS.__exit__
  File "/usr/local/lib/python3.9/site-packages/middlewared/plugins/zfs.py", line 226, in __zfs_vdev_operation
    op(target, *args)
  File "/usr/local/lib/python3.9/site-packages/middlewared/plugins/zfs.py", line 242, in <lambda>
    self.__zfs_vdev_operation(name, label, lambda target: target.offline())
  File "libzfs.pyx", line 2116, in libzfs.ZFSVdev.offline
libzfs.ZFSException: no valid replicas

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/concurrent/futures/process.py", line 243, in _process_worker
    r = call_item.fn(*call_item.args, **call_item.kwargs)
  File "/usr/local/lib/python3.9/site-packages/middlewared/worker.py", line 111, in main_worker
    res = MIDDLEWARE._run(*call_args)
  File "/usr/local/lib/python3.9/site-packages/middlewared/worker.py", line 45, in _run
    return self._call(name, serviceobj, methodobj, args, job=job)
  File "/usr/local/lib/python3.9/site-packages/middlewared/worker.py", line 39, in _call
    return methodobj(*params)
  File "/usr/local/lib/python3.9/site-packages/middlewared/worker.py", line 39, in _call
    return methodobj(*params)
  File "/usr/local/lib/python3.9/site-packages/middlewared/schema.py", line 979, in nf
    return f(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/middlewared/plugins/zfs.py", line 242, in offline
    self.__zfs_vdev_operation(name, label, lambda target: target.offline())
  File "/usr/local/lib/python3.9/site-packages/middlewared/plugins/zfs.py", line 228, in __zfs_vdev_operation
    raise CallError(str(e), e.code)
middlewared.service_exception.CallError: [EZFS_NOREPLICAS] no valid replicas
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/middlewared/main.py", line 138, in call_method
    result = await self.middleware._call(message['method'], serviceobj, methodobj, params, app=self,
  File "/usr/local/lib/python3.9/site-packages/middlewared/main.py", line 1213, in _call
    return await methodobj(*prepared_call.args)
  File "/usr/local/lib/python3.9/site-packages/middlewared/schema.py", line 975, in nf
    return await f(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/middlewared/plugins/pool.py", line 1115, in offline
    await self.middleware.call('zfs.pool.offline', pool['name'], found[1]['guid'])
  File "/usr/local/lib/python3.9/site-packages/middlewared/main.py", line 1256, in call
    return await self._call(
  File "/usr/local/lib/python3.9/site-packages/middlewared/main.py", line 1221, in _call
    return await self._call_worker(name, *prepared_call.args)
  File "/usr/local/lib/python3.9/site-packages/middlewared/main.py", line 1227, in _call_worker
    return await self.run_in_proc(main_worker, name, args, job)
  File "/usr/local/lib/python3.9/site-packages/middlewared/main.py", line 1154, in run_in_proc
    return await self.run_in_executor(self.__procpool, method, *args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/middlewared/main.py", line 1128, in run_in_executor
    return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs))
middlewared.service_exception.CallError: [EZFS_NOREPLICAS] no valid replicas

Outputof zpool status:

I see there is a command zpool remove gptid/eabc444e-f353-11e8-b13f-c08c6015a3

Is anyone familiar with this command?

 

[danb35]

Unfortunately I cannot offline the disk from the gui

No, you wouldn't be able to do that. You'd want to remove it, and it looks like the GUI doesn't allow doing that for single disks. So instead, from the CLI, you'd do zpool remove poolname diskid. Once that process completes, you can (if desired) remove the disk from your system.

 

[internat_user]

And the

Code:

zpool remove poolname diskid

command will evacuate the data on the disk to other mirrors in the pool, correct? The disk is a 10tb drive with 5.1tb of data on it that we cannot afford to lose.

 

[danb35]

command will evacuate the data on the disk to other mirrors in the pool, correct?

Correct. So you don't want to remove the disk until that evacuation process is complete.

 

[internat_user]

Okay, sorry for asking, I just want to be 100% clear on this. When executing the zpool remove poolname diskid it will evacuate the data AND remove the disk? Is there another step that needs to be performed in the GUI or another command that needs to be executed for the removal process to complete?

 

[danb35]

zpool remove will both evacuate the data and remove the disk from the pool.

 

[NugentS]

BUt please, out of an abundnace of caution make a backup first

 

[Patrick M. Hausen]

A zpool checkpoint might be a good idea in this case.

 

[internat_user]

Does a zpool checkpoint work the same way as a snapshot? In that I will need to have the same amount of space available to hold this data if I want to revert back? The pool itself is quite large.

I guess what I am trying to ask, if I did a zpool checkpoint would I need to have the equivalent amount of space to hold onto this 'checkpointed' (for lack of a better word) state of the pool?

 

[Patrick M. Hausen]

A zpool checkpoint of which you can only have one freezes the state of the entire pool including disk additions, removals, etc.
Please check the documentation for details, I would need to look it up myself.

 

[internat_user]

Per this documentation from OpenZFS you are unable to make device changes / vdev configuration changes. I would think with that said a zpool checkpoint is out of the option, although this is a feature I was unaware of and happy to have in my toolkit now.

https://openzfs.org/w/images/b/b6/Storage_Pool_Checkpoint.pdf

 

[Patrick M. Hausen]

Code:

DESCRIPTION
     Checkpoints the current state of pool , which can be later restored by
     zpool import --rewind-to-checkpoint.  The existence of a checkpoint in a
     pool prohibits the following zpool subcommands: remove, attach, detach,
     split, and reguid.  In addition, it may break reservation boundaries if
     the pool lacks free space.  The zpool status command indicates the
     existence of a checkpoint or the progress of discarding a checkpoint from
     a pool.  zpool list can be used to check how much space the checkpoint
     takes from the pool.

So you are right, sorry. I did use checkpoints before adding a new vdev which seems not to fall under this restriction. Weird.

 

[internat_user]

This has been resolved. I was able to successfully remove the single top level vdev from the mirrored pool. A few things I checked to confirm this would be completed without error:

1. Verify that the pool is striped
2. Confirm that the sector size is the same for all disks; I did this easily using a script provided in this thread: https://www.truenas.com/community/t...-devices-disktype-serial-num-multipath.59319/
3. Confirm that the feature@device_removal feature was enabled for the pool using zpool get all poolname | grep feature@
4. I then removed the disk using zpool remove poolname gptid/xxxx-xxx-xxxxx

I did take a snapshot of the pool before proceeding with the removal. The entire process took approximately 27h53m. I did have backup jobs running to SMB and iSCSI extents on this pool from 2 separate backup product, these all completed without error during the device removal process and seemed to not have major impact on the removal process (perhaps slowing it down some).

I appreciate all of the assistance with this!