Procedure to shrink vdev - any gotchyas?

[Stolen_Walnut]

I'm trying to add a device to vdev, but am getting a device is too small error

Code:

root@tardinas[/]# zpool attach np b77e82f4-3759-4e3d-bfed-9dc70a5e6d79 /dev/nvme0n1
cannot attach /dev/nvme0n1 to b77e82f4-3759-4e3d-bfed-9dc70a5e6d79: device is too small

Context... my np pool is two mirrored 2TB NVMe drives... unfortunately i wasn't careful when I ordered them so one is actually 2048 GB & the other is 2000 GB. (Long term, I plan to replace the 2048 with a 2000 drive, but in the interim...). I wanted to remove one of the drives from the pool to do some quick performance testing on it & then add it back to the pool.

I removed the 2000GB drive and did the performance testing easy enough. Then came time to re-add the drive... Didn't see where to do this in the GUI, so went to the command line which is when I got this error.

I dug around the forums & basically saw that there was no way to do this and that shrinking isn't supported (unless I didn't hunt hard enough). The general consensus was to create a new pool, and then replicate to that.

I have a VM running out of this pool, so that sounded cumbersome to stop services / VM / etc.

So what I did was this:

1. (starting)
   1. Vdevs:
      1. stripe: {nvme1(2024GB)}
2. GUI: Add nvme0
   1. Vdevs:
      1. stripe: {nvme1(2024GB)}
      2. stripe: {nvme0(2000GB)}
3. CLI: zpool remove np b77e82f4-3759-4e3d-bfed-9dc70a5e6d79
   1. (Data evacuated off of nvme1)
   2. Vdevs:
      1. stripe: {nvme0(2000GB)}
4. zpool attach np 28503698-8f41-4df9-b1c8-34d55915f3e9 b77e82f4-3759-4e3d-bfed-9dc70a5e6d79
   1. (285...3e9 is the id created on nvme0(2000GB) from adding in the GUI)
   2. Vdevs:
      1. mirror-1: {nvme0(2000GB), nvme1(2024GB)}

Obviously it would have been ideal to have some redundancy while i tested & while I did this dance to "shrink" the vdev (although technically I guess it's a different vdev), but the VM is new enough to where it wouldn't have been the end of the world if I did lose everything in there.

More over, it looks like this could have been executed in a manner to yield redundancy throughout the process (add a temp drive before removing the one of interest, add that drive back in with another scratch disk, so redundancy is present during the evacuation).

Other than that -- are there any gotchya's or no-nos that I should be aware in this process?

Show : OS/HW

"tardinas"
OS Version: Currently TrueNAS-SCALE-22.12.0, but flexible if something makes more sense
Motherboard: MSI MEG Z690I UNIFY
CPU: Intel Core i5-12500
RAM: G.Skill Ripjaws S5 64GB (2 x 32GB) DDR5-5200
Drives

• Pools
  • boot-pool: SATA SSD - Samsung 870 EVO Series 2.5" 250GB
  • np:
    • mirror-0
      • NVMe: Inland Premium 2TB SSD M.2
      • NVMe: Crucial - P3 2TB Internal SSD
      • Note: I'm aware these are different sizes... One will be replaced with identical size before continuing

Hard disk controllers

• 3x integrated/on-board SSD/NVMe controllers:
  • lspci -nnn | grep -E "0106|0108"
    • 00:17.0 SATA controller [0106]: Intel Corporation Device [8086:7ae2] (rev 11)
    • 09:00.0 Non-Volatile memory controller [0108]: Micron/Crucial Technology Device [c0a9:540a] (rev 01)
    • 13:00.0 Non-Volatile memory controller [0108]: Phison Electronics Corporation PS5013 E13 NVMe Controller [1987:5013] (rev 01)

Network cards

• 2x:
  • lspci -nnn | grep 0200
    • 08:00.0 Ethernet controller [0200]: Aquantia Corp. AQC107 NBase-T/IEEE 802.3bz Ethernet Controller [AQtion] [1d6a:07b1] (rev 02)
    • 0b:00.0 Ethernet controller [0200]: Intel Corporation Ethernet Controller I225-V [8086:15f3] (rev 03)

 

[sretalla]

Obviously it would have been ideal to have some redundancy while i tested & while I did this dance to "shrink" the vdev

Better would have been:

Existing Pool with NVME1. Stop VMs.

Create new Pool with NVME0

snapshot | zfs send | recv from NVME1 to NVME0 pool.

Do your checks (data now on both pools).

Destroy NVME1 pool then rename NVME0 pool to same as NVME1 pool

Attach NVME1 to NVME0 pool.

Done without having a striped pool.

 

[danb35]

Then came time to re-add the drive... Didn't see where to do this in the GUI,

On the Storage page, click Manage Devices on the card for the pool in question. Click the relevant vdev, and on the right, use the Extend button to add a device.

 

[Stolen_Walnut]

On the Storage page, click Manage Devices on the card for the pool in question. Click the relevant vdev, and on the right, use the Extend button to add a device.

Ahhh. Total oversight on my part. Still picking up the lingo. I saw "extend" and was thinking that would be adding more space to the pool (add vdev), but wouldn't ya know it -- that's the "Add VDEV" button. Makes sense. Thanks

Better would have been:

Existing Pool with NVME1. Stop VMs.

Create new Pool with NVME0

snapshot | zfs send | recv from NVME1 to NVME0 pool.

Do your checks (data now on both pools).

Destroy NVME1 pool then rename NVME0 pool to same as NVME1 pool

Attach NVME1 to NVME0 pool.

Done without having a striped pool.

Hmmmmm. Interesting approach, but I'm struggling with the "better" terminology -- Is this in the name of not temporarily having a striped pool?

The main thing I was after was a procedure that doesn't involve stopping the VM.

Let me clarify: Are there any holes to poke in the following procedure? (sidesteps sensitivity to redundancy concerns)

Aim: "Add" 2000GB drive B to mirror vdev with drive A & C without stopping pool / services using the pool:

1. start
   1. np pool
      1. mirror1: {drive A, drive C} - both 2048GB (wouldn't be able to directly add 2000GB drive B here directly
2. add D & B for redundancy
   1. np pool
      1. mirror1: {drive A, drive C} - both 2048GB
      2. mirror2: {drive D, drive B} - both 2000GB
3. remove mirror1 (which evacuates data to mirror2)
   1. np pool
      1. mirror2: {drive D, drive B} - both 2000GB
4. add A & C
   1. np pool
      1. mirror2: {drive D(2000), drive B(2000), drive A(2048), drive C(2048)}
5. remove temp drives
   1. np pool
      1. mirror2: {drive B(2000), drive A(2048), drive C(2048)}

As I see it, the main con is that it requires another drive (or drives depending on topology) and zaps some life from the temp drives. Seems like a small price to pay in order to not disrupt services. Are there any cons I'm missing? Is there something inherently bad about introducing a stripe that I'm missing?

Thanks

 

[sretalla]

Is this in the name of not temporarily having a striped pool?

yes.

I see that you wanted to keep the VM running though, so fair point.

If something happens while in a stripe (and no separate copy), you'll lose the pool.

 

[Stolen_Walnut]

Got it -- makes sense. I appreciate the input.