Taking every disk in a vdev offline? What happens?

[KirklandIceCream]

I'm playing a game of musical chairs with an old server with data, and a new server, and moving the HDDs around etc.

I have a server that had 2 vdevs, each 3 disks. I had bought 3 more HDDs to replace the 3 disks in 1 vdev then move the HDDs over, HOWEVER my dumb dumb self added the new HDDs as a new vdev. Now I have 3 vdevs, each at 3 disks.

What happens if I were to just take a disk offline from 1 vdev, do a manual scrub, and repeat that until all the disks in that 1 vdev were now offline? Would the data be resilvered to the other vdevs?

The goal is to use the old 3 HDDs (from the old server), into the new server, then transfer the data over, then destroy the entire pool from the old server and create a new pool in the old server.

Does that strategy work, or any other solutions?

 

[HoneyBadger]

Welcome.

The short version of "what happens if every disk in a vdev goes offline" is "the vdev fails, and with it, your pool"

Can you show the view from Storage -> Pools -> (Gear Icon) -> Status?

If you have RAIDZ1 vdevs, you're unfortunately stuck with them now, as there's no way to remove that type of vdev. If you have mirror vdevs though, those can be removed.

 

[KirklandIceCream]

Welcome.

The short version of "what happens if every disk in a vdev goes offline" is "the vdev fails, and with it, your pool"

Can you show the view from Storage -> Pools -> (Gear Icon) -> Status?

If you have RAIDZ1 vdevs, you're unfortunately stuck with them now, as there's no way to remove that type of vdev. If you have mirror vdevs though, those can be removed.

It's Raidz1. I know I can't remove the vdev, but I'm wondering if I can offline each disk, have it resliver across the other disks in the pool (other VDEVS), then since the data is now just on other disks on otehr vdevs, I can just physically remove the hard drives that were all offline, put them in the new server?

What I'm basically asking is can every disk in a vdev be offlined, assuming the remaining vdevs have sufficient storage to hold the data from that offlined vdev?

 

[Patrick M. Hausen]

No. Redundancy is per vdev. So if you are willing to take that risk you can offline one disk of each RAIDZ1 vdev and still access your data as long as no other disk fails. ZFS will not move around/resilver across vdevs.

 

[Ericloewe]

If you have mirror vdevs though, those can be removed.

If you only have mirror vdevs.

 

[HoneyBadger]

What I'm basically asking is can every disk in a vdev be offlined, assuming the remaining vdevs have sufficient storage to hold the data from that offlined vdev?

Unfortunately, no. Data is spread across all vdevs in a pool, and only mirror vdevs can be "evacuated" in the manner you're describing; and only to other non-RAIDZ vdevs. Your top-level vdevs must be mirrors or single devices - adding a RAIDZ vdev anywhere into the pool results in the zpool-remove feature being unavailable.

No. Redundancy is per vdev. So if you are willing to take that risk you can offline one disk of each RAIDZ1 vdev and still access your data as long as no other disk fails. ZFS will not move around/resilver across vdevs.

This would work, with the "Here Be Dragons" caveat that you would be degrading all three of your existing vdevs in order to build a three-drive vdev on the new server. A subsequent drive failure in the main unit would be rather catastrophic.

If you only have mirror vdevs.

Yes; apologies if that wasn't clear. I was hoping that @KirklandIceCream was using 3-way mirrors for their other vdevs.

 

[KirklandIceCream]

No. Redundancy is per vdev. So if you are willing to take that risk you can offline one disk of each RAIDZ1 vdev and still access your data as long as no other disk fails. ZFS will not move around/resilver across vdevs.

Thanks everyone! I am going to do this solsution. 1 HDD from each pool, move to new server, transfer the data, and then destroy the old pool.

Lesson learned. I'm printing a little paper that says "REPLACE. DON'T CREATE A NEW VDEV" next to my server lol

 

[Whattteva]

Thanks everyone! I am going to do this solsution. 1 HDD from each pool, move to new server, transfer the data, and then destroy the old pool.

Lesson learned. I'm printing a little paper that says "REPLACE. DON'T CREATE A NEW VDEV" next to my server lol

Since it sounds like you have two servers (old and new), why not just set up a ZFS replication?