Move HDD between VDEVs

[The_Saver]

Hello everyone,

I am thinking about switching to TrueNAS and ZFS permanently. I have installed it twice to test a few things including upgrading HBA firmware.
In the following setup with 8+8+8=24 HDD I would like to hear if what I am planning to do is a smart choice.


VDEV 1:
8 * 1 TB HDD in RAIDZ-3

VDEV 2:
8 * 2 TB HDD in RAIDZ-3

VDEV 3:
8 * 3 TB HDD in RAIDZ-3


In a scenario where disk in VDEV 3 dies I would replace it with a 4 TB HDD. If all 8 disks in VDEV 3 are replaced by 4 TB HDDs I will have 33% additional storage capacity going from 3 to 4 TB per HDD. In other words, a 3 TB HDD gets thrown out and is directly replaced by a 4 TB HDD.

If a disk from VDEV 2 dies, I would remove it from the system and buy a 4 TB HDD. I would then proceed by unplugging a 3 TB HDD from VDEV 3 and then reformat it on another computer. Then I will use that 3 TB HDD and replace the failed disk in VDEV 2. Then I will add the newly bought 4 TB HDD and add it to VDEV 3. In other words a 2 TB HDD gets thrown out (from VDEV 2) and a 3 TB HDD (from VDEV 3) takes over its spot. The 3 TB HDD is taken over by the newly bought 4 TB HDD (new HDD added to VDEV 3).

If a disk from VDEV 1 dies, the same happens as with VDEV 2, but just with one more step. A disk from VDEV 2 will be wiped and made to fill the void in VDEV 1. Then the void from VDEV 2 will be filled by an HDD from VDEV 3. Lastly, the void in VDEV 3 will be filled by a new 4 TB disk.

In the future I can move up to 5 TB HDDs, then 6 TB and so on.

In this manner, I can continually upgrade my storage capacity while still sticking to 24 disks and one server and having great redundancy, even when moving disks from VDEVs.

As I have never used ZFS before I would like to hear if you have any suggestions and see any problems with this approach. Perhaps you know of someone who has done the same thing? I have tried to search for people in a similar situation but without luck - maybe my idea is a bit outside the overton window.

 

[sretalla]

In a scenario where disk in VDEV 3 dies I would replace it with a 4 TB HDD. If all 8 disks in VDEV 3 are replaced by 4 TB HDDs I will have 33% additional storage capacity going from 3 to 4 TB per HDD.

Seems fine.

If a disk from VDEV 2 dies, I would remove it from the system and buy a 4 TB HDD. I would then proceed by unplugging a 3 TB HDD from VDEV 3 and then reformat it on another computer. Then I will use that 3 TB HDD and replace the failed disk in VDEV 2. Then I will add the newly bought 4 TB HDD and add it to VDEV 3. In other words a 2 TB HDD gets thrown out (from VDEV 2) and a 3 TB HDD (from VDEV 3) takes over its spot.

Also fine... no need to wipe it outside, just use Wipe from Storage | Disks

If a disk from VDEV 1 dies, the same happens as with VDEV 2, but just with one more step. A disk from VDEV 2 will be wiped and made to fill the void in VDEV 1. Then the void from VDEV 2 will be filled by an HDD from VDEV 3. Lastly, the void in VDEV 3 will be filled by a new 4 TB disk.

Again, fine.

 

[The_Saver]

Seems fine.


Also fine... no need to wipe it outside, just use Wipe from Storage | Disks


Again, fine.

Thank you for confirming my thoughts, Wizened Sage/sretalla!

I suppose that this could be added as a feature in the future, given that is it fine to do.

Thanks for the tip with wiping directly under Storage | Disks.

 

[sretalla]

I'm not sure exactly how the "feature" would work other than the steps to perform... like offline disk from VDEV3, wipe disk, remove from chassis and insert new disk, start replace/resilver... then repeat. All those functions are there in the GUI. It's all documented.

Maybe you're thinking of enclosure management, which would be a feature related to iX hardware only.

 

[The_Saver]

I'm not sure exactly how the "feature" would work other than the steps to perform... like offline disk from VDEV3, wipe disk, remove from chassis and insert new disk, start replace/resilver... then repeat. All those functions are there in the GUI. It's all documented.

Maybe you're thinking of enclosure management, which would be a feature related to iX hardware only.

I imagine that there would be an option in the given VDEV: "Move disk to another VDEV" that would perform all these steps (with appropriate warnings).

But it seems to be very easy to do, I just couldn't find anything about doing all of these steps combined, hence why I created this thread

Once again, thanks for the help.

 

[The_Saver]

I imagine that there would be an option in the given VDEV: "Move disk to another VDEV" that would perform all these steps (with appropriate warnings).

But it seems to be very easy to do, I just couldn't find anything about doing all of these steps combined, hence why I created this thread

Once again, thanks for the help.

I can't edit my post. I mean when moving a 100% working disk from VDEV 3 to VDEV 2, where VDEV 2 has just lost a disk so that 7 out of 8 are plugged in the server. Here the new physical disk you add would go to VDEV 3 (and not VDEV 2). This way VDEV 3 will always have the largest and newest disks and it is possible to continually increase the capacity of the pool.

 

[sretalla]

I would say that your proposed actions are not something that would be common... even if they are fine to do, I don't think it's something that would make it to a feature due to so few people needing to do it and the time and consideration needed for making such a feature clear enough to be well understood.

If I were to do something like that (and actually I have done it in the case of 3 disks), I would need to take into account the position in the chassis as I stagger my VDEVs across the chassis (for heat distribution) and 6 backplanes to avoid loss of any one VDEV in the case of a backplane unit failure:

My 3 VDEVs are RAIDZ2 in case anyone is wondering.