Change zfs pool layout

[joncy92]

Hi All,

I'm sure this question has been answered previously but I'll admit I wasn't sure what to search.

I currently have a ZFS pool with 2 data VDEVs or mirrors.
VDEV 1 has 2 6TB disks in RAID1
VDEV 2 has 2 2TB disks in RAID1.

zpool status:

root@freenas[/]# zpool status
pool: Nasuni
state: ONLINE
scan: scrub repaired 0 in 0 days 14:02:07 with 0 errors on Sun May 10 14:02:08 2020
config:

NAME STATE READ WRITE CKSUM
Nasuni ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
gptid/4086212e-16e7-11ea-865d-109836a0cf27 ONLINE 0 0 0
gptid/83f3c00f-1d32-11ea-bb3b-109836a0cf27 ONLINE 0 0 0
mirror-1 ONLINE 0 0 0
gptid/f79ba1ea-1d33-11ea-bb3b-109836a0cf27 ONLINE 0 0 0
gptid/044d1ca3-1d34-11ea-bb3b-109836a0cf27 ONLINE 0 0 0

The pool is then of course striped across the two VDEVs.

I'm now running out of capacity so I've ordered 2 new 6TB disks which are the same as the two I currently have to replace the two 2TB disks (they're old and my chassis only has room for 4 disks).
I understand I can replace one of the 2TB disks, resilver, then add the next 6TB and resilver again, but that will only leave me with 12TB of usuable storage.

What I want to do is "rebuild" the pool so that I can have RAIDZ parity across the four 6TB disks.
How do I "re-arrange" the zpool?

Any help appreciated! Thanks!

 

[Samuel Tai]

Unfortunately, you'll have to backup your data, destroy your current pool, and recreate it in the layout you want.

 

[joncy92]

This is the only way? The issue is I don't have 7TB of storage somewhere to back it all up :(

 

[danb35]

This is the only way?

Yes, that is the only way. And you should really re-think doing RAIDZ on 6TB disks.

 

[joncy92]

Yes, that is the only way. And you should really re-think doing RAIDZ on 6TB disks.

Thanks - I'm honestly at a loss with ZFS really. I've read through documentation and posts but I still don't really understand it that well.
Do you mean I should be using one mirror with all 4 disks?

 

[danb35]

Do you mean I should be using one mirror with all 4 disks?

No, and this isn't really anything unique to ZFS. Single-parity RAID (i.e., RAID5/RAIDZ1) isn't really recommended for disks larger than about 1 TB due to the risk of a read error during a rebuild with disks of that size. It's unlikely, with ZFS, that such an error would be fatal to your pool, but it could still result in data loss.

 

[joncy92]

No, and this isn't really anything unique to ZFS. Single-parity RAID (i.e., RAID5/RAIDZ1) isn't really recommended for disks larger than about 1 TB due to the risk of a read error during a rebuild with disks of that size. It's unlikely, with ZFS, that such an error would be fatal to your pool, but it could still result in data loss.

Thanks I think I understand - so if one disk fails in parity, another disk could fail during the rebuild and destroy the entire pool.
What would be the best thing to do now with my 4 6TB disks?

 

[danb35]

so if one disk fails in parity, another disk could fail during the rebuild and destroy the entire pool.

If another disk fails, yes, the entire pool is lost. But if there's a read failure on one of the remaining disks, even without a complete failure of the disk, that will cause data loss.

What would be the best thing to do now with my 4 6TB disks?

The best I can suggest is to proceed with your original plan--replace the 2 TB disks, one at a time, to get the extra space.

 

[joncy92]

If another disk fails, yes, the entire pool is lost. But if there's a read failure on one of the remaining disks, even without a complete failure of the disk, that will cause data loss.

The best I can suggest is to proceed with your original plan--replace the 2 TB disks, one at a time, to get the extra space.

Great, thanks very much for the clarification!
Just out of interest and for future reference, if I have 6 or more disks of the same capacity, what would be the best layout for maximum storage with redundency?

 

[Redcoat]

If you were in the US I'd suggest that you went to Best Buy and purchased a 10TB WD Easystore for $199.99 to back up your data to it as @Samuel Tai suggested.

 

[danb35]

what would be the best layout for maximum storage with redundency?

For groups of 6-10 disks, RAIDZ2 works well. But to expand that vdev, you now need to replace 6-10 disks, not 2.

 

[elorimer]

Also, if these are WD Red 6TB drives check out the SMR thread before using them.

 

[joncy92]

Also, if these are WD Red 6TB drives check out the SMR thread before using them.

Thanks very much for highlighting that as I was completely unaware. The drives were indeed EFAX models so I've cancelled the order.
What would be recommended for a 6TB disk size?

For groups of 6-10 disks, RAIDZ2 works well. But to expand that vdev, you now need to replace 6-10 disks, not 2.

So I understand that a pool of 4 disks in RAIDZ may be partially or completely lost if another disk fails during replacement, but isn't that also the case with two mirrors with one disk redundancy? The second disk my fail during resilver?
What's the best option when starting from scratch with 4 disks for redundancy, performance and capacity?

 

[Samuel Tai]

You have your choice of RAIDZ1, Z2, or Z3. For 4x 6TB drives, Z1 will result in 6 TB unavailable, reserved for parity, so the pool will end up with 18 TB. Z2 has 2x parity drives, so available space will be 12 TB. Z3 uses 3 disks for parity, so total space will be 6 TB.

I run Z2 as the best compromise between safety and space. I can lose up to 2x drives before a resilver risks destroying my pool.

 

[elorimer]

This is the only way? The issue is I don't have 7TB of storage somewhere to back it all up :(

This suggests something else for you to think about: a backup strategy for your pool. The SMR problem was so wicked because it made any RAID strategy untenable: you had to assume the loss of the RAID Vdev and the need to restore from backup.

You don't need to backup the whole pool, you only need to back up datasets. So you can spread your backups over smaller drives if you have them, and each dataset can have a different backup strategy. For some datasets you may not need a backup at all if you can tolerate the loss of all the data in the dataset (like a dataset that itself is a backup). The backup also can use most of the capacity in those drives, too, because you don't need to hold the datasets below 50-80% for performance reasons and snapshotting.

Personally, when I added a fourth drive, I blew up my 3x 4tb RAIDZ1 pool and made 2 mirrored vdevs, restoring from my backups where I needed to. I ended up with the same 8Tb storage, but in this configuration expanding the pool requires adding 2 drives of larger capacity, not 4, and with the faster mirroring process rather than the full RAID resilvering.

So I second the idea of a USB drive of significant capacity as a backup pool, in a size that reflects whatever you need to accommodate your backup strategy, and then using mirrors. For you, expanding the mirrors is easy. If you are going from a four drive Z1 pool to mirrors, another possibility is to get two USB drives in the size of what you want for expansion, destroy your pool, create it in the new configuration, restore it from the USB drives, shuck them, then put them into the mirrors, and put the old drives into the USB housings for backup going forward.

 

[joncy92]

Thanks all I

This suggests something else for you to think about: a backup strategy for your pool. The SMR problem was so wicked because it made any RAID strategy untenable: you had to assume the loss of the RAID Vdev and the need to restore from backup.

You don't need to backup the whole pool, you only need to back up datasets. So you can spread your backups over smaller drives if you have them, and each dataset can have a different backup strategy. For some datasets you may not need a backup at all if you can tolerate the loss of all the data in the dataset (like a dataset that itself is a backup). The backup also can use most of the capacity in those drives, too, because you don't need to hold the datasets below 50-80% for performance reasons and snapshotting.

Personally, when I added a fourth drive, I blew up my 3x 4tb RAIDZ1 pool and made 2 mirrored vdevs, restoring from my backups where I needed to. I ended up with the same 8Tb storage, but in this configuration expanding the pool requires adding 2 drives of larger capacity, not 4, and with the faster mirroring process rather than the full RAID resilvering.

So I second the idea of a USB drive of significant capacity as a backup pool, in a size that reflects whatever you need to accommodate your backup strategy, and then using mirrors. For you, expanding the mirrors is easy. If you are going from a four drive Z1 pool to mirrors, another possibility is to get two USB drives in the size of what you want for expansion, destroy your pool, create it in the new configuration, restore it from the USB drives, shuck them, then put them into the mirrors, and put the old drives into the USB housings for backup going forward.

I do actually have a few spare disks lying around but the largest dataset is 5.9TiB.
So the ideal way forward would be order a 12TB Disk for backup and backup eveything, then rebuild the pool from the 4 6TB disks and then restore the backup?

Once that's done I can expand the RAIDZ2 by adding 2 6TB disks each time?

 

[danb35]

Once that's done I can expand the RAIDZ2 by adding 2 6TB disks each time?

No, that's a drawback of any flavor of RAIDZn. Once you've created a RAIDZn vdev, you can neither add disk to, nor remove disks from, it. You'd expand the pool by adding another RAIDZ2 vdev, which would have no fewer than four disks. This point favors mirrors--you can easily expand the pool by adding a pair of disks at a time.

 

[joncy92]

No, that's a drawback of any flavor of RAIDZn. Once you've created a RAIDZn vdev, you can neither add disk to, nor remove disks from, it. You'd expand the pool by adding another RAIDZ2 vdev, which would have no fewer than four disks. This point favors mirrors--you can easily expand the pool by adding a pair of disks at a time.

Okay, hence your recommendation to just add the 6TB disk, resilver and then repeat?
While thinking about backups then, with mirrors having just one disk fault tolerance, how do I improve on fault tolerance? Just have an external backup nightly to a very large disk?

 

[elorimer]

I do actually have a few spare disks lying around but the largest dataset is 5.9TiB.

I suppose there are two ways of approaching this. If you are transitioning from one pool design to another, you need to be able to backup and then restore your data. So if your 5.9TiB dataset is a bunch of movies, you can copy off your movies to any number of places, create the new dataset on the new pool, and then copy the movies back. So you don't need to deal with the dataset as a unit.

If you are looking to maintain a current backup on an ongoing basis, then replicating a dataset in whole is easier. If your dataset is a bunch of movies, you may not need snapshots covering a long period of time--not more than it would take to recognize you got ransomwared.

So the ideal way forward would be order a 12TB Disk for backup and backup eveything, then rebuild the pool from the 4 6TB disks and then restore the backup?

This is a great opportunity to plan out things. By way of example, in my pool I have several datasets. One is a dataset of backups of PCs in my network. It has grown over time. If I pruned it to the last backup the dataset would be much smaller. If a deleted all this data in the short term, I would still have the PCs themselves. So for a short term transition I might not worry about that dataset. Another dataset is my own backup of Google Drive and OneDrive setups. Those I can certainly trash in the short term and recreate later.

I went in the direction of a 4 drive mirror of two vdevs, so I had one level of protection, with the ability to expand the pool with a pair of bigger drives. Then I have a backup on a USB drive for the portion I need, excluding the portion I don't really need. I found the Raid vs mirror thread really useful. YMMV

 

[joncy92]

Thanks everyone for you input, it's been invaluable.

I've decided to replace the 2TB drives with 8TB WD Reds (because they are still CMR) and continue with a pool of 2 mirrors. I'll have to accept the risk of single disk failure tolerance per mirror for now and when I can afford them, I'll purchase another 8TB and 6TB disk to have a 3disk mirror in each vdev for greater resilience.

Please do share your thoughts on this approach.

Thanks again