Swap disks and merge volumes

[sgaar]

All,

I have a 8 disk system running TrueNAS 12.0 U3. The system has 8 disks split between 2 volumes both configured as RaidZ1. I am seeing a disk stating to fail and with the overall age of the disks I am thinking about changing all of them out. With the amount of data I have, there really isn't a possibility to simply move the data somewhere and then redo the setup.

Vol1 are 4*3TB and Vol2 are 4*2TB. Vol1 is pretty full at 83% while Vol2 is only 20% utilized. The new disks will all be 4TB disks so as they get swapped out, there will be more space available on the volume.

The question I have is around how TrueNAS will behave when I start swapping out the drives. My thought is to start swapping in new drive in Vol1 and have the system re-silver the disks and just work through the 4 until it is done. Will TrueNAS see the additional space and automatically expand the volume after the last of the disks are swapped, or will it need to be 'manually' expanded? -if manually; what would the process be for that and are there any risks?

To deal with Vol2, I am thinking to simply copy over the 20% that is used to Vol1, remove Vol2, add new disks and add those new disks to Vol1 and have the system rebuild and rebalance the data. How does that sound? Will it work or are there gaping holes in my plan that will cause it to fail.

Please comment.

 

[sgaar]

I guess this has not been done before?

 

[sretalla]

My thought is to start swapping in new drive in Vol1 and have the system re-silver the disks and just work through the 4 until it is done. Will TrueNAS see the additional space and automatically expand the volume after the last of the disks are swapped, or will it need to be 'manually' expanded?

It's Automatic and will work as you expect... risk being that while you swap out a disk, you lose your redundancy, so a second lost disk in that pool is pool death.

To deal with Vol2, I am thinking to simply copy over the 20% that is used to Vol1, remove Vol2, add new disks and add those new disks to Vol1 and have the system rebuild and rebalance the data. How does that sound? Will it work or are there gaping holes in my plan that will cause it to fail.

Depends on how you think "adding those disks" will work...

You can't change the number of disks in a VDEV of RAIDZ type (at least not for some time/years), so the "rebuilding and rebalancing" you're thinking of won't happen that way.

If you're proposing adding a second VDEV to vol1, that will "work" but as you go to bigger disks and a bigger pool in total, you'll be taking additional risk (RAIDZ1 is already considered too great a risk for disks over 1TB).

What would be recommended (as a final setup) is a single pool with a single RAIDZ2 VDEV of 8 disks.

That doesn't tell you how to get there and that will in itself be difficult as it would need you to be able to fit all of your data on no more than 2 of your disks in order to do an in-place shuffle to a new pool without a backup/alternate storage location to help with an interim step.

Also, these parts from your build seem to be problematic:
ServerRAID M1015 (seems to be a RAID card... see here: https://www.truenas.com/community/t...s-and-why-cant-i-use-a-raid-controller.81931/ )
2 x 120GB SSD Mirrored Cache drives (you don't need to mirror L2ARC, it should be striped if anything... but you don't really have enough RAM to be thinking about L2ARC... 64GB should be the low end)

 

[ddaenen1]

I guess this has not been done before?

I have done this before. Just put the disk you want to change "offline", replace with empty disk and then "replace". You continue this until all are replaced. Please be aware that your pool size will not expand until the last disk has been changed.

 

[sgaar]

It's Automatic and will work as you expect... risk being that while you swap out a disk, you lose your redundancy, so a second lost disk in that pool is pool death.

Yeah, that is the gamble...

If you're proposing adding a second VDEV to vol1, that will "work" but as you go to bigger disks and a bigger pool in total, you'll be taking additional risk (RAIDZ1 is already considered too great a risk for disks over 1TB).

I was hoping it would be able to expand....time for plan B

What would be recommended (as a final setup) is a single pool with a single RAIDZ2 VDEV of 8 disks.

That doesn't tell you how to get there and that will in itself be difficult as it would need you to be able to fit all of your data on no more than 2 of your disks in order to do an in-place shuffle to a new pool without a backup/alternate storage location to help with an interim step.

I just might have to move data around quite a bit to get this to work. I have a second NAS where I can move things to clear this one. That will allow me to wipe both Volumes and create one from scratch as RAIDZ2 with all 8 new drives.

Also, these parts from your build seem to be problematic:
ServerRAID M1015 (seems to be a RAID card... see here: https://www.truenas.com/community/t...s-and-why-cant-i-use-a-raid-controller.81931/ )

The M1015 has been crossflashed to put it in IT mode. I cannot remember what rev of the FW but I believe it is 10.16. Likewise, when I run smartctl against one of my drives I do see the device information:

=== START OF INFORMATION SECTION ===
Model Family: Western Digital Red
Device Model: WDC WD30EFRX-68EUZN0
Serial Number: WD-WCC4N2RF53T5
LU WWN Device Id: 5 0014ee 2b6be350a
Firmware Version: 82.00A82
User Capacity: 3,000,592,982,016 bytes [3.00 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Rotation Rate: 5400 rpm
Device is: In smartctl database [for details use: -P show]
ATA Version is: ACS-2 (minor revision not indicated)
SATA Version is: SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is: Tue Apr 27 11:54:21 2021 EDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

That should make it OK, correct?

2 x 120GB SSD Mirrored Cache drives (you don't need to mirror L2ARC, it should be striped if anything... but you don't really have enough RAM to be thinking about L2ARC... 64GB should be the low end)

For the 2 SSD drives, they are configured as 'special' when looking at the zpool status. I configured then as VDEVs:

root@NAS2:~ # zpool status
pool: VOL1
state: ONLINE
scan: scrub repaired 0B in 06:42:23 with 0 errors on Sun Apr 11 06:42:24 2021
config:

NAME STATE READ WRITE CKSUM
VOL1 ONLINE 0 0 0
raidz1-0 ONLINE 0 0 0
gptid/0991ac25-944a-11e7-bbeb-d05099c06b76 ONLINE 0 0 0
gptid/0a790abe-944a-11e7-bbeb-d05099c06b76 ONLINE 0 0 0
gptid/0b61527c-944a-11e7-bbeb-d05099c06b76 ONLINE 0 0 0
gptid/0c46fa96-944a-11e7-bbeb-d05099c06b76 ONLINE 0 0 0
special
gptid/156da2fd-13a9-11eb-98b7-d05099c2bdf1 ONLINE 0 0 0

errors: No known data errors

pool: VOL2
state: ONLINE
scan: scrub repaired 0B in 01:17:37 with 0 errors on Tue Mar 23 11:12:23 2021
config:

NAME STATE READ WRITE CKSUM
VOL2 ONLINE 0 0 0
raidz1-0 ONLINE 0 0 0
gptid/35dd4ccc-944a-11e7-bbeb-d05099c06b76 ONLINE 0 0 0
gptid/36cfaab7-944a-11e7-bbeb-d05099c06b76 ONLINE 0 0 0
gptid/37a79cec-944a-11e7-bbeb-d05099c06b76 ONLINE 0 0 0
gptid/3889244d-944a-11e7-bbeb-d05099c06b76 ONLINE 0 0 0
special
gptid/d854964b-13a8-11eb-98b7-d05099c2bdf1 ONLINE 0 0 0

errors: No known data errors

pool: freenas-boot
state: ONLINE
status: Some supported features are not enabled on the pool. The pool can
still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
the pool may no longer be accessible by software that does not support
the features. See zpool-features(5) for details.
scan: scrub repaired 0B in 00:00:33 with 0 errors on Sun Apr 25 03:45:33 2021
config:

NAME STATE READ WRITE CKSUM
freenas-boot ONLINE 0 0 0
ada2p2 ONLINE 0 0 0

errors: No known data errors

 

[sretalla]

If those are special VDEVs (perhaps metadata) and not L2ARC (so not cache), then those drives are pool integral and your pool is dead if either of them is lost respectively.

You could benefit in the new setup by using them in a mirror (although strictly to match RAIDZ2 you need a 3-way mirror).

 

[sgaar]

OK, thank you. So maybe it would be better to simply not have them as part of the setup? I have a hard time seeing any actual benefit to having them when you look at the transfer rates and cache hit percentage.

 

[sretalla]

So maybe it would be better to simply not have them as part of the setup?

For what you described as your needs, maybe that's true.

You could also consider adding one of them as L2ARC with your pool set to metadata only for L2ARC, which can help in cases where metadata speed matters, but without the risk of a pool-integral VDEV.