Drive Failure in large vdev setup. Is resilvering time an issue with Fast SAS 3 Flash systems

[Ryannology]

Hi, general question about drive failure in a large Vdev with all Flash.

20 disk SAS 3 Flash
1 Vdev of 20disks Raidz2
latest Truenas Core version

With fast modern SAS Flash is reslivering really going to be an issue taking too long? This system writes and reads maxing out my X8 PCie bus. I can't believe drive rebuild will take that long.

I am Building a backup server so I want the largest pool I can have with 20 enterprise flash disk.

Thank you

 

[Ryannology]

Hi, general question about drive failure in a large Vdev with all Flash.

20 disk SAS 3 Flash
1 Vdev of 20disks Raidz2
latest Truenas Core version

With fast modern SAS Flash is reslivering really going to be an issue taking too long? This system writes and reads maxing out my X8 PCie bus. I can't believe drive rebuild will take that long.

I am Building a backup server so I want the largest pool I can have with 20 enterprise flash disk.

Thank you

There could be up to 50 to 90 TB of data on this array at some point.

 

[Arwen]

Larger width vDevs are not recommended because of long rebuild times, and potentially longer scrub times. About 10 to 12 storage devices in a RAID-Z2/3 is considered the maximum, depending on hardware configuration. Meaning if you have exactly 12 slots, it might make sense to use them all in a single RAID-Zx vDev. Then accept the slightly slower rebuild / scrub times.

Now I would think SAS 3 flash devices would be fast enough. But, part of the problem is that if a ZFS block stripes across all SSDs in a vDev, during rebuild of a failed SSD you have to read 17 data SSDs and 1 parity SSD, 18 reads total. For a single write to the replacement SSD. That is the problem.

It is worse if the SSDs are on a single controller through a SAS Expander, which means that the single controller is going to be quite busy. Plus, it then maters if you have 4 or 8 SAS channels from the SAS controller to the SAS Expander. And if the SAS controller has 8 PCIe lanes. (And if those lanes are PCIe v2, v3 or v4, depending on what the SAS controller supports...)

Or, it might just not mater in your case.

 

[samarium]

I would be doing some estimations based on available bandwidth for each device, card, etc in the chain and look at the bottlenecks. Then assume only 80% efficient, then benchmark with a reduced data set and just do a scrub, and see how that compares. Scrub should be less intense that actual disk replacement. Then compare theoretical speed vs observed speed. If possible simulate disk failure by taking a flash device offline, erasing it, and replacing it in the pool. Compare to scrub time and estimated time. Adjust estimates and then extrapolate to full capacity.