-
Important Announcement for The TrueNAS Community.
The TrueNAS Community has now been moved. This forum will now become READ-ONLY for historical purposes. Please feel free to join us on the new TrueNAS Community Forums
Mirrored vdevs: 3x2 vdevs vs 2x3 Sr
- Thread starter Phase
- Start date
- Joined
- Jan 1, 2016
- Messages
- 9,703
Take the scenario of a 2 disk loss:
2x3 RAIDZ1
A B C + D E F
vs
3x2 Mirrors
A B + C D + E F
If drives A & F are lost...
Both scenarios are still OK, both VDEVs degraded in the first case and 2/3 VDEVs degraded in the second.
Lose drives C & D and in the first scenario, still OK, but in the second, the pool is dead.
Lose B & C, the first scenario you lose the pool, 2 degraded VDEVs in the second.
Actually this is quite easy to play out in a matrix to show it visually
Red is a lost pool, yellow is one or more degraded VDEVs.
As you can see the RAIDZ1 scenario lands you in a lost pool twice as often as the mirrors do.
In all cases, a single lost drive will leave one VDEV with no redundancy. In the RAIDZ scenario, there's around twice the chance that the second failure is in that same VDEV
I'm sure somebody will point out that this doesn't need to be a complete matrix as it double-counts everything, so you can stop reading at the N/A on each row and still come to the same conclusion. (A + B fail is the same as B + A fail... etc.)
2x3 RAIDZ1
A B C + D E F
vs
3x2 Mirrors
A B + C D + E F
If drives A & F are lost...
Both scenarios are still OK, both VDEVs degraded in the first case and 2/3 VDEVs degraded in the second.
Lose drives C & D and in the first scenario, still OK, but in the second, the pool is dead.
Lose B & C, the first scenario you lose the pool, 2 degraded VDEVs in the second.
Actually this is quite easy to play out in a matrix to show it visually
Red is a lost pool, yellow is one or more degraded VDEVs.
As you can see the RAIDZ1 scenario lands you in a lost pool twice as often as the mirrors do.
In all cases, a single lost drive will leave one VDEV with no redundancy. In the RAIDZ scenario, there's around twice the chance that the second failure is in that same VDEV
I'm sure somebody will point out that this doesn't need to be a complete matrix as it double-counts everything, so you can stop reading at the N/A on each row and still come to the same conclusion. (A + B fail is the same as B + A fail... etc.)
Last edited:
A ha!
Thank you very much for the thoughtful reply!
I thought that if and entire stripe went down it could be rebuilt from the other.
This morning I was looking at other options and I found a thread in another forum (not sure is posting to another forum is allowed) where the person ("John4200") calculated the chance of loosing a pool based on the probability of a single drive failing during a rebuild. He did that for RAID10 (n/2 vdevs in RAID1), RAIDZ1, , RAIDZ2 and , RAIDZ3. The results were quite surprising.
We are now doing 8+1 disks -- 8 disks in the pool with 1 hot spare. All disks are exactly the same model, Toshibas 14TB MG07ACA14TE.
So I plugged his formulas in excel and keeping the pool at 8 disks and assuming a 5% probability of a second drive failing during a rebuilt the numbers came up to the following (the storage figures are not 100% accurate, but I do not know how to calculate them... but they should be directional:
I do not know if 5% is a reasonable figure for a single drive failing during the rebuild, but that is what the guy used.
I attached the spreadsheet where I put his formulas.
So based on the above I think we are going with an 8-disk RAIDZ3 with the 9th disk being a hot spare.
Does that sound right?
Thank you very much for the thoughtful reply!
I thought that if and entire stripe went down it could be rebuilt from the other.
This morning I was looking at other options and I found a thread in another forum (not sure is posting to another forum is allowed) where the person ("John4200") calculated the chance of loosing a pool based on the probability of a single drive failing during a rebuild. He did that for RAID10 (n/2 vdevs in RAID1), RAIDZ1, , RAIDZ2 and , RAIDZ3. The results were quite surprising.
We are now doing 8+1 disks -- 8 disks in the pool with 1 hot spare. All disks are exactly the same model, Toshibas 14TB MG07ACA14TE.
So I plugged his formulas in excel and keeping the pool at 8 disks and assuming a 5% probability of a second drive failing during a rebuilt the numbers came up to the following (the storage figures are not 100% accurate, but I do not know how to calculate them... but they should be directional:
I do not know if 5% is a reasonable figure for a single drive failing during the rebuild, but that is what the guy used.
I attached the spreadsheet where I put his formulas.
So based on the above I think we are going with an 8-disk RAIDZ3 with the 9th disk being a hot spare.
Does that sound right?
- Joined
- Jan 1, 2016
- Messages
- 9,703
Well, it looks like you're very keen to avoid any risk of pool loss... RAIDZ3 is the best option you have for that other than multi-way mirrors (which are very costly).
You might want to consider more factors than fault tolerance and disk failure during rebuilds... like your backup strategy and operational procedures.
If you monitor your disks properly with SMART tests, you will be able to do disk replacement operations long before any disk actually fails.
If you have enough spare bays, you can replace a failing disk while it's still online (reducing the load on the system and other disks in the pool during the resilver).
Hot-spares are only really good for the case where the disk has sudden (not detected with SMART first) and complete (becomes unavailable to the Operating System) failure, where a substitution and resilver are automatically started. SMART data indicating a failing disk is not enough to automatically trigger a hot-spare replacement. Which is to say that you might actually do better to have that disk burned-in and on a shelf not spinning away useful hours for your operations to use in replacing the disk as soon as you see SMART data pointing to a failing drive.
You might want to consider more factors than fault tolerance and disk failure during rebuilds... like your backup strategy and operational procedures.
If you monitor your disks properly with SMART tests, you will be able to do disk replacement operations long before any disk actually fails.
If you have enough spare bays, you can replace a failing disk while it's still online (reducing the load on the system and other disks in the pool during the resilver).
Hot-spares are only really good for the case where the disk has sudden (not detected with SMART first) and complete (becomes unavailable to the Operating System) failure, where a substitution and resilver are automatically started. SMART data indicating a failing disk is not enough to automatically trigger a hot-spare replacement. Which is to say that you might actually do better to have that disk burned-in and on a shelf not spinning away useful hours for your operations to use in replacing the disk as soon as you see SMART data pointing to a failing drive.
Thanks!
Will keep disk #9 off line. The original plan was to have a second server with the same data non-raid. But if you are not monitoring if the data goes bad it is just a hope and a prayer. I was not expecting this to get this complicated.
The one concern I have with raidz3 is whether I'll have enough CPU/memory. I'm using an i7-960 with 24GB of ram.
What are the best practices regarding "monitor your disks properly with SMART tests" -- I'm pretty sure looking at the dashboard everyday will not last past week 1, LOL.
The deed is done. Now moving data back into the new pool.
I was shocked!!! Absolutely shocked!!! That a TB was 10^12 as opposed to 2^40. I thought I was getting 14 * 1024 * 1024 * 1024 * 1024 and they are 12.7 * 1024 * 1024 * 1024 * 1024. That is such a ripoff. Freedom units man... freedom units!!! SI is a ripoff, LOL.
Anyway the 9 x 12.733 TiB in RaidZ3 yielded 68.220 TiB usable -- i.e. 10.7% loss on housekeeping and sizes of this vs. sizes of that (39.5% if you count parity)
The biggest headache is moving the data in and out over a 1Gb network... over 2 hrs per TiB.
I was shocked!!! Absolutely shocked!!! That a TB was 10^12 as opposed to 2^40. I thought I was getting 14 * 1024 * 1024 * 1024 * 1024 and they are 12.7 * 1024 * 1024 * 1024 * 1024. That is such a ripoff. Freedom units man... freedom units!!! SI is a ripoff, LOL.
Anyway the 9 x 12.733 TiB in RaidZ3 yielded 68.220 TiB usable -- i.e. 10.7% loss on housekeeping and sizes of this vs. sizes of that (39.5% if you count parity)
The biggest headache is moving the data in and out over a 1Gb network... over 2 hrs per TiB.
Similar threads
- Locked
