14X4TB Z3 vs. two 7X4TB vdevs

[tom123]

Supermicro X10SL7-F with 14 SATA ports flashed to IT mode
Intel Pentium G3220, should be fast enough for CIFS only use
32GB ECC RAM

Originally I was going to use 11(2^3 +3) 4TB Seagate NAS drives in Z3. I'll say the nominal capacity of this config is 32TB.

Then I decided that the twelfth slot in my 4U chassis was sad because it didn't have a drive in it.

So I bought another drive. 12 drives, 36TB.

Now I'm leaning towards getting another 2 drives to fill up SATA ports, 44TB. I wonder if 14 drives is pushing it for Z3.

Performance is secondary to reliability in this application which will be used to store media files, backups, etc.

What about two vdevs each with 7 drives Z2? 40TB

I assume performance would be better since data would be striped across the two vdevs but I'm not sure about reliability.

Z3 could handle the loss of any 3 drives but in the two vdev config data would be lost if 3 drives failed in a particular vdev. Not sure what the probability of this is.

Any thoughts?

 

[Stux]

I'd go with 2x7 drive raidz2. Twice the random performance, faster resilvering.

You should have a backup anyway.

 

[tom123]

Thanks for the response. Performance is far secondary to reliability.

I'm wondering about the 2 X 7 Z2 config. Since the data is striped across the 2 vdevs, I believe(not an expert) that it is half as reliable as a single Z2 vdev.

I've been looking at https://jsfiddle.net/Biduleohm/paq5u7z5/1/embedded/result/

With the default values, a 14 drive Z3 has a MTTDL of 5.9 x 10^11 hour.

A 7 drive Z2 has a MTTDL of 4.6 x 10^10 hour. Divided by 2 for the stripped VDEVs, a 2X7 Z2 would have a MTTDL of 2.3 x 10^10 hour.

So that would mean the Z3 config has a 25 times greater MTTDL than the 2X7 Z2 config. If my math is correct!

But maybe the 2X7 Z2 config is statistically reliable enough that I'll be long dead before the array is likely to fail.

Resilvering time is 172 hour compared to 114 hour.

 

[gpsguy]

IMHO 14 drives is too wide for a single vdev. Take Stux's advice and go with 2 7 disk RAIDz2 vdev's.

 

[Mirfster]

Now I'm leaning towards getting another 2 drives to fill up SATA ports, 44TB. I wonder if 14 drives is pushing it for Z3.

Per @Bidule0hm 's Calculator; 14 x 4TB RaidZ3 equates to ~34.5TB.

What about two vdevs each with 7 drives Z2? 40TB

7 x 4TB RaidZ2 equals 15.7TB each for ~ 30.1TB total, not 40TB.

Also, I agree with the others that 2 x 7 x 4TB vDevs is the better choice. I have heard of others running a single vDev using more than 12 drives, but I personally wouldn't.

 

[tom123]

Good catch on the capacity. For this post I used what I called "nominal capacity". Ignoring any TiB-TB differences, ZFS overhead and the 80% used recommendation.

I also don't know what the rebuild speed would be on Seagate NAS drives, so I kept the default value on the calculator. Resulting in the MTTR values of 6.8 days for the Z3 versus 4.7 days for the a single 7 drive Z2. But I think the Z3 config would be more reliable that the 2X7 Z2.

I was thinking of 14 drives because I would fill the SATA ports and an additional 2 drives is reasonable in terms of cost.

 

[Stux]

Start with 8 drives, and add the other 8 when you need them. Maybe bigger drives will be cheaper then.

I think you've already bought 12 right?

Alternatively go with multiples of 6. But then you're losing 33%

 

[tom123]

Yeah, I already got 12. A couple more jumped into my Amazon cart wanting to be bought.

 

[Mirfster]

Yeah, I already got 12. A couple more jumped into my Amazon cart wanting to be bought.

Lol, yeah that happens to a lot of us. ;)

 

[JohnDigital]

Forgive me. Im bored today and chiming in on random threads.

I run a 12 drive Z3 and I couldn't be happier. "scrub repaired 0 in 12h17m with 0 errors on Fri Oct 7 14:17:16 2016" I have not noticed any performance hits in this config. Reliability is what it is, 3 drives to fail of 12 brand new is a bit of a stretch IF you are vetting them, SMARTing them, scrubbing them as you should, and alerts going out when things are not copacetic.

My real world (Freenas reported) usable pool usable space is 38.4 TB with 12 5's in Z3. So I would expect yours to be 26.4TB using 12 4TB drives in Z3. Remember this is Freenas reported space (simply pool used space + pool free space). Hopefully by the time I we need more space, 8TB will be a hundred bucks.

 

[Stux]

Forgive me. Im bored today and chiming in on random threads.

I run a 12 drive Z3 and I couldn't be happier. "scrub repaired 0 in 12h17m with 0 errors on Fri Oct 7 14:17:16 2016" I have not noticed any performance hits in this config. Reliability is what it is, 3 drives to fail of 12 brand new is a bit of a stretch IF you are vetting them, SMARTing them, scrubbing them as you should, and alerts going out when things are not copacetic.

My real world (Freenas reported) usable pool usable space is 38.4 TB with 12 5's in Z3. So I would expect yours to be 26.4TB using 12 4TB drives in Z3. Remember this is Freenas reported space (simply pool used space + pool free space). Hopefully by the time I we need more space, 8TB will be a hundred bucks.

FreeNAS reports TiBs

 

[tom123]

Thanks for the responses.

For this post I used what I called "nominal capacity". This was just to give a rough estimate of capacity to compare different configurations. It was not meant to be an exact capacity figure.

The main point was to ask everybody's opinions on a 14 drive Z3 versus 2X7 drive Z2 vdevs. Mainly in terms of reliability or instability caused by such a wide Z3 vdev.

If John Digital is happy with 12 drive Z3, maybe I would be ok with a 14 drive Z3. Even though it is wider than recommended.

 

[JohnDigital]

I say do it, test it out for a few days. 12, 14 whats the difference? You can always just blow it away and start over before you get too far along if things are not as they should be. Its my green money and Id bet it that you will be just fine for many years. No matter what you decide run the Solnet Array Testing on them. Any drives that are not real similar in speed, Id RMA/Amazon Prime return them right away, also mentioning the ones that fail the pre Solnet SMART test of course these get RMA'd right away as well.

 

[depasseg]

"scrub repaired 0 in 12h17m with 0 errors on Fri Oct 7 14:17:16 2016"

How much data is in this pool?

 

[JohnDigital]

30TB

 

[nojohnny101]

While it may be technically possible to run vdevs wider than 12 drives, it is not recommended per the FreeNAS manual:

• Using more than 12 disks per vdev is not recommended. The recommended number of disks per vdev is between 3 and 9. If you have more disks, use multiple vdevs.
• Some older ZFS documentation recommends that a certain number of disks is needed for each type of RAIDZ in order to achieve optimal performance. On systems using LZ4 compression, which is the default for FreeNAS® 9.2.1 and higher, this is no longer true. See ZFS RAIDZ stripe width, or: How I Learned to Stop Worrying and Love RAIDZ for details.

@Stux @gpsguy @Mirfster have all given sound advice, 2 vdevs will suit you better.

The main point was to ask everybody's opinions on a 14 drive Z3 versus 2X7 drive Z2 vdevs. Mainly in terms of reliability or instability caused by such a wide Z3 vdev.

I think you have your consensus.