RAIDZ2 number of disks.

[mskenderian]

The FreeNAS Wiki says the following:

When determining how many disks to use in a RAIDZ, the following configurations provide optimal performance. Array sizes beyond 12 disks are not recommended.
Start a RAIDZ1 at at 3, 5, or 9, disks.
Start a RAIDZ2 at 4, 6, or 10 disks.
Start a RAIDZ3 at 5, 7, or 11 disks.
The recommended number of disks per group is between 3 and 9. If you have more disks, use multiple groups.

for RAIDZ2, why isnt 8 disks optimal? Its between the 3-9..

thanx

 

[jonnn]

The FreeNAS Wiki says the following:



for RAIDZ2, why isnt 8 disks optimal? Its between the 3-9..

thanx

Has to to with wasted space.

It has something to do with the way ZFS stripes data, you'll end up with a significant amount of wasted space. Perhaps on the order of 1 - 1.5TB for an array of 3TB drives.

I have seen thorough comparsion somewhere on the internet... try to find it. Sorry I didnt save the link.

All indications show the performance will be OK though.

 

[cyberjock]

Actually, it has to do with performance striping over 4k sector drives. For most home users its not that big of a deal. But for Enterprise users it can be a big deal.

 

[mskenderian]

Actually, it has to do with performance striping over 4k sector drives.

iam using WD RED

Sector Sizes: 512 bytes logical, 4096 bytes physical

so what is better i want to use 8 drives, or should i stick with 6 for better performance either will work at this point

 

[survive]

Hi mskenderian,

The "optimal configurations" you refer to in your first post get down to the math that's involved with calculating how the data is split up & the parity is calculated when you write to the file system. Simply put, 4 data drives & 2 parity drives are "cleaner" mathematically than 6 data drives & 2 parity drives.

I run an 8 drive raidz2 and when I was testing things I found I lost ~40MB/s using raidz2.....it took me from ~480MB/s to 440MB/s when I was "dd"ing a big file from the command line to the pool, so the loss in performance was totally worth the gains I got going to z2.

-Will

 

[jonnn]

iam using WD RED


so what is better i want to use 8 drives, or should i stick with 6 for better performance either will work at this point

The performance impact is small and seems to affect read only. overall performance would probably be slightly higher since there are more disks...
http://hardforum.com/showpost.php?p=1036395804&postcount=61

other sources:
https://blogs.oracle.com/roch/entry/when_to_and_not_to


The larger issue is wasted space. Over 1 TB for the RAIDZ2 @ 8 Drives.
Read here:
http://www.opendevs.org/ritk/zfs-4k-aligned-space-overhead.html
6: 0.2602885812520981
7: 1.1858475767076015
8: 1.149622268974781

You'll "waste" 1.149 - 0.26 TB of space (~1TB) by choosing 8 disks over 6. This is with 3TB disks.

The above #'s are with ashift = 12 (4k sectors)
with 512b sectors (ashift = 9) the wasted space with 6 drives is almost exactly the same (no penalty for 4k drives) but there is a penalty at 10 drives.

The most interesting comparison comes at 17-18 disks:
17: 3.5806130981072783
18: 0.7912140190601349

You'll gain almost 6TB of space by going from 17 to 18 drives because 17 drives is so wasteful!

 

[greuh]

I want to upgrade my 9.2.1.5 FreeNAS running on HP Proliant N40L server.
I have only one disk today, and SMART errors are increasing. Hence, after good reading here ;) and elsewhere, I want to create a RAID-Z2 vdev.
But I need to copy the content of my disk to the new vdev. I only have 6 SATA ports on HP N40L. So I wanted to have 5 disks in a RAID-Z2 vdev. (5 * 3TB disks)
But it seems that it is not advised not to use 4, 6 or 10 disks in a RAID-Z2.
However, thanks to the above messages, I saw no impact performance with 5 disks
http://hardforum.com/showpost.php?p=1036395804&postcount=61

And I calculated the overhead with 5 disks thanks to this link:
http://www.opendevs.org/ritk/zfs-4k-aligned-space-overhead.html

So the calculation is:
128k / 4k = 32 sectors,
32 sectors / 3 data disks = 32/3 sectors per disk, (float is around 10.666)
32/3 sectors per disk * (3 data disks + 2 parity disks) = 53.333 sectors.
53.333 is not a multiple of 3 so 0.666 sector padding is added. (1.2333% overhead)

Is my calculation correct ?
The "loss" would then only be of 3*3*10^12*0.012333 / 2^40 = 100MB

 

[Ericloewe]

I want to upgrade my 9.2.1.5 FreeNAS running on HP Proliant N40L server.
I have only one disk today, and SMART errors are increasing. Hence, after good reading here ;) and elsewhere, I want to create a RAID-Z2 vdev.
But I need to copy the content of my disk to the new vdev. I only have 6 SATA ports on HP N40L. So I wanted to have 5 disks in a RAID-Z2 vdev. (5 * 3TB disks)
But it seems that it is not advised not to use 4, 6 or 10 disks in a RAID-Z2.
However, thanks to the above messages, I saw no impact performance with 5 disks
http://hardforum.com/showpost.php?p=1036395804&postcount=61

And I calculated the overhead with 5 disks thanks to this link:
http://www.opendevs.org/ritk/zfs-4k-aligned-space-overhead.html

So the calculation is:
128k / 4k = 32 sectors,
32 sectors / 3 data disks = 32/3 sectors per disk, (float is around 10.666)
32/3 sectors per disk * (3 data disks + 2 parity disks) = 53.333 sectors.
53.333 is not a multiple of 3 so 0.666 sector padding is added. (1.2333% overhead)

Is my calculation correct ?
The "loss" would then only be of 3*3*10^12*0.012333 / 2^40 = 100MB

Honestly, a lot has been written on the subject and it's hard to figure out just how big the problem is.

One thing is certain: Everyone tends to agree that 6-disk RAIDZ2 vdevs are an (one of the) optimal configuration(s).

 

[greuh]

OK, but regardless of the extra cost for a 6th disk, how will I copy my existing disk to a new 6 disk vdev knowing that I only have 6 SATA ports (the truth is 5 SATA ports and 1 eSATA port) ?
I have several jails and datasets on this disk.

 

[dlavigne]

We will be changing both the docs and the Volume Manager recommendations for 9.3 due to new information: http://blog.delphix.com/matt/2014/06/06/zfs-stripe-width/.

 

[greuh]

Thank you, this is very enriching.
Hence I really have to use 6 disks in my RAIDZ2 (4+2).
Could I then plug my current disk with an USB-SATA adapter so that I could have 7 disks on my system to be able to copy from the former one to the new vdev ?