RaidZ3 Drive Configuration & Performance

[engmsf]

I was reading a post from the FreeBSD forum user MadCATZ and a few questions comes to mind.
1) The 8-disk RaidZ3 is also bad?
2) What is the main significant of whole number Kb?
3) What is the expected performance difference between 6-disk RaidZ2 and 7-disk RaidZ3? Marginal to go to 7-disk Z3? Even noticeable by media server streaming to 2 high def devices (possibly 3 in the future).

https://forums.freebsd.org/viewtopic.php?&t=34558

3-disk RAID-Z = 128KiB / 2 = 64KiB = good
4-disk RAID-Z = 128KiB / 3 = ~43KiB = BAD!
5-disk RAID-Z = 128KiB / 4 = 32KiB = good
9-disk RAID-Z = 128KiB / 8 = 16KiB = good

4-disk RAID-Z2 = 128 / 2 = 64KiB
6-disk RAID-Z2 = 128 / 4 = 32KiB
8-disk RAID-Z2 = 128 / 6 = 21.3kiB = bad!
10-disk RAID-Z2 = 128 / 8 = 16KiB

4-disk RAID-Z3 = 128 / 1 = 128
5-disk RAID-Z3 = 128 / 2 = 64
7-disk RAID-Z3 = 128 / 4 = 32
8-disk RAID-Z3 = 128 / 5 = 25.6
11-disk RAID-Z3 = 128 / 8 = 16
16-disk RAID-Z3 = 128 / 13 = 9.8 = bad!

 

[dlavigne]

Did you ever find an answer to this?

 

[Ericloewe]

I was reading a post from the FreeBSD forum user MadCATZ and a few questions comes to mind.
1) The 8-disk RaidZ3 is also bad?
2) What is the main significant of whole number Kb?
3) What is the expected performance difference between 6-disk RaidZ2 and 7-disk RaidZ3? Marginal to go to 7-disk Z3? Even noticeable by media server streaming to 2 high def devices (possibly 3 in the future).

https://forums.freebsd.org/viewtopic.php?&t=34558

3-disk RAID-Z = 128KiB / 2 = 64KiB = good
4-disk RAID-Z = 128KiB / 3 = ~43KiB = BAD!
5-disk RAID-Z = 128KiB / 4 = 32KiB = good
9-disk RAID-Z = 128KiB / 8 = 16KiB = good

4-disk RAID-Z2 = 128 / 2 = 64KiB
6-disk RAID-Z2 = 128 / 4 = 32KiB
8-disk RAID-Z2 = 128 / 6 = 21.3kiB = bad!
10-disk RAID-Z2 = 128 / 8 = 16KiB

4-disk RAID-Z3 = 128 / 1 = 128 Not possible! You need at least 2 drives + parity drives!
5-disk RAID-Z3 = 128 / 2 = 64
7-disk RAID-Z3 = 128 / 4 = 32
8-disk RAID-Z3 = 128 / 5 = 25.6 this is also bad!
11-disk RAID-Z3 = 128 / 8 = 16
16-disk RAID-Z3 = 128 / 13 = 9.8 = bad!

If you have weird block sizes, they'll end up not fitting exactly in Hard Drive sectors, which means accesses will have greater latency, lower throughput and due to the way ZFS handles fragmentation, you'll run out of usable space sooner.
You want numbers that are divided by 4k (4096).

Specific performance losses are hard to estimate, but I'd personally stay on the safe, conservative side.

 

[engmsf]

Did you ever find an answer to this?

Looks like Ericloewe helped answered my question, thanks!

I have been running a plex server in RaidZ1 for 1.5 years now with no issues on consumer level hardware.

As my important data is growing, I am considering setting up another server, but this time with server grade hardware. By the looks of things, either a 6-disk RaidZ2 or a 7-disk RaidZ3.

 

[Arman]

If you have weird block sizes, they'll end up not fitting exactly in Hard Drive sectors, which means accesses will have greater latency, lower throughput and due to the way ZFS handles fragmentation, you'll run out of usable space sooner.
You want numbers that are divided by 4k (4096).

Specific performance losses are hard to estimate, but I'd personally stay on the safe, conservative side.

So is an 8-drive raidz3 configuration bad?

 

[Ericloewe]

Compression is said to make this a non-issue.

 

[Arman]

Compression is said to make this a non-issue.

Sorry, could you explain a little bit further? I'm kind of confused about all this... I can only put a maximum of 8 disks in my system and I want to know what raid configuration gives me the best balance between redundancy and performance. Raidz3 came to mind after a lot of research... I'm still confused though.

 

[mattbbpl]

Compression is said to make this a non-issue.

Yeah, that's what they say. It still makes me pause, though. For instance, the vast (VAST!) majority of my data is already compressed, so it won't be compressed further by FreeNAS. It would still be an issue in that case, correct?

 

[Ericloewe]

Sorry, could you explain a little bit further? I'm kind of confused about all this... I can only put a maximum of 8 disks in my system and I want to know what raid configuration gives me the best balance between redundancy and performance. Raidz3 came to mind after a lot of research... I'm still confused though.

That would be two 4-disk RAIDZ2 vdevs, at a whopping 50% parity.

Yeah, that's what they say. It still makes me pause, though. For instance, the vast (VAST!) majority of my data is already compressed, so it won't be compressed further by FreeNAS. It would still be an issue in that case, correct?

Yeah, to some extent.

 

[Arman]

That would be two 4-disk RAIDZ2 vdevs, at a whopping 50% parity.


Yeah, to some extent.

What about only having a single 8 drive vdev in raidz3?

 

[Ericloewe]

What about only having a single 8 drive vdev in raidz3?

It works. Not much more to say. It depends on what you're using it for.

 

[Arman]

It works. Not much more to say. It depends on what you're using it for.

Just using it for backups... And occasional movie streaming or something...
So what's up with this: 8-disk RAID-Z3 = 128 / 5 = 25.6 ? I don't understand block sizes/sectors and stuff. Do I need to worry about it? Might it cause me problems in the future?

 

[Ericloewe]

General advice is don't worry about it.

 

[Bidule0hm]

This thread: https://forums.freenas.org/index.php?threads/misaligned-pools-and-lost-space.40288/ will probably be useful.

The short answer is that if you use a recordsize value (at the dataset creation in the advanced mode) of 1 M and you leave compression to default (lz4) you'll not loose space ;)

And a 8 drives RAID-Z3 will be able to saturate a gigabit link easily so it'll not be a bottleneck, the network will.