Recommended RaidZx depth

[DEMONmachine]

Studing iXuniversuty Module 101.

It says:
Suggested depth for....
RaidZ1 up to 9 devices
RaidZ2 up to 10
RaidZ3 up to 11

For more disks create a vdevs and stripe it.

But I can't understand any reson for it.
For example:
16 devs with RaidZ2 and stripe 2x 8dev RaidZ1
In both cases read/write perfomance will be sitributed over 14 devs.
OK, one reason is that in case of stipe there will be faster resilvering after drive replace.
But IMO 1st setup is more fault tolerance than 2nd.
And it is more simple in configuration / operations / maintenance.

So why?

 

[sretalla]

In both cases read/write perfomance will be sitributed over 14 devs.

No.

In all versions of RAIDZ, the expected performance of a single VDEV is approximately that of the slowest member disk of the VDEV. So in the case of a single VDEV of 16, you get the performance of 1 disk... with 2x8, you get twice the IOs.

OK, one reason is that in case of stipe there will be faster resilvering after drive replace.

I don't know where you get that impression either. I don't think it's correct.

RAIDZ1 is almost dead in the sense that nobody uses 1TB disks anymore, so all RAIDZ is recommended to be 2 or 3 to allow for continued redundancy while failed/resilvering.

But IMO 1st setup is more fault tolerance than 2nd.

That is correct. In the 2x8 case, you lose the wrong 2 disks (both in the same VDEV) and your pool is dead.

And it is more simple in configuration / operations / maintenance.

I don't think that's really the case. There's not a lot of difference.

 

[DEMONmachine]

I don't know where you get that impression either. I don't think it's correct.

I thoungt in case 2x 8dev vs 1x 16dev resilvering will be faster because only failed vdev will be rebuilded. So rebuild over 8 devs should be faster than over 16, Isnt' right?

In all versions of RAIDZ, the expected performance of a single VDEV is approximately that of the slowest member disk of the VDEV

In classical RAID5 or 6 you are theoretically getting (n-1)*s or (n-2)*s read/write performance (where n is number of divices in goup, and s is speed of the slowest device in gorup) because of each device reading/writing its own slice simultaneously. This rule isn't applicable for RAIDZ? With RAIDZ vdev you are really geting performance only s without dependent of n ?

 

[anmnz]

I thoungt in case 2x 8dev vs 1x 16dev resilvering will be faster because only failed vdev will be rebuilded.

Resilvering doesn't rebuild the whole vdev. If a disk fails and you replace it, only that disk gets resilvered.

 

[DEMONmachine]

So, please, can anyone confirm or disprove my unredstanding:

N - number of disks
P - parity level
S - speed of a single disk
I - IOPS of a single disk

so....

In classic RAID:
Speed: (N-P)*S
IOPS: (N-P)*I

RAIDZ:
Speed: (N-P)*S
IOPS: I

Is it right? If yes, how explained this ZFS limitation?

 

[sretalla]

Resilvering doesn't rebuild the whole vdev. If a disk fails and you replace it, only that disk gets resilvered.

Resilvering triggers a scrub, so all disks in the pool are hit, all VDEVs. I have seen some talk about this being a hangover from the old days when it was necessary, but I have not specifically seen a plan to change it.

This rule isn't applicable for RAIDZ? With RAIDZ vdev you are really geting performance only s without dependent of n ?

Yes, RAIDZ1/2 are not the same as RAID5/6... it's similar in the number of pariity drives lost, but that's pretty much where the similarity ends.

Is it right? If yes, how explained this ZFS limitation?

You should look for some youtube videos of @Allan Jude if you want to understand this (the inner workings of RAIDZ).

One of the three key design considerations for ZFS is data integrity. Checksums are written and read back all over the show, so comparing it to a filesystem that doesn't care about that is apples and oranges.

 

[anmnz]

Resilvering triggers a scrub, so all disks in the pool are hit, all VDEVs.

Yep. The point I was addressing was the mistaken idea that resilvering "rebuilds a vdev" and that it therefore would be faster to "rebuild" a smaller vdev.

If yes, how explained this ZFS limitation?

Checksums are written and read back all over the show, so comparing it to a filesystem that doesn't care about that is apples and oranges.

Yep again! I'd also point out that the "classic RAID" IOPS calculation above ("(N-P)*I") is a theoretical upper bound for a read-only workload that is likely unachievable in practice. While the RAIDZ "vdev is as performant as its slowest disk" rule-of-thumb is a slightly pessimistic but safe practical lower bound that is suitable for real-world capacity planning. Apples and oranges, again.

 

[sretalla]

Yep. The point I was addressing was the mistaken idea that resilvering "rebuilds a vdev" and that it therefore would be faster to "rebuild" a smaller vdev.

I have a really old thread out there somewhere where I hit this problem with the same assumption and was confused when I saw it hitting all VDEVs despite resilvering a single mirrored VDEV in the pool.

That's when I discovered the scrub was part of the story.

 

[sretalla]

And here it is...https://www.ixsystems.com/community...ing-all-disks-in-multi-vdev-mirror-set.61954/

Re-reading it, technically it's not a scrub, but shared code between resilver and scrub that means the same IO pattern.