RAIDZ Calculator. Does ZFS ever give write speed gains depending on layout?

[rigel]

Drives: 8 SATA HDD
Drive capacity: 4TB

Hi, I'm trying to calculate my write and read speed gains by changing layouts on the RAIDZ Calculator and no matter what layout I choose write speed is always unchanged. Does it mean that no matter the layout ZFS only provides the maximum speed of a single drive? Assume there are no any cache or special devices and RAM is limited to 32GB.

This is so much different with what I experienced with RAID10 before where for the same number of drives you can get up to 8x read and 4x write speed gain. I'm not comparing how much more reliable ZFS raid to normal raid is, I'm just curious why ZFS does not give any speed gains just like normal raid without buying 1TB of RAM and high endurance drives (10 DWPD) for special or cache devices.

2 VDEVs in RAIDz1 with 4 drives each vdev:

4 VDEVs in Mirror with 2 drives each vdev:

RAID10 with 8 drives:

 

[Davvo]

Please read the following resource.

ZFS Storage Pool Layout

This amazing document, created by iXsystems in February 2022 as a "White Paper", cleanly explains how to qualify pool performance touching briefly on how ZFS stores data and presents the advantages, performance and disadvantages of each pool...

www.truenas.com

TL;DR: Using RAIDZ# you get improved write performance, as well as with multiple VDEVs.

 

[glide]

I tried the same as @rigel with this calculator, not being able to see how to improve the performance.
I have 16 x 1.5TB SAS-3 SSD plugged into a SAS-3 LSI HBA on a 16x 3.0 PCIe slot.
Also tried several RAIDZx configs on TrueNAS Scale, the best write speed I reached was 650MB/s with a single stripped vdev, which seems quite unexpected to me.
This server I'm testing on only has 8MB of RAM though, could it be the bottleneck ?
Sorry I'm really discovering the RAID world here.

 

[sretalla]

Low amounts of RAM (and 8GB is the absolute minimum) will certainly cause writes to not benefit from acceleration.

Learning the difference between sync and async writes may also be important for you depending on the workload.

A good amount of RAM (again depending on the workload) would generally be considered 64GB (at which point it makes sense to start thinking about things which may improve reading speed as well like L2ARC).

 

[Davvo]

This server I'm testing on only has 8MB of RAM though, could it be the bottleneck ?

Really hope you mean GB and yeah, could be a reason. How did you test that?

 

[glide]

Haha yes indeed, I meant 8GB

How did you test that?

Locally with a simple rsync on a single 4GB file, I also used hdparm with several params combinations which gives me about the same numbers

Low amounts of RAM (and 8GB is the absolute minimum) will certainly cause writes to not benefit from acceleration.

So you think it could be the first limiting factor ?

Can we know the expected average write speed with my configuration ? With RAID-Z1 and 2 vdevs for example

16 x 1.5TB SAS-3 SSD plugged into a SAS-3 LSI HBA on a 16x 3.0 PCIe slot

 

[sretalla]

What you specified doesn't tell us the performance of the disks... no way to know other than saying a SAS3 interface is capable of 12Gbit/sec, so that x 16 is the absolute highest number that can be had based on what you told us, but reality of disks has a lot more to say about it, so likely to be a much smaller number.

 

[glide]

Yeah I guess we won't be able to reach the 12Gb/s
Well the disks are Toshiba's 160SMB02PX seem's like I'm already exceeding the announced 410MB/s write speed (constant 650MB/s), the wierd thing is it's when I'm using a stripe config.
I'm actually trying to be close to my 10Gb/s NIC, if you have any advice on that, don't hesitate to suggest.

 

[Davvo]

You could try jgreco's solnet array to assess your drives' performance.

solnet-array-test

Back in the late '90's, I was managing a bunch of large whitebox storage servers. For the largest of these, I had the pleasure of building and deploying a massive storage server, 8 shelves of 9 drives each, Seagate ST173404LW 73GB drives, a...

www.truenas.com

 

[glide]

That's interesting, I'll give it a try, thanks.

 

[Davvo]

I'm actually trying to be close to my 10Gb/s NIC, if you have any advice on that, don't hesitate to suggest.

I would suggest opening your own thread.

 

[glide]

Yes I agree, I'm a little bit off topic :)