Maximize write performance to NAS with 6x HDD - single raidz1 versus vdev group of 2x 3HDD?

[flashdrive]

Hello,

what would be the preferable option for a homelab setup with 6x HDD:


a) single raiz1
b) vdev group of 2x 3HDD?

How would one setup option b) to have only a single shareable storage space?

Received advice:

ZFS SLOG not speeding up writes, and recommended size for 10 GBit LAN

Having added a 128 GByte SLOG device (Samsung 830 SSD) in TrueNAS CORE and SCALE I do not see any write speed improvements during 10 GB LAN write operations. How can I make sure the the SSD is being utilised? The IO stat for sure. Is there any tuning to be done? In my understanding the SLOG...

www.truenas.com

 

[sretalla]

You're calling it a "lab"... what's your expectation regarding reliability and data integrity?

The "best" option for performance (both IOPS and throughput) is a striped pool.

If you don't mind keeping backups and having "downtime" while restoring occasionally while accepting lost "work" since your last backup, that's already enough said.

If you're keeping important data on it (then I would argue it's not a lab), you will probably want some redundancy to avoid losses between the last backup and disk failure, so then we come to your question on the "best use" of 6 disks for that.

I saw in your other thread that you're using exclusively SMB and Windows clients, so you may not really be seeking sync writes, but you may (depending on what files you're transferring/writing) need IOPS.

In order to have maximum IOPS, you would use mirrors, which mean that you're not restricted to the IOPS of just one HDD. In the case of 3 mirrors, you get the IOPS of 3 disks.

If it's throughput you need (fewer files, but larger size per file), then RAIDZ2 should work out fine.

Have a look at some of the thinking in this thread in terms of disk numbers:

SOLVED - Sanity check of performance, RaidZ2 server 50% of RaidZ1 server

So I have two servers (full specs in my signature). They are both SuperMicro 2U servers with 10GbE, one main one backup... My main server has 6x14TB in RaidZ2 My backup server has 7x8TB in RaidZ1 I am getting wildly different speeds for Read/Write for sustained large single files On my main...

www.truenas.com

 

[flashdrive]

Hello @sretalla

Thank you for your post and link.

> You're calling it a "lab"... what's your expectation regarding reliability and data integrity?

It is currently a lab to check out what can be done. I am investigation bot CORE and SCALE.

Clients:

MacOS and Windows, both using SMB

The main purpose is a storage for mostly larger file sizes. Windows and Time Machine backups as well.
Home usage, less than 5 clients accessing at the same time.

> If it's throughput you need (fewer files, but larger size per file), then RAIDZ2 should work out fine.

Yes, I am currently looking for throughput.

What is to say against 2x RAIDZ1 with 6 disks?

This is also the ongoing discussion in the other thread.

Hence my question if option b) is a feasable one and how to set this up.

 

[Kuro Houou]

Joining this thread as I am in the same boat… my RiadZ2 6 disk config is abysmal in performance… hitting a max of roughly 220MB/s for write speeds and around 300MB/s for reads on my 10G network :(. I have a 7 disk Raidz1 pool (I know not apples to apples) on another server that gets 2X those speeds.. so trying to figure out this question myself ;)

 

[rvassar]

Joining this thread as I am in the same boat… my RiadZ2 6 disk config is abysmal in performance… hitting a max of roughly 220MB/s for write speeds and around 300MB/s for reads on my 10G network :(. I have a 7 disk Raidz1 pool (I know not apples to apples) on another server that gets 2X those speeds.. so trying to figure out this question myself ;)

It's the vDev's. You only have 6 drives, so your RAIDz2 is one vDev. When you write to it, it behaves as if you're writing to a single disk. All the drives perform the I/O in lockstep, and it's not acknowledged as complete until the slowest drive returns success. The RAIDz1 pool likely has two vDev's and issues IOPS in round-robin between them. It will issue I/O to the first vDev, and then move to the second and issue more, hop back to the first, etc... In order to have two vDev's in RAIDz2 you have to meet the fault tolerance requirement for each vDev, so you'd need 8 drives (and could suffer 2 failures in each vDev).

 

[Kuro Houou]

It's the vDev's. You only have 6 drives, so your RAIDz2 is one vDev. When you write to it, it behaves as if you're writing to a single disk. All the drives perform the I/O in lockstep, and it's not acknowledged as complete until the slowest drive returns success. The RAIDz1 pool likely has two vDev's and issues IOPS in round-robin between them. It will issue I/O to the first vDev, and then move to the second and issue more, hop back to the first, etc... In order to have two vDev's in RAIDz2 you have to meet the fault tolerance requirement for each vDev, so you'd need 8 drives (and could suffer 2 failures in each vDev).

My RaidZ1 pool is one vdev, all 7 disks in one.. no mirroring or striping of vdevs going on. But yeah, wasn’t planning on trying raidz2 with two vdevs, was gonna try 2 raidz1 vdevs, which only needs 6 disks.

 

[sretalla]

Be aware that in RAIDZ, heavy load is placed on the remaining disks when resilvering (which can take a long time with larger disks ... over 2Tb as a guide), which could land you in the situation of a second failed disk... if that happens, consider this:

Yellow cells represent Pool is degraded, Red cells indicate Pool is dead (i.e. you can't get any data from it at all)

The choice is yours on the risks of the approach.

For Mirrors, you lose half your capacity to redundancy... some folks see that as too much.

For RAIDZ2, you lose 2 disks to parity, so more disks is better value for capacity, but performance (as discussed in the other thread) isn't necessarily better as you consider wider VDEVs (at very least, there's no linear link to wider=more throughput... and the recommended maximum width is 12)

For mirrors, you can get more IOPS for every additional VDEV you add (and you can add them as you go... if maximum IOPS isn't your highest objective... if it is, you need to erase and re-populate the pool to balance the contents on the new and old VDEVs).