Do I need same sized disks to extend a mirror and profit from more speed through sttriping?

Supose

Dabbler
Joined
Dec 23, 2020
Messages
23
Hi! I am new to TrueNAS. I installed FreeNAS a while ago and set up a mirror of two disks (each 6 TB). As now I need more space, I consider buying two more drives. I want to set them up as a mirror again and add them to the pool. Will this automatically create a striping configuration, so I gain speed as well? Would I need 6 TB disks for that or can I get bigger disks?
 

Supose

Dabbler
Joined
Dec 23, 2020
Messages
23
I just played around with a FreeNAS installation in a virtual machine (see below image).

Imagine da1p2 und da2p2 are my two existing 6 TB drives.

Now, when I add two more 6 TB drives (da3p2 and da4p2) it should double my storage AND increase speed (because it is striped), right?

If I use 10 TB drives for da3p2 and da4p2 does that increase the speed as well?

Code:
pool (STRIPE)
  MIRROR
    6TB
    6TB
  MIRROR
    10TB
    10TB


stripe.png
 

Chris Moore

Hall of Famer
Joined
May 2, 2015
Messages
10,080
The throughput to the pool is based on the number of vdevs. The capacity of the pool is based on the size of the vdevs. So the capacity increases based on the drive size, but the speed is only connected to the number of vdevs in the pool, not the size.
Individual vdevs provide performance to the pool based on the speed of the constituent drives, so slow drives make slow vdevs and fast drives make fast vdevs.
 

Supose

Dabbler
Joined
Dec 23, 2020
Messages
23
Thank you! I found contradictory info about the performance increase for RAID-Z2 vs. striped mirrors when using four 6 TB drives.

Which configuration has which performance benefits?
 

sretalla

Powered by Neutrality
Moderator
Joined
Jan 1, 2016
Messages
9,702
Now, when I add two more 6 TB drives (da3p2 and da4p2) it should double my storage AND increase speed (because it is striped), right?
Not exactly, but maybe.

Increasing the number of VDEVs in a pool that already has data on it will clearly increase the capacity of the pool by the capacity of the added VDEV... no mysteries there.

However, because the existing VDEV has existing data, ZFS will (or at least may) favor the new VDEV when making decisions about where to write things, so the IOPS boost you might expect may only be evident some of the time (if at all) until the VDEVs have evenly distributed data.

The same could be said for the aspect of throughput (more simply equated with speed), which will also only show up as an increase if data is written across both VDEVs.

Note that when having VDEVs of different sizes (although this is fine in terms of data safety and pool functionality), you have built in the factor of unbalanced VDEVs just by the nature of their size, so you will eventually lose any IOPS and throughput advantage when the smaller VDEV becomes full (or near to it).

If you have evenly sized VDEVs, the way to "hurry along" the process of evening out the VDEVs is either to clean out the pool and re-fill it from a backup or move the data to another dataset (which can subsequently be renamed to the saem thing after the original is removed).

I found contradictory info about the performance increase for RAID-Z2 vs. striped mirrors when using four 6 TB drives.

Which configuration has which performance benefits?
There should be no contradictions. Just pay attention to the difference between IOPS and "speed" (=throughput). Lots of little reads aren't the same as one single large read in that many more IOPS are required to deal with lots of small files (or parts of files in block storage).

People often mix the two concepts together in particular when discussing the requirements for block storage (more IOPS is better, so "more speed" for block storage actually means more IOPS in many people's language)

RAIDZ2 will have the IOPS of a single disk and the throughput of somewhere around 1 disk for read (it's worth noting that with such a small RAIDZ2 pool it's fighting its own rules here, needing to do parity calculation for half of the reads and writes) and 2 disks for write.

Mirrored pairs will have the IOPS of 2 disks and the throughput of 3 disks (maybe a little more) for read, about 2 disks for write.

Have a look at the tests done here for some more detail into the differences in performance... There are a lot of factors to take into consideration, so there's no single magic answer of one being better than the other in all cases or even scenarios (during the running operations down the track as pools fill up).

The testing done on that site is rather simplistic, so don't take it to be a final say on what's good, it's just some data to look at.

Making the comparison for 4 disk pools based on "performance" isn't a great thing to do due to RAIDZ2 only really becoming sensible from 6 disks and up.
 

SweetAndLow

Sweet'NASty
Joined
Nov 6, 2013
Messages
6,421
Having different sized drives for each vdev is totally normal and almost unavoidable if you're building a system over time like most users would do.
 
Last edited:

Chris Moore

Hall of Famer
Joined
May 2, 2015
Messages
10,080

tip0666

Cadet
Joined
Oct 24, 2023
Messages
7
Not exactly, but maybe.

Increasing the number of VDEVs in a pool that already has data on it will clearly increase the capacity of the pool by the capacity of the added VDEV... no mysteries there.

However, because the existing VDEV has existing data, ZFS will (or at least may) favor the new VDEV when making decisions about where to write things, so the IOPS boost you might expect may only be evident some of the time (if at all) until the VDEVs have evenly distributed data.

The same could be said for the aspect of throughput (more simply equated with speed), which will also only show up as an increase if data is written across both VDEVs.

Note that when having VDEVs of different sizes (although this is fine in terms of data safety and pool functionality), you have built in the factor of unbalanced VDEVs just by the nature of their size, so you will eventually lose any IOPS and throughput advantage when the smaller VDEV becomes full (or near to it).

If you have evenly sized VDEVs, the way to "hurry along" the process of evening out the VDEVs is either to clean out the pool and re-fill it from a backup or move the data to another dataset (which can subsequently be renamed to the saem thing after the original is removed).


There should be no contradictions. Just pay attention to the difference between IOPS and "speed" (=throughput). Lots of little reads aren't the same as one single large read in that many more IOPS are required to deal with lots of small files (or parts of files in block storage).

People often mix the two concepts together in particular when discussing the requirements for block storage (more IOPS is better, so "more speed" for block storage actually means more IOPS in many people's language)

RAIDZ2 will have the IOPS of a single disk and the throughput of somewhere around 1 disk for read (it's worth noting that with such a small RAIDZ2 pool it's fighting its own rules here, needing to do parity calculation for half of the reads and writes) and 2 disks for write.

Mirrored pairs will have the IOPS of 2 disks and the throughput of 3 disks (maybe a little more) for read, about 2 disks for write.

Have a look at the tests done here for some more detail into the differences in performance... There are a lot of factors to take into consideration, so there's no single magic answer of one being better than the other in all cases or even scenarios (during the running operations down the track as pools fill up).

The testing done on that site is rather simplistic, so don't take it to be a final say on what's good, it's just some data to look at.

Making the comparison for 4 disk pools based on "performance" isn't a great thing to do due to RAIDZ2 only really becoming sensible from 6 disks and up.
Always thought speed was always faster in mirror
 

Etorix

Wizard
Joined
Dec 30, 2020
Messages
2,110
Only to the extent that speed comes having enough IOPS, i.e. random speed.
For sequential speed, 3*2-way mirrors would have the theoretical read speed of 6 disks but the write speed of 3; a single 6-wide raidz2 would read and write as 4 disks—raidz2 wins on writes. For IOPS, it's 3 (striped mirrors) to 1 (single raidz2), so not even close.
 

sretalla

Powered by Neutrality
Moderator
Joined
Jan 1, 2016
Messages
9,702
Speed can be measured in IOPS and / or throughput.

Mirrors don’t always provide more throughput per VDEV, but almost always win on IOPS.
 
Top