RAID-Z2 vs Striped Mirror (RAID10) Performance comparison

[andrwhmmr]

Hi,

I just wanted to leave some benchmarks regarding the performance delta of RAID-Z2 vs RAID10.
I think this forum is a great resource, so if anyone wants to have some more input for making their decision I hope this might help.

I am by no means a FIO/ZFS Expert so, any input is greatly appreciated, I will continue to test more before this pool goes into production.

RAID-Z2 was created as one vdev with 6 devices
RAID10 was created with 3 vdevs á 2 mirrored devices
Everything on default.

Hardware:
6x WD RED Pro 4TB (WD4003FFBX)
Broadcom 9207-8i SAS2308
i5-6500
24GB RAM (non-ECC)

First Comparison:

fio --filename=/mnt/test1/file1 --size=500GB --direct=1 --rw=randrw --bs=64k --ioengine=posixaio --iodepth=64 --runtime=120 --numjobs=4 --time_based --group_reporting --name=throughput-test-job --eta-newline=1

Show : RAID-Z2

throughput-test-job: (groupid=0, jobs=4): err= 0: pid=36165: Fri Jun 4 22:15:45 2021
read: IOPS=239, BW=14.0MiB/s (15.7MB/s)(1807MiB/120801msec)
slat (nsec): min=512, max=705169, avg=3088.44, stdev=14625.70
clat (usec): min=96, max=2569.6k, avg=540329.69, stdev=439879.97
lat (usec): min=108, max=2569.6k, avg=540332.77, stdev=439877.40
clat percentiles (usec):
| 1.00th=[ 441], 5.00th=[ 2212], 10.00th=[ 29754],
| 20.00th=[ 62129], 30.00th=[ 143655], 40.00th=[ 283116],
| 50.00th=[ 526386], 60.00th=[ 725615], 70.00th=[ 843056],
| 80.00th=[ 943719], 90.00th=[1082131], 95.00th=[1249903],
| 99.00th=[1635779], 99.50th=[1803551], 99.90th=[2038432],
| 99.95th=[2122318], 99.99th=[2264925]
bw ( KiB/s): min= 1504, max=206639, per=99.28%, avg=15203.07, stdev=5211.73, samples=945
iops : min= 20, max= 3227, avg=235.13, stdev=81.50, samples=945
write: IOPS=242, BW=15.1MiB/s (15.9MB/s)(1829MiB/120801msec)
slat (nsec): min=940, max=878599, avg=5217.39, stdev=14775.89
clat (usec): min=79, max=2408.6k, avg=521712.68, stdev=448748.78
lat (usec): min=122, max=2408.6k, avg=521717.90, stdev=448746.31
clat percentiles (usec):
| 1.00th=[ 441], 5.00th=[ 2073], 10.00th=[ 27657],
| 20.00th=[ 57934], 30.00th=[ 105382], 40.00th=[ 208667],
| 50.00th=[ 484443], 60.00th=[ 717226], 70.00th=[ 834667],
| 80.00th=[ 935330], 90.00th=[1082131], 95.00th=[1249903],
| 99.00th=[1652556], 99.50th=[1803551], 99.90th=[2088764],
| 99.95th=[2197816], 99.99th=[2332034]
bw ( KiB/s): min= 1506, max=216789, per=99.35%, avg=15397.98, stdev=5496.92, samples=944
iops : min= 20, max= 3385, avg=238.19, stdev=85.94, samples=944
lat (usec) : 100=0.01%, 250=0.43%, 500=0.72%, 750=0.63%, 1000=0.57%
lat (msec) : 2=2.42%, 4=1.06%, 10=0.52%, 20=1.65%, 50=8.77%
lat (msec) : 100=11.22%, 250=12.61%, 500=9.03%, 750=12.28%, 1000=23.21%
lat (msec) : 2000=14.72%, >=2000=0.15%
cpu : usr=0.07%, sys=0.16%, ctx=54644, majf=0, minf=4
IO depths : 1=0.1%, 2=0.2%, 4=0.3%, 8=0.7%, 16=1.6%, 32=84.1%, >=64=13.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=93.7%, 8=2.4%, 16=2.6%, 32=1.1%, 64=0.1%, >=64=0.0%
issued rwts: total=28906,29256,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
READ: bw=14.0MiB/s (15.7MB/s), 14.0MiB/s-14.0MiB/s (15.7MB/s-15.7MB/s), io=1807MiB (1894MB), run=120801-120801msec
WRITE: bw=15.1MiB/s (15.9MB/s), 15.1MiB/s-15.1MiB/s (15.9MB/s-15.9MB/s), io=1829MiB (1917MB), run=120801-120801msec

Show : Striped Mirror (RAID10)

throughput-test-job: (groupid=0, jobs=4): err= 0: pid=38066: Fri Jun 4 22:32:31 2021
read: IOPS=516, BW=32.3MiB/s (33.9MB/s)(3893MiB/120523msec)
slat (nsec): min=501, max=555586, avg=921.89, stdev=2454.24
clat (usec): min=181, max=1480.2k, avg=246170.93, stdev=132586.16
lat (usec): min=184, max=1480.2k, avg=246171.85, stdev=132586.16
clat percentiles (msec):
| 1.00th=[ 68], 5.00th=[ 97], 10.00th=[ 114], 20.00th=[ 138],
| 30.00th=[ 161], 40.00th=[ 184], 50.00th=[ 213], 60.00th=[ 247],
| 70.00th=[ 288], 80.00th=[ 342], 90.00th=[ 422], 95.00th=[ 502],
| 99.00th=[ 684], 99.50th=[ 751], 99.90th=[ 953], 99.95th=[ 1003],
| 99.99th=[ 1133]
bw ( KiB/s): min=12239, max=54232, per=100.00%, avg=33178.15, stdev=1909.96, samples=948
iops : min= 188, max= 846, avg=516.53, stdev=29.86, samples=948
write: IOPS=519, BW=32.5MiB/s (34.1MB/s)(3917MiB/120523msec)
slat (nsec): min=997, max=62810, avg=3470.49, stdev=1796.50
clat (usec): min=175, max=1294.6k, avg=246773.24, stdev=132660.24
lat (usec): min=193, max=1294.6k, avg=246776.71, stdev=132660.21
clat percentiles (msec):
| 1.00th=[ 69], 5.00th=[ 97], 10.00th=[ 114], 20.00th=[ 138],
| 30.00th=[ 161], 40.00th=[ 186], 50.00th=[ 213], 60.00th=[ 247],
| 70.00th=[ 288], 80.00th=[ 347], 90.00th=[ 426], 95.00th=[ 502],
| 99.00th=[ 684], 99.50th=[ 743], 99.90th=[ 944], 99.95th=[ 1003],
| 99.99th=[ 1099]
bw ( KiB/s): min=11679, max=51400, per=100.00%, avg=33379.88, stdev=1831.65, samples=948
iops : min= 180, max= 801, avg=519.65, stdev=28.64, samples=948
lat (usec) : 250=0.01%, 500=0.01%
lat (msec) : 4=0.01%, 10=0.01%, 20=0.01%, 50=0.31%, 100=5.30%
lat (msec) : 250=55.26%, 500=34.01%, 750=4.61%, 1000=0.44%, 2000=0.05%
cpu : usr=0.16%, sys=0.35%, ctx=123966, majf=0, minf=4
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.2%, 32=87.1%, >=64=12.7%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, throughput-test-job: (groupid=0, jobs=4): err= 0: pid=38066: Fri Jun 4 22:32:31 2021
read: IOPS=516, BW=32.3MiB/s (33.9MB/s)(3893MiB/120523msec)
slat (nsec): min=501, max=555586, avg=921.89, stdev=2454.24
clat (usec): min=181, max=1480.2k, avg=246170.93, stdev=132586.16
lat (usec): min=184, max=1480.2k, avg=246171.85, stdev=132586.16
clat percentiles (msec):
| 1.00th=[ 68], 5.00th=[ 97], 10.00th=[ 114], 20.00th=[ 138],
| 30.00th=[ 161], 40.00th=[ 184], 50.00th=[ 213], 60.00th=[ 247],
| 70.00th=[ 288], 80.00th=[ 342], 90.00th=[ 422], 95.00th=[ 502],
| 99.00th=[ 684], 99.50th=[ 751], 99.90th=[ 953], 99.95th=[ 1003],
| 99.99th=[ 1133]
bw ( KiB/s): min=12239, max=54232, per=100.00%, avg=33178.15, stdev=1909.96, samples=948
iops : min= 188, max= 846, avg=516.53, stdev=29.86, samples=948
write: IOPS=519, BW=32.5MiB/s (34.1MB/s)(3917MiB/120523msec)
slat (nsec): min=997, max=62810, avg=3470.49, stdev=1796.50
clat (usec): min=175, max=1294.6k, avg=246773.24, stdev=132660.24
lat (usec): min=193, max=1294.6k, avg=246776.71, stdev=132660.21
clat percentiles (msec):
| 1.00th=[ 69], 5.00th=[ 97], 10.00th=[ 114], 20.00th=[ 138],
| 30.00th=[ 161], 40.00th=[ 186], 50.00th=[ 213], 60.00th=[ 247],
| 70.00th=[ 288], 80.00th=[ 347], 90.00th=[ 426], 95.00th=[ 502],
| 99.00th=[ 684], 99.50th=[ 743], 99.90th=[ 944], 99.95th=[ 1003],
| 99.99th=[ 1099]
bw ( KiB/s): min=11679, max=51400, per=100.00%, avg=33379.88, stdev=1831.65, samples=948
iops : min= 180, max= 801, avg=519.65, stdev=28.64, samples=948
lat (usec) : 250=0.01%, 500=0.01%
lat (msec) : 4=0.01%, 10=0.01%, 20=0.01%, 50=0.31%, 100=5.30%
lat (msec) : 250=55.26%, 500=34.01%, 750=4.61%, 1000=0.44%, 2000=0.05%
cpu : usr=0.16%, sys=0.35%, ctx=123966, majf=0, minf=4
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.2%, 32=87.1%, >=64=12.7%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=93.2%, 8=2.8%, 16=2.7%, 32=1.2%, 64=0.1%, >=64=0.0%
issued rwts: total=62290,62666,0,0 shthroughput-test-job: (groupid=0, jobs=4): err= 0: pid=38066: Fri Jun 4 22:32:31 2021
read: IOPS=516, BW=32.3MiB/s (33.9MB/s)(3893MiB/120523msec)
slat (nsec): min=501, max=555586, avg=921.89, stdev=2454.24
clat (usec): min=181, max=1480.2k, avg=246170.93, stdev=132586.16
lat (usec): min=184, max=1480.2k, avg=246171.85, stdev=132586.16
clat percentiles (msec):
| 1.00th=[ 68], 5.00th=[ 97], 10.00th=[ 114], 20.00th=[ 138],
| 30.00th=[ 161], 40.00th=[ 184], 50.00th=[ 213], 60.00th=[ 247],
| 70.00th=[ 288], 80.00th=[ 342], 90.00th=[ 422], 95.00th=[ 502],
| 99.00th=[ 684], 99.50th=[ 751], 99.90th=[ 953], 99.95th=[ 1003],
| 99.99th=[ 1133]
bw ( KiB/s): min=12239, max=54232, per=100.00%, avg=33178.15, stdev=1909.96, samples=948
iops : min= 188, max= 846, avg=516.53, stdev=29.86, samples=948
write: IOPS=519, BW=32.5MiB/s (34.1MB/s)(3917MiB/120523msec)
slat (nsec): min=997, max=62810, avg=3470.49, stdev=1796.50
clat (usec): min=175, max=1294.6k, avg=246773.24, stdev=132660.24
lat (usec): min=193, max=1294.6k, avg=246776.71, stdev=132660.21
clat percentiles (msec):
| 1.00th=[ 69], 5.00th=[ 97], 10.00th=[ 114], 20.00th=[ 138],
| 30.00th=[ 161], 40.00th=[ 186], 50.00th=[ 213], 60.00th=[ 247],
| 70.00th=[ 288], 80.00th=[ 347], 90.00th=[ 426], 95.00th=[ 502],
| 99.00th=[ 684], 99.50th=[ 743], 99.90th=[ 944], 99.95th=[ 1003],
| 99.99th=[ 1099]
bw ( KiB/s): min=11679, max=51400, per=100.00%, avg=33379.88, stdev=1831.65, samples=948
iops : min= 180, max= 801, avg=519.65, stdev=28.64, samples=948
lat (usec) : 250=0.01%, 500=0.01%
lat (msec) : 4=0.01%, 10=0.01%, 20=0.01%, 50=0.31%, 100=5.30%
lat (msec) : 250=55.26%, 500=34.01%, 750=4.61%, 1000=0.44%, 2000=0.05%
cpu : usr=0.16%, sys=0.35%, ctx=123966, majf=0, minf=4
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.2%, 32=87.1%, >=64=12.7%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=93.2%, 8=2.8%, 16=2.7%, 32=1.2%, 64=0.1%, >=64=0.0%
issued rwts: total=62290,62666,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
READ: bw=32.3MiB/s (33.9MB/s), 32.3MiB/s-32.3MiB/s (33.9MB/s-33.9MB/s), io=3893MiB (4082MB), run=120523-120523msec
WRITE: bw=32.5MiB/s (34.1MB/s), 32.5MiB/s-32.5MiB/s (34.1MB/s-34.1MB/s), io=3917MiB (4107MB), run=120523-120523msec
ort=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
READ: bw=32.3MiB/s (33.9MB/s), 32.3MiB/s-32.3MiB/s (33.9MB/s-33.9MB/s), io=3893MiB (4082MB), run=120523-120523msec
WRITE: bw=32.5MiB/s (34.1MB/s), 32.5MiB/s-32.5MiB/s (34.1MB/s-34.1MB/s), io=3917MiB (4107MB), run=120523-120523msec
>=64=0.0%
complete : 0=0.0%, 4=93.2%, 8=2.8%, 16=2.7%, 32=1.2%, 64=0.1%, >=64=0.0%
issued rwts: total=62290,62666,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
READ: bw=32.3MiB/s (33.9MB/s), 32.3MiB/s-32.3MiB/s (33.9MB/s-33.9MB/s), io=3893MiB (4082MB), run=120523-120523msec
WRITE: bw=32.5MiB/s (34.1MB/s), 32.5MiB/s-32.5MiB/s (34.1MB/s-34.1MB/s), io=3917MiB (4107MB), run=120523-120523msec

Summary:
RAID-Z2: READ: bw=14.0MiB/s (15.7MB/s), WRITE: bw=15.1MiB/s (15.9MB/s), write: IOPS=242 , read: IOPS=239
RAID10: READ: bw=32.3MiB/s (33.9MB/s), WRITE: bw=32.5MiB/s (34.1MB/s), write: IOPS=519, read: IOPS=516

Second Comparison:

fio --filename=/mnt/test1/file1 --size=500GB --direct=1 --rw=randrw --bs=4k --ioengine=posixaio --iodepth=256 --runtime=120 --numjobs=4 --time_based --group_reporting --name=iops-test-job --eta-newline=1

Show : RAID-Z2

iops-test-job: (groupid=0, jobs=4): err= 0: pid=32315: Fri Jun 4 20:42:29 2021
read: IOPS=100, BW=403KiB/s (413kB/s)(48.9MiB/124181msec)
slat (nsec): min=504, max=227325, avg=1052.51, stdev=3727.74
clat (msec): min=188, max=10689, avg=4947.31, stdev=986.77
lat (msec): min=188, max=10689, avg=4947.31, stdev=986.77
clat percentiles (msec):
| 1.00th=[ 1754], 5.00th=[ 3473], 10.00th=[ 3775], 20.00th=[ 4245],
| 30.00th=[ 4665], 40.00th=[ 4799], 50.00th=[ 5000], 60.00th=[ 5201],
| 70.00th=[ 5470], 80.00th=[ 5671], 90.00th=[ 5873], 95.00th=[ 6208],
| 99.00th=[ 7886], 99.50th=[ 8356], 99.90th=[ 9329], 99.95th=[ 9866],
| 99.99th=[10671]
bw ( KiB/s): min= 52, max= 3871, per=100.00%, avg=909.08, stdev=215.44, samples=421
iops : min= 10, max= 965, avg=225.30, stdev=53.96, samples=421
write: IOPS=104, BW=417KiB/s (427kB/s)(50.6MiB/124181msec)
slat (nsec): min=565, max=223761, avg=1246.43, stdev=4330.20
clat (usec): min=880, max=10693k, avg=4947211.52, stdev=967140.05
lat (usec): min=904, max=10693k, avg=4947212.76, stdev=967140.31
clat percentiles (msec):
| 1.00th=[ 1770], 5.00th=[ 3473], 10.00th=[ 3809], 20.00th=[ 4245],
| 30.00th=[ 4665], 40.00th=[ 4799], 50.00th=[ 5000], 60.00th=[ 5201],
| 70.00th=[ 5403], 80.00th=[ 5671], 90.00th=[ 5873], 95.00th=[ 6208],
| 99.00th=[ 7483], 99.50th=[ 8154], 99.90th=[ 8792], 99.95th=[ 9463],
| 99.99th=[10537]
bw ( KiB/s): min= 44, max= 4105, per=100.00%, avg=936.72, stdev=223.37, samples=423
iops : min= 8, max= 1024, avg=232.15, stdev=55.93, samples=423
lat (usec) : 1000=0.01%
lat (msec) : 2=0.01%, 250=0.01%, 500=0.06%, 750=0.12%, 1000=0.16%
lat (msec) : 2000=0.91%, >=2000=98.73%
cpu : usr=0.02%, sys=0.29%, ctx=25559, majf=0, minf=4
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.5%, 32=2.1%, >=64=97.1%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=97.7%, 8=0.4%, 16=0.5%, 32=0.6%, 64=0.4%, >=64=0.4%
issued rwts: total=12519,12943,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=256

Run status group 0 (all jobs):
READ: bw=403KiB/s (413kB/s), 403KiB/s-403KiB/s (413kB/s-413kB/s), io=48.9MiB (51.3MB), run=124181-124181msec
WRITE: bw=417KiB/s (427kB/s), 417KiB/s-417KiB/s (427kB/s-427kB/s), io=50.6MiB (53.0MB), run=124181-124181msec

Show : Striped Mirror (RAID10)

iops-test-job: (groupid=0, jobs=4): err= 0: pid=30062: Fri Jun 4 20:20:11 2021
read: IOPS=678, BW=2714KiB/s (2779kB/s)(321MiB/121018msec)
slat (nsec): min=512, max=397318k, avg=20017.75, stdev=1444869.34
clat (usec): min=627, max=3179.7k, avg=749743.34, stdev=415494.24
lat (usec): min=677, max=3179.7k, avg=749763.35, stdev=415463.36
clat percentiles (msec):
| 1.00th=[ 4], 5.00th=[ 14], 10.00th=[ 28], 20.00th=[ 144],
| 30.00th=[ 726], 40.00th=[ 793], 50.00th=[ 844], 60.00th=[ 894],
| 70.00th=[ 961], 80.00th=[ 1036], 90.00th=[ 1167], 95.00th=[ 1284],
| 99.00th=[ 1687], 99.50th=[ 1804], 99.90th=[ 2265], 99.95th=[ 2534],
| 99.99th=[ 2635]
bw ( KiB/s): min= 323, max=46289, per=100.00%, avg=2718.67, stdev=1014.76, samples=929
iops : min= 78, max=11571, avg=678.24, stdev=253.70, samples=929
write: IOPS=681, BW=2725KiB/s (2790kB/s)(322MiB/121018msec)
slat (nsec): min=561, max=213209k, avg=38457.73, stdev=1115054.67
clat (usec): min=750, max=3035.4k, avg=747149.32, stdev=413514.91
lat (usec): min=876, max=3035.4k, avg=747187.78, stdev=413451.97
clat percentiles (msec):
| 1.00th=[ 5], 5.00th=[ 15], 10.00th=[ 29], 20.00th=[ 150],
| 30.00th=[ 726], 40.00th=[ 793], 50.00th=[ 844], 60.00th=[ 894],
| 70.00th=[ 953], 80.00th=[ 1028], 90.00th=[ 1167], 95.00th=[ 1284],
| 99.00th=[ 1687], 99.50th=[ 1804], 99.90th=[ 2265], 99.95th=[ 2534],
| 99.99th=[ 2534]
bw ( KiB/s): min= 275, max=47915, per=100.00%, avg=2725.19, stdev=1035.74, samples=929
iops : min= 66, max=11977, avg=679.85, stdev=258.94, samples=929
lat (usec) : 750=0.01%, 1000=0.01%
lat (msec) : 2=0.21%, 4=0.75%, 10=2.58%, 20=3.54%, 50=7.48%
lat (msec) : 100=3.41%, 250=2.92%, 500=1.19%, 750=10.61%, 1000=43.25%
lat (msec) : 2000=23.81%, >=2000=0.24%
cpu : usr=0.12%, sys=1.57%, ctx=161777, majf=1, minf=7
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.3%, 16=0.8%, 32=2.0%, >=64=96.8%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=97.7%, 8=0.3%, 16=0.4%, 32=0.6%, 64=0.6%, >=64=0.4%
issued rwts: total=82116,82439,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=256

Run status group 0 (all jobs):
READ: bw=2714KiB/s (2779kB/s), 2714KiB/s-2714KiB/s (2779kB/s-2779kB/s), io=321MiB (336MB), run=121018-121018msec
WRITE: bw=2725KiB/s (2790kB/s), 2725KiB/s-2725KiB/s (2790kB/s-2790kB/s), io=322MiB (338MB), run=121018-121018msec

Summary:
RAID-Z2: READ: bw=403KiB/s (413kB/s), WRITE: bw=417KiB/s (427kB/s) write: IOPS=104 , read: IOPS=100
RAID10: READ: bw=2714KiB/s (2779kB/s), WRITE: bw=2725KiB/s (2790kB/s), write: IOPS=681 ,read: IOPS=678

Have a good one.

 

[mistermanko]

I should've gone mirrored vdevs instead of raidz2.

Sacrificing 16% space for an almost 50% performance uplift is a no-brainer. Sadly nobody recommended that here.
There is unfortunately a lack of information for newbies regarding pool sizing and performance from officials. Newbies are let to ask here in the forums and get mixed answers. Clearly the use case is what matters and 1 in 3 new users here (my gut feeling) want a nextcloud instance or a plex media server up in running, so in both usecases a 2-way-mirrored vdev pool of no matter the drive count, is always the better choice, according to the numbers above. And yet there is still lot of threads here to be found, from 2015, with discussions about wether raidz1 or raidz2 is better. It's confusing for newcomers, there should be something done about it. And I don't think the documentation is sufficient in that regard.

Review Storage Needs
It is strongly recommended to review the available system resources and plan the storage use case before creating a storage pool.

• When storing critical information, more drives allocated to the pool increases redundancy.
• Maximizing total available storage at the expense of redundancy or performance means allocating large volume disks and configuring the pool for minimal redundancy.
• Maximizing pool performance means installing and allocating high-speed SSD drives to the pool.

Determining your specific storage requirements is a critical step before creating a pool.

 

[rungekutta]

Then I’m afraid I have to contribute to that confusion by adding contrary advise. ;-)

I do not agree that pools of 2-way mirror vdevs is always the answer. Even in your own example - the typical nextcloud/plex server! Let’s say user has 4 or maybe 6 disk. Striped mirrors yields the best performance but at only 50% space efficiency AND you are also vulnerable to the wrong 2 disks breaking at the same time, taking down your entire pool. Or the mirror disk breaks while you resilver. I.e exactly why RaidZ1 is usually not recommended. RaidZ2 otoh provides more resilience (any 2 disks can break) and in the case of 6 disks, 67% space efficiency (vs 50%). Performance should also be more than enough for nextcloud and plex.

There are very strong cases for pools of mirror vdevs for sure, but it’s not the be-all-end-all.

(Says the guy who runs 6 disks in RaidZ2. ;-) )

 

[mistermanko]

you are also vulnerable to the wrong 2 disks breaking at the same time, taking down your entire pool.

That could also happen with raidz2, same probability. The only risk is the resilver stress for a disk in a mirrored vdev. MTTDF of 6,214,561.67 hours, with a pessimistic estimate.

source: https://www.servethehome.com/raid-calculator/raid-reliability-calculator-simple-mttdl-model/

RAID is no backup. It's not statet often enough imho. Both data loss scenarios you described can be mitigated by the simple 3 - 2 - 1 rule, an offsite backup is always more safe than any raid config.

Performance should also be more than enough for nextcloud and plex.

What is enough? I believe nextcloud would highly benefit from the higher iops and the high rw numbers.

 

[rungekutta]

That could also happen with raidz2, same probability. The only risk is the resilver stress for a disk in a mirrored vdev. MTTDF of 6,214,561.67 hours, with a pessimistic estimate.

Nope, not same probability. If you lose a disk in a mirror, 1 and exactly 1 other disk has got the data that you need to recover. So you are vulnerable to failure on that disk during the resilver process. In contrast to RaidZ2, where you can lose *any* combination of 2 disks in your pool without anything affected. And in addition, with more than 4 disks, it is also more space efficient. So both more resilient and more cost/space efficient. But less performant.

RAID is no backup. It's not statet often enough imho. Both data loss scenarios you described can be mitigated by the simple 3 - 2 - 1 rule, an offsite backup is always more safe than any raid config.

True and I haven’t suggested anything else. So if you don’t care about resiliency at all, why even bother with mirrors? Why not just stripe everything..? (Yes, because it’s all a trade off…)

What is enough? I believe nextcloud would highly benefit from the higher iops and the high rw numbers.

Exactly. It depends on the use case. Of course there will be nextcloud installations that “need” mirror pools to fly. Or even all-flash arrays and/or tons of memory. But also, and again, the typical use case you alluded to initially (“newbies” who want nextcloud or plex “up and running”) probably doesn’t. Neither Nextcloud or particularly Plex is particularly prone to large amounts of 4k random read/write in the typical use case.

Ymmv.

 

[mistermanko]

Nope, not same probability.

The probability of two disk failing at the same time is equal across all array configs. Resilver stress failure is another scenario, and yes, it's the achilles heel of RAID10. Constant disk monitoring is key and maybe switching drives occasionally between mirrors for better age distribution.

Neither Nextcloud or particularly Plex is particularly prone to large amounts of 4k random read/write in the typical use case.

blocksize is irrelevant because RAID10 will prevail with any blocksize, op tested 64k and 4k. Will nextcloud benefit from it? Well, yes. Latency.

As I have stated earlier, RAID 0 and RAID 10 have, effectively, no system overhead to consider. The mirroring operation requires essentially no computational effort and is, for all intents and purposes, immeasurably small. Parity RAID does have computational overhead and this results in latency at the storage layer and system resources being consumed.

Understanding RAID Performance at Various Levels

Choosing a RAID level is an exercise in balancing many factors, including cost, reliability, capacity, and performance. RAID performance and speed can be challenging to understand, mainly as distinct

blog.storagecraft.com

 

[rungekutta]

The probability of two disk failing at the same time is equal across all array configs. Resilver stress failure is another scenario, and yes, it's the achilles heel of RAID10. Constant disk monitoring is key and maybe switching drives occasionally between mirrors for better age distribution.

Yes possibility of 2 disk failure is the same irrespective of layout but as I explained before, in mirrors if the wrong 2 disks fail at the same time your pool is dead, whereas in RaidZ2 *any* two disks can fail and you are still ok (until a 3rd one dies!). So the probabilities are not the same and how this all compares more or less favorably varies with total amount of disks etc. But no, it’s not all the same.

blocksize is irrelevant because RAID10 will prevail with any blocksize, op tested 64k and 4k. Will nextcloud benefit from it? Well, yes. Latency.

Understanding RAID Performance at Various Levels

Choosing a RAID level is an exercise in balancing many factors, including cost, reliability, capacity, and performance. RAID performance and speed can be challenging to understand, mainly as distinct

blog.storagecraft.com

Again, depends. But in any case, I think that point has already been well made by now.

 

[blanchet]

It depends of the type of array.

• For an array with spinning disks, a strip of mirrors may be the better choice because you have large disks but few IOPS. So sacrificing space for more IOPS is very interesting.
• For an all-flash-array, a 6-wide RAIDz2 may be the better choice because you have a lot of IOPS but smaller disks. So sacrificing IOPS for space is very interesting.

 

[jgreco]

Nope, not same probability. If you lose a disk in a mirror, 1 and exactly 1 other disk has got the data that you need to recover.

Just to be precise, that's if you lose a disk in a two-disk mirror. For environments where loss of redundancy due to a single drive failure is unacceptable, the standard has always been a three-way mirror or better, and in that case, two or more other disks will have the data if you lose a disk.

 

[rungekutta]

Just to be precise, that's if you lose a disk in a two-disk mirror. For environments where loss of redundancy due to a single drive failure is unacceptable, the standard has always been a three-way mirror or better, and in that case, two or more other disks will have the data if you lose a disk.

Fair and yes you’re right, that was of course in reference to two-disk mirrors.

 

[jgreco]

Fair and yes you’re right, that was of course in reference to two-disk mirrors.

Right, but then why is the comparison between a double-redundancy solution (RAIDZ2) and a single-redundancy solution (two-way mirrors)? The parallel IOPS performance of a three-way mirror for reads is gobsmacking and shouldn't be dismissed offhand. It is, of course, very expensive in terms of drive count though.

 

[rungekutta]

You can read back a bit for context. This was all in response (objection) to post #2 and specifically (my bold) "Clearly the use case is what matters and 1 in 3 new users here (my gut feeling) want a nextcloud instance or a plex media server up in running, so in both usecases a 2-way-mirrored vdev pool of no matter the drive count, is always the better choice"

 

[jgreco]

I suppose, but in the context of the totality of the thread, it still feels like we should not assume mirror == 2-way, as that is an apples-to-oranges comparison.

 

[mikeyscott]

In the process of setting up TrueNAS Core and tempted with 4 x 3TB in 2 x mirror config over RAIDZ2 setup. Disks can be quickly purchased these days and equally I may purchase an additional disk for an online spare. The T340 I have can take 8 x 3.5" disk / 2.5" with caddies

 

[pschatz100]

The discussion of 4 disk RaidZ2 versus 2 mirrored pairs has carried on for years. As summarized in the old threads above: RaidZ2 will give somewhat better resiliency while two mirrored pairs will give somewhat better performance.

Folks often think they need the best possible performance - but this might not be correct. What do you need for your use case?

If your system is primarily used for serving movies and music (Plex comes to mind here), then you do not need maximum performance: the bit rate of your movies will not exceed the performance on any RaidZ2 pool - therefore the faster performance of mirrored pools will not give any advantage. However, if your work load involves something like editing files in real-time then you might want to configure the system for performance. Also, don't forget that your network speed will impose limits on what you can do.

My system is used primarily for Plex and for back-ups of three windows desktops. I have a couple of jails, but no VM's. I use a 4x4Tb RaidZ2. Performance of the pool has never been a bottleneck.

Regardless of the topology you decide to go with, remember that neither RaidZ2 or mirroring are a substitute for a reliable backup strategy.