andrwhmmr
Dabbler
- Joined
- Oct 8, 2017
- Messages
- 13
Hi,
I just wanted to leave some benchmarks regarding the performance delta of RAID-Z2 vs RAID10.
I think this forum is a great resource, so if anyone wants to have some more input for making their decision I hope this might help.
I am by no means a FIO/ZFS Expert so, any input is greatly appreciated, I will continue to test more before this pool goes into production.
RAID-Z2 was created as one vdev with 6 devices
RAID10 was created with 3 vdevs á 2 mirrored devices
Everything on default.
Hardware:
6x WD RED Pro 4TB (WD4003FFBX)
Broadcom 9207-8i SAS2308
i5-6500
24GB RAM (non-ECC)
First Comparison:
Summary:
RAID-Z2: READ: bw=14.0MiB/s (15.7MB/s), WRITE: bw=15.1MiB/s (15.9MB/s), write: IOPS=242 , read: IOPS=239
RAID10: READ: bw=32.3MiB/s (33.9MB/s), WRITE: bw=32.5MiB/s (34.1MB/s), write: IOPS=519, read: IOPS=516
Second Comparison:
Summary:
RAID-Z2: READ: bw=403KiB/s (413kB/s), WRITE: bw=417KiB/s (427kB/s) write: IOPS=104 , read: IOPS=100
RAID10: READ: bw=2714KiB/s (2779kB/s), WRITE: bw=2725KiB/s (2790kB/s), write: IOPS=681 ,read: IOPS=678
Have a good one.
I just wanted to leave some benchmarks regarding the performance delta of RAID-Z2 vs RAID10.
I think this forum is a great resource, so if anyone wants to have some more input for making their decision I hope this might help.
I am by no means a FIO/ZFS Expert so, any input is greatly appreciated, I will continue to test more before this pool goes into production.
RAID-Z2 was created as one vdev with 6 devices
RAID10 was created with 3 vdevs á 2 mirrored devices
Everything on default.
Hardware:
6x WD RED Pro 4TB (WD4003FFBX)
Broadcom 9207-8i SAS2308
i5-6500
24GB RAM (non-ECC)
First Comparison:
fio --filename=/mnt/test1/file1 --size=500GB --direct=1 --rw=randrw --bs=64k --ioengine=posixaio --iodepth=64 --runtime=120 --numjobs=4 --time_based --group_reporting --name=throughput-test-job --eta-newline=1
throughput-test-job: (groupid=0, jobs=4): err= 0: pid=36165: Fri Jun 4 22:15:45 2021
read: IOPS=239, BW=14.0MiB/s (15.7MB/s)(1807MiB/120801msec)
slat (nsec): min=512, max=705169, avg=3088.44, stdev=14625.70
clat (usec): min=96, max=2569.6k, avg=540329.69, stdev=439879.97
lat (usec): min=108, max=2569.6k, avg=540332.77, stdev=439877.40
clat percentiles (usec):
| 1.00th=[ 441], 5.00th=[ 2212], 10.00th=[ 29754],
| 20.00th=[ 62129], 30.00th=[ 143655], 40.00th=[ 283116],
| 50.00th=[ 526386], 60.00th=[ 725615], 70.00th=[ 843056],
| 80.00th=[ 943719], 90.00th=[1082131], 95.00th=[1249903],
| 99.00th=[1635779], 99.50th=[1803551], 99.90th=[2038432],
| 99.95th=[2122318], 99.99th=[2264925]
bw ( KiB/s): min= 1504, max=206639, per=99.28%, avg=15203.07, stdev=5211.73, samples=945
iops : min= 20, max= 3227, avg=235.13, stdev=81.50, samples=945
write: IOPS=242, BW=15.1MiB/s (15.9MB/s)(1829MiB/120801msec)
slat (nsec): min=940, max=878599, avg=5217.39, stdev=14775.89
clat (usec): min=79, max=2408.6k, avg=521712.68, stdev=448748.78
lat (usec): min=122, max=2408.6k, avg=521717.90, stdev=448746.31
clat percentiles (usec):
| 1.00th=[ 441], 5.00th=[ 2073], 10.00th=[ 27657],
| 20.00th=[ 57934], 30.00th=[ 105382], 40.00th=[ 208667],
| 50.00th=[ 484443], 60.00th=[ 717226], 70.00th=[ 834667],
| 80.00th=[ 935330], 90.00th=[1082131], 95.00th=[1249903],
| 99.00th=[1652556], 99.50th=[1803551], 99.90th=[2088764],
| 99.95th=[2197816], 99.99th=[2332034]
bw ( KiB/s): min= 1506, max=216789, per=99.35%, avg=15397.98, stdev=5496.92, samples=944
iops : min= 20, max= 3385, avg=238.19, stdev=85.94, samples=944
lat (usec) : 100=0.01%, 250=0.43%, 500=0.72%, 750=0.63%, 1000=0.57%
lat (msec) : 2=2.42%, 4=1.06%, 10=0.52%, 20=1.65%, 50=8.77%
lat (msec) : 100=11.22%, 250=12.61%, 500=9.03%, 750=12.28%, 1000=23.21%
lat (msec) : 2000=14.72%, >=2000=0.15%
cpu : usr=0.07%, sys=0.16%, ctx=54644, majf=0, minf=4
IO depths : 1=0.1%, 2=0.2%, 4=0.3%, 8=0.7%, 16=1.6%, 32=84.1%, >=64=13.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=93.7%, 8=2.4%, 16=2.6%, 32=1.1%, 64=0.1%, >=64=0.0%
issued rwts: total=28906,29256,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64
Run status group 0 (all jobs):
READ: bw=14.0MiB/s (15.7MB/s), 14.0MiB/s-14.0MiB/s (15.7MB/s-15.7MB/s), io=1807MiB (1894MB), run=120801-120801msec
WRITE: bw=15.1MiB/s (15.9MB/s), 15.1MiB/s-15.1MiB/s (15.9MB/s-15.9MB/s), io=1829MiB (1917MB), run=120801-120801msec
read: IOPS=239, BW=14.0MiB/s (15.7MB/s)(1807MiB/120801msec)
slat (nsec): min=512, max=705169, avg=3088.44, stdev=14625.70
clat (usec): min=96, max=2569.6k, avg=540329.69, stdev=439879.97
lat (usec): min=108, max=2569.6k, avg=540332.77, stdev=439877.40
clat percentiles (usec):
| 1.00th=[ 441], 5.00th=[ 2212], 10.00th=[ 29754],
| 20.00th=[ 62129], 30.00th=[ 143655], 40.00th=[ 283116],
| 50.00th=[ 526386], 60.00th=[ 725615], 70.00th=[ 843056],
| 80.00th=[ 943719], 90.00th=[1082131], 95.00th=[1249903],
| 99.00th=[1635779], 99.50th=[1803551], 99.90th=[2038432],
| 99.95th=[2122318], 99.99th=[2264925]
bw ( KiB/s): min= 1504, max=206639, per=99.28%, avg=15203.07, stdev=5211.73, samples=945
iops : min= 20, max= 3227, avg=235.13, stdev=81.50, samples=945
write: IOPS=242, BW=15.1MiB/s (15.9MB/s)(1829MiB/120801msec)
slat (nsec): min=940, max=878599, avg=5217.39, stdev=14775.89
clat (usec): min=79, max=2408.6k, avg=521712.68, stdev=448748.78
lat (usec): min=122, max=2408.6k, avg=521717.90, stdev=448746.31
clat percentiles (usec):
| 1.00th=[ 441], 5.00th=[ 2073], 10.00th=[ 27657],
| 20.00th=[ 57934], 30.00th=[ 105382], 40.00th=[ 208667],
| 50.00th=[ 484443], 60.00th=[ 717226], 70.00th=[ 834667],
| 80.00th=[ 935330], 90.00th=[1082131], 95.00th=[1249903],
| 99.00th=[1652556], 99.50th=[1803551], 99.90th=[2088764],
| 99.95th=[2197816], 99.99th=[2332034]
bw ( KiB/s): min= 1506, max=216789, per=99.35%, avg=15397.98, stdev=5496.92, samples=944
iops : min= 20, max= 3385, avg=238.19, stdev=85.94, samples=944
lat (usec) : 100=0.01%, 250=0.43%, 500=0.72%, 750=0.63%, 1000=0.57%
lat (msec) : 2=2.42%, 4=1.06%, 10=0.52%, 20=1.65%, 50=8.77%
lat (msec) : 100=11.22%, 250=12.61%, 500=9.03%, 750=12.28%, 1000=23.21%
lat (msec) : 2000=14.72%, >=2000=0.15%
cpu : usr=0.07%, sys=0.16%, ctx=54644, majf=0, minf=4
IO depths : 1=0.1%, 2=0.2%, 4=0.3%, 8=0.7%, 16=1.6%, 32=84.1%, >=64=13.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=93.7%, 8=2.4%, 16=2.6%, 32=1.1%, 64=0.1%, >=64=0.0%
issued rwts: total=28906,29256,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64
Run status group 0 (all jobs):
READ: bw=14.0MiB/s (15.7MB/s), 14.0MiB/s-14.0MiB/s (15.7MB/s-15.7MB/s), io=1807MiB (1894MB), run=120801-120801msec
WRITE: bw=15.1MiB/s (15.9MB/s), 15.1MiB/s-15.1MiB/s (15.9MB/s-15.9MB/s), io=1829MiB (1917MB), run=120801-120801msec
throughput-test-job: (groupid=0, jobs=4): err= 0: pid=38066: Fri Jun 4 22:32:31 2021
read: IOPS=516, BW=32.3MiB/s (33.9MB/s)(3893MiB/120523msec)
slat (nsec): min=501, max=555586, avg=921.89, stdev=2454.24
clat (usec): min=181, max=1480.2k, avg=246170.93, stdev=132586.16
lat (usec): min=184, max=1480.2k, avg=246171.85, stdev=132586.16
clat percentiles (msec):
| 1.00th=[ 68], 5.00th=[ 97], 10.00th=[ 114], 20.00th=[ 138],
| 30.00th=[ 161], 40.00th=[ 184], 50.00th=[ 213], 60.00th=[ 247],
| 70.00th=[ 288], 80.00th=[ 342], 90.00th=[ 422], 95.00th=[ 502],
| 99.00th=[ 684], 99.50th=[ 751], 99.90th=[ 953], 99.95th=[ 1003],
| 99.99th=[ 1133]
bw ( KiB/s): min=12239, max=54232, per=100.00%, avg=33178.15, stdev=1909.96, samples=948
iops : min= 188, max= 846, avg=516.53, stdev=29.86, samples=948
write: IOPS=519, BW=32.5MiB/s (34.1MB/s)(3917MiB/120523msec)
slat (nsec): min=997, max=62810, avg=3470.49, stdev=1796.50
clat (usec): min=175, max=1294.6k, avg=246773.24, stdev=132660.24
lat (usec): min=193, max=1294.6k, avg=246776.71, stdev=132660.21
clat percentiles (msec):
| 1.00th=[ 69], 5.00th=[ 97], 10.00th=[ 114], 20.00th=[ 138],
| 30.00th=[ 161], 40.00th=[ 186], 50.00th=[ 213], 60.00th=[ 247],
| 70.00th=[ 288], 80.00th=[ 347], 90.00th=[ 426], 95.00th=[ 502],
| 99.00th=[ 684], 99.50th=[ 743], 99.90th=[ 944], 99.95th=[ 1003],
| 99.99th=[ 1099]
bw ( KiB/s): min=11679, max=51400, per=100.00%, avg=33379.88, stdev=1831.65, samples=948
iops : min= 180, max= 801, avg=519.65, stdev=28.64, samples=948
lat (usec) : 250=0.01%, 500=0.01%
lat (msec) : 4=0.01%, 10=0.01%, 20=0.01%, 50=0.31%, 100=5.30%
lat (msec) : 250=55.26%, 500=34.01%, 750=4.61%, 1000=0.44%, 2000=0.05%
cpu : usr=0.16%, sys=0.35%, ctx=123966, majf=0, minf=4
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.2%, 32=87.1%, >=64=12.7%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, throughput-test-job: (groupid=0, jobs=4): err= 0: pid=38066: Fri Jun 4 22:32:31 2021
read: IOPS=516, BW=32.3MiB/s (33.9MB/s)(3893MiB/120523msec)
slat (nsec): min=501, max=555586, avg=921.89, stdev=2454.24
clat (usec): min=181, max=1480.2k, avg=246170.93, stdev=132586.16
lat (usec): min=184, max=1480.2k, avg=246171.85, stdev=132586.16
clat percentiles (msec):
| 1.00th=[ 68], 5.00th=[ 97], 10.00th=[ 114], 20.00th=[ 138],
| 30.00th=[ 161], 40.00th=[ 184], 50.00th=[ 213], 60.00th=[ 247],
| 70.00th=[ 288], 80.00th=[ 342], 90.00th=[ 422], 95.00th=[ 502],
| 99.00th=[ 684], 99.50th=[ 751], 99.90th=[ 953], 99.95th=[ 1003],
| 99.99th=[ 1133]
bw ( KiB/s): min=12239, max=54232, per=100.00%, avg=33178.15, stdev=1909.96, samples=948
iops : min= 188, max= 846, avg=516.53, stdev=29.86, samples=948
write: IOPS=519, BW=32.5MiB/s (34.1MB/s)(3917MiB/120523msec)
slat (nsec): min=997, max=62810, avg=3470.49, stdev=1796.50
clat (usec): min=175, max=1294.6k, avg=246773.24, stdev=132660.24
lat (usec): min=193, max=1294.6k, avg=246776.71, stdev=132660.21
clat percentiles (msec):
| 1.00th=[ 69], 5.00th=[ 97], 10.00th=[ 114], 20.00th=[ 138],
| 30.00th=[ 161], 40.00th=[ 186], 50.00th=[ 213], 60.00th=[ 247],
| 70.00th=[ 288], 80.00th=[ 347], 90.00th=[ 426], 95.00th=[ 502],
| 99.00th=[ 684], 99.50th=[ 743], 99.90th=[ 944], 99.95th=[ 1003],
| 99.99th=[ 1099]
bw ( KiB/s): min=11679, max=51400, per=100.00%, avg=33379.88, stdev=1831.65, samples=948
iops : min= 180, max= 801, avg=519.65, stdev=28.64, samples=948
lat (usec) : 250=0.01%, 500=0.01%
lat (msec) : 4=0.01%, 10=0.01%, 20=0.01%, 50=0.31%, 100=5.30%
lat (msec) : 250=55.26%, 500=34.01%, 750=4.61%, 1000=0.44%, 2000=0.05%
cpu : usr=0.16%, sys=0.35%, ctx=123966, majf=0, minf=4
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.2%, 32=87.1%, >=64=12.7%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=93.2%, 8=2.8%, 16=2.7%, 32=1.2%, 64=0.1%, >=64=0.0%
issued rwts: total=62290,62666,0,0 shthroughput-test-job: (groupid=0, jobs=4): err= 0: pid=38066: Fri Jun 4 22:32:31 2021
read: IOPS=516, BW=32.3MiB/s (33.9MB/s)(3893MiB/120523msec)
slat (nsec): min=501, max=555586, avg=921.89, stdev=2454.24
clat (usec): min=181, max=1480.2k, avg=246170.93, stdev=132586.16
lat (usec): min=184, max=1480.2k, avg=246171.85, stdev=132586.16
clat percentiles (msec):
| 1.00th=[ 68], 5.00th=[ 97], 10.00th=[ 114], 20.00th=[ 138],
| 30.00th=[ 161], 40.00th=[ 184], 50.00th=[ 213], 60.00th=[ 247],
| 70.00th=[ 288], 80.00th=[ 342], 90.00th=[ 422], 95.00th=[ 502],
| 99.00th=[ 684], 99.50th=[ 751], 99.90th=[ 953], 99.95th=[ 1003],
| 99.99th=[ 1133]
bw ( KiB/s): min=12239, max=54232, per=100.00%, avg=33178.15, stdev=1909.96, samples=948
iops : min= 188, max= 846, avg=516.53, stdev=29.86, samples=948
write: IOPS=519, BW=32.5MiB/s (34.1MB/s)(3917MiB/120523msec)
slat (nsec): min=997, max=62810, avg=3470.49, stdev=1796.50
clat (usec): min=175, max=1294.6k, avg=246773.24, stdev=132660.24
lat (usec): min=193, max=1294.6k, avg=246776.71, stdev=132660.21
clat percentiles (msec):
| 1.00th=[ 69], 5.00th=[ 97], 10.00th=[ 114], 20.00th=[ 138],
| 30.00th=[ 161], 40.00th=[ 186], 50.00th=[ 213], 60.00th=[ 247],
| 70.00th=[ 288], 80.00th=[ 347], 90.00th=[ 426], 95.00th=[ 502],
| 99.00th=[ 684], 99.50th=[ 743], 99.90th=[ 944], 99.95th=[ 1003],
| 99.99th=[ 1099]
bw ( KiB/s): min=11679, max=51400, per=100.00%, avg=33379.88, stdev=1831.65, samples=948
iops : min= 180, max= 801, avg=519.65, stdev=28.64, samples=948
lat (usec) : 250=0.01%, 500=0.01%
lat (msec) : 4=0.01%, 10=0.01%, 20=0.01%, 50=0.31%, 100=5.30%
lat (msec) : 250=55.26%, 500=34.01%, 750=4.61%, 1000=0.44%, 2000=0.05%
cpu : usr=0.16%, sys=0.35%, ctx=123966, majf=0, minf=4
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.2%, 32=87.1%, >=64=12.7%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=93.2%, 8=2.8%, 16=2.7%, 32=1.2%, 64=0.1%, >=64=0.0%
issued rwts: total=62290,62666,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64
Run status group 0 (all jobs):
READ: bw=32.3MiB/s (33.9MB/s), 32.3MiB/s-32.3MiB/s (33.9MB/s-33.9MB/s), io=3893MiB (4082MB), run=120523-120523msec
WRITE: bw=32.5MiB/s (34.1MB/s), 32.5MiB/s-32.5MiB/s (34.1MB/s-34.1MB/s), io=3917MiB (4107MB), run=120523-120523msec
ort=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64
Run status group 0 (all jobs):
READ: bw=32.3MiB/s (33.9MB/s), 32.3MiB/s-32.3MiB/s (33.9MB/s-33.9MB/s), io=3893MiB (4082MB), run=120523-120523msec
WRITE: bw=32.5MiB/s (34.1MB/s), 32.5MiB/s-32.5MiB/s (34.1MB/s-34.1MB/s), io=3917MiB (4107MB), run=120523-120523msec
>=64=0.0%
complete : 0=0.0%, 4=93.2%, 8=2.8%, 16=2.7%, 32=1.2%, 64=0.1%, >=64=0.0%
issued rwts: total=62290,62666,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64
Run status group 0 (all jobs):
READ: bw=32.3MiB/s (33.9MB/s), 32.3MiB/s-32.3MiB/s (33.9MB/s-33.9MB/s), io=3893MiB (4082MB), run=120523-120523msec
WRITE: bw=32.5MiB/s (34.1MB/s), 32.5MiB/s-32.5MiB/s (34.1MB/s-34.1MB/s), io=3917MiB (4107MB), run=120523-120523msec
read: IOPS=516, BW=32.3MiB/s (33.9MB/s)(3893MiB/120523msec)
slat (nsec): min=501, max=555586, avg=921.89, stdev=2454.24
clat (usec): min=181, max=1480.2k, avg=246170.93, stdev=132586.16
lat (usec): min=184, max=1480.2k, avg=246171.85, stdev=132586.16
clat percentiles (msec):
| 1.00th=[ 68], 5.00th=[ 97], 10.00th=[ 114], 20.00th=[ 138],
| 30.00th=[ 161], 40.00th=[ 184], 50.00th=[ 213], 60.00th=[ 247],
| 70.00th=[ 288], 80.00th=[ 342], 90.00th=[ 422], 95.00th=[ 502],
| 99.00th=[ 684], 99.50th=[ 751], 99.90th=[ 953], 99.95th=[ 1003],
| 99.99th=[ 1133]
bw ( KiB/s): min=12239, max=54232, per=100.00%, avg=33178.15, stdev=1909.96, samples=948
iops : min= 188, max= 846, avg=516.53, stdev=29.86, samples=948
write: IOPS=519, BW=32.5MiB/s (34.1MB/s)(3917MiB/120523msec)
slat (nsec): min=997, max=62810, avg=3470.49, stdev=1796.50
clat (usec): min=175, max=1294.6k, avg=246773.24, stdev=132660.24
lat (usec): min=193, max=1294.6k, avg=246776.71, stdev=132660.21
clat percentiles (msec):
| 1.00th=[ 69], 5.00th=[ 97], 10.00th=[ 114], 20.00th=[ 138],
| 30.00th=[ 161], 40.00th=[ 186], 50.00th=[ 213], 60.00th=[ 247],
| 70.00th=[ 288], 80.00th=[ 347], 90.00th=[ 426], 95.00th=[ 502],
| 99.00th=[ 684], 99.50th=[ 743], 99.90th=[ 944], 99.95th=[ 1003],
| 99.99th=[ 1099]
bw ( KiB/s): min=11679, max=51400, per=100.00%, avg=33379.88, stdev=1831.65, samples=948
iops : min= 180, max= 801, avg=519.65, stdev=28.64, samples=948
lat (usec) : 250=0.01%, 500=0.01%
lat (msec) : 4=0.01%, 10=0.01%, 20=0.01%, 50=0.31%, 100=5.30%
lat (msec) : 250=55.26%, 500=34.01%, 750=4.61%, 1000=0.44%, 2000=0.05%
cpu : usr=0.16%, sys=0.35%, ctx=123966, majf=0, minf=4
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.2%, 32=87.1%, >=64=12.7%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, throughput-test-job: (groupid=0, jobs=4): err= 0: pid=38066: Fri Jun 4 22:32:31 2021
read: IOPS=516, BW=32.3MiB/s (33.9MB/s)(3893MiB/120523msec)
slat (nsec): min=501, max=555586, avg=921.89, stdev=2454.24
clat (usec): min=181, max=1480.2k, avg=246170.93, stdev=132586.16
lat (usec): min=184, max=1480.2k, avg=246171.85, stdev=132586.16
clat percentiles (msec):
| 1.00th=[ 68], 5.00th=[ 97], 10.00th=[ 114], 20.00th=[ 138],
| 30.00th=[ 161], 40.00th=[ 184], 50.00th=[ 213], 60.00th=[ 247],
| 70.00th=[ 288], 80.00th=[ 342], 90.00th=[ 422], 95.00th=[ 502],
| 99.00th=[ 684], 99.50th=[ 751], 99.90th=[ 953], 99.95th=[ 1003],
| 99.99th=[ 1133]
bw ( KiB/s): min=12239, max=54232, per=100.00%, avg=33178.15, stdev=1909.96, samples=948
iops : min= 188, max= 846, avg=516.53, stdev=29.86, samples=948
write: IOPS=519, BW=32.5MiB/s (34.1MB/s)(3917MiB/120523msec)
slat (nsec): min=997, max=62810, avg=3470.49, stdev=1796.50
clat (usec): min=175, max=1294.6k, avg=246773.24, stdev=132660.24
lat (usec): min=193, max=1294.6k, avg=246776.71, stdev=132660.21
clat percentiles (msec):
| 1.00th=[ 69], 5.00th=[ 97], 10.00th=[ 114], 20.00th=[ 138],
| 30.00th=[ 161], 40.00th=[ 186], 50.00th=[ 213], 60.00th=[ 247],
| 70.00th=[ 288], 80.00th=[ 347], 90.00th=[ 426], 95.00th=[ 502],
| 99.00th=[ 684], 99.50th=[ 743], 99.90th=[ 944], 99.95th=[ 1003],
| 99.99th=[ 1099]
bw ( KiB/s): min=11679, max=51400, per=100.00%, avg=33379.88, stdev=1831.65, samples=948
iops : min= 180, max= 801, avg=519.65, stdev=28.64, samples=948
lat (usec) : 250=0.01%, 500=0.01%
lat (msec) : 4=0.01%, 10=0.01%, 20=0.01%, 50=0.31%, 100=5.30%
lat (msec) : 250=55.26%, 500=34.01%, 750=4.61%, 1000=0.44%, 2000=0.05%
cpu : usr=0.16%, sys=0.35%, ctx=123966, majf=0, minf=4
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.2%, 32=87.1%, >=64=12.7%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=93.2%, 8=2.8%, 16=2.7%, 32=1.2%, 64=0.1%, >=64=0.0%
issued rwts: total=62290,62666,0,0 shthroughput-test-job: (groupid=0, jobs=4): err= 0: pid=38066: Fri Jun 4 22:32:31 2021
read: IOPS=516, BW=32.3MiB/s (33.9MB/s)(3893MiB/120523msec)
slat (nsec): min=501, max=555586, avg=921.89, stdev=2454.24
clat (usec): min=181, max=1480.2k, avg=246170.93, stdev=132586.16
lat (usec): min=184, max=1480.2k, avg=246171.85, stdev=132586.16
clat percentiles (msec):
| 1.00th=[ 68], 5.00th=[ 97], 10.00th=[ 114], 20.00th=[ 138],
| 30.00th=[ 161], 40.00th=[ 184], 50.00th=[ 213], 60.00th=[ 247],
| 70.00th=[ 288], 80.00th=[ 342], 90.00th=[ 422], 95.00th=[ 502],
| 99.00th=[ 684], 99.50th=[ 751], 99.90th=[ 953], 99.95th=[ 1003],
| 99.99th=[ 1133]
bw ( KiB/s): min=12239, max=54232, per=100.00%, avg=33178.15, stdev=1909.96, samples=948
iops : min= 188, max= 846, avg=516.53, stdev=29.86, samples=948
write: IOPS=519, BW=32.5MiB/s (34.1MB/s)(3917MiB/120523msec)
slat (nsec): min=997, max=62810, avg=3470.49, stdev=1796.50
clat (usec): min=175, max=1294.6k, avg=246773.24, stdev=132660.24
lat (usec): min=193, max=1294.6k, avg=246776.71, stdev=132660.21
clat percentiles (msec):
| 1.00th=[ 69], 5.00th=[ 97], 10.00th=[ 114], 20.00th=[ 138],
| 30.00th=[ 161], 40.00th=[ 186], 50.00th=[ 213], 60.00th=[ 247],
| 70.00th=[ 288], 80.00th=[ 347], 90.00th=[ 426], 95.00th=[ 502],
| 99.00th=[ 684], 99.50th=[ 743], 99.90th=[ 944], 99.95th=[ 1003],
| 99.99th=[ 1099]
bw ( KiB/s): min=11679, max=51400, per=100.00%, avg=33379.88, stdev=1831.65, samples=948
iops : min= 180, max= 801, avg=519.65, stdev=28.64, samples=948
lat (usec) : 250=0.01%, 500=0.01%
lat (msec) : 4=0.01%, 10=0.01%, 20=0.01%, 50=0.31%, 100=5.30%
lat (msec) : 250=55.26%, 500=34.01%, 750=4.61%, 1000=0.44%, 2000=0.05%
cpu : usr=0.16%, sys=0.35%, ctx=123966, majf=0, minf=4
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.2%, 32=87.1%, >=64=12.7%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=93.2%, 8=2.8%, 16=2.7%, 32=1.2%, 64=0.1%, >=64=0.0%
issued rwts: total=62290,62666,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64
Run status group 0 (all jobs):
READ: bw=32.3MiB/s (33.9MB/s), 32.3MiB/s-32.3MiB/s (33.9MB/s-33.9MB/s), io=3893MiB (4082MB), run=120523-120523msec
WRITE: bw=32.5MiB/s (34.1MB/s), 32.5MiB/s-32.5MiB/s (34.1MB/s-34.1MB/s), io=3917MiB (4107MB), run=120523-120523msec
ort=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64
Run status group 0 (all jobs):
READ: bw=32.3MiB/s (33.9MB/s), 32.3MiB/s-32.3MiB/s (33.9MB/s-33.9MB/s), io=3893MiB (4082MB), run=120523-120523msec
WRITE: bw=32.5MiB/s (34.1MB/s), 32.5MiB/s-32.5MiB/s (34.1MB/s-34.1MB/s), io=3917MiB (4107MB), run=120523-120523msec
>=64=0.0%
complete : 0=0.0%, 4=93.2%, 8=2.8%, 16=2.7%, 32=1.2%, 64=0.1%, >=64=0.0%
issued rwts: total=62290,62666,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64
Run status group 0 (all jobs):
READ: bw=32.3MiB/s (33.9MB/s), 32.3MiB/s-32.3MiB/s (33.9MB/s-33.9MB/s), io=3893MiB (4082MB), run=120523-120523msec
WRITE: bw=32.5MiB/s (34.1MB/s), 32.5MiB/s-32.5MiB/s (34.1MB/s-34.1MB/s), io=3917MiB (4107MB), run=120523-120523msec
RAID-Z2: READ: bw=14.0MiB/s (15.7MB/s), WRITE: bw=15.1MiB/s (15.9MB/s), write: IOPS=242 , read: IOPS=239
RAID10: READ: bw=32.3MiB/s (33.9MB/s), WRITE: bw=32.5MiB/s (34.1MB/s), write: IOPS=519, read: IOPS=516
Second Comparison:
fio --filename=/mnt/test1/file1 --size=500GB --direct=1 --rw=randrw --bs=4k --ioengine=posixaio --iodepth=256 --runtime=120 --numjobs=4 --time_based --group_reporting --name=iops-test-job --eta-newline=1
iops-test-job: (groupid=0, jobs=4): err= 0: pid=32315: Fri Jun 4 20:42:29 2021
read: IOPS=100, BW=403KiB/s (413kB/s)(48.9MiB/124181msec)
slat (nsec): min=504, max=227325, avg=1052.51, stdev=3727.74
clat (msec): min=188, max=10689, avg=4947.31, stdev=986.77
lat (msec): min=188, max=10689, avg=4947.31, stdev=986.77
clat percentiles (msec):
| 1.00th=[ 1754], 5.00th=[ 3473], 10.00th=[ 3775], 20.00th=[ 4245],
| 30.00th=[ 4665], 40.00th=[ 4799], 50.00th=[ 5000], 60.00th=[ 5201],
| 70.00th=[ 5470], 80.00th=[ 5671], 90.00th=[ 5873], 95.00th=[ 6208],
| 99.00th=[ 7886], 99.50th=[ 8356], 99.90th=[ 9329], 99.95th=[ 9866],
| 99.99th=[10671]
bw ( KiB/s): min= 52, max= 3871, per=100.00%, avg=909.08, stdev=215.44, samples=421
iops : min= 10, max= 965, avg=225.30, stdev=53.96, samples=421
write: IOPS=104, BW=417KiB/s (427kB/s)(50.6MiB/124181msec)
slat (nsec): min=565, max=223761, avg=1246.43, stdev=4330.20
clat (usec): min=880, max=10693k, avg=4947211.52, stdev=967140.05
lat (usec): min=904, max=10693k, avg=4947212.76, stdev=967140.31
clat percentiles (msec):
| 1.00th=[ 1770], 5.00th=[ 3473], 10.00th=[ 3809], 20.00th=[ 4245],
| 30.00th=[ 4665], 40.00th=[ 4799], 50.00th=[ 5000], 60.00th=[ 5201],
| 70.00th=[ 5403], 80.00th=[ 5671], 90.00th=[ 5873], 95.00th=[ 6208],
| 99.00th=[ 7483], 99.50th=[ 8154], 99.90th=[ 8792], 99.95th=[ 9463],
| 99.99th=[10537]
bw ( KiB/s): min= 44, max= 4105, per=100.00%, avg=936.72, stdev=223.37, samples=423
iops : min= 8, max= 1024, avg=232.15, stdev=55.93, samples=423
lat (usec) : 1000=0.01%
lat (msec) : 2=0.01%, 250=0.01%, 500=0.06%, 750=0.12%, 1000=0.16%
lat (msec) : 2000=0.91%, >=2000=98.73%
cpu : usr=0.02%, sys=0.29%, ctx=25559, majf=0, minf=4
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.5%, 32=2.1%, >=64=97.1%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=97.7%, 8=0.4%, 16=0.5%, 32=0.6%, 64=0.4%, >=64=0.4%
issued rwts: total=12519,12943,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=256
Run status group 0 (all jobs):
READ: bw=403KiB/s (413kB/s), 403KiB/s-403KiB/s (413kB/s-413kB/s), io=48.9MiB (51.3MB), run=124181-124181msec
WRITE: bw=417KiB/s (427kB/s), 417KiB/s-417KiB/s (427kB/s-427kB/s), io=50.6MiB (53.0MB), run=124181-124181msec
read: IOPS=100, BW=403KiB/s (413kB/s)(48.9MiB/124181msec)
slat (nsec): min=504, max=227325, avg=1052.51, stdev=3727.74
clat (msec): min=188, max=10689, avg=4947.31, stdev=986.77
lat (msec): min=188, max=10689, avg=4947.31, stdev=986.77
clat percentiles (msec):
| 1.00th=[ 1754], 5.00th=[ 3473], 10.00th=[ 3775], 20.00th=[ 4245],
| 30.00th=[ 4665], 40.00th=[ 4799], 50.00th=[ 5000], 60.00th=[ 5201],
| 70.00th=[ 5470], 80.00th=[ 5671], 90.00th=[ 5873], 95.00th=[ 6208],
| 99.00th=[ 7886], 99.50th=[ 8356], 99.90th=[ 9329], 99.95th=[ 9866],
| 99.99th=[10671]
bw ( KiB/s): min= 52, max= 3871, per=100.00%, avg=909.08, stdev=215.44, samples=421
iops : min= 10, max= 965, avg=225.30, stdev=53.96, samples=421
write: IOPS=104, BW=417KiB/s (427kB/s)(50.6MiB/124181msec)
slat (nsec): min=565, max=223761, avg=1246.43, stdev=4330.20
clat (usec): min=880, max=10693k, avg=4947211.52, stdev=967140.05
lat (usec): min=904, max=10693k, avg=4947212.76, stdev=967140.31
clat percentiles (msec):
| 1.00th=[ 1770], 5.00th=[ 3473], 10.00th=[ 3809], 20.00th=[ 4245],
| 30.00th=[ 4665], 40.00th=[ 4799], 50.00th=[ 5000], 60.00th=[ 5201],
| 70.00th=[ 5403], 80.00th=[ 5671], 90.00th=[ 5873], 95.00th=[ 6208],
| 99.00th=[ 7483], 99.50th=[ 8154], 99.90th=[ 8792], 99.95th=[ 9463],
| 99.99th=[10537]
bw ( KiB/s): min= 44, max= 4105, per=100.00%, avg=936.72, stdev=223.37, samples=423
iops : min= 8, max= 1024, avg=232.15, stdev=55.93, samples=423
lat (usec) : 1000=0.01%
lat (msec) : 2=0.01%, 250=0.01%, 500=0.06%, 750=0.12%, 1000=0.16%
lat (msec) : 2000=0.91%, >=2000=98.73%
cpu : usr=0.02%, sys=0.29%, ctx=25559, majf=0, minf=4
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.5%, 32=2.1%, >=64=97.1%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=97.7%, 8=0.4%, 16=0.5%, 32=0.6%, 64=0.4%, >=64=0.4%
issued rwts: total=12519,12943,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=256
Run status group 0 (all jobs):
READ: bw=403KiB/s (413kB/s), 403KiB/s-403KiB/s (413kB/s-413kB/s), io=48.9MiB (51.3MB), run=124181-124181msec
WRITE: bw=417KiB/s (427kB/s), 417KiB/s-417KiB/s (427kB/s-427kB/s), io=50.6MiB (53.0MB), run=124181-124181msec
iops-test-job: (groupid=0, jobs=4): err= 0: pid=30062: Fri Jun 4 20:20:11 2021
read: IOPS=678, BW=2714KiB/s (2779kB/s)(321MiB/121018msec)
slat (nsec): min=512, max=397318k, avg=20017.75, stdev=1444869.34
clat (usec): min=627, max=3179.7k, avg=749743.34, stdev=415494.24
lat (usec): min=677, max=3179.7k, avg=749763.35, stdev=415463.36
clat percentiles (msec):
| 1.00th=[ 4], 5.00th=[ 14], 10.00th=[ 28], 20.00th=[ 144],
| 30.00th=[ 726], 40.00th=[ 793], 50.00th=[ 844], 60.00th=[ 894],
| 70.00th=[ 961], 80.00th=[ 1036], 90.00th=[ 1167], 95.00th=[ 1284],
| 99.00th=[ 1687], 99.50th=[ 1804], 99.90th=[ 2265], 99.95th=[ 2534],
| 99.99th=[ 2635]
bw ( KiB/s): min= 323, max=46289, per=100.00%, avg=2718.67, stdev=1014.76, samples=929
iops : min= 78, max=11571, avg=678.24, stdev=253.70, samples=929
write: IOPS=681, BW=2725KiB/s (2790kB/s)(322MiB/121018msec)
slat (nsec): min=561, max=213209k, avg=38457.73, stdev=1115054.67
clat (usec): min=750, max=3035.4k, avg=747149.32, stdev=413514.91
lat (usec): min=876, max=3035.4k, avg=747187.78, stdev=413451.97
clat percentiles (msec):
| 1.00th=[ 5], 5.00th=[ 15], 10.00th=[ 29], 20.00th=[ 150],
| 30.00th=[ 726], 40.00th=[ 793], 50.00th=[ 844], 60.00th=[ 894],
| 70.00th=[ 953], 80.00th=[ 1028], 90.00th=[ 1167], 95.00th=[ 1284],
| 99.00th=[ 1687], 99.50th=[ 1804], 99.90th=[ 2265], 99.95th=[ 2534],
| 99.99th=[ 2534]
bw ( KiB/s): min= 275, max=47915, per=100.00%, avg=2725.19, stdev=1035.74, samples=929
iops : min= 66, max=11977, avg=679.85, stdev=258.94, samples=929
lat (usec) : 750=0.01%, 1000=0.01%
lat (msec) : 2=0.21%, 4=0.75%, 10=2.58%, 20=3.54%, 50=7.48%
lat (msec) : 100=3.41%, 250=2.92%, 500=1.19%, 750=10.61%, 1000=43.25%
lat (msec) : 2000=23.81%, >=2000=0.24%
cpu : usr=0.12%, sys=1.57%, ctx=161777, majf=1, minf=7
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.3%, 16=0.8%, 32=2.0%, >=64=96.8%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=97.7%, 8=0.3%, 16=0.4%, 32=0.6%, 64=0.6%, >=64=0.4%
issued rwts: total=82116,82439,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=256
Run status group 0 (all jobs):
READ: bw=2714KiB/s (2779kB/s), 2714KiB/s-2714KiB/s (2779kB/s-2779kB/s), io=321MiB (336MB), run=121018-121018msec
WRITE: bw=2725KiB/s (2790kB/s), 2725KiB/s-2725KiB/s (2790kB/s-2790kB/s), io=322MiB (338MB), run=121018-121018msec
read: IOPS=678, BW=2714KiB/s (2779kB/s)(321MiB/121018msec)
slat (nsec): min=512, max=397318k, avg=20017.75, stdev=1444869.34
clat (usec): min=627, max=3179.7k, avg=749743.34, stdev=415494.24
lat (usec): min=677, max=3179.7k, avg=749763.35, stdev=415463.36
clat percentiles (msec):
| 1.00th=[ 4], 5.00th=[ 14], 10.00th=[ 28], 20.00th=[ 144],
| 30.00th=[ 726], 40.00th=[ 793], 50.00th=[ 844], 60.00th=[ 894],
| 70.00th=[ 961], 80.00th=[ 1036], 90.00th=[ 1167], 95.00th=[ 1284],
| 99.00th=[ 1687], 99.50th=[ 1804], 99.90th=[ 2265], 99.95th=[ 2534],
| 99.99th=[ 2635]
bw ( KiB/s): min= 323, max=46289, per=100.00%, avg=2718.67, stdev=1014.76, samples=929
iops : min= 78, max=11571, avg=678.24, stdev=253.70, samples=929
write: IOPS=681, BW=2725KiB/s (2790kB/s)(322MiB/121018msec)
slat (nsec): min=561, max=213209k, avg=38457.73, stdev=1115054.67
clat (usec): min=750, max=3035.4k, avg=747149.32, stdev=413514.91
lat (usec): min=876, max=3035.4k, avg=747187.78, stdev=413451.97
clat percentiles (msec):
| 1.00th=[ 5], 5.00th=[ 15], 10.00th=[ 29], 20.00th=[ 150],
| 30.00th=[ 726], 40.00th=[ 793], 50.00th=[ 844], 60.00th=[ 894],
| 70.00th=[ 953], 80.00th=[ 1028], 90.00th=[ 1167], 95.00th=[ 1284],
| 99.00th=[ 1687], 99.50th=[ 1804], 99.90th=[ 2265], 99.95th=[ 2534],
| 99.99th=[ 2534]
bw ( KiB/s): min= 275, max=47915, per=100.00%, avg=2725.19, stdev=1035.74, samples=929
iops : min= 66, max=11977, avg=679.85, stdev=258.94, samples=929
lat (usec) : 750=0.01%, 1000=0.01%
lat (msec) : 2=0.21%, 4=0.75%, 10=2.58%, 20=3.54%, 50=7.48%
lat (msec) : 100=3.41%, 250=2.92%, 500=1.19%, 750=10.61%, 1000=43.25%
lat (msec) : 2000=23.81%, >=2000=0.24%
cpu : usr=0.12%, sys=1.57%, ctx=161777, majf=1, minf=7
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.3%, 16=0.8%, 32=2.0%, >=64=96.8%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=97.7%, 8=0.3%, 16=0.4%, 32=0.6%, 64=0.6%, >=64=0.4%
issued rwts: total=82116,82439,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=256
Run status group 0 (all jobs):
READ: bw=2714KiB/s (2779kB/s), 2714KiB/s-2714KiB/s (2779kB/s-2779kB/s), io=321MiB (336MB), run=121018-121018msec
WRITE: bw=2725KiB/s (2790kB/s), 2725KiB/s-2725KiB/s (2790kB/s-2790kB/s), io=322MiB (338MB), run=121018-121018msec
RAID-Z2: READ: bw=403KiB/s (413kB/s), WRITE: bw=417KiB/s (427kB/s) write: IOPS=104 , read: IOPS=100
RAID10: READ: bw=2714KiB/s (2779kB/s), WRITE: bw=2725KiB/s (2790kB/s), write: IOPS=681 ,read: IOPS=678
Have a good one.