I got the new LSI 9305-16i controller now and could upgrade it to latest firmware (IT mode) successfully.
The performance results with fio are slightly slower than the HP as it seems (sync=always and same settings for ZFS pool) on:
Code:
Random write test for IOP/s
==================
HP:
root@truenas[/mnt/tank/nas/test]# sync;fio --randrepeat=1 --ioengine=libaio --direct=1 --name=test --filename=test --bs=4k --size=4G --readwrite=randwrite --ramp_time=4
test: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
fio-3.25
Starting 1 process
test: Laying out IO file (1 file / 4096MiB)
Jobs: 1 (f=1): [w(1)][98.1%][w=27.9MiB/s][w=7142 IOPS][eta 00m:03s]
test: (groupid=0, jobs=1): err= 0: pid=1998556: Fri Feb 4 16:35:13 2022
write: IOPS=6839, BW=26.7MiB/s (28.0MB/s)(4027MiB/150724msec); 0 zone resets
slat (usec): min=62, max=473862, avg=140.91, stdev=702.07
clat (nsec): min=1070, max=536429, avg=2585.70, stdev=2237.90
lat (usec): min=64, max=473877, avg=143.93, stdev=702.16
clat percentiles (nsec):
| 1.00th=[ 1448], 5.00th=[ 1576], 10.00th=[ 1672], 20.00th=[ 1912],
| 30.00th=[ 2480], 40.00th=[ 2608], 50.00th=[ 2672], 60.00th=[ 2704],
| 70.00th=[ 2768], 80.00th=[ 2896], 90.00th=[ 3088], 95.00th=[ 3344],
| 99.00th=[ 4640], 99.50th=[ 5344], 99.90th=[12096], 99.95th=[15296],
| 99.99th=[27008]
bw ( KiB/s): min= 9539, max=35480, per=100.00%, avg=27361.58, stdev=3657.79, samples=301
iops : min= 2384, max= 8870, avg=6840.31, stdev=914.46, samples=301
lat (usec) : 2=22.35%, 4=76.07%, 10=1.45%, 20=0.11%, 50=0.02%
lat (usec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%
cpu : usr=4.37%, sys=49.85%, ctx=2066823, majf=1, minf=971
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,1030819,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
WRITE: bw=26.7MiB/s (28.0MB/s), 26.7MiB/s-26.7MiB/s (28.0MB/s-28.0MB/s), io=4027MiB (4222MB), run=150724-150724msec
LSI:
sync;fio --randrepeat=1 --ioengine=libaio --direct=1 --name=test --filename=testnew2 --bs=4k --size=4G --readwrite=randwrite --ramp_time=4
test: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
fio-3.25
Starting 1 process
test: Laying out IO file (1 file / 4096MiB)
Jobs: 1 (f=1): [w(1)][97.9%][w=27.0MiB/s][w=6914 IOPS][eta 00m:04s]
test: (groupid=0, jobs=1): err= 0: pid=1067055: Thu Feb 10 12:13:05 2022
write: IOPS=5748, BW=22.5MiB/s (23.5MB/s)(4019MiB/178987msec); 0 zone resets
slat (usec): min=65, max=31422, avg=168.23, stdev=140.11
clat (nsec): min=1154, max=656163, avg=2589.97, stdev=1856.37
lat (usec): min=66, max=31436, avg=171.28, stdev=140.37
clat percentiles (nsec):
| 1.00th=[ 1352], 5.00th=[ 1496], 10.00th=[ 1656], 20.00th=[ 2192],
| 30.00th=[ 2448], 40.00th=[ 2512], 50.00th=[ 2576], 60.00th=[ 2736],
| 70.00th=[ 2832], 80.00th=[ 2928], 90.00th=[ 3120], 95.00th=[ 3376],
| 99.00th=[ 4512], 99.50th=[ 5088], 99.90th=[11328], 99.95th=[14656],
| 99.99th=[24704]
bw ( KiB/s): min=15432, max=30200, per=99.95%, avg=22982.62, stdev=2107.80, samples=357
iops : min= 3858, max= 7550, avg=5745.57, stdev=526.97, samples=357
lat (usec) : 2=17.91%, 4=80.46%, 10=1.51%, 20=0.10%, 50=0.02%
lat (usec) : 100=0.01%, 500=0.01%, 750=0.01%
cpu : usr=4.00%, sys=55.41%, ctx=2059230, majf=0, minf=1081
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,1028904,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
WRITE: bw=22.5MiB/s (23.5MB/s), 22.5MiB/s-22.5MiB/s (23.5MB/s-23.5MB/s), io=4019MiB (4214MB), run=178987-178987msec
Random Read test for IOP/s
==================
HP:
root@truenas[/mnt/tank/nas/test]# sync;fio --randrepeat=1 --ioengine=libaio --direct=1 --name=test --filename=test --bs=4k --size=4G --readwrite=randread --ramp_time=4
test: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
fio-3.25
Starting 1 process
Jobs: 1 (f=1): [r(1)][81.0%][r=398MiB/s][r=102k IOPS][eta 00m:04s]
test: (groupid=0, jobs=1): err= 0: pid=2866151: Fri Feb 4 16:37:05 2022
read: IOPS=65.7k, BW=257MiB/s (269MB/s)(3247MiB/12657msec)
slat (usec): min=3, max=549, avg=13.95, stdev=18.16
clat (nsec): min=604, max=444275, avg=731.07, stdev=1274.23
lat (usec): min=3, max=551, avg=14.77, stdev=18.35
clat percentiles (nsec):
| 1.00th=[ 628], 5.00th=[ 636], 10.00th=[ 644], 20.00th=[ 644],
| 30.00th=[ 652], 40.00th=[ 652], 50.00th=[ 660], 60.00th=[ 668],
| 70.00th=[ 684], 80.00th=[ 876], 90.00th=[ 924], 95.00th=[ 948],
| 99.00th=[ 1032], 99.50th=[ 1224], 99.90th=[ 2160], 99.95th=[ 6752],
| 99.99th=[16320]
bw ( KiB/s): min=196032, max=535048, per=98.18%, avg=257947.96, stdev=70103.51, samples=25
iops : min=49008, max=133762, avg=64486.88, stdev=17525.91, samples=25
lat (nsec) : 750=78.40%, 1000=20.19%
lat (usec) : 2=1.23%, 4=0.11%, 10=0.01%, 20=0.05%, 50=0.01%
lat (usec) : 500=0.01%
cpu : usr=8.81%, sys=91.17%, ctx=28, majf=0, minf=63
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=831353,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=257MiB/s (269MB/s), 257MiB/s-257MiB/s (269MB/s-269MB/s), io=3247MiB (3405MB), run=12657-12657msec
LSI:
root@truenas[/mnt/tank/nas/test]# sync;fio --randrepeat=1 --ioengine=libaio --direct=1 --name=test --filename=test --bs=4k --size=4G --readwrite=randread --ramp_time=4
test: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
fio-3.25
Starting 1 process
Jobs: 1 (f=1): [r(1)][90.0%][r=108MiB/s][r=27.7k IOPS][eta 00m:05s]
test: (groupid=0, jobs=1): err= 0: pid=859569: Thu Feb 10 10:07:33 2022
read: IOPS=23.4k, BW=91.3MiB/s (95.7MB/s)(3751MiB/41095msec)
slat (usec): min=3, max=583, avg=40.94, stdev=13.31
clat (nsec): min=616, max=455495, avg=976.73, stdev=1084.00
lat (usec): min=3, max=585, avg=42.05, stdev=13.47
clat percentiles (nsec):
| 1.00th=[ 644], 5.00th=[ 700], 10.00th=[ 916], 20.00th=[ 932],
| 30.00th=[ 948], 40.00th=[ 956], 50.00th=[ 964], 60.00th=[ 972],
| 70.00th=[ 980], 80.00th=[ 996], 90.00th=[ 1020], 95.00th=[ 1064],
| 99.00th=[ 1288], 99.50th=[ 1400], 99.90th=[ 6880], 99.95th=[15680],
| 99.99th=[16768]
bw ( KiB/s): min=81912, max=172240, per=99.60%, avg=93098.37, stdev=10122.34, samples=82
iops : min=20478, max=43060, avg=23274.46, stdev=2530.60, samples=82
lat (nsec) : 750=5.82%, 1000=77.17%
lat (usec) : 2=16.71%, 4=0.18%, 10=0.02%, 20=0.09%, 50=0.01%
lat (usec) : 500=0.01%
cpu : usr=4.82%, sys=95.18%, ctx=98, majf=0, minf=64
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=960345,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=91.3MiB/s (95.7MB/s), 91.3MiB/s-91.3MiB/s (95.7MB/s-95.7MB/s), io=3751MiB (3934MB), run=41095-41095msec
Mixed Random Workload
================
root@truenas[/mnt/tank/nas/test]# sync;fio --randrepeat=1 --ioengine=libaio --direct=1 --name=test --filename=test --bs=4k --size=4G --readwrite=readwrite --ramp_time=4
HP:
test: (g=0): rw=rw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
fio-3.25
Starting 1 process
Jobs: 1 (f=1): [M(1)][93.8%][r=28.7MiB/s,w=28.3MiB/s][r=7355,w=7246 IOPS][eta 00m:05s]
test: (groupid=0, jobs=1): err= 0: pid=2867096: Fri Feb 4 16:39:01 2022
read: IOPS=6896, BW=26.9MiB/s (28.2MB/s)(1936MiB/71868msec)
slat (usec): min=3, max=43247, avg=12.07, stdev=62.48
clat (nsec): min=679, max=452581, avg=1694.52, stdev=1770.21
lat (usec): min=4, max=43250, avg=13.96, stdev=62.53
clat percentiles (nsec):
| 1.00th=[ 860], 5.00th=[ 1012], 10.00th=[ 1624], 20.00th=[ 1656],
| 30.00th=[ 1672], 40.00th=[ 1688], 50.00th=[ 1704], 60.00th=[ 1736],
| 70.00th=[ 1752], 80.00th=[ 1768], 90.00th=[ 1800], 95.00th=[ 1832],
| 99.00th=[ 2480], 99.50th=[ 2544], 99.90th=[ 3120], 99.95th=[ 9536],
| 99.99th=[13888]
bw ( KiB/s): min=18032, max=33200, per=99.92%, avg=27564.61, stdev=1953.09, samples=143
iops : min= 4508, max= 8300, avg=6891.09, stdev=488.27, samples=143
write: IOPS=6886, BW=26.9MiB/s (28.2MB/s)(1933MiB/71868msec); 0 zone resets
slat (usec): min=59, max=170528, avg=125.27, stdev=510.20
clat (nsec): min=1142, max=446374, avg=2709.78, stdev=2012.39
lat (usec): min=60, max=170540, avg=128.44, stdev=510.24
clat percentiles (nsec):
| 1.00th=[ 1496], 5.00th=[ 1816], 10.00th=[ 2576], 20.00th=[ 2640],
| 30.00th=[ 2672], 40.00th=[ 2704], 50.00th=[ 2704], 60.00th=[ 2736],
| 70.00th=[ 2768], 80.00th=[ 2832], 90.00th=[ 2928], 95.00th=[ 3056],
| 99.00th=[ 3600], 99.50th=[ 3856], 99.90th=[ 5984], 99.95th=[11456],
| 99.99th=[15808]
bw ( KiB/s): min=17960, max=34048, per=99.90%, avg=27519.25, stdev=1861.64, samples=143
iops : min= 4490, max= 8512, avg=6879.76, stdev=465.42, samples=143
lat (nsec) : 750=0.02%, 1000=2.37%
lat (usec) : 2=49.44%, 4=47.95%, 10=0.16%, 20=0.06%, 50=0.01%
lat (usec) : 100=0.01%, 250=0.01%, 500=0.01%
cpu : usr=6.88%, sys=45.60%, ctx=989819, majf=0, minf=91
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=495619,494904,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=26.9MiB/s (28.2MB/s), 26.9MiB/s-26.9MiB/s (28.2MB/s-28.2MB/s), io=1936MiB (2030MB), run=71868-71868msec
WRITE: bw=26.9MiB/s (28.2MB/s), 26.9MiB/s-26.9MiB/s (28.2MB/s-28.2MB/s), io=1933MiB (2027MB), run=71868-71868msec
LSI:
sync;fio --randrepeat=1 --ioengine=libaio --direct=1 --name=test --filename=test --bs=4k --size=4G --readwrite=readwrite --ramp_time=4
test: (g=0): rw=rw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
fio-3.25
Starting 1 process
Jobs: 1 (f=0): [f(1)][100.0%][r=26.3MiB/s,w=26.8MiB/s][r=6722,w=6858 IOPS][eta 00m:00s]
test: (groupid=0, jobs=1): err= 0: pid=862535: Thu Feb 10 10:10:46 2022
read: IOPS=6889, BW=26.9MiB/s (28.2MB/s)(1944MiB/72228msec)
slat (usec): min=3, max=34604, avg=13.06, stdev=67.59
clat (nsec): min=668, max=433936, avg=1630.22, stdev=1734.46
lat (usec): min=4, max=34607, avg=14.87, stdev=67.66
clat percentiles (nsec):
| 1.00th=[ 820], 5.00th=[ 1020], 10.00th=[ 1576], 20.00th=[ 1592],
| 30.00th=[ 1592], 40.00th=[ 1608], 50.00th=[ 1624], 60.00th=[ 1640],
| 70.00th=[ 1656], 80.00th=[ 1672], 90.00th=[ 1720], 95.00th=[ 1816],
| 99.00th=[ 2320], 99.50th=[ 2416], 99.90th=[ 3152], 99.95th=[ 7008],
| 99.99th=[12480]
bw ( KiB/s): min=18504, max=33144, per=100.00%, avg=27566.56, stdev=1869.06, samples=144
iops : min= 4626, max= 8286, avg=6891.58, stdev=467.27, samples=144
write: IOPS=6880, BW=26.9MiB/s (28.2MB/s)(1941MiB/72228msec); 0 zone resets
slat (usec): min=60, max=187177, avg=125.11, stdev=421.97
clat (nsec): min=1063, max=438415, avg=2344.43, stdev=1667.71
lat (usec): min=62, max=187185, avg=127.98, stdev=422.01
clat percentiles (nsec):
| 1.00th=[ 1224], 5.00th=[ 1672], 10.00th=[ 2192], 20.00th=[ 2256],
| 30.00th=[ 2288], 40.00th=[ 2288], 50.00th=[ 2320], 60.00th=[ 2352],
| 70.00th=[ 2384], 80.00th=[ 2416], 90.00th=[ 2512], 95.00th=[ 2736],
| 99.00th=[ 3376], 99.50th=[ 3504], 99.90th=[ 5024], 99.95th=[10560],
| 99.99th=[14528]
bw ( KiB/s): min=18232, max=32200, per=100.00%, avg=27530.99, stdev=1749.62, samples=144
iops : min= 4558, max= 8050, avg=6882.69, stdev=437.40, samples=144
lat (nsec) : 750=0.01%, 1000=2.43%
lat (usec) : 2=48.62%, 4=48.79%, 10=0.10%, 20=0.04%, 50=0.01%
lat (usec) : 500=0.01%
cpu : usr=6.07%, sys=45.98%, ctx=994047, majf=0, minf=120
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=497605,496956,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=26.9MiB/s (28.2MB/s), 26.9MiB/s-26.9MiB/s (28.2MB/s-28.2MB/s), io=1944MiB (2038MB), run=72228-72228msec
WRITE: bw=26.9MiB/s (28.2MB/s), 26.9MiB/s-26.9MiB/s (28.2MB/s-28.2MB/s), io=1941MiB (2036MB), run=72228-72228msec
Sequential write test for throughput
=======================
HP:
root@truenas[/mnt/tank/nas/test]# sync;fio --randrepeat=1 --ioengine=libaio --direct=1 --name=test --filename=test --bs=4M --size=4G --readwrite=write --ramp_time=4
test: (g=0): rw=write, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=libaio, iodepth=1
fio-3.25
Starting 1 process
Jobs: 1 (f=1): [W(1)][42.9%][w=621MiB/s][w=155 IOPS][eta 00m:08s]
test: (groupid=0, jobs=1): err= 0: pid=3065934: Fri Feb 4 16:39:36 2022
write: IOPS=148, BW=595MiB/s (624MB/s)(1140MiB/1917msec); 0 zone resets
slat (msec): min=4, max=129, avg= 6.94, stdev= 8.81
clat (nsec): min=1989, max=28696, avg=4774.37, stdev=2519.47
lat (msec): min=4, max=129, avg= 6.74, stdev= 8.10
clat percentiles (nsec):
| 1.00th=[ 2064], 5.00th=[ 2640], 10.00th=[ 2896], 20.00th=[ 3376],
| 30.00th=[ 3664], 40.00th=[ 4128], 50.00th=[ 4512], 60.00th=[ 4704],
| 70.00th=[ 4832], 80.00th=[ 5152], 90.00th=[ 6432], 95.00th=[ 9280],
| 99.00th=[16064], 99.50th=[22912], 99.90th=[28800], 99.95th=[28800],
| 99.99th=[28800]
bw ( KiB/s): min=614400, max=729088, per=100.00%, avg=666720.33, stdev=58000.40, samples=3
iops : min= 150, max= 178, avg=162.67, stdev=14.19, samples=3
lat (usec) : 2=0.70%, 4=37.32%, 10=58.45%, 20=2.82%, 50=0.70%
cpu : usr=3.91%, sys=65.97%, ctx=1218, majf=0, minf=61
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,284,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
WRITE: bw=595MiB/s (624MB/s), 595MiB/s-595MiB/s (624MB/s-624MB/s), io=1140MiB (1195MB), run=1917-1917msec
LSI:
sync;fio --randrepeat=1 --ioengine=libaio --direct=1 --name=test --filename=test --bs=4M --size=4G --readwrite=write --ramp_time=4
test: (g=0): rw=write, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=libaio, iodepth=1
fio-3.25
Starting 1 process
Jobs: 1 (f=1): [W(1)][46.2%][w=629MiB/s][w=157 IOPS][eta 00m:07s]
test: (groupid=0, jobs=1): err= 0: pid=1086055: Thu Feb 10 10:11:51 2022
write: IOPS=178, BW=718MiB/s (753MB/s)(1408MiB/1961msec); 0 zone resets
slat (msec): min=3, max=117, avg= 5.58, stdev= 7.38
clat (nsec): min=1817, max=446102, avg=5034.25, stdev=23656.84
lat (msec): min=3, max=117, avg= 5.58, stdev= 7.39
clat percentiles (usec):
| 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 3], 20.00th=[ 3],
| 30.00th=[ 4], 40.00th=[ 4], 50.00th=[ 4], 60.00th=[ 4],
| 70.00th=[ 4], 80.00th=[ 5], 90.00th=[ 5], 95.00th=[ 7],
| 99.00th=[ 11], 99.50th=[ 12], 99.90th=[ 445], 99.95th=[ 445],
| 99.99th=[ 445]
bw ( KiB/s): min=442368, max=876544, per=96.64%, avg=710515.00, stdev=234410.46, samples=3
iops : min= 108, max= 214, avg=173.33, stdev=57.14, samples=3
lat (usec) : 2=0.57%, 4=74.07%, 10=23.36%, 20=1.71%, 500=0.28%
cpu : usr=3.26%, sys=78.17%, ctx=788, majf=0, minf=63
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,351,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
WRITE: bw=718MiB/s (753MB/s), 718MiB/s-718MiB/s (753MB/s-753MB/s), io=1408MiB (1476MB), run=1961-1961msec
Sequential Read test for throughput
=======================
HP:
root@truenas[/mnt/tank/nas/test]# sync;fio --randrepeat=1 --ioengine=libaio --direct=1 --name=test --filename=test --bs=4M --size=4G --readwrite=read --ramp_time=4
test: (g=0): rw=read, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=libaio, iodepth=1
fio-3.25
Starting 1 process
Jobs: 1 (f=0)
test: (groupid=0, jobs=1): err= 0: pid=3120437: Fri Feb 4 16:39:58 2022
read: IOPS=507, BW=2030MiB/s (2128MB/s)(4096MiB/2018msec)
cpu : usr=0.35%, sys=99.50%, ctx=2, majf=0, minf=526
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1024,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=2030MiB/s (2128MB/s), 2030MiB/s-2030MiB/s (2128MB/s-2128MB/s), io=4096MiB (4295MB), run=2018-2018msec
LSI:
root@truenas[/mnt/tank/nas/test]# sync;fio --randrepeat=1 --ioengine=libaio --direct=1 --name=test --filename=test --bs=4M --size=4G --readwrite=read --ramp_time=4
test: (g=0): rw=read, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=libaio, iodepth=1
fio-3.25
Starting 1 process
Jobs: 1 (f=0)
test: (groupid=0, jobs=1): err= 0: pid=1140319: Thu Feb 10 10:12:54 2022
read: IOPS=517, BW=2069MiB/s (2169MB/s)(4096MiB/1980msec)
cpu : usr=0.00%, sys=99.90%, ctx=4, majf=0, minf=523
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1024,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=2069MiB/s (2169MB/s), 2069MiB/s-2069MiB/s (2169MB/s-2169MB/s), io=4096MiB (4295MB), run=1980-1980msec
Is the performance I get with the LSI card attached that slow because of TrueNAS Scale not tuned yet (read this somewhere, don't know if correct)? Would it better to give TrueNAS Core a try?
The LSI controller UEFI allows me to enable Disk Write Caching. Should this be done or left disabled, whats the common practice here?
The server boots slower that is true. However, for me this is not a reason to not use it and I rather think its better to have the opportunity to control the card via UEFI directly although the slower boot.