Some performance tests...
Note that the FreeNas box has a significant amount of memory so will have cached a bit of that (But thats fine since we are looking into network performance here).
gstat showed only ~55% utilization of the drives though, not sure what was limiting them on higher T/J counts
Code:
Z:\>fio --direct=1 --refill_buffers --norandommap --randrepeat=0 --group_reporting --ioengine=windowsaio --size=100G --bs=128k --iodepth=1 --numjobs=1 --rw=write --filename=fio.test -name win1
fio: this platform does not support process shared mutexes, forcing use of threads. Use the 'thread' option to get rid of this warning.
win1: (g=0): rw=write, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=windowsaio, iodepth=1
fio-3.16
Starting 1 thread
win1: Laying out IO file (1 file / 102400MiB)
Jobs: 1 (f=1): [W(1)][100.0%][w=817MiB/s][w=6533 IOPS][eta 00m:00s]
win1: (groupid=0, jobs=1): err= 0: pid=5284: Sun Dec 29 19:36:16 2019
write: IOPS=6708, BW=839MiB/s (879MB/s)(100GiB/122116msec)
slat (usec): min=15, max=604377, avg=113.73, stdev=671.36
clat (nsec): min=241, max=391319, avg=8071.63, stdev=4381.48
lat (usec): min=88, max=604394, avg=121.80, stdev=671.44
clat percentiles (nsec):
| 1.00th=[ 6176], 5.00th=[ 6368], 10.00th=[ 6368], 20.00th=[ 6432],
| 30.00th=[ 6496], 40.00th=[ 6496], 50.00th=[ 6560], 60.00th=[ 6688],
| 70.00th=[ 6944], 80.00th=[ 7328], 90.00th=[13248], 95.00th=[19328],
| 99.00th=[24960], 99.50th=[29312], 99.90th=[39680], 99.95th=[46336],
| 99.99th=[59136]
bw ( KiB/s): min=751904, max=962304, per=100.00%, avg=861146.40, stdev=48995.71, samples=243
iops : min= 5874, max= 7518, avg=6727.46, stdev=382.73, samples=243
lat (nsec) : 250=0.01%, 500=0.01%, 1000=0.01%
lat (usec) : 2=0.01%, 4=0.05%, 10=87.80%, 20=7.24%, 50=4.86%
lat (usec) : 100=0.03%, 500=0.01%
cpu : usr=19.65%, sys=16.38%, ctx=0, majf=0, minf=0
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,819200,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
WRITE: bw=839MiB/s (879MB/s), 839MiB/s-839MiB/s (879MB/s-879MB/s), io=100GiB (107GB), run=122116-122116msec
Z:\>fio --direct=1 --refill_buffers --norandommap --randrepeat=0 --group_reporting --ioengine=windowsaio --size=100G --bs=128k --iodepth=16 --numjobs=16 --rw=write --filename=fio.test -name win1
fio: this platform does not support process shared mutexes, forcing use of threads. Use the 'thread' option to get rid of this warning.
win1: (g=0): rw=write, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=windowsaio, iodepth=16
...
fio-3.16
Starting 16 threads
Jobs: 16 (f=16): [W(16)][99.8%][w=3307MiB/s][w=26.5k IOPS][eta 00m:01s]
win1: (groupid=0, jobs=16): err= 0: pid=6004: Sun Dec 29 19:45:30 2019
write: IOPS=27.8k, BW=3476MiB/s (3645MB/s)(1600GiB/471327msec)
slat (usec): min=3, max=4089, avg=11.43, stdev= 6.29
clat (usec): min=110, max=362591, avg=9160.72, stdev=2262.43
lat (usec): min=125, max=362602, avg=9172.15, stdev=2262.35
clat percentiles (usec):
| 1.00th=[ 7898], 5.00th=[ 8160], 10.00th=[ 8291], 20.00th=[ 8455],
| 30.00th=[ 8586], 40.00th=[ 8717], 50.00th=[ 8848], 60.00th=[ 9110],
| 70.00th=[ 9241], 80.00th=[ 9634], 90.00th=[10421], 95.00th=[11076],
| 99.00th=[12387], 99.50th=[12911], 99.90th=[13566], 99.95th=[13829],
| 99.99th=[14746]
bw ( MiB/s): min= 1406, max= 3969, per=99.99%, avg=3475.80, stdev=10.18, samples=15072
iops : min=11254, max=31755, avg=27805.70, stdev=81.47, samples=15072
lat (usec) : 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
lat (msec) : 2=0.02%, 4=0.01%, 10=84.75%, 20=15.22%, 50=0.01%
lat (msec) : 500=0.01%
cpu : usr=5.16%, sys=1.91%, ctx=0, majf=0, minf=0
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.2%, 16=99.8%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,13107200,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16
Run status group 0 (all jobs):
WRITE: bw=3476MiB/s (3645MB/s), 3476MiB/s-3476MiB/s (3645MB/s-3645MB/s), io=16
00GiB (1718GB), run=471327-471327msec
Z:\>fio --direct=1 --refill_buffers --norandommap --randrepeat=0 --group_reporting --ioengine=windowsaio --size=100G --bs=128k --iodepth=64 --numjobs=64 --rw=write --filename=fio.test -name win1
fio: this platform does not support process shared mutexes, forcing use of threads. Use the 'thread' option to get rid of this warning.
win1: (g=0): rw=write, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=windowsaio, iodepth=64
...
fio-3.16
Starting 64 threads
Jobs: 64 (f=64): [W(64)][13.9%][w=3352MiB/s][w=26.8k IOPS][eta 28m:11s]
fio: terminating on signal 2
win1: (groupid=0, jobs=64): err= 0: pid=5252: Sun Dec 29 19:51:50 2019
write: IOPS=26.8k, BW=3347MiB/s (3510MB/s)(891GiB/272726msec)
slat (usec): min=3, max=70025, avg=14.82, stdev=49.38
clat (usec): min=168, max=303734, avg=152738.44, stdev=5037.43
lat (usec): min=423, max=303745, avg=152753.26, stdev=5027.99
clat percentiles (msec):
| 1.00th=[ 148], 5.00th=[ 148], 10.00th=[ 150], 20.00th=[ 150],
| 30.00th=[ 153], 40.00th=[ 153], 50.00th=[ 153], 60.00th=[ 155],
| 70.00th=[ 155], 80.00th=[ 155], 90.00th=[ 157], 95.00th=[ 157],
| 99.00th=[ 159], 99.50th=[ 161], 99.90th=[ 163], 99.95th=[ 167],
| 99.99th=[ 268]
bw ( MiB/s): min= 2742, max= 4828, per=100.00%, avg=3349.49, stdev= 4.24, samples=34837
iops : min=21935, max=38625, avg=26795.24, stdev=33.89, samples=34837
lat (usec) : 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.03%, 50=0.05%
lat (msec) : 100=0.04%, 250=99.87%, 500=0.02%
cpu : usr=1.42%, sys=0.41%, ctx=0, majf=0, minf=0
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.2%, >=64=99.7%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.1%, 32=0.1%, 64=0.1%, >=64=0.0%
issued rwts: total=0,7302503,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64
Run status group 0 (all jobs):
WRITE: bw=3347MiB/s (3510MB/s), 3347MiB/s-3347MiB/s (3510MB/s-3510MB/s), io=891GiB (957GB), run=272726-272726msec