root@truenas[~]# fio --filename=./rawrite --direct=1 --rw=randrw --refill_buffers --norandommap --randrepeat=0 --bs=8k --rwmixread=70 --iodepth=16 --numjobs=16 --runtime=60 --group_reporting --name=8k7030test --size=4G
8k7030test: (g=0): rw=randrw, bs=(R) 8192B-8192B, (W) 8192B-8192B, (T) 8192B-8192B, ioengine=psync, iodepth=16
...
fio-3.28
Starting 16 processes
8k7030test: Laying out IO file (1 file / 4096MiB)
Jobs: 16 (f=16): [m(16)][100.0%][r=278MiB/s,w=118MiB/s][r=35.6k,w=15.1k IOPS][eta 00m:00s]
8k7030test: (groupid=0, jobs=16): err= 0: pid=9329: Mon Dec 25 14:54:25 2023
read: IOPS=43.7k, BW=342MiB/s (358MB/s)(20.0GiB/60002msec)
clat (usec): min=2, max=2651, avg=14.50, stdev=20.75
lat (usec): min=2, max=2651, avg=14.61, stdev=20.76
clat percentiles (usec):
| 1.00th=[ 4], 5.00th=[ 4], 10.00th=[ 5], 20.00th=[ 5],
| 30.00th=[ 5], 40.00th=[ 6], 50.00th=[ 7], 60.00th=[ 9],
| 70.00th=[ 10], 80.00th=[ 21], 90.00th=[ 44], 95.00th=[ 50],
| 99.00th=[ 73], 99.50th=[ 82], 99.90th=[ 119], 99.95th=[ 184],
| 99.99th=[ 668]
bw ( KiB/s): min=210904, max=1301635, per=100.00%, avg=350329.69, stdev=7627.49, samples=1888
iops : min=26357, max=162696, avg=43785.01, stdev=953.41, samples=1888
write: IOPS=18.8k, BW=147MiB/s (154MB/s)(8794MiB/60002msec); 0 zone resets
clat (usec): min=9, max=2927, avg=813.49, stdev=296.16
lat (usec): min=9, max=2927, avg=813.62, stdev=296.18
clat percentiles (usec):
| 1.00th=[ 21], 5.00th=[ 153], 10.00th=[ 478], 20.00th=[ 611],
| 30.00th=[ 668], 40.00th=[ 734], 50.00th=[ 807], 60.00th=[ 898],
| 70.00th=[ 1004], 80.00th=[ 1106], 90.00th=[ 1188], 95.00th=[ 1237],
| 99.00th=[ 1303], 99.50th=[ 1319], 99.90th=[ 1352], 99.95th=[ 1352],
| 99.99th=[ 1385]
bw ( KiB/s): min=99644, max=558695, per=100.00%, avg=150265.31, stdev=3251.28, samples=1888
iops : min=12447, max=69831, avg=18777.53, stdev=406.38, samples=1888
lat (usec) : 4=5.32%, 10=46.45%, 20=4.31%, 50=11.17%, 100=4.02%
lat (usec) : 250=0.21%, 500=1.79%, 750=9.55%, 1000=8.11%
lat (msec) : 2=9.07%, 4=0.01%
cpu : usr=0.92%, sys=7.75%, ctx=1073548, majf=0, minf=0
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=2624309,1125639,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16
Run status group 0 (all jobs):
READ: bw=342MiB/s (358MB/s), 342MiB/s-342MiB/s (358MB/s-358MB/s), io=20.0GiB (21.5GB), run=60002-60002msec
WRITE: bw=147MiB/s (154MB/s), 147MiB/s-147MiB/s (154MB/s-154MB/s), io=8794MiB (9221MB), run=60002-60002msec