Are these performance numbers right?

[Caffe Mocha]

Hello,
Could someone please check for me if my performance is normal. the specs are as below. the first & second quote are fio test within freeNAS. third and fourth are fio result shared Ubuntu server. I understand NFS within ESXI without slog write performance is going to be bad. still the read and write performance within freenas itself is much lower than expected. read performance in NFS doesn't look so great either.
I had better performance when the set up is 2 vdev in raidz 1 or 2 vdeve in raidz2 among 8 ssds.
if anyone need me to run more tests please let me know. thanks


Motherboard X10DRH-CT with intel E5-2650Lv4 with 8 vcpu dedicated to FreeNAS
RAM: 64G dedicated ram
HBA: IBM M1215 flashed to IT mode x 2
SSD Micron S630DC 960G x 16 2vdev raidz2

within FreeNAS
fio --filename=./test --direct=1 --rw=randrw --refill_buffers --norandommap --randrepeat=0 --bs=4k --rwmixread=100 --iodepth=16 --numjobs=16 --runtime=60 --group_reporting --name=4ktest --size=4G


4ktest: Laying out IO file (1 file / 4096MiB)
Jobs: 16 (f=16): [r(16)][100.0%][r=507MiB/s,w=0KiB/s][r=130k,w=0 IOPS][eta 00m:00s]
4ktest: (groupid=0, jobs=16): err= 0: pid=21211: Tue Jan 30 13:46:10 2018
read: IOPS=129k, BW=502MiB/s (527MB/s)(29.4GiB/60002msec)
clat (usec): min=10, max=205230, avg=123.06, stdev=1173.78
lat (usec): min=10, max=205230, avg=123.19, stdev=1173.91
clat percentiles (usec):
| 1.00th=[ 13], 5.00th=[ 55], 10.00th=[ 56], 20.00th=[ 56],
| 30.00th=[ 57], 40.00th=[ 57], 50.00th=[ 57], 60.00th=[ 58],
| 70.00th=[ 58], 80.00th=[ 59], 90.00th=[ 63], 95.00th=[ 74],
| 99.00th=[ 318], 99.50th=[ 3392], 99.90th=[15270], 99.95th=[23200],
| 99.99th=[50070]
bw ( KiB/s): min=12289, max=52108, per=6.08%, avg=31258.36, stdev=5300.34, samples=1904
iops : min= 3072, max=13027, avg=7814.22, stdev=1325.08, samples=1904
lat (usec) : 20=2.30%, 50=0.17%, 100=95.73%, 250=0.74%, 500=0.28%
lat (usec) : 750=0.05%, 1000=0.01%
lat (msec) : 2=0.10%, 4=0.16%, 10=0.27%, 20=0.13%, 50=0.05%
lat (msec) : 100=0.01%, 250=0.01%
cpu : usr=1.58%, sys=45.65%, ctx=106269, majf=4, minf=28
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwt: total=7716685,0,0, short=0,0,0, dropped=0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
READ: bw=502MiB/s (527MB/s), 502MiB/s-502MiB/s (527MB/s-527MB/s), io=29.4GiB (31.6GB), run=60002-60002msec

Within FreeNAS
fio --filename=./rawrite --direct=1 --rw=randrw --refill_buffers --norandommap --randrepeat=0 --bs=8k --rwmixread=70 --iodepth=16 --numjobs=16 --runtime=60 --group_reporting --name=8k7030test --size=4G
fio-3.0
8k7030test: Laying out IO file (1 file / 4096MiB)
Jobs: 16 (f=16): [m(16)][100.0%][r=154MiB/s,w=67.1MiB/s][r=19.8k,w=8586 IOPS][eta 00m:00s]
8k7030test: (groupid=0, jobs=16): err= 0: pid=21604: Tue Jan 30 13:50:03 2018
read: IOPS=28.4k, BW=222MiB/s (232MB/s)(12.0GiB/60027msec)
clat (usec): min=10, max=290249, avg=149.14, stdev=1549.48
lat (usec): min=10, max=290249, avg=149.52, stdev=1550.27
clat percentiles (usec):
| 1.00th=[ 12], 5.00th=[ 12], 10.00th=[ 13], 20.00th=[ 13],
| 30.00th=[ 14], 40.00th=[ 14], 50.00th=[ 15], 60.00th=[ 51],
| 70.00th=[ 53], 80.00th=[ 56], 90.00th=[ 83], 95.00th=[ 178],
| 99.00th=[ 2606], 99.50th=[ 5669], 99.90th=[16712], 99.95th=[22676],
| 99.99th=[50070]
bw ( KiB/s): min= 3171, max=56298, per=6.09%, avg=13807.54, stdev=9258.56, samples=1905
iops : min= 396, max= 7037, avg=1725.52, stdev=1157.33, samples=1905
write: IOPS=12.2k, BW=95.1MiB/s (99.7MB/s)(5707MiB/60027msec)
clat (usec): min=16, max=309529, avg=943.43, stdev=3701.98
lat (usec): min=16, max=309529, avg=943.89, stdev=3702.59
clat percentiles (usec):
| 1.00th=[ 20], 5.00th=[ 63], 10.00th=[ 93], 20.00th=[ 149],
| 30.00th=[ 202], 40.00th=[ 251], 50.00th=[ 306], 60.00th=[ 371],
| 70.00th=[ 469], 80.00th=[ 660], 90.00th=[ 1516], 95.00th=[ 3687],
| 99.00th=[ 12387], 99.50th=[ 17433], 99.90th=[ 39584], 99.95th=[ 61604],
| 99.99th=[141558]
bw ( KiB/s): min= 1500, max=23090, per=6.09%, avg=5925.22, stdev=3951.97, samples=1905
iops : min= 187, max= 2886, avg=740.22, stdev=494.01, samples=1905
lat (usec) : 20=37.70%, 50=4.62%, 100=25.60%, 250=10.99%, 500=10.74%
lat (usec) : 750=3.46%, 1000=1.41%
lat (msec) : 2=2.18%, 4=1.42%, 10=1.27%, 20=0.45%, 50=0.13%
lat (msec) : 100=0.02%, 250=0.01%, 500=0.01%
cpu : usr=0.91%, sys=14.31%, ctx=1437406, majf=0, minf=0
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwt: total=1702059,730548,0, short=0,0,0, dropped=0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
READ: bw=222MiB/s (232MB/s), 222MiB/s-222MiB/s (232MB/s-232MB/s), io=12.0GiB (13.9GB), run=60027-60027msec
WRITE: bw=95.1MiB/s (99.7MB/s), 95.1MiB/s-95.1MiB/s (99.7MB/s-99.7MB/s), io=5707MiB (5985MB), run=60027-60027msec

VM NFS in Ubuntu
fio --filename=./test --direct=1 --rw=randrw --refill_buffers --norandommap --randrepeat=0 --bs=4k --rwmixread=100 --iodepth=16 --numjobs=16 --runtime=60 --group_reporting --name=4ktest --size=4G

4ktest: (groupid=0, jobs=16): err= 0: pid=2582: Wed Jan 31 12:55:13 2018
read : io=5766.2MB, bw=98405KB/s, iops=24601, runt= 60001msec
clat (usec): min=175, max=7686, avg=646.98, stdev=207.81
lat (usec): min=176, max=7686, avg=647.24, stdev=207.85
clat percentiles (usec):
| 1.00th=[ 282], 5.00th=[ 342], 10.00th=[ 402], 20.00th=[ 482],
| 30.00th=[ 540], 40.00th=[ 588], 50.00th=[ 636], 60.00th=[ 692],
| 70.00th=[ 740], 80.00th=[ 788], 90.00th=[ 868], 95.00th=[ 956],
| 99.00th=[ 1208], 99.50th=[ 1336], 99.90th=[ 2480], 99.95th=[ 2992],
| 99.99th=[ 3696]
bw (KB /s): min= 4416, max= 7152, per=6.25%, avg=6154.56, stdev=564.33
lat (usec) : 250=0.14%, 500=22.51%, 750=49.42%, 1000=24.32%
lat (msec) : 2=3.48%, 4=0.12%, 10=0.01%
cpu : usr=0.76%, sys=6.68%, ctx=1483321, majf=0, minf=146
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued : total=r=1476099/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
latency : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
READ: io=5766.2MB, aggrb=98404KB/s, minb=98404KB/s, maxb=98404KB/s, mint=6000 1msec, maxt=60001msec

Disk stats (read/write):
dm-0: ios=1475864/18, merge=0/0, ticks=873920/64, in_queue=875992, util=99.9 6%, aggrios=1476099/16, aggrmerge=0/2, aggrticks=884328/56, aggrin_queue=884256, aggrutil=99.91%
sda: ios=1476099/16, merge=0/2, ticks=884328/56, in_queue=884256, util=99.91%

VM NFS in Ubuntu
fio --filename=./rawrite --direct=1 --rw=randrw --refill_buffers --norandommap --randrepeat=0 --bs=8k --rwmixread=70 --iodepth=16 --numjobs=16 --runtime=60 --group_reporting --name=8k7030test --size=4G


fio-2.2.10
Starting 16 processes
8k7030test: Laying out IO file(s) (1 file(s) / 4096MB)
Jobs: 16 (f=16): [m(16)] [100.0% done] [9054KB/4123KB/0KB /s] [1131/515/0 iops] [eta 00m:00s]
8k7030test: (groupid=0, jobs=16): err= 0: pid=2604: Wed Jan 31 12:57:23 2018
read : io=576432KB, bw=9604.7KB/s, iops=1200, runt= 60016msec
clat (usec): min=211, max=64421, avg=8819.74, stdev=7245.99
lat (usec): min=211, max=64421, avg=8820.01, stdev=7246.00
clat percentiles (usec):
| 1.00th=[ 278], 5.00th=[ 302], 10.00th=[ 326], 20.00th=[ 402],
| 30.00th=[ 572], 40.00th=[ 7584], 50.00th=[ 9408], 60.00th=[11072],
| 70.00th=[12736], 80.00th=[14528], 90.00th=[17536], 95.00th=[20096],
| 99.00th=[28800], 99.50th=[34048], 99.90th=[44288], 99.95th=[48384],
| 99.99th=[59648]
bw (KB /s): min= 444, max= 801, per=6.26%, avg=600.84, stdev=59.27
write: io=247808KB, bw=4129.4KB/s, iops=516, runt= 60016msec
clat (msec): min=1, max=63, avg=10.47, stdev= 7.33
lat (msec): min=1, max=63, avg=10.47, stdev= 7.33
clat percentiles (usec):
| 1.00th=[ 1512], 5.00th=[ 1592], 10.00th=[ 1656], 20.00th=[ 1752],
| 30.00th=[ 4832], 40.00th=[ 9024], 50.00th=[10944], 60.00th=[12480],
| 70.00th=[14144], 80.00th=[16064], 90.00th=[19072], 95.00th=[21888],
| 99.00th=[32128], 99.50th=[37120], 99.90th=[47360], 99.95th=[50432],
| 99.99th=[58624]
bw (KB /s): min= 31, max= 534, per=6.25%, avg=257.95, stdev=73.08
lat (usec) : 250=0.07%, 500=19.47%, 750=2.32%, 1000=0.13%
lat (msec) : 2=8.46%, 4=0.89%, 10=20.08%, 20=42.65%, 50=5.87%
lat (msec) : 100=0.04%
cpu : usr=0.11%, sys=0.88%, ctx=198853, majf=0, minf=168
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued : total=r=72054/w=30976/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
latency : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
READ: io=576432KB, aggrb=9604KB/s, minb=9604KB/s, maxb=9604KB/s, mint=60016msec, maxt=60016msec
WRITE: io=247808KB, aggrb=4129KB/s, minb=4129KB/s, maxb=4129KB/s, mint=60016msec, maxt=60016msec

Disk stats (read/write):
dm-0: ios=71927/30985, merge=0/0, ticks=28112/54736, in_queue=82888, util=96.70%, aggrios=72054/31008, aggrmerge=0/43, aggrticks=28396/54700, aggrin_queue=83052, aggrutil=96.57%
sda: ios=72054/31008, merge=0/43, ticks=28396/54700, in_queue=83052, util=96.57%

 

[SMnasMAN]

i was hoping to see a reply / some info as i read through your post. personally I can only speak to your FIO results in FN directly-

I think there is something wrong/off about FIO directly run on FN (even though it is an included app). pretty often im seeing FIO report results way higher than what im seeing from zpool iostat and gstat. I know for sure if you use the --numjobs=2 my FIO results litterally will double (while the iostat/gstat still show half of that).

also your NFS datastore results dont look that "off" to me, ive never run FIO through nfs to FN, but other benchmarks of NFS datastores with similar hardware, do look low, yes even with all flash drives. Until you add a nice SLOG (ie optane or p3700) - (even then though FN is not as fast as you maybe expecting it to be).

Also keep in mind you almost have to throw out any/all READ benchmarks, as they will often be coming from arc/cache. sure you can set the test file size to 10gb above your arc or ram size, but even then a majority of the test is STILL coming from arc. I have yet to find a method to bypass ARC on any type of R benchmark (i have 128gb ram).
(only option i have found is to use both:
hw.physmem 4294967296 (locks you to 4gb total ram)
and
sysctl vfs.zfs.arc_max=514128320 (force 512mb arc).

but even using those isnt really fair, as FN/ZFS does need alot of memory even for write / non cache ops.

bottom line, for non-sync write and non-cache reads- FN / ZFS is not fast, its frankly SLOW , even with all flash! (esp as compared to HW raid), but ppl are not here for speed, they are here for data integrity.

but for sync writes, everything is SLOW. (unless you have a proper SLOG, and then you can expect ok sync-w speeds)