NV1616 potential issues

[Rand]

Link over from https://www.ixsystems.com/community...g-and-finding-the-best-slog.63521/post-550323 to keep that one clean of technical discussions:

Issue - NV1616 - potentially slow small blocksize performance:

Code:

root@freenas[~]#  diskinfo -citwS /dev/nvd1 > nv1616.log
root@freenas[~]# cat nv1616.log
/dev/nvd1
        512             # sectorsize
        16493051904     # mediasize in bytes (15G)
        32212992        # mediasize in sectors
        0               # stripesize
        0               # stripeoffset
        MTR_MLC_TS_16GB # Disk descr.
        0400001C3A46    # Disk ident.
        No              # TRIM/UNMAP support
        0               # Rotation rate in RPM

I/O command overhead:
        time to read 10MB block      0.009230 sec       =    0.000 msec/sector
        time to read 20480 sectors   0.495170 sec       =    0.024 msec/sector
        calculated command overhead                     =    0.024 msec/sector

Seek times:
        Full stroke:      250 iter in   0.013656 sec =    0.055 msec
        Half stroke:      250 iter in   0.014394 sec =    0.058 msec
        Quarter stroke:   500 iter in   0.022849 sec =    0.046 msec
        Short forward:    400 iter in   0.020386 sec =    0.051 msec
        Short backward:   400 iter in   0.018663 sec =    0.047 msec
        Seq outer:       2048 iter in   0.077314 sec =    0.038 msec
        Seq inner:       2048 iter in   0.089136 sec =    0.044 msec

Transfer rates:
        outside:       102400 kbytes in   0.102964 sec =   994522 kbytes/sec
        middle:        102400 kbytes in   0.097748 sec =  1047592 kbytes/sec
        inside:        102400 kbytes in   0.095201 sec =  1075619 kbytes/sec

Asynchronous random reads:
        sectorsize:    811230 ops in    3.000059 sec =   270405 IOPS
        4 kbytes:      800086 ops in    3.000065 sec =   266690 IOPS
        32 kbytes:     462154 ops in    3.000798 sec =   154010 IOPS
        128 kbytes:    117525 ops in    3.002186 sec =    39146 IOPS

Synchronous random writes:
         0.5 kbytes:    150.3 usec/IO =      3.2 Mbytes/s
           1 kbytes:    201.8 usec/IO =      4.8 Mbytes/s
           2 kbytes:    151.0 usec/IO =     12.9 Mbytes/s
           4 kbytes:    152.2 usec/IO =     25.7 Mbytes/s
           8 kbytes:    152.0 usec/IO =     51.4 Mbytes/s
          16 kbytes:    159.6 usec/IO =     97.9 Mbytes/s
          32 kbytes:    169.1 usec/IO =    184.8 Mbytes/s
          64 kbytes:    190.5 usec/IO =    328.0 Mbytes/s
         128 kbytes:    229.0 usec/IO =    545.9 Mbytes/s
         256 kbytes:    240.6 usec/IO =   1039.3 Mbytes/s
         512 kbytes:    294.8 usec/IO =   1696.3 Mbytes/s
        1024 kbytes:    368.9 usec/IO =   2711.1 Mbytes/s
        2048 kbytes:    553.1 usec/IO =   3615.8 Mbytes/s
        4096 kbytes:    969.4 usec/IO =   4126.2 Mbytes/s
        8192 kbytes:   1790.0 usec/IO =   4469.2 Mbytes/s

This runs on FreeNAS-11.2-U6, the NV1616 has a capacitator attached. It resides in a x16 slot on an Intel S2600STB (dual board with a single CPU running atm)

sysctl -a | grep -E "machdep.tsc_freq|kern.smp.cpus|hw.model"
kern.smp.cpus: 36
hw.model: Intel(R) Xeon(R) Gold 6150 CPU @ 2.70GHz
machdep.tsc_freq: 2693743674


The card was plugged in straight from the parcel, it was brand new. Will need to see if there is a newer FW.

 

[HoneyBadger]

Check your power saving settings, if it's sleeping the device or link then that can crank up latency massively.

 

[Rand]

Slightly better with everything power saving turned off

Code:

Synchronous random writes:
         0.5 kbytes:     78.4 usec/IO =      6.2 Mbytes/s
           1 kbytes:     78.5 usec/IO =     12.4 Mbytes/s
           2 kbytes:     78.8 usec/IO =     24.8 Mbytes/s
           4 kbytes:     79.5 usec/IO =     49.1 Mbytes/s
           8 kbytes:     80.5 usec/IO =     97.1 Mbytes/s
          16 kbytes:     86.5 usec/IO =    180.7 Mbytes/s
          32 kbytes:     92.9 usec/IO =    336.2 Mbytes/s
          64 kbytes:    106.7 usec/IO =    585.6 Mbytes/s
         128 kbytes:    136.4 usec/IO =    916.7 Mbytes/s
         256 kbytes:    151.0 usec/IO =   1655.7 Mbytes/s
         512 kbytes:    197.0 usec/IO =   2537.5 Mbytes/s
        1024 kbytes:    291.9 usec/IO =   3425.4 Mbytes/s
        2048 kbytes:    496.0 usec/IO =   4032.6 Mbytes/s
        4096 kbytes:    922.4 usec/IO =   4336.3 Mbytes/s
        8192 kbytes:   1748.8 usec/IO =   4574.6 Mbytes/s

I saw on the other post that you had significantly better results on linux, but the discussion wandered off at some point.
Have not gotten access to Microsemi's page yet, but from your comments it does not look like there is firmware available?

 

[HoneyBadger]

Yes, the NVMe driver under CentOS seemed to deliver better results, but I'd want to run exact A:B tests to be sure.

Regarding the Microsemi site, no, I haven't been able to get access to the restricted downloads. Thinking I will be reselling my card to someone who already has the access and needs a spare unit (and supercap pack)

 

[Rand]

Still trying to get access to documentation and firmware - not looking too bad, just slow progress:)

 

[Rand]

So got access to the documentation but there is no firmware that I can see.
But Microsemi is not really willing to help since the drives have been EOL'ed (not even commenting on whether there is an update or not...)