Resource icon

SLOG benchmarking and finding the best SLOG

HoneyBadger

actually does care
Administrator
Moderator
iXsystems
Joined
Feb 6, 2014
Messages
5,110
You don't need to forget that 0815 NAND with PLP might be enough for 99% of the home systems postet here.
Spot on. While overkill is fun to benchmark, it's important to build appropriate solutions.

We'd all love to be able to set up all-flash systems with NVDIMM SLOG devices and Optane special vdevs, connected via 40GbE and watch how fast they go - but many organizations and users don't need (or don't want to pay for) that kind of speed, and would be just as well served by a simpler and much less expensive setup.

Part of which is why I'm rather annoyed by the rumblings that Intel is discontinuing consumer-grade Optane devices, outside of the H20 hybrid/QLC model.


The little M10 sticks and the 900p/905p were great for the home-lab and "prosumer" workspace.
 

Herr_Merlin

Patron
Joined
Oct 25, 2019
Messages
200
Hahah we only have 10G network here. Only the switch uplinks are 40G. Most people tend to forget that small I/O does not equal large network traffic.. looking at all those database clusters it's all small I/O
 

Ericloewe

Server Wrangler
Moderator
Joined
Feb 15, 2014
Messages
20,175
Not sure if this is a factor of controller overhead or firmware design
Probably both. NAND SSDs have had monstrous amounts of engineering thrown at them to make them fast, which I don't imagine DRAM SSDs got.
 

Herr_Merlin

Patron
Joined
Oct 25, 2019
Messages
200
Question did anyone tried the Optane DIMMs yet?
Another question - any idea how to pass them to a VM?
I might have a stupid idea for my privat ESXi and TrueNAS all in one box...
 

Rand

Guru
Joined
Dec 30, 2013
Messages
906
Just got a link to this document the other day: https://lenovopress.com/lp1085-analyzing-the-performance-of-dcpmm-storage-appdirect-mode

They are mixed baggage, fast reads, slower writes. NVDimm-N is significantly faster for writes (as expected).

Edit - some numbers for optane Dimms from https://jira.ixsystems.com/browse/NAS-108510
Code:
/dev/pmem0
512 # sectorsize
270582935552 # mediasize in bytes (252G)
528482296 # mediasize in sectors
0 # stripesize
0 # stripeoffset
PMEM region 252GB # Disk descr.
E60A9A2579EB399E # Disk ident.
No # TRIM/UNMAP support
0 # Rotation rate in RPM

I/O command overhead:
time to read 10MB block 0.002218 sec = 0.000 msec/sector
time to read 20480 sectors 0.016406 sec = 0.001 msec/sector
calculated command overhead = 0.001 msec/sector

Seek times:
Full stroke: 250 iter in 0.000500 sec = 0.002 msec
Half stroke: 250 iter in 0.000486 sec = 0.002 msec
Quarter stroke: 500 iter in 0.000997 sec = 0.002 msec
Short forward: 400 iter in 0.000766 sec = 0.002 msec
Short backward: 400 iter in 0.000822 sec = 0.002 msec
Seq outer: 2048 iter in 0.001842 sec = 0.001 msec
Seq inner: 2048 iter in 0.001862 sec = 0.001 msec

Transfer rates:
outside: 102400 kbytes in 0.013630 sec = 7512839 kbytes/sec
middle: 102400 kbytes in 0.013790 sec = 7425671 kbytes/sec
inside: 102400 kbytes in 0.013994 sec = 7317422 kbytes/sec

Asynchronous random reads:
sectorsize: 1425829 ops in 3.000028 sec = 475272 IOPS
4 kbytes: 864612 ops in 3.000029 sec = 288201 IOPS
32 kbytes: 945080 ops in 3.000400 sec = 314985 IOPS
128 kbytes: 164956 ops in 3.002401 sec = 54941 IOPS

Synchronous random writes:
0.5 kbytes: 2.0 usec/IO = 241.0 Mbytes/s
1 kbytes: 2.1 usec/IO = 466.8 Mbytes/s
2 kbytes: 2.3 usec/IO = 848.2 Mbytes/s
4 kbytes: 2.6 usec/IO = 1510.5 Mbytes/s
8 kbytes: 5.1 usec/IO = 1544.3 Mbytes/s
16 kbytes: 10.8 usec/IO = 1452.0 Mbytes/s
32 kbytes: 20.8 usec/IO = 1503.5 Mbytes/s
64 kbytes: 37.7 usec/IO = 1658.9 Mbytes/s
128 kbytes: 79.2 usec/IO = 1577.3 Mbytes/s
256 kbytes: 157.2 usec/IO = 1590.8 Mbytes/s
512 kbytes: 310.1 usec/IO = 1612.5 Mbytes/s
1024 kbytes: 665.0 usec/IO = 1503.7 Mbytes/s
2048 kbytes: 1364.2 usec/IO = 1466.0 Mbytes/s
4096 kbytes: 2800.3 usec/IO = 1428.4 Mbytes/s
8192 kbytes: 5639.6 usec/IO = 1418.5 Mbytes/s
 
Last edited:

Rand

Guru
Joined
Dec 30, 2013
Messages
906
SS530 Read intensive

Code:
smartctl 7.1 2019-12-30 r5022 [FreeBSD 12.2-RELEASE-p3 amd64] (local build)
Copyright (C) 2002-19, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Vendor:               WDC
Product:              WUSTR6416ASS204
Revision:             C925
Compliance:           SPC-5
User Capacity:        1,600,321,314,816 bytes [1.60 TB]
Logical block size:   512 bytes
Physical block size:  4096 bytes
LU is resource provisioned, LBPRZ=1
Rotation Rate:        Solid State Device
Form Factor:          2.5 inches
Logical Unit id:      0x5000cca0ab7483dc
Serial number:        
Device type:          disk
Transport protocol:   SAS (SPL-3)
Local Time is:        Sat Feb 13 16:51:23 2021 CET
SMART support is:     Available - device has SMART capability.
SMART support is:     Enabled
Temperature Warning:  Enabled

 512             # sectorsize
        1600321314816   # mediasize in bytes (1.5T)
        3125627568      # mediasize in sectors
        4096            # stripesize
        0               # stripeoffset
        194561          # Cylinders according to firmware.
        255             # Heads according to firmware.
        63              # Sectors according to firmware.
        WDC WUSTR6416ASS204     # Disk descr.
        WXX22G3A        # Disk ident.
        Yes             # TRIM/UNMAP support
        0               # Rotation rate in RPM
        Not_Zoned       # Zone Mode

Synchronous random writes:
         0.5 kbytes:    127.5 usec/IO =      3.8 Mbytes/s
           1 kbytes:    128.1 usec/IO =      7.6 Mbytes/s
           2 kbytes:    129.8 usec/IO =     15.0 Mbytes/s
           4 kbytes:    133.0 usec/IO =     29.4 Mbytes/s
           8 kbytes:    141.5 usec/IO =     55.2 Mbytes/s
          16 kbytes:    159.9 usec/IO =     97.7 Mbytes/s
          32 kbytes:    188.1 usec/IO =    166.1 Mbytes/s
          64 kbytes:    253.6 usec/IO =    246.5 Mbytes/s
         128 kbytes:    375.7 usec/IO =    332.7 Mbytes/s
         256 kbytes:    612.4 usec/IO =    408.3 Mbytes/s
         512 kbytes:   1083.0 usec/IO =    461.7 Mbytes/s
        1024 kbytes:   2040.3 usec/IO =    490.1 Mbytes/s
        2048 kbytes:   3942.1 usec/IO =    507.3 Mbytes/s
        4096 kbytes:   7720.6 usec/IO =    518.1 Mbytes/s
        8192 kbytes:  15359.9 usec/IO =    520.8 Mbytes/s
 

qbix

Cadet
Joined
Dec 6, 2020
Messages
4
INTEL D3-S4610 960GB TLC

Code:
root@nas[~]# smartctl -a /dev/da0
smartctl 7.1 2019-12-30 r5022 [FreeBSD 12.2-RELEASE-p3 amd64] (local build)
Copyright (C) 2002-19, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:       Intel S4510/S4610/S4500/S4600 Series SSDs
Device Model:       INTEL SSDSC2KG960G801
Serial Number:      BTYG93130G0W960CGN
LU WWN Device Id:   5 5cd2e4 1514d10e3
Firmware Version:   XCV10110
User Capacity:      960,197,124,096 bytes [960 GB]
Sector Sizes:       512 bytes logical, 4096 bytes physical
Rotation Rate:      Solid State Device
Form Factor:        2.5 inches
Device is:          In smartctl database [for details use: -P show]
ATA Version is:     ACS-3 T13/2161-D revision 5
SATA Version is:    SATA 3.2, 6.0 Gb/s (current: 6.0 Gb/s)



Code:
root@nas[~]# diskinfo -wS /dev/da0
/dev/da0
        512             # sectorsize
        960197124096    # mediasize in bytes (894G)
        1875385008      # mediasize in sectors
        4096            # stripesize
        0               # stripeoffset
        116737          # Cylinders according to firmware.
        255             # Heads according to firmware.
        63              # Sectors according to firmware.
        ATA INTEL SSDSC2KG96    # Disk descr.
        BTYG93130G0W960CGN      # Disk ident.
        Yes             # TRIM/UNMAP support
        0               # Rotation rate in RPM
        Not_Zoned       # Zone Mode

Synchronous random writes:
         0.5 kbytes:    115.2 usec/IO =      4.2 Mbytes/s
           1 kbytes:    115.8 usec/IO =      8.4 Mbytes/s
           2 kbytes:    116.9 usec/IO =     16.7 Mbytes/s
           4 kbytes:    116.9 usec/IO =     33.4 Mbytes/s
           8 kbytes:    125.2 usec/IO =     62.4 Mbytes/s
          16 kbytes:    142.5 usec/IO =    109.6 Mbytes/s
          32 kbytes:    178.1 usec/IO =    175.5 Mbytes/s
          64 kbytes:    248.1 usec/IO =    251.9 Mbytes/s
         128 kbytes:    400.5 usec/IO =    312.1 Mbytes/s
         256 kbytes:    664.7 usec/IO =    376.1 Mbytes/s
         512 kbytes:   1193.4 usec/IO =    419.0 Mbytes/s
        1024 kbytes:   2233.4 usec/IO =    447.7 Mbytes/s
        2048 kbytes:   4344.4 usec/IO =    460.4 Mbytes/s
        4096 kbytes:   8619.1 usec/IO =    464.1 Mbytes/s
        8192 kbytes:  16871.2 usec/IO =    474.2 Mbytes/s
 
Last edited:

douglasg

Cadet
Joined
Mar 24, 2021
Messages
7
I didn't see the 983 DCT here so here are my results:
Code:
root@truenas[~]# smartctl -a /dev/nvme2
smartctl 7.1 2019-12-30 r5022 [FreeBSD 12.2-RELEASE-p3 amd64] (local build)
Copyright (C) 2002-19, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Number:                       Samsung SSD 983 DCT M.2 960GB
Serial Number:                      S48ENC0N501364K
Firmware Version:                   EDA7602Q
PCI Vendor/Subsystem ID:            0x144d
IEEE OUI Identifier:                0x002538
Total NVM Capacity:                 960,197,124,096 [960 GB]
Unallocated NVM Capacity:           0
Controller ID:                      4
Number of Namespaces:               1
Namespace 1 Size/Capacity:          960,197,124,096 [960 GB]
Namespace 1 Utilization:            227,024,896 [227 MB]
Namespace 1 Formatted LBA Size:     512
Local Time is:                      Thu Mar 25 00:05:12 2021 EDT
Firmware Updates (0x17):            3 Slots, Slot 1 R/O, no Reset required
Optional Admin Commands (0x000f):   Security Format Frmw_DL NS_Mngmt
Optional NVM Commands (0x001f):     Comp Wr_Unc DS_Mngmt Wr_Zero Sav/Sel_Feat
Maximum Data Transfer Size:         512 Pages
Warning  Comp. Temp. Threshold:     86 Celsius
Critical Comp. Temp. Threshold:     87 Celsius
Namespace 1 Features (0x02):        NA_Fields


Code:
root@truenas[~]# diskinfo -wS /dev/nvd2
/dev/nvd2
        512             # sectorsize
        960197124096    # mediasize in bytes (894G)
        1875385008      # mediasize in sectors
        0               # stripesize
        0               # stripeoffset
        Samsung SSD 983 DCT M.2 960GB   # Disk descr.
        S48ENC0N501364K # Disk ident.
        Yes             # TRIM/UNMAP support
        0               # Rotation rate in RPM

Synchronous random writes:
         0.5 kbytes:     17.5 usec/IO =     28.0 Mbytes/s
           1 kbytes:     17.6 usec/IO =     55.6 Mbytes/s
           2 kbytes:     18.1 usec/IO =    107.9 Mbytes/s
           4 kbytes:     19.0 usec/IO =    205.3 Mbytes/s
           8 kbytes:     21.8 usec/IO =    358.5 Mbytes/s
          16 kbytes:     27.2 usec/IO =    575.0 Mbytes/s
          32 kbytes:     41.2 usec/IO =    759.2 Mbytes/s
          64 kbytes:     70.6 usec/IO =    885.8 Mbytes/s
         128 kbytes:    147.1 usec/IO =    849.5 Mbytes/s
         256 kbytes:    260.2 usec/IO =    960.8 Mbytes/s
         512 kbytes:    459.4 usec/IO =   1088.3 Mbytes/s
        1024 kbytes:    863.5 usec/IO =   1158.1 Mbytes/s
        2048 kbytes:   1667.3 usec/IO =   1199.6 Mbytes/s
        4096 kbytes:   3244.0 usec/IO =   1233.0 Mbytes/s
        8192 kbytes:   6503.6 usec/IO =   1230.1 Mbytes/s


I also had a 950 Pro in my system so here are those results:
Code:
root@truenas[~]# smartctl -a /dev/nvme0
smartctl 7.1 2019-12-30 r5022 [FreeBSD 12.2-RELEASE-p3 amd64] (local build)
Copyright (C) 2002-19, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Number:                       Samsung SSD 950 PRO 512GB
Serial Number:                      S2GMNX0H609029E
Firmware Version:                   1B0QBXX7
PCI Vendor/Subsystem ID:            0x144d
IEEE OUI Identifier:                0x002538
Controller ID:                      1
Number of Namespaces:               1
Namespace 1 Size/Capacity:          512,110,190,592 [512 GB]
Namespace 1 Utilization:            104,878,292,992 [104 GB]
Namespace 1 Formatted LBA Size:     512
Namespace 1 IEEE EUI-64:            002538 5661b053ec
Local Time is:                      Wed Mar 24 23:57:59 2021 EDT
Firmware Updates (0x06):            3 Slots
Optional Admin Commands (0x0007):   Security Format Frmw_DL
Optional NVM Commands (0x001f):     Comp Wr_Unc DS_Mngmt Wr_Zero Sav/Sel_Feat
Maximum Data Transfer Size:         32 Pages


Code:
root@truenas[~]# diskinfo -wS /dev/nvd0
/dev/nvd0
        512             # sectorsize
        512110190592    # mediasize in bytes (477G)
        1000215216      # mediasize in sectors
        0               # stripesize
        0               # stripeoffset
        Samsung SSD 950 PRO 512GB       # Disk descr.
        S2GMNX0H609029E # Disk ident.
        Yes             # TRIM/UNMAP support
        0               # Rotation rate in RPM

Synchronous random writes:
         0.5 kbytes:   1675.6 usec/IO =      0.3 Mbytes/s
           1 kbytes:   1692.3 usec/IO =      0.6 Mbytes/s
           2 kbytes:   1663.1 usec/IO =      1.2 Mbytes/s
           4 kbytes:   1740.6 usec/IO =      2.2 Mbytes/s
           8 kbytes:   1616.9 usec/IO =      4.8 Mbytes/s
          16 kbytes:   1842.3 usec/IO =      8.5 Mbytes/s
          32 kbytes:   1887.4 usec/IO =     16.6 Mbytes/s
          64 kbytes:   2178.0 usec/IO =     28.7 Mbytes/s
         128 kbytes:   2759.6 usec/IO =     45.3 Mbytes/s
         256 kbytes:   3612.3 usec/IO =     69.2 Mbytes/s
         512 kbytes:   3644.1 usec/IO =    137.2 Mbytes/s
        1024 kbytes:   3695.2 usec/IO =    270.6 Mbytes/s
        2048 kbytes:   3622.2 usec/IO =    552.1 Mbytes/s
        4096 kbytes:   5484.4 usec/IO =    729.3 Mbytes/s
        8192 kbytes:   8916.5 usec/IO =    897.2 Mbytes/s
 

Ralms

Dabbler
Joined
Jan 28, 2019
Messages
29
Sharing the results of a Samsung 960 EVO 250GB


Code:
root@freenas[~]# smartctl -a /dev/nvme0
smartctl 7.1 2019-12-30 r5022 [FreeBSD 12.2-RELEASE-p3 amd64] (local build)
Copyright (C) 2002-19, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Number:                       Samsung SSD 960 EVO 250GB
Serial Number:                      S3ESNX0K298763Z
Firmware Version:                   3B7QCXE7
PCI Vendor/Subsystem ID:            0x144d
IEEE OUI Identifier:                0x002538
Total NVM Capacity:                 250,059,350,016 [250 GB]
Unallocated NVM Capacity:           0
Controller ID:                      2
Number of Namespaces:               1
Namespace 1 Size/Capacity:          250,059,350,016 [250 GB]
Namespace 1 Utilization:            250,058,170,368 [250 GB]
Namespace 1 Formatted LBA Size:     512
Namespace 1 IEEE EUI-64:            002538 5281b2d7eb
Local Time is:                      Sun Apr  4 12:29:01 2021 WEST
Firmware Updates (0x16):            3 Slots, no Reset required
Optional Admin Commands (0x0007):   Security Format Frmw_DL
Optional NVM Commands (0x001f):     Comp Wr_Unc DS_Mngmt Wr_Zero Sav/Sel_Feat
Maximum Data Transfer Size:         512 Pages
Warning  Comp. Temp. Threshold:     77 Celsius
Critical Comp. Temp. Threshold:     79 Celsius


Code:
root@freenas[~]# diskinfo -wS /dev/nvd0
/dev/nvd0
        512             # sectorsize
        250059350016    # mediasize in bytes (233G)
        488397168       # mediasize in sectors
        0               # stripesize
        0               # stripeoffset
        Samsung SSD 960 EVO 250GB       # Disk descr.
        S3ESNX0K298763Z # Disk ident.
        Yes             # TRIM/UNMAP support
        0               # Rotation rate in RPM

Synchronous random writes:
         0.5 kbytes:    727.9 usec/IO =      0.7 Mbytes/s
           1 kbytes:    695.8 usec/IO =      1.4 Mbytes/s
           2 kbytes:    721.1 usec/IO =      2.7 Mbytes/s
           4 kbytes:    674.6 usec/IO =      5.8 Mbytes/s
           8 kbytes:    725.0 usec/IO =     10.8 Mbytes/s
          16 kbytes:    829.8 usec/IO =     18.8 Mbytes/s
          32 kbytes:    869.4 usec/IO =     35.9 Mbytes/s
          64 kbytes:    818.4 usec/IO =     76.4 Mbytes/s
         128 kbytes:    913.7 usec/IO =    136.8 Mbytes/s
         256 kbytes:   1108.7 usec/IO =    225.5 Mbytes/s
         512 kbytes:   1564.5 usec/IO =    319.6 Mbytes/s
        1024 kbytes:   2257.3 usec/IO =    443.0 Mbytes/s
        2048 kbytes:   3702.2 usec/IO =    540.2 Mbytes/s
        4096 kbytes:   6467.6 usec/IO =    618.5 Mbytes/s
        8192 kbytes:  11691.3 usec/IO =    684.3 Mbytes/s
 

ehsab

Dabbler
Joined
Aug 2, 2020
Messages
45
The RMS-300 seems to stand quite good against the Optane NVDIMMs.

Optane NVDIMM

64 kbytes: 37.7 usec/IO = 1658.9 Mbytes/s
128 kbytes: 79.2 usec/IO = 1577.3 Mbytes/s
256 kbytes: 157.2 usec/IO = 1590.8 Mbytes/s
512 kbytes: 310.1 usec/IO = 1612.5 Mbytes/s
1024 kbytes: 665.0 usec/IO = 1503.7 Mbytes/s


RMS-300

64 kbytes: 45.8 usec/IO = 1363.3 Mbytes/s
128 kbytes: 80.6 usec/IO = 1550.3 Mbytes/s
256 kbytes: 99.0 usec/IO = 2525.0 Mbytes/s
512 kbytes: 147.6 usec/IO = 3388.3 Mbytes/s
1024 kbytes: 256.2 usec/IO = 3902.8 Mbytes/s


Optane NVDIMM looks really fast on small writes, where the RMS-300 performs better with 256K or larger writes.
 

Ericloewe

Server Wrangler
Moderator
Joined
Feb 15, 2014
Messages
20,175
I assume those Optane numbers are from a block device configuration and not a memory mode configuration, correct?
 

Sirius

Dabbler
Joined
Mar 1, 2018
Messages
41
I'm currently using 2 x mirrored Optane 900p as SLOGs, what would be a good step up from that?

Maybe the 905p or the P4800X? I also wonder if a striped mirror of 900p would work while still retaining some degree of data integrity.

Or is something more exotic like a 16gb Radian card (assuming I can find one, ideally two, for mirroring) the way to go? NVDIMM is a bit too much for my budget. Would 16GB be enough for 100gbE? I assume it'd be enough for 56gbE.
 

mpyusko

Dabbler
Joined
Jul 5, 2019
Messages
49
Intel® Optane™ Memory M10 Series (64GB, M.2 80mm PCIe 3.0, 20nm, 3D XPoint™)

Code:
root@cygnus[~]# smartctl -a /dev/nvd0
smartctl 7.2 2020-12-30 r5155 [FreeBSD 12.2-RELEASE-p6 amd64] (local build)
Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org

/dev/nvd0: To monitor NVMe disks use /dev/nvme* device names
Please specify device type with the -d option.

Use smartctl -h to get a usage summary

root@cygnus[~]# smartctl -a /dev/nvme0
smartctl 7.2 2020-12-30 r5155 [FreeBSD 12.2-RELEASE-p6 amd64] (local build)
Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Number:                       INTEL MEMPEK1J064GA
Serial Number:                      PHBT9074037A064Q
Firmware Version:                   K4110440
PCI Vendor/Subsystem ID:            0x8086
IEEE OUI Identifier:                0x5cd2e4
Controller ID:                      0
NVMe Version:                       <1.2
Number of Namespaces:               1
Namespace 1 Size/Capacity:          58,977,157,120 [58.9 GB]
Namespace 1 Formatted LBA Size:     512
Namespace 1 IEEE EUI-64:            5cd2e4 10457d0100
Local Time is:                      Wed Jun  9 13:12:46 2021 EDT
Firmware Updates (0x02):            1 Slot
Optional Admin Commands (0x0006):   Format Frmw_DL
Optional NVM Commands (0x0046):     Wr_Unc DS_Mngmt Timestmp
Log Page Attributes (0x02):         Cmd_Eff_Lg
Maximum Data Transfer Size:         32 Pages

Supported Power States
St Op     Max   Active     Idle   RL RT WL WT  Ent_Lat  Ex_Lat
 0 +     3.60W       -        -    0  0  0  0  1000000   50000
 1 +     2.50W       -        -    0  1  0  1  1000000   50000
 2 +     1.80W       -        -    0  2  0  2  1000000   50000
 3 -   0.0080W       -        -    0  0  0  0  1150000   50000

Supported LBA Sizes (NSID 0x1)
Id Fmt  Data  Metadt  Rel_Perf
 0 +     512       0         2

=== START OF SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

SMART/Health Information (NVMe Log 0x02)
Critical Warning:                   0x00
Temperature:                        46 Celsius
Available Spare:                    100%
Available Spare Threshold:          0%
Percentage Used:                    0%
Data Units Read:                    17 [8.70 MB]
Data Units Written:                 1,591,528 [814 GB]
Host Read Commands:                 596
Host Write Commands:                28,538,941
Controller Busy Time:               0
Power Cycles:                       2
Power On Hours:                     135
Unsafe Shutdowns:                   0
Media and Data Integrity Errors:    0
Error Information Log Entries:      0

Error Information (NVMe Log 0x01, 16 of 64 entries)
No Errors Logged

root@cygnus[~]#


Code:
root@cygnus[~]# diskinfo -wS /dev/nvd0
/dev/nvd0
        512             # sectorsize
        58977157120     # mediasize in bytes (55G)
        115189760       # mediasize in sectors
        0               # stripesize
        0               # stripeoffset
        INTEL MEMPEK1J064GA     # Disk descr.
        PHBT9074037A064Q        # Disk ident.
        Yes             # TRIM/UNMAP support
        0               # Rotation rate in RPM

Synchronous random writes:
         0.5 kbytes:     10.1 usec/IO =     48.5 Mbytes/s
           1 kbytes:     10.6 usec/IO =     92.4 Mbytes/s
           2 kbytes:     12.1 usec/IO =    161.2 Mbytes/s
           4 kbytes:     15.1 usec/IO =    258.6 Mbytes/s
           8 kbytes:     21.4 usec/IO =    364.5 Mbytes/s
          16 kbytes:     34.7 usec/IO =    450.5 Mbytes/s
          32 kbytes:     60.0 usec/IO =    520.8 Mbytes/s
          64 kbytes:    111.4 usec/IO =    560.9 Mbytes/s
         128 kbytes:    213.9 usec/IO =    584.4 Mbytes/s
         256 kbytes:    418.6 usec/IO =    597.2 Mbytes/s
         512 kbytes:    819.4 usec/IO =    610.2 Mbytes/s
        1024 kbytes:   1633.8 usec/IO =    612.1 Mbytes/s
        2048 kbytes:   3249.8 usec/IO =    615.4 Mbytes/s
        4096 kbytes:   6511.7 usec/IO =    614.3 Mbytes/s
        8192 kbytes:  12996.7 usec/IO =    615.5 Mbytes/s
root@cygnus[~]#
 

mpyusko

Dabbler
Joined
Jul 5, 2019
Messages
49
Intel® Optane™ Memory M10 Series (32GB, M.2 80mm PCIe 3.0, 20nm, 3D XPoint™)
Code:
root@palomino[~]# smartctl -a /dev/nvme2
smartctl 7.2 2020-12-30 r5155 [FreeBSD 12.2-RELEASE-p6 amd64] (local build)
Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Number:                       INTEL MEMPEK1W032GA
Serial Number:                      PHBT8170016V032E
Firmware Version:                   K3110310
PCI Vendor/Subsystem ID:            0x8086
IEEE OUI Identifier:                0x5cd2e4
Controller ID:                      0
NVMe Version:                       <1.2
Number of Namespaces:               1
Namespace 1 Size/Capacity:          29,260,513,280 [29.2 GB]
Namespace 1 Formatted LBA Size:     512
Local Time is:                      Wed Jun  9 13:20:46 2021 EDT
Firmware Updates (0x02):            1 Slot
Optional Admin Commands (0x0006):   Format Frmw_DL
Optional NVM Commands (0x0046):     Wr_Unc DS_Mngmt Timestmp
Log Page Attributes (0x02):         Cmd_Eff_Lg
Maximum Data Transfer Size:         32 Pages

Supported Power States
St Op     Max   Active     Idle   RL RT WL WT  Ent_Lat  Ex_Lat
 0 +     4.50W       -        -    0  0  0  0        0       0

Supported LBA Sizes (NSID 0x1)
Id Fmt  Data  Metadt  Rel_Perf
 0 +     512       0         2

=== START OF SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

SMART/Health Information (NVMe Log 0x02)
Critical Warning:                   0x00
Temperature:                        49 Celsius
Available Spare:                    100%
Available Spare Threshold:          0%
Percentage Used:                    1%
Data Units Read:                    139 [71.1 MB]
Data Units Written:                 5,865,439 [3.00 TB]
Host Read Commands:                 3,180
Host Write Commands:                212,255,430
Controller Busy Time:               0
Power Cycles:                       23
Power On Hours:                     3,198
Unsafe Shutdowns:                   0
Media and Data Integrity Errors:    0
Error Information Log Entries:      0

Error Information (NVMe Log 0x01, 16 of 64 entries)
No Errors Logged

root@palomino[~]#

Code:
root@palomino[~]# diskinfo -wS /dev/nvd2
/dev/nvd2
        512             # sectorsize
        29260513280     # mediasize in bytes (27G)
        57149440        # mediasize in sectors
        0               # stripesize
        0               # stripeoffset
        INTEL MEMPEK1W032GA     # Disk descr.
        PHBT8170016V032E        # Disk ident.
        Yes             # TRIM/UNMAP support
        0               # Rotation rate in RPM

Synchronous random writes:
         0.5 kbytes:     15.9 usec/IO =     30.7 Mbytes/s
           1 kbytes:     16.4 usec/IO =     59.7 Mbytes/s
           2 kbytes:     19.4 usec/IO =    100.7 Mbytes/s
           4 kbytes:     24.5 usec/IO =    159.4 Mbytes/s
           8 kbytes:     39.7 usec/IO =    196.7 Mbytes/s
          16 kbytes:     70.1 usec/IO =    222.9 Mbytes/s
          32 kbytes:    162.8 usec/IO =    192.0 Mbytes/s
          64 kbytes:    351.0 usec/IO =    178.1 Mbytes/s
         128 kbytes:    581.8 usec/IO =    214.9 Mbytes/s
         256 kbytes:   1019.3 usec/IO =    245.3 Mbytes/s
         512 kbytes:   1912.9 usec/IO =    261.4 Mbytes/s
        1024 kbytes:   3640.1 usec/IO =    274.7 Mbytes/s
        2048 kbytes:   7134.7 usec/IO =    280.3 Mbytes/s
        4096 kbytes:  14144.0 usec/IO =    282.8 Mbytes/s
        8192 kbytes:  28235.3 usec/IO =    283.3 Mbytes/s
root@palomino[~]#
 

Kailee71

Contributor
Joined
Jul 8, 2018
Messages
110
Hey,

here's one of my two Optane M10 32G (but different model number to mpyusko? J vs W...)

Code:
truenas# smartctl -x /dev/nvme1
smartctl 7.2 2020-12-30 r5155 [FreeBSD 12.2-RELEASE-p6 amd64] (local build)
Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Number:                       INTEL MEMPEK1J032GA
Serial Number:                      PHBT809300ZU032P
Firmware Version:                   K4110400
PCI Vendor/Subsystem ID:            0x8086
IEEE OUI Identifier:                0x5cd2e4
Controller ID:                      0
NVMe Version:                       <1.2
Number of Namespaces:               1
Namespace 1 Size/Capacity:          29,260,513,280 [29.2 GB]
Namespace 1 Formatted LBA Size:     512
Namespace 1 IEEE EUI-64:            5cd2e4 12501e0100
Local Time is:                      Sun Jul 18 00:37:23 2021 CEST
Firmware Updates (0x02):            1 Slot
Optional Admin Commands (0x0006):   Format Frmw_DL
Optional NVM Commands (0x0046):     Wr_Unc DS_Mngmt Timestmp
Log Page Attributes (0x02):         Cmd_Eff_Lg
Maximum Data Transfer Size:         32 Pages

Supported Power States
St Op     Max   Active     Idle   RL RT WL WT  Ent_Lat  Ex_Lat
 0 +     2.80W       -        -    0  0  0  0  1000000   30000
 1 +     2.20W       -        -    0  1  0  1  1000000   30000
 2 +     1.80W       -        -    0  2  0  2  1000000   30000
 3 -   0.0080W       -        -    0  0  0  0  1150000   30000

Supported LBA Sizes (NSID 0x1)
Id Fmt  Data  Metadt  Rel_Perf
 0 +     512       0         2

=== START OF SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

SMART/Health Information (NVMe Log 0x02)
Critical Warning:                   0x00
Temperature:                        49 Celsius
Available Spare:                    100%
Available Spare Threshold:          0%
Percentage Used:                    0%
Data Units Read:                    19,247 [9.85 GB]
Data Units Written:                 517,312 [264 GB]
Host Read Commands:                 179,388
Host Write Commands:                4,083,315
Controller Busy Time:               0
Power Cycles:                       11
Power On Hours:                     67
Unsafe Shutdowns:                   0
Media and Data Integrity Errors:    0
Error Information Log Entries:      0

Error Information (NVMe Log 0x01, 16 of 64 entries)
No Errors Logged


Code:
truenas# diskinfo -wS /dev/nvd1
/dev/nvd1
        512             # sectorsize
        29260513280     # mediasize in bytes (27G)
        57149440        # mediasize in sectors
        0               # stripesize
        0               # stripeoffset
        INTEL MEMPEK1J032GA     # Disk descr.
        PHBT809300ZU032P        # Disk ident.
        Yes             # TRIM/UNMAP support
        0               # Rotation rate in RPM

Synchronous random writes:
         0.5 kbytes:     42.6 usec/IO =     11.5 Mbytes/s
           1 kbytes:     34.8 usec/IO =     28.1 Mbytes/s
           2 kbytes:     37.5 usec/IO =     52.0 Mbytes/s
           4 kbytes:     51.4 usec/IO =     76.0 Mbytes/s
           8 kbytes:     87.7 usec/IO =     89.0 Mbytes/s
          16 kbytes:    136.7 usec/IO =    114.3 Mbytes/s
          32 kbytes:    175.4 usec/IO =    178.1 Mbytes/s
          64 kbytes:    257.7 usec/IO =    242.6 Mbytes/s
         128 kbytes:    468.2 usec/IO =    267.0 Mbytes/s
         256 kbytes:    823.4 usec/IO =    303.6 Mbytes/s
         512 kbytes:   1581.9 usec/IO =    316.1 Mbytes/s
        1024 kbytes:   3089.7 usec/IO =    323.7 Mbytes/s
        2048 kbytes:   6032.5 usec/IO =    331.5 Mbytes/s
        4096 kbytes:  11991.3 usec/IO =    333.6 Mbytes/s
        8192 kbytes:  23832.1 usec/IO =    335.7 Mbytes/s

Note the low numbers <32KiB. Will investigate if BIOS on my DL380p g8 has performance or efficiency settings active (am doing burn-in currently so will have to wait a day or two). Could alternatively also be a PCIe passthrough issue as I'm running on ESXi.

Using them both in parallel as logs essentially doubles transfer bandwidth in SMB. Any way to test this with diskinfo? Make a temp zfs stripe?

Also - and an important point for me currently; as zfs starts throttling at 10% of Memsize with my 32GiB VM it's getting throttled at just over 3GiB, which basically dumps transfers down to spinning speeds (for me ~220MiB/s on a stripe of 2 mirrors of WD40EFRX). Will start playing with more mem on the VM and also ZFS tuning through sysctl.
 

Kailee71

Contributor
Joined
Jul 8, 2018
Messages
110
I just pinned the TrueNAS VM CPUs to the socket that serves /dev/nvd1;
Code:
truenas# diskinfo -wS /dev/nvd1
/dev/nvd1
        512             # sectorsize
        29260513280     # mediasize in bytes (27G)
        57149440        # mediasize in sectors
        0               # stripesize
        0               # stripeoffset
        INTEL MEMPEK1J032GA     # Disk descr.
        PHBT809300ZU032P        # Disk ident.
        Yes             # TRIM/UNMAP support
        0               # Rotation rate in RPM

Synchronous random writes:
         0.5 kbytes:     28.9 usec/IO =     16.9 Mbytes/s
           1 kbytes:     25.9 usec/IO =     37.7 Mbytes/s
           2 kbytes:     27.7 usec/IO =     70.5 Mbytes/s
           4 kbytes:     38.6 usec/IO =    101.3 Mbytes/s
           8 kbytes:     57.7 usec/IO =    135.3 Mbytes/s
          16 kbytes:    126.0 usec/IO =    124.0 Mbytes/s
          32 kbytes:    164.6 usec/IO =    189.9 Mbytes/s
          64 kbytes:    254.7 usec/IO =    245.4 Mbytes/s
         128 kbytes:    443.8 usec/IO =    281.7 Mbytes/s
         256 kbytes:    809.4 usec/IO =    308.9 Mbytes/s
         512 kbytes:   1538.2 usec/IO =    325.0 Mbytes/s
        1024 kbytes:   3026.1 usec/IO =    330.5 Mbytes/s
        2048 kbytes:   5986.5 usec/IO =    334.1 Mbytes/s
        4096 kbytes:  11881.6 usec/IO =    336.7 Mbytes/s
        8192 kbytes:  23689.6 usec/IO =    337.7 Mbytes/s


So bandwidth went up considerably for 0.5KiB < XFer < 16KiB. No luck yet on the latency, but will reboot with different PCIe BIOS setting and report back.

+++++ EDIT +++++
BIOS settings made no discernable difference
Making the *correct* BIOS change did make a difference (thanks @HoneyBadger);
Code:
truenas# diskinfo -wS /dev/nvd1
/dev/nvd1
        512             # sectorsize
        29260513280     # mediasize in bytes (27G)
        57149440        # mediasize in sectors
        0               # stripesize
        0               # stripeoffset
        INTEL MEMPEK1J032GA     # Disk descr.
        PHBT809300ZU032P        # Disk ident.
        Yes             # TRIM/UNMAP support
        0               # Rotation rate in RPM

Synchronous random writes:
         0.5 kbytes:     24.4 usec/IO =     20.0 Mbytes/s
           1 kbytes:     27.2 usec/IO =     36.0 Mbytes/s
           2 kbytes:     27.7 usec/IO =     70.5 Mbytes/s
           4 kbytes:     35.0 usec/IO =    111.5 Mbytes/s
           8 kbytes:     45.9 usec/IO =    170.4 Mbytes/s
          16 kbytes:     75.6 usec/IO =    206.8 Mbytes/s
          32 kbytes:    128.6 usec/IO =    242.9 Mbytes/s
          64 kbytes:    219.7 usec/IO =    284.4 Mbytes/s
         128 kbytes:    406.1 usec/IO =    307.8 Mbytes/s
         256 kbytes:    772.0 usec/IO =    323.8 Mbytes/s
         512 kbytes:   1494.6 usec/IO =    334.5 Mbytes/s
        1024 kbytes:   2953.0 usec/IO =    338.6 Mbytes/s
        2048 kbytes:   5860.1 usec/IO =    341.3 Mbytes/s
        4096 kbytes:  11738.1 usec/IO =    340.8 Mbytes/s
        8192 kbytes:  23331.2 usec/IO =    342.9 Mbytes/s

Boy did that help! Will now need to sift through ESXi settings to see which might help further.
+++++ /EDIT +++++

+++++ EDIT 2 +++++
Setting BIOS to "OS Control Mode" power regulator allows for these things to be adjusted on the fly in ESXi. The following settings in ESXi 7.0U2, within a custom power mgmt profile, come closest to setting the all-out performance profile in BIOS;
Code:
Power.PerfBiasEnable=0
Power.TimerHz=1000
Power.UseCStates=0
Power.UsePStates=0

With this it's then easy to switch between a power-saving profile and the custom profile on-the-fly (even while the VMs are powered on).
+++++ /EDIT 2 +++++
 
Last edited:

HoneyBadger

actually does care
Administrator
Moderator
iXsystems
Joined
Feb 6, 2014
Messages
5,110
BIOS settings made no discernable difference
Did you check the settings that pertained to power savings on the CPU as well as the PCIe link state/ASPM? Having this be a VM passthrough scenario might also necessitate that you change your hypervisor power policy to Custom/High performance. Small recordsizes are especially sensitive to this.
 

Kailee71

Contributor
Joined
Jul 8, 2018
Messages
110
And just because it's a bit of fun, here are 2x Samsung 850 250GiB SSDs striped on my (battery backed) 1GiB cache P420i; will of course not use this config but it was an interesting experiment;

Code:
truenas# diskinfo -wS /dev/da1
/dev/da1
        512             # sectorsize
        409532891136    # mediasize in bytes (381G)
        799868928       # mediasize in sectors
        0               # stripesize
        0               # stripeoffset
        49789           # Cylinders according to firmware.
        255             # Heads according to firmware.
        63              # Sectors according to firmware.
        HP LOGICAL VOLUME       # Disk descr.
        0014380347BFEC0         # Disk ident.
        No              # TRIM/UNMAP support
        0               # Rotation rate in RPM
        Not_Zoned       # Zone Mode

Synchronous random writes:
         0.5 kbytes:    213.0 usec/IO =      2.3 Mbytes/s
           1 kbytes:    146.6 usec/IO =      6.7 Mbytes/s
           2 kbytes:    149.5 usec/IO =     13.1 Mbytes/s
           4 kbytes:    161.9 usec/IO =     24.1 Mbytes/s
           8 kbytes:    148.5 usec/IO =     52.6 Mbytes/s
          16 kbytes:    160.1 usec/IO =     97.6 Mbytes/s
          32 kbytes:    170.2 usec/IO =    183.6 Mbytes/s
          64 kbytes:    365.3 usec/IO =    171.1 Mbytes/s
         128 kbytes:    493.0 usec/IO =    253.5 Mbytes/s
         256 kbytes:    599.7 usec/IO =    416.9 Mbytes/s
         512 kbytes:    872.2 usec/IO =    573.2 Mbytes/s
        1024 kbytes:   2030.2 usec/IO =    492.6 Mbytes/s
        2048 kbytes:   3473.4 usec/IO =    575.8 Mbytes/s
        4096 kbytes:   6384.8 usec/IO =    626.5 Mbytes/s
        8192 kbytes:  15578.5 usec/IO =    513.5 Mbytes/s


Kinda sucks at small xfers but not so bad with big ones.
 

Kailee71

Contributor
Joined
Jul 8, 2018
Messages
110
Hi again. Quick question; when I have power management set to balanced and not performance, I get these kind of messages when running the diskinfo test:

Code:
interrupt storm detected on "irq18:"; throttling interrupt source


And looking at vmstat this is for sure the nvme disk;

Code:
truenas# vmstat -i
interrupt                          total       rate
irq1: atkbd0                           2          0
irq17: nvme1                      265837        402
irq18: nvme0                      472298        714


This happens on the smaller block sizes; anything up to 8-16KiB. I'm now thinking it is this throttling that might be hampering the performance with those small block sizes, as for the larger block sizes the performance is comparable to when power management is set to performance. Is there a way to circumvent the throttling other than changing the power mngmt setting to test this hypothesis? I'm hoping I'd be able to have power management set balanced and still have the higher performing log devices (total power draw goes up from ~140W to ~250W on idle)...

Cheers,

Kai.
 
Top