Register for the iXsystems Community to get an ad-free experience and exclusive discounts in our eBay Store.
Resource icon

SLOG benchmarking and finding the best SLOG

HoneyBadger

Mushroom! Mushroom!
Joined
Feb 6, 2014
Messages
3,171
You don't need to forget that 0815 NAND with PLP might be enough for 99% of the home systems postet here.
Spot on. While overkill is fun to benchmark, it's important to build appropriate solutions.

We'd all love to be able to set up all-flash systems with NVDIMM SLOG devices and Optane special vdevs, connected via 40GbE and watch how fast they go - but many organizations and users don't need (or don't want to pay for) that kind of speed, and would be just as well served by a simpler and much less expensive setup.

Part of which is why I'm rather annoyed by the rumblings that Intel is discontinuing consumer-grade Optane devices, outside of the H20 hybrid/QLC model.


The little M10 sticks and the 900p/905p were great for the home-lab and "prosumer" workspace.
 

Herr_Merlin

Member
Joined
Oct 25, 2019
Messages
188
Hahah we only have 10G network here. Only the switch uplinks are 40G. Most people tend to forget that small I/O does not equal large network traffic.. looking at all those database clusters it's all small I/O
 

Ericloewe

Not-very-passive-but-aggressive
Moderator
Joined
Feb 15, 2014
Messages
16,950
Not sure if this is a factor of controller overhead or firmware design
Probably both. NAND SSDs have had monstrous amounts of engineering thrown at them to make them fast, which I don't imagine DRAM SSDs got.
 

Herr_Merlin

Member
Joined
Oct 25, 2019
Messages
188
Question did anyone tried the Optane DIMMs yet?
Another question - any idea how to pass them to a VM?
I might have a stupid idea for my privat ESXi and TrueNAS all in one box...
 

Rand

Neophyte Sage
Joined
Dec 30, 2013
Messages
885
Just got a link to this document the other day: https://lenovopress.com/lp1085-analyzing-the-performance-of-dcpmm-storage-appdirect-mode

They are mixed baggage, fast reads, slower writes. NVDimm-N is significantly faster for writes (as expected).

Edit - some numbers for optane Dimms from https://jira.ixsystems.com/browse/NAS-108510
Code:
/dev/pmem0
512 # sectorsize
270582935552 # mediasize in bytes (252G)
528482296 # mediasize in sectors
0 # stripesize
0 # stripeoffset
PMEM region 252GB # Disk descr.
E60A9A2579EB399E # Disk ident.
No # TRIM/UNMAP support
0 # Rotation rate in RPM

I/O command overhead:
time to read 10MB block 0.002218 sec = 0.000 msec/sector
time to read 20480 sectors 0.016406 sec = 0.001 msec/sector
calculated command overhead = 0.001 msec/sector

Seek times:
Full stroke: 250 iter in 0.000500 sec = 0.002 msec
Half stroke: 250 iter in 0.000486 sec = 0.002 msec
Quarter stroke: 500 iter in 0.000997 sec = 0.002 msec
Short forward: 400 iter in 0.000766 sec = 0.002 msec
Short backward: 400 iter in 0.000822 sec = 0.002 msec
Seq outer: 2048 iter in 0.001842 sec = 0.001 msec
Seq inner: 2048 iter in 0.001862 sec = 0.001 msec

Transfer rates:
outside: 102400 kbytes in 0.013630 sec = 7512839 kbytes/sec
middle: 102400 kbytes in 0.013790 sec = 7425671 kbytes/sec
inside: 102400 kbytes in 0.013994 sec = 7317422 kbytes/sec

Asynchronous random reads:
sectorsize: 1425829 ops in 3.000028 sec = 475272 IOPS
4 kbytes: 864612 ops in 3.000029 sec = 288201 IOPS
32 kbytes: 945080 ops in 3.000400 sec = 314985 IOPS
128 kbytes: 164956 ops in 3.002401 sec = 54941 IOPS

Synchronous random writes:
0.5 kbytes: 2.0 usec/IO = 241.0 Mbytes/s
1 kbytes: 2.1 usec/IO = 466.8 Mbytes/s
2 kbytes: 2.3 usec/IO = 848.2 Mbytes/s
4 kbytes: 2.6 usec/IO = 1510.5 Mbytes/s
8 kbytes: 5.1 usec/IO = 1544.3 Mbytes/s
16 kbytes: 10.8 usec/IO = 1452.0 Mbytes/s
32 kbytes: 20.8 usec/IO = 1503.5 Mbytes/s
64 kbytes: 37.7 usec/IO = 1658.9 Mbytes/s
128 kbytes: 79.2 usec/IO = 1577.3 Mbytes/s
256 kbytes: 157.2 usec/IO = 1590.8 Mbytes/s
512 kbytes: 310.1 usec/IO = 1612.5 Mbytes/s
1024 kbytes: 665.0 usec/IO = 1503.7 Mbytes/s
2048 kbytes: 1364.2 usec/IO = 1466.0 Mbytes/s
4096 kbytes: 2800.3 usec/IO = 1428.4 Mbytes/s
8192 kbytes: 5639.6 usec/IO = 1418.5 Mbytes/s
 
Last edited:

Rand

Neophyte Sage
Joined
Dec 30, 2013
Messages
885
SS530 Read intensive

Code:
smartctl 7.1 2019-12-30 r5022 [FreeBSD 12.2-RELEASE-p3 amd64] (local build)
Copyright (C) 2002-19, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Vendor:               WDC
Product:              WUSTR6416ASS204
Revision:             C925
Compliance:           SPC-5
User Capacity:        1,600,321,314,816 bytes [1.60 TB]
Logical block size:   512 bytes
Physical block size:  4096 bytes
LU is resource provisioned, LBPRZ=1
Rotation Rate:        Solid State Device
Form Factor:          2.5 inches
Logical Unit id:      0x5000cca0ab7483dc
Serial number:        
Device type:          disk
Transport protocol:   SAS (SPL-3)
Local Time is:        Sat Feb 13 16:51:23 2021 CET
SMART support is:     Available - device has SMART capability.
SMART support is:     Enabled
Temperature Warning:  Enabled

 512             # sectorsize
        1600321314816   # mediasize in bytes (1.5T)
        3125627568      # mediasize in sectors
        4096            # stripesize
        0               # stripeoffset
        194561          # Cylinders according to firmware.
        255             # Heads according to firmware.
        63              # Sectors according to firmware.
        WDC WUSTR6416ASS204     # Disk descr.
        WXX22G3A        # Disk ident.
        Yes             # TRIM/UNMAP support
        0               # Rotation rate in RPM
        Not_Zoned       # Zone Mode

Synchronous random writes:
         0.5 kbytes:    127.5 usec/IO =      3.8 Mbytes/s
           1 kbytes:    128.1 usec/IO =      7.6 Mbytes/s
           2 kbytes:    129.8 usec/IO =     15.0 Mbytes/s
           4 kbytes:    133.0 usec/IO =     29.4 Mbytes/s
           8 kbytes:    141.5 usec/IO =     55.2 Mbytes/s
          16 kbytes:    159.9 usec/IO =     97.7 Mbytes/s
          32 kbytes:    188.1 usec/IO =    166.1 Mbytes/s
          64 kbytes:    253.6 usec/IO =    246.5 Mbytes/s
         128 kbytes:    375.7 usec/IO =    332.7 Mbytes/s
         256 kbytes:    612.4 usec/IO =    408.3 Mbytes/s
         512 kbytes:   1083.0 usec/IO =    461.7 Mbytes/s
        1024 kbytes:   2040.3 usec/IO =    490.1 Mbytes/s
        2048 kbytes:   3942.1 usec/IO =    507.3 Mbytes/s
        4096 kbytes:   7720.6 usec/IO =    518.1 Mbytes/s
        8192 kbytes:  15359.9 usec/IO =    520.8 Mbytes/s
 

qbix

Neophyte
Joined
Dec 6, 2020
Messages
4
INTEL D3-S4610 960GB TLC

Code:
root@nas[~]# smartctl -a /dev/da0
smartctl 7.1 2019-12-30 r5022 [FreeBSD 12.2-RELEASE-p3 amd64] (local build)
Copyright (C) 2002-19, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:       Intel S4510/S4610/S4500/S4600 Series SSDs
Device Model:       INTEL SSDSC2KG960G801
Serial Number:      BTYG93130G0W960CGN
LU WWN Device Id:   5 5cd2e4 1514d10e3
Firmware Version:   XCV10110
User Capacity:      960,197,124,096 bytes [960 GB]
Sector Sizes:       512 bytes logical, 4096 bytes physical
Rotation Rate:      Solid State Device
Form Factor:        2.5 inches
Device is:          In smartctl database [for details use: -P show]
ATA Version is:     ACS-3 T13/2161-D revision 5
SATA Version is:    SATA 3.2, 6.0 Gb/s (current: 6.0 Gb/s)



Code:
root@nas[~]# diskinfo -wS /dev/da0
/dev/da0
        512             # sectorsize
        960197124096    # mediasize in bytes (894G)
        1875385008      # mediasize in sectors
        4096            # stripesize
        0               # stripeoffset
        116737          # Cylinders according to firmware.
        255             # Heads according to firmware.
        63              # Sectors according to firmware.
        ATA INTEL SSDSC2KG96    # Disk descr.
        BTYG93130G0W960CGN      # Disk ident.
        Yes             # TRIM/UNMAP support
        0               # Rotation rate in RPM
        Not_Zoned       # Zone Mode

Synchronous random writes:
         0.5 kbytes:    115.2 usec/IO =      4.2 Mbytes/s
           1 kbytes:    115.8 usec/IO =      8.4 Mbytes/s
           2 kbytes:    116.9 usec/IO =     16.7 Mbytes/s
           4 kbytes:    116.9 usec/IO =     33.4 Mbytes/s
           8 kbytes:    125.2 usec/IO =     62.4 Mbytes/s
          16 kbytes:    142.5 usec/IO =    109.6 Mbytes/s
          32 kbytes:    178.1 usec/IO =    175.5 Mbytes/s
          64 kbytes:    248.1 usec/IO =    251.9 Mbytes/s
         128 kbytes:    400.5 usec/IO =    312.1 Mbytes/s
         256 kbytes:    664.7 usec/IO =    376.1 Mbytes/s
         512 kbytes:   1193.4 usec/IO =    419.0 Mbytes/s
        1024 kbytes:   2233.4 usec/IO =    447.7 Mbytes/s
        2048 kbytes:   4344.4 usec/IO =    460.4 Mbytes/s
        4096 kbytes:   8619.1 usec/IO =    464.1 Mbytes/s
        8192 kbytes:  16871.2 usec/IO =    474.2 Mbytes/s
 
Last edited:

douglasg

Newbie
Joined
Mar 24, 2021
Messages
2
I didn't see the 983 DCT here so here are my results:
Code:
root@truenas[~]# smartctl -a /dev/nvme2
smartctl 7.1 2019-12-30 r5022 [FreeBSD 12.2-RELEASE-p3 amd64] (local build)
Copyright (C) 2002-19, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Number:                       Samsung SSD 983 DCT M.2 960GB
Serial Number:                      S48ENC0N501364K
Firmware Version:                   EDA7602Q
PCI Vendor/Subsystem ID:            0x144d
IEEE OUI Identifier:                0x002538
Total NVM Capacity:                 960,197,124,096 [960 GB]
Unallocated NVM Capacity:           0
Controller ID:                      4
Number of Namespaces:               1
Namespace 1 Size/Capacity:          960,197,124,096 [960 GB]
Namespace 1 Utilization:            227,024,896 [227 MB]
Namespace 1 Formatted LBA Size:     512
Local Time is:                      Thu Mar 25 00:05:12 2021 EDT
Firmware Updates (0x17):            3 Slots, Slot 1 R/O, no Reset required
Optional Admin Commands (0x000f):   Security Format Frmw_DL NS_Mngmt
Optional NVM Commands (0x001f):     Comp Wr_Unc DS_Mngmt Wr_Zero Sav/Sel_Feat
Maximum Data Transfer Size:         512 Pages
Warning  Comp. Temp. Threshold:     86 Celsius
Critical Comp. Temp. Threshold:     87 Celsius
Namespace 1 Features (0x02):        NA_Fields


Code:
root@truenas[~]# diskinfo -wS /dev/nvd2
/dev/nvd2
        512             # sectorsize
        960197124096    # mediasize in bytes (894G)
        1875385008      # mediasize in sectors
        0               # stripesize
        0               # stripeoffset
        Samsung SSD 983 DCT M.2 960GB   # Disk descr.
        S48ENC0N501364K # Disk ident.
        Yes             # TRIM/UNMAP support
        0               # Rotation rate in RPM

Synchronous random writes:
         0.5 kbytes:     17.5 usec/IO =     28.0 Mbytes/s
           1 kbytes:     17.6 usec/IO =     55.6 Mbytes/s
           2 kbytes:     18.1 usec/IO =    107.9 Mbytes/s
           4 kbytes:     19.0 usec/IO =    205.3 Mbytes/s
           8 kbytes:     21.8 usec/IO =    358.5 Mbytes/s
          16 kbytes:     27.2 usec/IO =    575.0 Mbytes/s
          32 kbytes:     41.2 usec/IO =    759.2 Mbytes/s
          64 kbytes:     70.6 usec/IO =    885.8 Mbytes/s
         128 kbytes:    147.1 usec/IO =    849.5 Mbytes/s
         256 kbytes:    260.2 usec/IO =    960.8 Mbytes/s
         512 kbytes:    459.4 usec/IO =   1088.3 Mbytes/s
        1024 kbytes:    863.5 usec/IO =   1158.1 Mbytes/s
        2048 kbytes:   1667.3 usec/IO =   1199.6 Mbytes/s
        4096 kbytes:   3244.0 usec/IO =   1233.0 Mbytes/s
        8192 kbytes:   6503.6 usec/IO =   1230.1 Mbytes/s


I also had a 950 Pro in my system so here are those results:
Code:
root@truenas[~]# smartctl -a /dev/nvme0
smartctl 7.1 2019-12-30 r5022 [FreeBSD 12.2-RELEASE-p3 amd64] (local build)
Copyright (C) 2002-19, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Number:                       Samsung SSD 950 PRO 512GB
Serial Number:                      S2GMNX0H609029E
Firmware Version:                   1B0QBXX7
PCI Vendor/Subsystem ID:            0x144d
IEEE OUI Identifier:                0x002538
Controller ID:                      1
Number of Namespaces:               1
Namespace 1 Size/Capacity:          512,110,190,592 [512 GB]
Namespace 1 Utilization:            104,878,292,992 [104 GB]
Namespace 1 Formatted LBA Size:     512
Namespace 1 IEEE EUI-64:            002538 5661b053ec
Local Time is:                      Wed Mar 24 23:57:59 2021 EDT
Firmware Updates (0x06):            3 Slots
Optional Admin Commands (0x0007):   Security Format Frmw_DL
Optional NVM Commands (0x001f):     Comp Wr_Unc DS_Mngmt Wr_Zero Sav/Sel_Feat
Maximum Data Transfer Size:         32 Pages


Code:
root@truenas[~]# diskinfo -wS /dev/nvd0
/dev/nvd0
        512             # sectorsize
        512110190592    # mediasize in bytes (477G)
        1000215216      # mediasize in sectors
        0               # stripesize
        0               # stripeoffset
        Samsung SSD 950 PRO 512GB       # Disk descr.
        S2GMNX0H609029E # Disk ident.
        Yes             # TRIM/UNMAP support
        0               # Rotation rate in RPM

Synchronous random writes:
         0.5 kbytes:   1675.6 usec/IO =      0.3 Mbytes/s
           1 kbytes:   1692.3 usec/IO =      0.6 Mbytes/s
           2 kbytes:   1663.1 usec/IO =      1.2 Mbytes/s
           4 kbytes:   1740.6 usec/IO =      2.2 Mbytes/s
           8 kbytes:   1616.9 usec/IO =      4.8 Mbytes/s
          16 kbytes:   1842.3 usec/IO =      8.5 Mbytes/s
          32 kbytes:   1887.4 usec/IO =     16.6 Mbytes/s
          64 kbytes:   2178.0 usec/IO =     28.7 Mbytes/s
         128 kbytes:   2759.6 usec/IO =     45.3 Mbytes/s
         256 kbytes:   3612.3 usec/IO =     69.2 Mbytes/s
         512 kbytes:   3644.1 usec/IO =    137.2 Mbytes/s
        1024 kbytes:   3695.2 usec/IO =    270.6 Mbytes/s
        2048 kbytes:   3622.2 usec/IO =    552.1 Mbytes/s
        4096 kbytes:   5484.4 usec/IO =    729.3 Mbytes/s
        8192 kbytes:   8916.5 usec/IO =    897.2 Mbytes/s
 

Ralms

Junior Member
Joined
Jan 28, 2019
Messages
15
Sharing the results of a Samsung 960 EVO 250GB


Code:
root@freenas[~]# smartctl -a /dev/nvme0
smartctl 7.1 2019-12-30 r5022 [FreeBSD 12.2-RELEASE-p3 amd64] (local build)
Copyright (C) 2002-19, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Number:                       Samsung SSD 960 EVO 250GB
Serial Number:                      S3ESNX0K298763Z
Firmware Version:                   3B7QCXE7
PCI Vendor/Subsystem ID:            0x144d
IEEE OUI Identifier:                0x002538
Total NVM Capacity:                 250,059,350,016 [250 GB]
Unallocated NVM Capacity:           0
Controller ID:                      2
Number of Namespaces:               1
Namespace 1 Size/Capacity:          250,059,350,016 [250 GB]
Namespace 1 Utilization:            250,058,170,368 [250 GB]
Namespace 1 Formatted LBA Size:     512
Namespace 1 IEEE EUI-64:            002538 5281b2d7eb
Local Time is:                      Sun Apr  4 12:29:01 2021 WEST
Firmware Updates (0x16):            3 Slots, no Reset required
Optional Admin Commands (0x0007):   Security Format Frmw_DL
Optional NVM Commands (0x001f):     Comp Wr_Unc DS_Mngmt Wr_Zero Sav/Sel_Feat
Maximum Data Transfer Size:         512 Pages
Warning  Comp. Temp. Threshold:     77 Celsius
Critical Comp. Temp. Threshold:     79 Celsius


Code:
root@freenas[~]# diskinfo -wS /dev/nvd0
/dev/nvd0
        512             # sectorsize
        250059350016    # mediasize in bytes (233G)
        488397168       # mediasize in sectors
        0               # stripesize
        0               # stripeoffset
        Samsung SSD 960 EVO 250GB       # Disk descr.
        S3ESNX0K298763Z # Disk ident.
        Yes             # TRIM/UNMAP support
        0               # Rotation rate in RPM

Synchronous random writes:
         0.5 kbytes:    727.9 usec/IO =      0.7 Mbytes/s
           1 kbytes:    695.8 usec/IO =      1.4 Mbytes/s
           2 kbytes:    721.1 usec/IO =      2.7 Mbytes/s
           4 kbytes:    674.6 usec/IO =      5.8 Mbytes/s
           8 kbytes:    725.0 usec/IO =     10.8 Mbytes/s
          16 kbytes:    829.8 usec/IO =     18.8 Mbytes/s
          32 kbytes:    869.4 usec/IO =     35.9 Mbytes/s
          64 kbytes:    818.4 usec/IO =     76.4 Mbytes/s
         128 kbytes:    913.7 usec/IO =    136.8 Mbytes/s
         256 kbytes:   1108.7 usec/IO =    225.5 Mbytes/s
         512 kbytes:   1564.5 usec/IO =    319.6 Mbytes/s
        1024 kbytes:   2257.3 usec/IO =    443.0 Mbytes/s
        2048 kbytes:   3702.2 usec/IO =    540.2 Mbytes/s
        4096 kbytes:   6467.6 usec/IO =    618.5 Mbytes/s
        8192 kbytes:  11691.3 usec/IO =    684.3 Mbytes/s
 

ehsab

Member
Joined
Aug 2, 2020
Messages
45
The RMS-300 seems to stand quite good against the Optane NVDIMMs.

Optane NVDIMM

64 kbytes: 37.7 usec/IO = 1658.9 Mbytes/s
128 kbytes: 79.2 usec/IO = 1577.3 Mbytes/s
256 kbytes: 157.2 usec/IO = 1590.8 Mbytes/s
512 kbytes: 310.1 usec/IO = 1612.5 Mbytes/s
1024 kbytes: 665.0 usec/IO = 1503.7 Mbytes/s


RMS-300

64 kbytes: 45.8 usec/IO = 1363.3 Mbytes/s
128 kbytes: 80.6 usec/IO = 1550.3 Mbytes/s
256 kbytes: 99.0 usec/IO = 2525.0 Mbytes/s
512 kbytes: 147.6 usec/IO = 3388.3 Mbytes/s
1024 kbytes: 256.2 usec/IO = 3902.8 Mbytes/s


Optane NVDIMM looks really fast on small writes, where the RMS-300 performs better with 256K or larger writes.
 

Ericloewe

Not-very-passive-but-aggressive
Moderator
Joined
Feb 15, 2014
Messages
16,950
I assume those Optane numbers are from a block device configuration and not a memory mode configuration, correct?
 
Top