Resource icon

SLOG benchmarking and finding the best SLOG

suhl

Cadet
Joined
Sep 8, 2020
Messages
4
Somebody was asking some time ago how the Optane P4801X 200G will change its performance when the sector size will be changed from the default 512 to 4096. I was looking for an answer myself and I was finally able to find out. Below is the result of my comparison. The difference is insignificant. This Optane drive has Variable Sector Size feature. The sector size can be changed and it is not like with the regular drives when setting the sector size too small to the physical sector size will result with high write penalty.
First with 4k sector:

Code:
root@nas02[~]# sysctl kern.geom.debugflags=16
kern.geom.debugflags: 0 -> 16
root@nas02[~]# diskinfo -wS /dev/nvd0
/dev/nvd0
        4096            # sectorsize
        200049647616    # mediasize in bytes (186G)
        48840246        # mediasize in sectors
        0               # stripesize
        0               # stripeoffset
        INTEL SSDPEL1K200GA     # Disk descr.
        PHKM9163002C200E        # Disk ident.
        Yes             # TRIM/UNMAP support
        0               # Rotation rate in RPM

Synchronous random writes:
           4 kbytes:     24.9 usec/IO =    156.9 Mbytes/s
           8 kbytes:     28.4 usec/IO =    275.1 Mbytes/s
          16 kbytes:     34.9 usec/IO =    447.2 Mbytes/s
          32 kbytes:     45.0 usec/IO =    694.9 Mbytes/s
          64 kbytes:     62.6 usec/IO =    998.7 Mbytes/s
         128 kbytes:     96.4 usec/IO =   1296.9 Mbytes/s
         256 kbytes:    161.0 usec/IO =   1552.5 Mbytes/s
         512 kbytes:    290.4 usec/IO =   1722.0 Mbytes/s
        1024 kbytes:    545.6 usec/IO =   1833.0 Mbytes/s
        2048 kbytes:   1055.4 usec/IO =   1895.0 Mbytes/s
        4096 kbytes:   2085.0 usec/IO =   1918.4 Mbytes/s
        8192 kbytes:   4125.8 usec/IO =   1939.0 Mbytes/s


512 sector for reference:

Code:
root@nas02[~]# diskinfo -wS /dev/nvd0
/dev/nvd0
        512             # sectorsize
        200049647616    # mediasize in bytes (186G)
        390721968       # mediasize in sectors
        0               # stripesize
        0               # stripeoffset
        INTEL SSDPEL1K200GA     # Disk descr.
        PHKM9163002C200E        # Disk ident.
        Yes             # TRIM/UNMAP support
        0               # Rotation rate in RPM

Synchronous random writes:
         0.5 kbytes:     27.7 usec/IO =     17.6 Mbytes/s
           1 kbytes:     28.4 usec/IO =     34.4 Mbytes/s
           2 kbytes:     29.1 usec/IO =     67.2 Mbytes/s
           4 kbytes:     25.6 usec/IO =    152.3 Mbytes/s
           8 kbytes:     29.4 usec/IO =    265.4 Mbytes/s
          16 kbytes:     35.7 usec/IO =    438.0 Mbytes/s
          32 kbytes:     47.4 usec/IO =    659.6 Mbytes/s
          64 kbytes:     64.5 usec/IO =    968.6 Mbytes/s
         128 kbytes:    101.0 usec/IO =   1238.2 Mbytes/s
         256 kbytes:    170.0 usec/IO =   1470.2 Mbytes/s
         512 kbytes:    302.1 usec/IO =   1655.2 Mbytes/s
        1024 kbytes:    577.1 usec/IO =   1732.7 Mbytes/s
        2048 kbytes:   1114.9 usec/IO =   1794.0 Mbytes/s
        4096 kbytes:   2190.1 usec/IO =   1826.4 Mbytes/s
        8192 kbytes:   4353.6 usec/IO =   1837.6 Mbytes/s


I'm researching this for the purpose of setting the right sector size for my mirrored Optanes as a SLOG device in my build. What I still didn't figure out is how the ashift of the SLOG should be alligned with the pool vdev ashift to get the best results for select recordsize parameters when working with pool vdev of 8 drives in RAIDz2 with lz4 compression. My main use is MySQL, so I'm testing recordsize of 16k right now. Can anybody suggest some good materials to read about how to properly do the sector size aligment between SLOG/vdev/dataset recordsize for best performance?
 

ee21

Dabbler
Joined
Nov 2, 2020
Messages
33
Sabrent Rocket PCie 3.0 NVMe SSD, 800TBW endurance rating at the 512GB size, $69 USD:

Code:
root@truenas[~]# diskinfo -wS /dev/nvd0
/dev/nvd0
        512             # sectorsize
        512110190592    # mediasize in bytes (477G)
        1000215216      # mediasize in sectors
        0               # stripesize
        0               # stripeoffset
        Sabrent         # Disk descr.
        203E0705081500197960    # Disk ident.
        Yes             # TRIM/UNMAP support
        0               # Rotation rate in RPM

Synchronous random writes:
         0.5 kbytes:   1022.8 usec/IO =      0.5 Mbytes/s
           1 kbytes:   1027.9 usec/IO =      1.0 Mbytes/s
           2 kbytes:   1032.4 usec/IO =      1.9 Mbytes/s
           4 kbytes:    994.3 usec/IO =      3.9 Mbytes/s
           8 kbytes:   1013.9 usec/IO =      7.7 Mbytes/s
          16 kbytes:   1014.0 usec/IO =     15.4 Mbytes/s
          32 kbytes:   1042.0 usec/IO =     30.0 Mbytes/s
          64 kbytes:   1212.3 usec/IO =     51.6 Mbytes/s
         128 kbytes:   1045.8 usec/IO =    119.5 Mbytes/s
         256 kbytes:   1375.5 usec/IO =    181.8 Mbytes/s
         512 kbytes:   1940.4 usec/IO =    257.7 Mbytes/s
        1024 kbytes:   2855.9 usec/IO =    350.1 Mbytes/s
        2048 kbytes:   5304.7 usec/IO =    377.0 Mbytes/s
        4096 kbytes:  10197.0 usec/IO =    392.3 Mbytes/s
        8192 kbytes:  19968.5 usec/IO =    400.6 Mbytes/s
 
Last edited:

ehsab

Dabbler
Joined
Aug 2, 2020
Messages
45
Samsung PM983 1.9TB U2

Code:
root@truenas[~]# diskinfo -wS /dev/nvd3
/dev/nvd3
        512             # sectorsize
        1920383410176   # mediasize in bytes (1.7T)
        3750748848      # mediasize in sectors
        0               # stripesize
        0               # stripeoffset
        SAMSUNG MZQLB1T9HAJR-00007      # Disk descr.
        S439NXXX901635  # Disk ident.
        Yes             # TRIM/UNMAP support
        0               # Rotation rate in RPM

Synchronous random writes:
         0.5 kbytes:     13.1 usec/IO =     37.3 Mbytes/s
           1 kbytes:     13.3 usec/IO =     73.5 Mbytes/s
           2 kbytes:     13.7 usec/IO =    142.9 Mbytes/s
           4 kbytes:     14.3 usec/IO =    274.1 Mbytes/s
           8 kbytes:     15.6 usec/IO =    499.9 Mbytes/s
          16 kbytes:     18.5 usec/IO =    843.2 Mbytes/s
          32 kbytes:     24.0 usec/IO =   1304.4 Mbytes/s
          64 kbytes:     34.2 usec/IO =   1826.7 Mbytes/s
         128 kbytes:     60.0 usec/IO =   2084.3 Mbytes/s
         256 kbytes:    117.3 usec/IO =   2132.0 Mbytes/s
         512 kbytes:    234.5 usec/IO =   2132.4 Mbytes/s
        1024 kbytes:    465.9 usec/IO =   2146.2 Mbytes/s
        2048 kbytes:    923.8 usec/IO =   2164.9 Mbytes/s
        4096 kbytes:   1840.1 usec/IO =   2173.8 Mbytes/s
        8192 kbytes:   3696.7 usec/IO =   2164.1 Mbytes/s
 

ehsab

Dabbler
Joined
Aug 2, 2020
Messages
45
Samsung PM1733 1.9TB U2

Code:
# diskinfo -wS /dev/nvd0
/dev/nvd0
        512             # sectorsize
        1920383410176   # mediasize in bytes (1.7T)
        3750748848      # mediasize in sectors
        0               # stripesize
        0               # stripeoffset
        SAMSUNG MZWLJ1T9HBJR-00007      # Disk descr.
        XXXXXXXXXXXXXX  # Disk ident.
        Yes             # TRIM/UNMAP support
        0               # Rotation rate in RPM

Synchronous random writes:
         0.5 kbytes:     20.6 usec/IO =     23.7 Mbytes/s
           1 kbytes:     20.3 usec/IO =     48.2 Mbytes/s
           2 kbytes:     19.1 usec/IO =    102.2 Mbytes/s
           4 kbytes:     13.8 usec/IO =    283.3 Mbytes/s
           8 kbytes:     15.3 usec/IO =    510.1 Mbytes/s
          16 kbytes:     17.2 usec/IO =    907.4 Mbytes/s
          32 kbytes:     20.7 usec/IO =   1508.6 Mbytes/s
          64 kbytes:     27.3 usec/IO =   2292.8 Mbytes/s
         128 kbytes:     50.4 usec/IO =   2481.3 Mbytes/s
         256 kbytes:    101.2 usec/IO =   2470.3 Mbytes/s
         512 kbytes:    202.2 usec/IO =   2473.0 Mbytes/s
        1024 kbytes:    403.0 usec/IO =   2481.3 Mbytes/s
        2048 kbytes:    813.3 usec/IO =   2459.2 Mbytes/s
        4096 kbytes:   1620.7 usec/IO =   2468.1 Mbytes/s
        8192 kbytes:   3258.2 usec/IO =   2455.3 Mbytes/s
 

kfason

Cadet
Joined
Jul 25, 2018
Messages
4
Here are mine to compare to others that already tested them. Mine are different. I am on R720xd E5-26760v2 with H710 mini HBA. Tests pasted from ESX TrueNAS VM however I did run on the hardware directly with very close results.

S3700. Dell version but newer firmware and forced to 600MB/sec negotiation.

Code:
=== START OF INFORMATION SECTION ===
Model Family:     Intel 730 and DC S35x0/3610/3700 Series SSDs
Device Model:     INTEL SSDSC2BA200G3T
Serial Number:    BTTV419500MN200GGN
LU WWN Device Id: 5 5cd2e4 04b6348a7
Add. Product Id:  DELL(tm)
Firmware Version: 5DV1DL08
User Capacity:    200,049,647,616 bytes [200 GB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    Solid State Device
Form Factor:      2.5 inches
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ACS-2 T13/2015-D revision 3
SATA Version is:  SATA 2.6, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Fri Jan  8 21:25:58 2021 MST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled


Code:
Synchronous random writes:
         0.5 kbytes:    225.2 usec/IO =      2.2 Mbytes/s
           1 kbytes:    247.8 usec/IO =      3.9 Mbytes/s
           2 kbytes:    258.8 usec/IO =      7.5 Mbytes/s
           4 kbytes:    191.6 usec/IO =     20.4 Mbytes/s
           8 kbytes:    185.3 usec/IO =     42.2 Mbytes/s
          16 kbytes:    201.1 usec/IO =     77.7 Mbytes/s
          32 kbytes:    232.4 usec/IO =    134.5 Mbytes/s
          64 kbytes:    307.9 usec/IO =    203.0 Mbytes/s
         128 kbytes:    455.3 usec/IO =    274.5 Mbytes/s
         256 kbytes:    784.9 usec/IO =    318.5 Mbytes/s
         512 kbytes:   1424.0 usec/IO =    351.1 Mbytes/s
        1024 kbytes:   2764.8 usec/IO =    361.7 Mbytes/s
        2048 kbytes:   5425.2 usec/IO =    368.7 Mbytes/s
        4096 kbytes:  10552.2 usec/IO =    379.1 Mbytes/s
        8192 kbytes:  21154.6 usec/IO =    378.2 Mbytes/s


800p. Strange the MB are double the S3700 yet the IO is half of it. What could cause that? The S3700 is faster in NFS VM crystalbenchmark tests because of it.

Code:
=== START OF INFORMATION SECTION ===
Model Number:                       INTEL SSDPEK1W060GA
Serial Number:                      PHBT9074069L064Q
Firmware Version:                   K4110440
PCI Vendor/Subsystem ID:            0x8086
IEEE OUI Identifier:                0x5cd2e4
Controller ID:                      0
Number of Namespaces:               1
Namespace 1 Size/Capacity:          58,977,157,120 [58.9 GB]
Namespace 1 Formatted LBA Size:     512
Namespace 1 IEEE EUI-64:            5cd2e4 d56d7d0100
Local Time is:                      Fri Jan  8 21:30:04 2021 MST
Firmware Updates (0x02):            1 Slot
Optional Admin Commands (0x0006):   Format Frmw_DL
Optional NVM Commands (0x0046):     Wr_Unc DS_Mngmt Timestmp
Maximum Data Transfer Size:         32 Pages


Code:
Synchronous random writes:
         0.5 kbytes:     11.4 usec/IO =     42.8 Mbytes/s
           1 kbytes:      9.6 usec/IO =    101.5 Mbytes/s
           2 kbytes:     11.9 usec/IO =    163.6 Mbytes/s
           4 kbytes:     15.0 usec/IO =    260.2 Mbytes/s
           8 kbytes:     21.1 usec/IO =    370.1 Mbytes/s
          16 kbytes:     32.8 usec/IO =    476.1 Mbytes/s
          32 kbytes:     63.6 usec/IO =    491.0 Mbytes/s
          64 kbytes:    121.5 usec/IO =    514.2 Mbytes/s
         128 kbytes:    224.4 usec/IO =    557.0 Mbytes/s
         256 kbytes:    428.3 usec/IO =    583.8 Mbytes/s
         512 kbytes:    888.3 usec/IO =    562.9 Mbytes/s
        1024 kbytes:   1700.5 usec/IO =    588.1 Mbytes/s
        2048 kbytes:   3328.5 usec/IO =    600.9 Mbytes/s
        4096 kbytes:   6632.9 usec/IO =    603.1 Mbytes/s
        8192 kbytes:  13094.5 usec/IO =    610.9 Mbytes/s
 

CLEsportsfan

Dabbler
Joined
Nov 23, 2020
Messages
20
Here's my Intel DC P3700 I'll be using as my SLOG. It's the 1.6TB version over provisioned to 400GB, running on a Dell R720, bare metal, not in a VM. Seems to have better performance than some other P3700's I saw here

Code:
# smartctl -a /dev/nvme0
smartctl 7.1 2019-12-30 r5022 [FreeBSD 12.2-RELEASE-p2 amd64] (local build)
Copyright (C) 2002-19, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Number:                       INTEL SSDPEDMD016T4
Serial Number:                      CVFTxxxxxxxxxx6DGN
Firmware Version:                   8DV101H0
PCI Vendor/Subsystem ID:            0x8086
IEEE OUI Identifier:                0x5cd2e4
Controller ID:                      0
Number of Namespaces:               1
Namespace 1 Size/Capacity:          400,080,326,656 [400 GB]
Namespace 1 Formatted LBA Size:     4096
Local Time is:                      Tue Jan 12 10:53:18 2021 EST
Firmware Updates (0x02):            1 Slot
Optional Admin Commands (0x0006):   Format Frmw_DL
Optional NVM Commands (0x0006):     Wr_Unc DS_Mngmt
Maximum Data Transfer Size:         32 Pages


Here is the performance test at factory-set 512 sector size when I had it OP'd to 800GB.
Code:
# diskinfo -wS /dev/nvd0
/dev/nvd0
        512             # sectorsize
        800160657408    # mediasize in bytes (745G)
        1562813784      # mediasize in sectors
        131072          # stripesize
        0               # stripeoffset
        INTEL SSDPEDMD016T4     # Disk descr.
        CVFTxxxxxxxxxx6DGN      # Disk ident.
        Yes             # TRIM/UNMAP support
        0               # Rotation rate in RPM

Synchronous random writes:
         0.5 kbytes:     15.0 usec/IO =     32.5 Mbytes/s
           1 kbytes:     14.7 usec/IO =     66.6 Mbytes/s
           2 kbytes:     13.9 usec/IO =    140.5 Mbytes/s
           4 kbytes:     12.2 usec/IO =    320.9 Mbytes/s
           8 kbytes:     14.4 usec/IO =    541.4 Mbytes/s
          16 kbytes:     19.5 usec/IO =    803.0 Mbytes/s
          32 kbytes:     25.3 usec/IO =   1237.2 Mbytes/s
          64 kbytes:     42.4 usec/IO =   1475.0 Mbytes/s
         128 kbytes:     82.1 usec/IO =   1521.7 Mbytes/s
         256 kbytes:    146.7 usec/IO =   1704.3 Mbytes/s
         512 kbytes:    294.5 usec/IO =   1698.1 Mbytes/s
        1024 kbytes:    587.1 usec/IO =   1703.3 Mbytes/s
        2048 kbytes:   1141.7 usec/IO =   1751.9 Mbytes/s
        4096 kbytes:   2278.9 usec/IO =   1755.2 Mbytes/s
        8192 kbytes:   4472.4 usec/IO =   1788.7 Mbytes/s


I then OverProvisioned more to 400GB and formatted with 4k sector size, and this is how it'll stay. Results are here:
Code:
# diskinfo -wS /dev/nvd0
/dev/nvd0
        4096            # sectorsize
        400080326656    # mediasize in bytes (373G)
        97675861        # mediasize in sectors
        131072          # stripesize
        0               # stripeoffset
        INTEL SSDPEDMD016T4     # Disk descr.
        CVFTxxxxxxxxxx6DGN      # Disk ident.
        Yes             # TRIM/UNMAP support
        0               # Rotation rate in RPM

Synchronous random writes:
           4 kbytes:     12.3 usec/IO =    317.0 Mbytes/s
           8 kbytes:     14.5 usec/IO =    538.4 Mbytes/s
          16 kbytes:     18.7 usec/IO =    837.2 Mbytes/s
          32 kbytes:     26.1 usec/IO =   1196.4 Mbytes/s
          64 kbytes:     41.7 usec/IO =   1498.0 Mbytes/s
         128 kbytes:     81.7 usec/IO =   1529.1 Mbytes/s
         256 kbytes:    150.1 usec/IO =   1665.9 Mbytes/s
         512 kbytes:    290.9 usec/IO =   1718.6 Mbytes/s
        1024 kbytes:    575.7 usec/IO =   1737.0 Mbytes/s
        2048 kbytes:   1159.6 usec/IO =   1724.8 Mbytes/s
        4096 kbytes:   2293.2 usec/IO =   1744.3 Mbytes/s
        8192 kbytes:   4527.2 usec/IO =   1767.1 Mbytes/s
 
Last edited:

CLEsportsfan

Dabbler
Joined
Nov 23, 2020
Messages
20
Toshiba PX04SVB040 SAS SSD 400gb. Drive limited here to 6gbps by my LSI SAS 9207-8i card. Over provisioned drive to 300gb, using them for metadata cache mirror and L2ARC.

Code:
# smartctl -a /dev/da9
smartctl 7.1 2019-12-30 r5022 [FreeBSD 12.2-RELEASE-p2 amd64] (local build)
Copyright (C) 2002-19, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Vendor:               TOSHIBA
Product:              PX04SVB040
Revision:             0106
Compliance:           SPC-4
User Capacity:        300,066,342,912 bytes [300 GB]
Logical block size:   512 bytes
Physical block size:  4096 bytes
LU is resource provisioned, LBPRZ=1
Rotation Rate:        Solid State Device
Form Factor:          2.5 inches
Logical Unit id:      0x58ce38e06c919cb1
Serial number:        77XxxxxxT43E
Device type:          disk
Transport protocol:   SAS (SPL-3)
Local Time is:        Tue Jan 12 14:30:21 2021 EST
SMART support is:     Available - device has SMART capability.
SMART support is:     Enabled
Temperature Warning:  Enabled

Performance before over provision:
Code:
# diskinfo -wS /dev/da9
/dev/da9
        512             # sectorsize
        400088457216    # mediasize in bytes (373G)
        781422768       # mediasize in sectors
        4096            # stripesize
        0               # stripeoffset
        48641           # Cylinders according to firmware.
        255             # Heads according to firmware.
        63              # Sectors according to firmware.
        TOSHIBA PX04SVB040      # Disk descr.
        77XxxxxxT43E    # Disk ident.
        Yes             # TRIM/UNMAP support
        0               # Rotation rate in RPM
        Not_Zoned       # Zone Mode

Synchronous random writes:
         0.5 kbytes:     76.3 usec/IO =      6.4 Mbytes/s
           1 kbytes:     77.7 usec/IO =     12.6 Mbytes/s
           2 kbytes:     75.2 usec/IO =     26.0 Mbytes/s
           4 kbytes:     88.3 usec/IO =     44.3 Mbytes/s
           8 kbytes:     94.8 usec/IO =     82.4 Mbytes/s
          16 kbytes:    107.1 usec/IO =    145.9 Mbytes/s
          32 kbytes:    139.1 usec/IO =    224.7 Mbytes/s
          64 kbytes:    198.8 usec/IO =    314.4 Mbytes/s
         128 kbytes:    319.0 usec/IO =    391.8 Mbytes/s
         256 kbytes:    546.2 usec/IO =    457.7 Mbytes/s
         512 kbytes:   1021.8 usec/IO =    489.3 Mbytes/s
        1024 kbytes:   1947.6 usec/IO =    513.4 Mbytes/s
        2048 kbytes:   3812.0 usec/IO =    524.7 Mbytes/s
        4096 kbytes:   7535.5 usec/IO =    530.8 Mbytes/s
        8192 kbytes:  14981.7 usec/IO =    534.0 Mbytes/s

Over provisioned to 300gb performance, basically no speed change
Code:
# diskinfo -wS /dev/da9
/dev/da9
        512             # sectorsize
        300066342912    # mediasize in bytes (279G)
        586067076       # mediasize in sectors
        4096            # stripesize
        0               # stripeoffset
        36480           # Cylinders according to firmware.
        255             # Heads according to firmware.
        63              # Sectors according to firmware.
        TOSHIBA PX04SVB040      # Disk descr.
        77XxxxxxT43E    # Disk ident.
        Yes             # TRIM/UNMAP support
        0               # Rotation rate in RPM
        Not_Zoned       # Zone Mode

Synchronous random writes:
         0.5 kbytes:     78.1 usec/IO =      6.3 Mbytes/s
           1 kbytes:     81.7 usec/IO =     11.9 Mbytes/s
           2 kbytes:     77.1 usec/IO =     25.3 Mbytes/s
           4 kbytes:     89.0 usec/IO =     43.9 Mbytes/s
           8 kbytes:     92.1 usec/IO =     84.8 Mbytes/s
          16 kbytes:    111.0 usec/IO =    140.7 Mbytes/s
          32 kbytes:    142.8 usec/IO =    218.8 Mbytes/s
          64 kbytes:    200.5 usec/IO =    311.7 Mbytes/s
         128 kbytes:    319.3 usec/IO =    391.4 Mbytes/s
         256 kbytes:    552.1 usec/IO =    452.8 Mbytes/s
         512 kbytes:   1018.8 usec/IO =    490.8 Mbytes/s
        1024 kbytes:   1951.1 usec/IO =    512.5 Mbytes/s
        2048 kbytes:   3809.3 usec/IO =    525.0 Mbytes/s
        4096 kbytes:   7532.6 usec/IO =    531.0 Mbytes/s
        8192 kbytes:  14977.1 usec/IO =    534.1 Mbytes/s
 
Last edited:

ee21

Dabbler
Joined
Nov 2, 2020
Messages
33
Intel P4801X 100GB running in full PCIe 3.0 x4 link width:

Code:
root@truenas[~]# smartctl -a /dev/nvme0
smartctl 7.1 2019-12-30 r5022 [FreeBSD 12.2-RELEASE-p2 amd64] (local build)
Copyright (C) 2002-19, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Number:                       INTEL SSDPEL1K100GA
Serial Number:                      PHKM926000AU100D
Firmware Version:                   E2010450
PCI Vendor/Subsystem ID:            0x8086
IEEE OUI Identifier:                0x5cd2e4
Controller ID:                      0
Number of Namespaces:               1
Namespace 1 Size/Capacity:          100,030,242,816 [100 GB]
Namespace 1 Formatted LBA Size:     512
Namespace 1 IEEE EUI-64:            5cd2e4 016ef20100
Local Time is:                      Wed Jan 13 22:35:59 2021 PST
Firmware Updates (0x02):            1 Slot
Optional Admin Commands (0x0007):   Security Format Frmw_DL
Optional NVM Commands (0x0006):     Wr_Unc DS_Mngmt
Maximum Data Transfer Size:         32 Pages

Supported Power States
St Op     Max   Active     Idle   RL RT WL WT  Ent_Lat  Ex_Lat
 0 +    10.00W       -        -    0  0  0  0        0       0

Supported LBA Sizes (NSID 0x1)
Id Fmt  Data  Metadt  Rel_Perf
 0 +     512       0         2
 1 -     512       8         2
 2 -     512      16         2
 3 -    4096       0         0
 4 -    4096       8         0
 5 -    4096      64         0
 6 -    4096     128         0

=== START OF SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

SMART/Health Information (NVMe Log 0x02)
Critical Warning:                   0x00
Temperature:                        53 Celsius
Available Spare:                    100%
Available Spare Threshold:          0%
Percentage Used:                    0%
Data Units Read:                    500 [256 MB]
Data Units Written:                 6,739,564 [3.45 TB]
Host Read Commands:                 11,886
Host Write Commands:                122,234,673
Controller Busy Time:               143
Power Cycles:                       16
Power On Hours:                     775
Unsafe Shutdowns:                   3
Media and Data Integrity Errors:    0
Error Information Log Entries:      0



Code:
root@truenas[~]# diskinfo -wS /dev/nvd0
/dev/nvd0
        512             # sectorsize
        100030242816    # mediasize in bytes (93G)
        195371568       # mediasize in sectors
        0               # stripesize
        0               # stripeoffset
        INTEL SSDPEL1K100GA     # Disk descr.
        PHKM926000AU100D        # Disk ident.
        Yes             # TRIM/UNMAP support
        0               # Rotation rate in RPM

Synchronous random writes:
         0.5 kbytes:     14.9 usec/IO =     32.9 Mbytes/s
           1 kbytes:     15.1 usec/IO =     64.8 Mbytes/s
           2 kbytes:     15.4 usec/IO =    127.2 Mbytes/s
           4 kbytes:     12.8 usec/IO =    304.3 Mbytes/s
           8 kbytes:     16.9 usec/IO =    462.9 Mbytes/s
          16 kbytes:     25.1 usec/IO =    623.3 Mbytes/s
          32 kbytes:     40.1 usec/IO =    780.3 Mbytes/s
          64 kbytes:     69.6 usec/IO =    897.4 Mbytes/s
         128 kbytes:    129.0 usec/IO =    969.3 Mbytes/s
         256 kbytes:    245.4 usec/IO =   1018.7 Mbytes/s
         512 kbytes:    478.5 usec/IO =   1044.9 Mbytes/s
        1024 kbytes:    946.1 usec/IO =   1056.9 Mbytes/s
        2048 kbytes:   1877.3 usec/IO =   1065.3 Mbytes/s
        4096 kbytes:   3747.6 usec/IO =   1067.4 Mbytes/s
        8192 kbytes:   7478.6 usec/IO =   1069.7 Mbytes/s
 

Constantin

Vampire Pig
Joined
May 19, 2017
Messages
1,828
That’s a much better result than what I was getting with a stock p4801x in its default condition on the low end. At 0.5kb/s throughout here was only 17Mb/s. For larger record sizes, the results are similar, so that suggests something odd is a bottleneck.

Have you made any changes from the default settings?
 

ee21

Dabbler
Joined
Nov 2, 2020
Messages
33
None. As mentioned I'm running in full PCIe 3.0 x4 link mode though, my Intel Atom board was only x2, as were both M.2 slots on my new X470D4U (well 3.0 x2, and 2.0 x4 respectively) which I was surprised to find out, I ended up buying a NVMe riser card that plugged into a standard PCIe x4 expansion slot, which I made sure had the full x4 link allocated to it.

Also, seriously doubt this has any impact on the test, but I did add some aluminum heatsinks to the NAND and controller chip.
 

Constantin

Vampire Pig
Joined
May 19, 2017
Messages
1,828
I found that adding heat sinks actually increased the temperature of the optane stick.

since the discrepancy is at the low end, I doubt it has to do with pcie lanes (which are allegedly 3.0x4 per SM) but rather something firmware related. I’ll check on that.
 

ee21

Dabbler
Joined
Nov 2, 2020
Messages
33
Based on the results below, when manually creating a partition on the drive to use for an SLOG (so I can overprovision the drive), would it be best to align to around 128k? This seems like the sweet spot for my drive, with the highest MB/s and lowest latency..

I found this command posted on another thread to do this, but they suggested (arbitrarily) using -a 1m:
gpart create -s GPT nvd0
gpart add -t freebsd-zfs -a 1m -l sloga -s 88G nvd0

zpool add TEST1 log nvd0p1


Code:
Synchronous random writes:
         0.5 kbytes:     14.9 usec/IO =     32.9 Mbytes/s
           1 kbytes:     15.1 usec/IO =     64.8 Mbytes/s
           2 kbytes:     15.4 usec/IO =    127.2 Mbytes/s
           4 kbytes:     12.8 usec/IO =    304.3 Mbytes/s
           8 kbytes:     16.9 usec/IO =    462.9 Mbytes/s
          16 kbytes:     25.1 usec/IO =    623.3 Mbytes/s
          32 kbytes:     40.1 usec/IO =    780.3 Mbytes/s
          64 kbytes:     69.6 usec/IO =    897.4 Mbytes/s
         128 kbytes:    129.0 usec/IO =    969.3 Mbytes/s
         256 kbytes:    245.4 usec/IO =   1018.7 Mbytes/s
         512 kbytes:    478.5 usec/IO =   1044.9 Mbytes/s
        1024 kbytes:    946.1 usec/IO =   1056.9 Mbytes/s
        2048 kbytes:   1877.3 usec/IO =   1065.3 Mbytes/s
        4096 kbytes:   3747.6 usec/IO =   1067.4 Mbytes/s
        8192 kbytes:   7478.6 usec/IO =   1069.7 Mbytes/s
 

Herr_Merlin

Patron
Joined
Oct 25, 2019
Messages
200
Did some testing:

ZeusIOPS STEC S842E800M2
Code:
# diskinfo -wS /dev/da5
/dev/da5
        512             # sectorsize
        800166076416    # mediasize in bytes (745G)
        1562824368      # mediasize in sectors
        0               # stripesize
        0               # stripeoffset
        97281           # Cylinders according to firmware.
        255             # Heads according to firmware.
        63              # Sectors according to firmware.
        STEC S842E800M2 # Disk descr.
        STM00019F50C    # Disk ident.
        id1,enc@n300705b00e019607/type@0/slot@10        # Physical path
        No              # TRIM/UNMAP support
        0               # Rotation rate in RPM
        Not_Zoned       # Zone Mode

Synchronous random writes:
         0.5 kbytes:     74.8 usec/IO =      6.5 Mbytes/s
           1 kbytes:     89.0 usec/IO =     11.0 Mbytes/s
           2 kbytes:     89.8 usec/IO =     21.8 Mbytes/s
           4 kbytes:     85.5 usec/IO =     45.7 Mbytes/s
           8 kbytes:     89.5 usec/IO =     87.3 Mbytes/s
          16 kbytes:    111.8 usec/IO =    139.7 Mbytes/s
          32 kbytes:    147.4 usec/IO =    212.1 Mbytes/s
          64 kbytes:    222.5 usec/IO =    280.9 Mbytes/s
         128 kbytes:    592.8 usec/IO =    210.9 Mbytes/s
         256 kbytes:    938.1 usec/IO =    266.5 Mbytes/s
         512 kbytes:   1624.5 usec/IO =    307.8 Mbytes/s
        1024 kbytes:   2901.1 usec/IO =    344.7 Mbytes/s
        2048 kbytes:   7006.5 usec/IO =    285.5 Mbytes/s
        4096 kbytes:  14221.6 usec/IO =    281.3 Mbytes/s
        8192 kbytes:  32827.3 usec/IO =    243.7 Mbytes/s

INTEL SSDPELKX010T8
Code:
#diskinfo -wS /dev/nvd0
/dev/nvd0
        512             # sectorsize
        1000204886016   # mediasize in bytes (932G)
        1953525168      # mediasize in sectors
        131072          # stripesize
        0               # stripeoffset
        INTEL SSDPELKX010T8     # Disk descr.
        BTLJ91310FDP1P0I        # Disk ident.
        Yes             # TRIM/UNMAP support
        0               # Rotation rate in RPM

Synchronous random writes:
         0.5 kbytes:     18.1 usec/IO =     27.0 Mbytes/s
           1 kbytes:     18.2 usec/IO =     53.7 Mbytes/s
           2 kbytes:     18.1 usec/IO =    108.1 Mbytes/s
           4 kbytes:     17.3 usec/IO =    225.2 Mbytes/s
           8 kbytes:     21.1 usec/IO =    371.0 Mbytes/s
          16 kbytes:     24.9 usec/IO =    628.2 Mbytes/s
          32 kbytes:     34.3 usec/IO =    910.6 Mbytes/s
          64 kbytes:     66.8 usec/IO =    935.7 Mbytes/s
         128 kbytes:    143.7 usec/IO =    869.7 Mbytes/s
         256 kbytes:    270.9 usec/IO =    922.8 Mbytes/s
         512 kbytes:    562.7 usec/IO =    888.6 Mbytes/s
        1024 kbytes:   1068.1 usec/IO =    936.3 Mbytes/s
        2048 kbytes:   2245.7 usec/IO =    890.6 Mbytes/s
        4096 kbytes:   4329.1 usec/IO =    924.0 Mbytes/s
        8192 kbytes:   8753.5 usec/IO =    913.9 Mbytes/s

Randian RMS-200
Code:
# diskinfo -wS /dev/nvd1
/dev/nvd1
        512             # sectorsize
        8581545984      # mediasize in bytes (8.0G)
        16760832        # mediasize in sectors
        0               # stripesize
        0               # stripeoffset
        RMS-200         # Disk descr.
        0032539         # Disk ident.
        Yes             # TRIM/UNMAP support
        0               # Rotation rate in RPM

Synchronous random writes:
         0.5 kbytes:     28.1 usec/IO =     17.4 Mbytes/s
           1 kbytes:     28.1 usec/IO =     34.8 Mbytes/s
           2 kbytes:     28.1 usec/IO =     69.4 Mbytes/s
           4 kbytes:     30.8 usec/IO =    126.8 Mbytes/s
           8 kbytes:     36.0 usec/IO =    216.8 Mbytes/s
          16 kbytes:     43.6 usec/IO =    358.1 Mbytes/s
          32 kbytes:     48.5 usec/IO =    644.1 Mbytes/s
          64 kbytes:     58.7 usec/IO =   1064.0 Mbytes/s
         128 kbytes:     97.9 usec/IO =   1276.6 Mbytes/s
         256 kbytes:    108.5 usec/IO =   2304.2 Mbytes/s
         512 kbytes:    160.2 usec/IO =   3120.3 Mbytes/s
        1024 kbytes:    283.6 usec/IO =   3526.6 Mbytes/s
        2048 kbytes:    503.3 usec/IO =   3973.4 Mbytes/s
        4096 kbytes:    950.1 usec/IO =   4210.2 Mbytes/s
        8192 kbytes:   1752.3 usec/IO =   4565.6 Mbytes/s
 

Herr_Merlin

Patron
Joined
Oct 25, 2019
Messages
200
And some more:

ZeusRAM
Code:
diskinfo -wS /dev/da2
/dev/da2
        512             # sectorsize
        8000000000      # mediasize in bytes (7.5G)
        15625000        # mediasize in sectors
        0               # stripesize
        0               # stripeoffset
        1914            # Cylinders according to firmware.
        255             # Heads according to firmware.
        32              # Sectors according to firmware.
        STEC ZeusRAM    # Disk descr.
        STM00016901B    # Disk ident.
        No              # TRIM/UNMAP support
        0               # Rotation rate in RPM
        Not_Zoned       # Zone Mode

Synchronous random writes:
         0.5 kbytes:    145.4 usec/IO =      3.4 Mbytes/s
           1 kbytes:    145.0 usec/IO =      6.7 Mbytes/s
           2 kbytes:    145.1 usec/IO =     13.5 Mbytes/s
           4 kbytes:    149.4 usec/IO =     26.1 Mbytes/s
           8 kbytes:    158.4 usec/IO =     49.3 Mbytes/s
          16 kbytes:    176.3 usec/IO =     88.6 Mbytes/s
          32 kbytes:    212.0 usec/IO =    147.4 Mbytes/s
          64 kbytes:    287.9 usec/IO =    217.1 Mbytes/s
         128 kbytes:    425.9 usec/IO =    293.5 Mbytes/s
         256 kbytes:    721.6 usec/IO =    346.4 Mbytes/s
         512 kbytes:   1314.2 usec/IO =    380.5 Mbytes/s
        1024 kbytes:   2496.1 usec/IO =    400.6 Mbytes/s
        2048 kbytes:   4878.8 usec/IO =    409.9 Mbytes/s
        4096 kbytes:   9713.1 usec/IO =    411.8 Mbytes/s
        8192 kbytes:  19240.1 usec/IO =    415.8 Mbytes/s


HGST HUSMM1680ASS200
Code:
diskinfo -wS /dev/da6
/dev/da6
        512             # sectorsize
        800166076416    # mediasize in bytes (745G)
        1562824368      # mediasize in sectors
        4096            # stripesize
        0               # stripeoffset
        191522          # Cylinders according to firmware.
        255             # Heads according to firmware.
        32              # Sectors according to firmware.
        HGST HUSMM1680ASS200    # Disk descr.
        0RX1EZYA        # Disk ident.
        Yes             # TRIM/UNMAP support
        0               # Rotation rate in RPM
        Not_Zoned       # Zone Mode

Synchronous random writes:
         0.5 kbytes:    225.9 usec/IO =      2.2 Mbytes/s
           1 kbytes:    178.4 usec/IO =      5.5 Mbytes/s
           2 kbytes:    179.0 usec/IO =     10.9 Mbytes/s
           4 kbytes:    179.4 usec/IO =     21.8 Mbytes/s
           8 kbytes:    196.4 usec/IO =     39.8 Mbytes/s
          16 kbytes:    214.2 usec/IO =     72.9 Mbytes/s
          32 kbytes:    245.6 usec/IO =    127.2 Mbytes/s
          64 kbytes:    311.7 usec/IO =    200.5 Mbytes/s
         128 kbytes:    430.4 usec/IO =    290.4 Mbytes/s
         256 kbytes:    681.7 usec/IO =    366.7 Mbytes/s
         512 kbytes:   1164.9 usec/IO =    429.2 Mbytes/s
        1024 kbytes:   2176.2 usec/IO =    459.5 Mbytes/s
        2048 kbytes:   4109.0 usec/IO =    486.7 Mbytes/s
        4096 kbytes:   8139.7 usec/IO =    491.4 Mbytes/s
        8192 kbytes:  16079.4 usec/IO =    497.5 Mbytes/s


For the Samsung SAMSUNG MZILS480HEGR/007 I get:
iskinfo -wS /dev/da4
diskinfo: /dev/da4: Operation not permitted
Any idea?
 

CLEsportsfan

Dabbler
Joined
Nov 23, 2020
Messages
20
Impressive results out of the Intel P4511 drive there.
I'm surprised my P3700 out performs it from a few posts up since it's 4 years older. Are only some of the P45xx series designed to be faster then?

That Radian RMS-200.... wow!
 

HoneyBadger

actually does care
Administrator
Moderator
iXsystems
Joined
Feb 6, 2014
Messages
5,110
I'm surprised my P3700 out performs it from a few posts up since it's 4 years older. Are only some of the P45xx series designed to be faster then?

It's about the intended purpose - the P3700 was designed to be a "write-intensive" drive, whereas the P45xx is more towards the "mixed-use" - you've also got a bigger drive (1.6T vs 1T) with much more overprovisioning.

That Radian RMS-200.... wow!

The NVRAM cards are stupidly fast at the top end, but it looks as if fast NAND is able to beat them in small blocks (4K/8K/16K) now. Not sure if this is a factor of controller overhead or firmware design, but as mentioned previously the "unlimited endurance" is a massive win for them in the SLOG use case - a few pages back there was someone with an RMS-200 that was sitting at 115PB lifetime writes - and that was in August.

 

Herr_Merlin

Patron
Joined
Oct 25, 2019
Messages
200
You don't need to forget that 0815 NAND with PLP might be enough for 99% of the home systems postet here.
I do use those drives at work in production system - there is a another level of uptime and reliability required and thus those NV devices came in handy.
Seeing that those INTEL SSDPELKX010T8 are about 300€ it would make more sense to buy optane P4x 100GB as those are about 140€. Anyways nearly all of the hardware we had in stock and those NV devices came quite cheap.
Workload depends per system and is only a few PB per year. Thus even an optane would have life some time. But those NV devices came at a fraction of the price.
 
Last edited:
Top