Resource icon

SLOG benchmarking and finding the best SLOG

Overview Reviews (1) Discussion
HoneyBadger

HoneyBadger

actually does care
Administrator
Moderator
iXsystems
Joined
Feb 6, 2014
Messages
5,112
ggiinnoo said:
Intel optane 900P 480GB:

Code:
           4 kbytes:     27.2 usec/IO =    143.7 Mbytes/s
           8 kbytes:     27.5 usec/IO =    283.8 Mbytes/s

           4 kbytes:     25.8 usec/IO =    151.6 Mbytes/s
           8 kbytes:     27.7 usec/IO =    281.7 Mbytes/s

           4 kbytes:     25.7 usec/IO =    152.2 Mbytes/s
           8 kbytes:     27.4 usec/IO =    284.7 Mbytes/s
Click to expand...

Feels like you've still got the handbrake on based on comparison to some other small recordsize speeds in the thread; let's see if we can get you going faster. PCIe devices are very sensitive to power-saving settings such as dynamic processor clocking and PCIe link state power management (ASPM) - check your BIOS and see if you can adjust or disable these power-saving features. That should improve your 4K/8K speeds, but at the cost of additional power consumption.
 
G

ggiinnoo

Dabbler
Joined
Sep 25, 2022
Messages
24
HoneyBadger said:
Feels like you've still got the handbrake on based on comparison to some other small recordsize speeds in the thread; let's see if we can get you going faster. PCIe devices are very sensitive to power-saving settings such as dynamic processor clocking and PCIe link state power management (ASPM) - check your BIOS and see if you can adjust or disable these power-saving features. That should improve your 4K/8K speeds, but at the cost of additional power consumption.
Click to expand...

I checked the bios for an option like this, and found the following:
`Platform Power Management` this is set to disabled, wich is good (?)

`ErP` is also off. And saw no futher power options.
Just so you know, i got a gigabyte Z370 AORUS GAMING 7-OP combined with a intel 8086K ( these were 'leftovers's' from my last build )
 
HoneyBadger

HoneyBadger

actually does care
Administrator
Moderator
iXsystems
Joined
Feb 6, 2014
Messages
5,112
Happy 2023, everyone!

First contestant of the new year is an HP-branded HGST Ultrastar SSD800MH "Write Intensive" unit, model number HUSMH8020ASS200 from the factory.

Code:
        512             # sectorsize
        200049647616    # mediasize in bytes (186G)
        390721968       # mediasize in sectors
        4096            # stripesize
        0               # stripeoffset
        24321           # Cylinders according to firmware.
        255             # Heads according to firmware.
        63              # Sectors according to firmware.
        HP EO0200JDVFA  # Disk descr.
        redacted        # Disk ident.
        mps0            # Attachment
        Yes             # TRIM/UNMAP support
        0               # Rotation rate in RPM
        Not_Zoned       # Zone Mode

Synchronous random writes:
         0.5 kbytes:     88.8 usec/IO =      5.5 Mbytes/s
           1 kbytes:     89.9 usec/IO =     10.9 Mbytes/s
           2 kbytes:     90.2 usec/IO =     21.7 Mbytes/s
           4 kbytes:     92.4 usec/IO =     42.3 Mbytes/s
           8 kbytes:    106.1 usec/IO =     73.6 Mbytes/s
          16 kbytes:    131.0 usec/IO =    119.2 Mbytes/s
          32 kbytes:    167.6 usec/IO =    186.5 Mbytes/s
          64 kbytes:    232.5 usec/IO =    268.8 Mbytes/s
         128 kbytes:    353.2 usec/IO =    353.9 Mbytes/s
         256 kbytes:    611.3 usec/IO =    408.9 Mbytes/s
         512 kbytes:   1109.8 usec/IO =    450.5 Mbytes/s
        1024 kbytes:   2072.9 usec/IO =    482.4 Mbytes/s
        2048 kbytes:   4161.9 usec/IO =    480.5 Mbytes/s
        4096 kbytes:   8064.5 usec/IO =    496.0 Mbytes/s
        8192 kbytes:  16041.7 usec/IO =    498.7 Mbytes/s


They're a bit slow for an SLOG in 2023, but with 25 DWPD rated write endurance (9.1PBW) I don't think I'd be at risk of burning these out any time soon if I were to put them into use as a special vdev.
 
mpyusko

mpyusko

Dabbler
Joined
Jul 5, 2019
Messages
49

Intel® Optane™ SSD P1600X Series (118GB, M.2 80mm PCIe 3.0 x4, 3D XPoint™) - Product Specifications | Intel

Intel® Optane™ SSD P1600X Series (118GB, M.2 80mm PCIe 3.0 x4, 3D XPoint™) quick reference with specifications, features, and technologies.
www.intel.com www.intel.com

Code:
root@cygnus[~]# smartctl -a /dev/nvme0
smartctl 7.2 2021-09-14 r5236 [FreeBSD 13.1-RELEASE-p7 amd64] (local build)
Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Number:                       INTEL SSDPEK1A118GA
Serial Number:                      BTOC14120VGD118B
Firmware Version:                   U5110550
PCI Vendor/Subsystem ID:            0x8086
IEEE OUI Identifier:                0x5cd2e4
Controller ID:                      0
NVMe Version:                       1.1
Number of Namespaces:               1
Namespace 1 Size/Capacity:          118,410,444,800 [118 GB]
Namespace 1 Formatted LBA Size:     512
Namespace 1 IEEE EUI-64:            5cd2e4 0694af0100
Local Time is:                      Mon Mar 20 04:14:38 2023 EDT
Firmware Updates (0x02):            1 Slot
Optional Admin Commands (0x0016):   Format Frmw_DL Self_Test
Optional NVM Commands (0x0056):     Wr_Unc DS_Mngmt Sav/Sel_Feat Timestmp
Log Page Attributes (0x03):         S/H_per_NS Cmd_Eff_Lg
Maximum Data Transfer Size:         32 Pages
Warning  Comp. Temp. Threshold:     70 Celsius
Critical Comp. Temp. Threshold:     78 Celsius

Supported Power States
St Op     Max   Active     Idle   RL RT WL WT  Ent_Lat  Ex_Lat
 0 +     5.50W       -        -    0  0  0  0     1000    4000
 1 +     4.60W       -        -    0  1  0  1     1000    4000
 2 +     3.30W       -        -    0  2  0  2     1000    4000

Supported LBA Sizes (NSID 0x1)
Id Fmt  Data  Metadt  Rel_Perf
 0 +     512       0         2
 1 -    4096       0         0

=== START OF SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

SMART/Health Information (NVMe Log 0x02)
Critical Warning:                   0x00
Temperature:                        41 Celsius
Available Spare:                    100%
Available Spare Threshold:          0%
Percentage Used:                    0%
Data Units Read:                    79 [40.4 MB]
Data Units Written:                 343,491 [175 GB]
Host Read Commands:                 839
Host Write Commands:                4,673,880
Controller Busy Time:               2
Power Cycles:                       2
Power On Hours:                     39
Unsafe Shutdowns:                   0
Media and Data Integrity Errors:    0
Error Information Log Entries:      0
Warning  Comp. Temperature Time:    0
Critical Comp. Temperature Time:    0

Error Information (NVMe Log 0x01, 16 of 64 entries)
No Errors Logged

root@cygnus[~]#

Code:
root@cygnus[~]# diskinfo -wS /dev/nvd0
diskinfo: /dev/nvd0: Operation not permitted
root@cygnus[~]# diskinfo -wS /dev/nvme0
diskinfo: /dev/nvme0: ioctl(DIOCGMEDIASIZE) failed, probably not a disk.
 
awasb

awasb

Patron
Joined
Jan 11, 2021
Messages
415
R

rdfreak

Dabbler
Joined
May 22, 2019
Messages
12
I'm trying to run the SLOG benchmark on a Samsung PM9A3 U.2 1.92 TB drive. I get the following error when I execute diskinfo:

Code:
Synchronous random writes:
        0.5 kbytes: diskinfo: Flush error: Inappropriate ioctl for device


The unusual thing about the drive is that I created 2 NVMe namespaces on it - installed TrueNAS Core on one of them, so that I wouldn't have to overwrite my existing TrueNAS Scale installation on the other, primary drive. The second NVMe namespace was created to run the benchmark. Neither namespace is shared.

How can I address the error and run the benchmark?
 
ZPrime

ZPrime

Cadet
Joined
Sep 19, 2019
Messages
6
Apologies for what seems like a dumb question here, but... the diskinfo command doesn't seem to exist on a standard installation of TrueNAS Scale (I'm on SCALE-22.12.2).

I know we're not supposed to use apt so I can't just add it myself... What's the proper path of action here?
 
NickF

NickF

Guru
Joined
Jun 12, 2014
Messages
763
Just because I haven't gone through the whole thread here...Is there a Google Sheet or equivelant that has a "TLDR" of peoples results? If not, once can be arranged.
 
NickF

NickF

Guru
Joined
Jun 12, 2014
Messages
763
IMG_1982.jpeg
Whoever is in the lead, I’m coming for you
 
NickF

NickF

Guru
Joined
Jun 12, 2014
Messages
763
Alright so in ascending order of awesomeness
First is a Samsung PM1721A:

Code:
=== START OF INFORMATION SECTION ===
Model Number:                       SAMSUNG MZWLL1T6HEHP-00003
Serial Number:                      S3HDNX0KB01248
Firmware Version:                   GPNA5B3Q
PCI Vendor/Subsystem ID:            0x144d
IEEE OUI Identifier:                0x002538
Total NVM Capacity:                 1,600,321,314,816 [1.60 TB]
Unallocated NVM Capacity:           0
Controller ID:                      33
NVMe Version:                       1.2
Number of Namespaces:               32
Local Time is:                      Tue May 16 21:29:58 2023 PDT
Firmware Updates (0x17):            3 Slots, Slot 1 R/O, no Reset required
Optional Admin Commands (0x000e):   Format Frmw_DL NS_Mngmt
Optional NVM Commands (0x003e):     Wr_Unc DS_Mngmt Wr_Zero Sav/Sel_Feat Resv
Log Page Attributes (0x02):         Cmd_Eff_Lg
Maximum Data Transfer Size:         32 Pages
Warning  Comp. Temp. Threshold:     78 Celsius
Critical Comp. Temp. Threshold:     85 Celsius

Supported Power States

Code:
root@truenas[~]# diskinfo -wS /dev/nvd0
/dev/nvd0
        512             # sectorsize
        1600321314816   # mediasize in bytes (1.5T)
        3125627568      # mediasize in sectors
        0               # stripesize
        0               # stripeoffset
        SAMSUNG MZWLL1T6HEHP-00003      # Disk descr.
        S3HDNX0KB01248  # Disk ident.
        nvme0           # Attachment
        Yes             # TRIM/UNMAP support
        0               # Rotation rate in RPM

Synchronous random writes:
         0.5 kbytes:     18.7 usec/IO =     26.1 Mbytes/s
           1 kbytes:     20.7 usec/IO =     47.2 Mbytes/s
           2 kbytes:     20.5 usec/IO =     95.4 Mbytes/s
           4 kbytes:     22.2 usec/IO =    175.9 Mbytes/s
           8 kbytes:     24.4 usec/IO =    320.1 Mbytes/s
          16 kbytes:     31.3 usec/IO =    499.7 Mbytes/s
          32 kbytes:     53.5 usec/IO =    583.9 Mbytes/s
          64 kbytes:    112.2 usec/IO =    556.8 Mbytes/s
         128 kbytes:    203.3 usec/IO =    614.9 Mbytes/s
         256 kbytes:    248.1 usec/IO =   1007.6 Mbytes/s
         512 kbytes:    387.5 usec/IO =   1290.5 Mbytes/s
        1024 kbytes:    695.7 usec/IO =   1437.4 Mbytes/s
        2048 kbytes:   1277.5 usec/IO =   1565.5 Mbytes/s
        4096 kbytes:   2472.8 usec/IO =   1617.6 Mbytes/s
        8192 kbytes:   4891.5 usec/IO =   1635.5 Mbytes/s
root@truenas[~]#


Next is Optane 900p 280GB
Code:
=== START OF INFORMATION SECTION ===
Model Number:                       INTEL SSDPE21D280GA
Serial Number:                      PHM2746000G3280AGN
Firmware Version:                   E2010603
PCI Vendor/Subsystem ID:            0x8086
IEEE OUI Identifier:                0x5cd2e4
Controller ID:                      0
NVMe Version:                       <1.2
Number of Namespaces:               1
Namespace 1 Size/Capacity:          280,065,171,456 [280 GB]
Namespace 1 Formatted LBA Size:     512
Local Time is:                      Tue May 16 21:32:52 2023 PDT
Firmware Updates (0x02):            1 Slot
Optional Admin Commands (0x0007):   Security Format Frmw_DL
Optional NVM Commands (0x0006):     Wr_Unc DS_Mngmt
Log Page Attributes (0x0a):         Cmd_Eff_Lg Telmtry_Lg
Maximum Data Transfer Size:         32 Pages


Which looks like
Code:
root@truenas[~]# diskinfo -wS /dev/nvd2
/dev/nvd2
        512             # sectorsize
        280065171456    # mediasize in bytes (261G)
        547002288       # mediasize in sectors
        0               # stripesize
        0               # stripeoffset
        INTEL SSDPE21D280GA     # Disk descr.
        PHM2746000G3280AGN      # Disk ident.
        nvme2           # Attachment
        Yes             # TRIM/UNMAP support
        0               # Rotation rate in RPM

Synchronous random writes:
         0.5 kbytes:     24.3 usec/IO =     20.1 Mbytes/s
           1 kbytes:     24.5 usec/IO =     39.8 Mbytes/s
           2 kbytes:     24.5 usec/IO =     79.6 Mbytes/s
           4 kbytes:     19.7 usec/IO =    198.5 Mbytes/s
           8 kbytes:     22.8 usec/IO =    343.3 Mbytes/s
          16 kbytes:     30.1 usec/IO =    518.5 Mbytes/s
          32 kbytes:     45.1 usec/IO =    693.6 Mbytes/s
          64 kbytes:     86.4 usec/IO =    723.2 Mbytes/s
         128 kbytes:    153.9 usec/IO =    812.2 Mbytes/s
         256 kbytes:    259.2 usec/IO =    964.4 Mbytes/s
         512 kbytes:    456.6 usec/IO =   1095.1 Mbytes/s
        1024 kbytes:    835.7 usec/IO =   1196.5 Mbytes/s
        2048 kbytes:   1601.9 usec/IO =   1248.5 Mbytes/s
        4096 kbytes:   3119.1 usec/IO =   1282.4 Mbytes/s
        8192 kbytes:   6066.3 usec/IO =   1318.8 Mbytes/s


And Finally an NVDIMM/PMEM Test
This is a 16GB DDR4-2400 Micon? PMEM module (ATM machine) backed by a fancy supercap.

Unfortunately SMART doesn't support this type...:wink:
Code:
root@truenas[~]# smartctl -a /dev/pmem0
smartctl 7.2 2021-09-14 r5236 [FreeBSD 13.1-RELEASE-p7 amd64] (local build)
Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org

/dev/pmem0: Unable to detect device type
Please specify device type with the -d option.

Use smartctl -h to get a usage summary

root@truenas[~]#


Drum rollllll
Code:
root@truenas[~]# diskinfo -wS /dev/pmem0
/dev/pmem0
        512             # sectorsize
        17179865088     # mediasize in bytes (16G)
        33554424        # mediasize in sectors
        0               # stripesize
        0               # stripeoffset
        PMEM region 16GB        # Disk descr.
        B70F6225EC9F760A        # Disk ident.
                        # Attachment
        No              # TRIM/UNMAP support
        0               # Rotation rate in RPM

Synchronous random writes:
         0.5 kbytes:      1.6 usec/IO =    310.8 Mbytes/s
           1 kbytes:      2.0 usec/IO =    487.7 Mbytes/s
           2 kbytes:      2.2 usec/IO =    899.3 Mbytes/s
           4 kbytes:      2.7 usec/IO =   1437.4 Mbytes/s
           8 kbytes:      3.8 usec/IO =   2046.7 Mbytes/s
          16 kbytes:      5.3 usec/IO =   2949.3 Mbytes/s
          32 kbytes:     12.1 usec/IO =   2583.6 Mbytes/s
          64 kbytes:     14.2 usec/IO =   4389.5 Mbytes/s
         128 kbytes:     21.4 usec/IO =   5851.7 Mbytes/s
         256 kbytes:     32.5 usec/IO =   7684.6 Mbytes/s
         512 kbytes:     53.7 usec/IO =   9314.6 Mbytes/s
        1024 kbytes:     99.1 usec/IO =  10087.5 Mbytes/s
        2048 kbytes:    189.0 usec/IO =  10584.3 Mbytes/s
        4096 kbytes:    361.8 usec/IO =  11056.3 Mbytes/s
        8192 kbytes:    710.4 usec/IO =  11261.5 Mbytes/s
root@truenas[~]#


I think I will hold the trophy of having the fastest SLOG on the forum for quite some time to come... :smile:
People out there are going to come in here and be look look at my Optane PMEM results! Then they are going to be sad when they see this post.

Whats sad is that in a world where NVME drive pools are faster than system memory, sync writes on NVME drive pools are gonna be slooowwwww writes vs reads, even at these insane speeds. I mean slow in relative terms of course. That microsecond latency tho lol
 
Last edited:
NickF

NickF

Guru
Joined
Jun 12, 2014
Messages
763
I STAND CORRECTED
I am only second best
I'll eat my words now..

SLOG benchmarking and finding the best SLOG

One more fun toy.... This is RMS-200 From Radiant Memory Systems === START OF SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED SMART/Health Information (NVMe Log 0x02, NSID 0xffffffff) Critical Warning: 0x00 Temperature...
www.truenas.com www.truenas.com
 
NickF

NickF

Guru
Joined
Jun 12, 2014
Messages
763
Was having a conversation with @jgreco in another thread that strayed towards a sidebar about NVME name spaces.

Seeking help with deteriorating performance of TrueNAS as a vSphere storage backend

I have a vSphere cluster that uses TrueNAS-SCALE and iSCSI as the storage backend, and usually I have 90 to 150 windows virtual machines running in this vSphere cluster. Before this problem occurred, it took me about 2 minutes to clone a 60 GB VM in vCenter.However, a few days ago, the time to...
www.truenas.com www.truenas.com

He linked me to an old thread from @HoneyBadger https://www.truenas.com/community/threads/slog-underprovisioning.38374/

Which got some wheels turning when I saw this graph from Micron:
performance-by-namespace.png


It's a fairly well known fact that NVME drives generally perform better at Queue-Depths greater than 1, and Micron's testing of creating multiple namespaces and scaling a workload out to 8 of them for whatever marketing application they used demonstrates that.

Would this be useful for a SLOG? I created 4 (4GB) namespaces on a 1.6TB SAMSUNG MZWLL1T6HEHP-00003 (PM1725A).

Man Page:

nvmecontrol(8)

man.freebsd.org man.freebsd.org

Code:
root@freenas[~]# nvmecontrol ns create -s 8192000 -c 8192000 -n 0 -L 0 -d 0 nvme1
namespace 1 created
root@freenas[~]# nvmecontrol ns create -s 8192000 -c 8192000 -n 1 -L 0 -d 0 nvme1
namespace 2 created
root@freenas[~]# nvmecontrol ns create -s 8192000 -c 8192000 -n 1 -L 0 -d 0 nvme1
namespace 3 created
root@freenas[~]# nvmecontrol ns create -s 8192000 -c 8192000 -n 1 -L 0 -d 0 nvme1
namespace 4 created
root@freenas[~]# nvmecontrol ns attach -n 2 nvme1
namespace 2 attached
root@freenas[~]# nvmecontrol ns attach -n 3 nvme1
namespace 3 attached
root@freenas[~]# nvmecontrol ns attach -n 4 nvme1


(don't ask me why 8192000= 4 Gig, I have no idea, I was just plugging numbers...)

I can see them now when I do a nvmecontrol devlist. I had to do a reboot for the UI to see them, I'd imagine it has something to do with glabel but I was too lazy to dig any deeper.
Code:
root@freenas[~]# nvmecontrol devlist
 nvme0: INTEL SSDPEK1A058GA
    nvme0ns1 (56245MB)
 nvme1: SAMSUNG MZWLL1T6HEHP-00003
    nvme1ns1 (4000MB)
    nvme1ns2 (4000MB)
    nvme1ns3 (4000MB)
    nvme1ns4 (4000MB)


1685416319666.png



Code:
root@freenas[~]# zpool list -v
NAME                                             SIZE  ALLOC   FREE  CK                                                                                                                                                                                                         POINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
freenas-boot                                     111G  3.50G   108G                                                                                                                                                                                                                 -         -     0%     3%  1.00x    ONLINE  -
  da24p2                                         112G  3.50G   108G                                                                                                                                                                                                                 -         -     0%  3.14%      -    ONLINE
test                                              87T   676G  86.3T                                                                                                                                                                                                                 -         -     0%     0%  1.00x    ONLINE  /mnt
  mirror-0                                      7.25T  56.2G  7.20T                                                                                                                                                                                                                 -         -     0%  0.75%      -    ONLINE
    gptid/4b89ccb5-fbe4-11ed-ac09-ac1f6b566eea  7.26T      -      -                                                                                                                                                                                                                 -         -      -      -      -    ONLINE
    gptid/4bf9b5a4-fbe4-11ed-ac09-ac1f6b566eea  7.26T      -      -                                                                                                                                                                                                                 -         -      -      -      -    ONLINE
  mirror-1                                      7.25T  56.3G  7.20T                                                                                                                                                                                                                 -         -     0%  0.75%      -    ONLINE
    gptid/4a55d9e3-fbe4-11ed-ac09-ac1f6b566eea  7.26T      -      -                                                                                                                                                                                                                 -         -      -      -      -    ONLINE
    gptid/4a63c5f1-fbe4-11ed-ac09-ac1f6b566eea  7.26T      -      -                                                                                                                                                                                                                 -         -      -      -      -    ONLINE
  mirror-2                                      7.25T  56.3G  7.20T                                                                                                                                                                                                                 -         -     0%  0.75%      -    ONLINE
    gptid/491da9f0-fbe4-11ed-ac09-ac1f6b566eea  7.26T      -      -                                                                                                                                                                                                                 -         -      -      -      -    ONLINE
    gptid/4be8eb99-fbe4-11ed-ac09-ac1f6b566eea  7.26T      -      -                                                                                                                                                                                                                 -         -      -      -      -    ONLINE
  mirror-3                                      7.25T  56.2G  7.20T                                                                                                                                                                                                                 -         -     0%  0.75%      -    ONLINE
    gptid/4a9f9c8b-fbe4-11ed-ac09-ac1f6b566eea  7.26T      -      -                                                                                                                                                                                                                 -         -      -      -      -    ONLINE
    gptid/4b9cf351-fbe4-11ed-ac09-ac1f6b566eea  7.26T      -      -                                                                                                                                                                                                                 -         -      -      -      -    ONLINE
  mirror-4                                      7.25T  56.1G  7.20T                                                                                                                                                                                                                 -         -     0%  0.75%      -    ONLINE
    gptid/4b7518a5-fbe4-11ed-ac09-ac1f6b566eea  7.26T      -      -                                                                                                                                                                                                                 -         -      -      -      -    ONLINE
    gptid/4b5e6c08-fbe4-11ed-ac09-ac1f6b566eea  7.26T      -      -                                                                                                                                                                                                                 -         -      -      -      -    ONLINE
  mirror-5                                      7.25T  56.2G  7.20T                                                                                                                                                                                                                 -         -     0%  0.75%      -    ONLINE
    gptid/4904fd2e-fbe4-11ed-ac09-ac1f6b566eea  7.26T      -      -                                                                                                                                                                                                                 -         -      -      -      -    ONLINE
    gptid/4ac53ef7-fbe4-11ed-ac09-ac1f6b566eea  7.26T      -      -                                                                                                                                                                                                                 -         -      -      -      -    ONLINE
  mirror-6                                      7.25T  56.4G  7.19T                                                                                                                                                                                                                 -         -     0%  0.76%      -    ONLINE
    gptid/495734c2-fbe4-11ed-ac09-ac1f6b566eea  7.26T      -      -                                                                                                                                                                                                                 -         -      -      -      -    ONLINE
    gptid/4bb00352-fbe4-11ed-ac09-ac1f6b566eea  7.26T      -      -                                                                                                                                                                                                                 -         -      -      -      -    ONLINE
  mirror-7                                      7.25T  56.4G  7.19T                                                                                                                                                                                                                 -         -     0%  0.75%      -    ONLINE
    gptid/4b6e4069-fbe4-11ed-ac09-ac1f6b566eea  7.26T      -      -                                                                                                                                                                                                                 -         -      -      -      -    ONLINE
    gptid/4b3d7786-fbe4-11ed-ac09-ac1f6b566eea  7.26T      -      -                                                                                                                                                                                                                 -         -      -      -      -    ONLINE
  mirror-8                                      7.25T  56.4G  7.19T                                                                                                                                                                                                                 -         -     0%  0.76%      -    ONLINE
    gptid/4e436d52-fbe4-11ed-ac09-ac1f6b566eea  7.26T      -      -                                                                                                                                                                                                                 -         -      -      -      -    ONLINE
    gptid/4f69d0d5-fbe4-11ed-ac09-ac1f6b566eea  7.26T      -      -                                                                                                                                                                                                                 -         -      -      -      -    ONLINE
  mirror-9                                      7.25T  56.4G  7.19T                                                                                                                                                                                                                 -         -     0%  0.75%      -    ONLINE
    gptid/4e68a61a-fbe4-11ed-ac09-ac1f6b566eea  7.26T      -      -                                                                                                                                                                                                                 -         -      -      -      -    ONLINE
    gptid/4f1cf63b-fbe4-11ed-ac09-ac1f6b566eea  7.26T      -      -                                                                                                                                                                                                                 -         -      -      -      -    ONLINE
  mirror-10                                     7.25T  56.9G  7.19T                                                                                                                                                                                                                 -         -     0%  0.76%      -    ONLINE
    gptid/4f63b958-fbe4-11ed-ac09-ac1f6b566eea  7.26T      -      -                                                                                                                                                                                                                 -         -      -      -      -    ONLINE
    gptid/4f801d45-fbe4-11ed-ac09-ac1f6b566eea  7.26T      -      -                                                                                                                                                                                                                 -         -      -      -      -    ONLINE
  mirror-11                                     7.25T  56.5G  7.19T                                                                                                                                                                                                                 -         -     0%  0.76%      -    ONLINE
    gptid/4f8ed2aa-fbe4-11ed-ac09-ac1f6b566eea  7.26T      -      -                                                                                                                                                                                                                 -         -      -      -      -    ONLINE
    gptid/4fe8597c-fbe4-11ed-ac09-ac1f6b566eea  7.26T      -      -                                                                                                                                                                                                                 -         -      -      -      -    ONLINE
logs                                                -      -      -                                                                                                                                                                                                                 -         -      -      -      -  -
  gptid/091e0445-fe98-11ed-8574-ac1f6b566eea    3.91G      0  3.75G                                                                                                                                                                                                                 -         -     0%  0.00%      -    ONLINE
  gptid/09201b82-fe98-11ed-8574-ac1f6b566eea    3.91G      0  3.75G                                                                                                                                                                                                                 -         -     0%  0.00%      -    ONLINE
  gptid/092a14ee-fe98-11ed-8574-ac1f6b566eea    3.91G      0  3.75G                                                                                                                                                                                                                 -         -     0%  0.00%      -    ONLINE
  gptid/0921a9dc-fe98-11ed-8574-ac1f6b566eea    3.91G      0  3.75G                                                                                                                                                                                                                 -         -     0%  0.00%      -    ONLINE


Super quick and dirty benching. I mounted a Samba share to my workstation (10G), and set SYNC=ALWAYS.
I am all ears on a better testing methodology also, any idea how I might use diskinfo to to test across multiple devices in a meaningful way to compare to a single NVME namespace?

Running CrystalDiskMark with the above pool layout I got the following:
1685417249543.png


I ran the same test after removing 3 of the 4 NVME Namespaces from the pool.
Code:
root@freenas[~]# zpool list -v
NAME                                             SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
freenas-boot                                     111G  3.50G   108G        -         -     0%     3%  1.00x    ONLINE  -
  da24p2                                         112G  3.50G   108G        -         -     0%  3.14%      -    ONLINE
test                                              87T   678G  86.3T        -         -     0%     0%  1.00x    ONLINE  /mnt
  mirror-0                                      7.25T  56.4G  7.19T        -         -     0%  0.75%      -    ONLINE
    gptid/4b89ccb5-fbe4-11ed-ac09-ac1f6b566eea  7.26T      -      -        -         -      -      -      -    ONLINE
    gptid/4bf9b5a4-fbe4-11ed-ac09-ac1f6b566eea  7.26T      -      -        -         -      -      -      -    ONLINE
  mirror-1                                      7.25T  56.5G  7.19T        -         -     0%  0.76%      -    ONLINE
    gptid/4a55d9e3-fbe4-11ed-ac09-ac1f6b566eea  7.26T      -      -        -         -      -      -      -    ONLINE
    gptid/4a63c5f1-fbe4-11ed-ac09-ac1f6b566eea  7.26T      -      -        -         -      -      -      -    ONLINE
  mirror-2                                      7.25T  56.4G  7.19T        -         -     0%  0.75%      -    ONLINE
    gptid/491da9f0-fbe4-11ed-ac09-ac1f6b566eea  7.26T      -      -        -         -      -      -      -    ONLINE
    gptid/4be8eb99-fbe4-11ed-ac09-ac1f6b566eea  7.26T      -      -        -         -      -      -      -    ONLINE
  mirror-3                                      7.25T  56.3G  7.20T        -         -     0%  0.75%      -    ONLINE
    gptid/4a9f9c8b-fbe4-11ed-ac09-ac1f6b566eea  7.26T      -      -        -         -      -      -      -    ONLINE
    gptid/4b9cf351-fbe4-11ed-ac09-ac1f6b566eea  7.26T      -      -        -         -      -      -      -    ONLINE
  mirror-4                                      7.25T  56.2G  7.20T        -         -     0%  0.75%      -    ONLINE
    gptid/4b7518a5-fbe4-11ed-ac09-ac1f6b566eea  7.26T      -      -        -         -      -      -      -    ONLINE
    gptid/4b5e6c08-fbe4-11ed-ac09-ac1f6b566eea  7.26T      -      -        -         -      -      -      -    ONLINE
  mirror-5                                      7.25T  56.4G  7.19T        -         -     0%  0.75%      -    ONLINE
    gptid/4904fd2e-fbe4-11ed-ac09-ac1f6b566eea  7.26T      -      -        -         -      -      -      -    ONLINE
    gptid/4ac53ef7-fbe4-11ed-ac09-ac1f6b566eea  7.26T      -      -        -         -      -      -      -    ONLINE
  mirror-6                                      7.25T  56.6G  7.19T        -         -     0%  0.76%      -    ONLINE
    gptid/495734c2-fbe4-11ed-ac09-ac1f6b566eea  7.26T      -      -        -         -      -      -      -    ONLINE
    gptid/4bb00352-fbe4-11ed-ac09-ac1f6b566eea  7.26T      -      -        -         -      -      -      -    ONLINE
  mirror-7                                      7.25T  56.5G  7.19T        -         -     0%  0.76%      -    ONLINE
    gptid/4b6e4069-fbe4-11ed-ac09-ac1f6b566eea  7.26T      -      -        -         -      -      -      -    ONLINE
    gptid/4b3d7786-fbe4-11ed-ac09-ac1f6b566eea  7.26T      -      -        -         -      -      -      -    ONLINE
  mirror-8                                      7.25T  56.6G  7.19T        -         -     0%  0.76%      -    ONLINE
    gptid/4e436d52-fbe4-11ed-ac09-ac1f6b566eea  7.26T      -      -        -         -      -      -      -    ONLINE
    gptid/4f69d0d5-fbe4-11ed-ac09-ac1f6b566eea  7.26T      -      -        -         -      -      -      -    ONLINE
  mirror-9                                      7.25T  56.6G  7.19T        -         -     0%  0.76%      -    ONLINE
    gptid/4e68a61a-fbe4-11ed-ac09-ac1f6b566eea  7.26T      -      -        -         -      -      -      -    ONLINE
    gptid/4f1cf63b-fbe4-11ed-ac09-ac1f6b566eea  7.26T      -      -        -         -      -      -      -    ONLINE
  mirror-10                                     7.25T  57.1G  7.19T        -         -     0%  0.76%      -    ONLINE
    gptid/4f63b958-fbe4-11ed-ac09-ac1f6b566eea  7.26T      -      -        -         -      -      -      -    ONLINE
    gptid/4f801d45-fbe4-11ed-ac09-ac1f6b566eea  7.26T      -      -        -         -      -      -      -    ONLINE
  mirror-11                                     7.25T  56.6G  7.19T        -         -     0%  0.76%      -    ONLINE
    gptid/4f8ed2aa-fbe4-11ed-ac09-ac1f6b566eea  7.26T      -      -        -         -      -      -      -    ONLINE
    gptid/4fe8597c-fbe4-11ed-ac09-ac1f6b566eea  7.26T      -      -        -         -      -      -      -    ONLINE
logs                                                -      -      -        -         -      -      -      -  -
  gptid/091e0445-fe98-11ed-8574-ac1f6b566eea    3.91G  3.32G   439M        -         -     0%  88.6%      -    ONLINE
root@freenas[~]#

1685417592313.png

FWIW there is currently nothing else talking to this (or running on) server:
Code:
root@freenas[~]# netstat | grep microsoft
tcp4       0   8360 10.69.40.19.microsoft- bartowski.50073        ESTABLISHED
 
Last edited:
NickF

NickF

Guru
Joined
Jun 12, 2014
Messages
763
Because I am literally insane and I am doing this at midnight...
I just repeated the steps above on a second PM1725a, now I am going to re-run the exact same test as above. Though now, I have 8 logical devices, on 2 physical devices, striped as my SLOG. Only other changed made was to re-boot the TrueNAS.

Code:
root@freenas[~]# zpool list -v
NAME                                             SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
freenas-boot                                     111G  3.50G   108G        -         -     0%     3%  1.00x    ONLINE  -
  da24p2                                         112G  3.50G   108G        -         -     0%  3.14%      -    ONLINE
test                                              87T   676G  86.3T        -         -     0%     0%  1.00x    ONLINE  /mnt
  mirror-0                                      7.25T  56.2G  7.20T        -         -     0%  0.75%      -    ONLINE
    gptid/4b89ccb5-fbe4-11ed-ac09-ac1f6b566eea  7.26T      -      -        -         -      -      -      -    ONLINE
    gptid/4bf9b5a4-fbe4-11ed-ac09-ac1f6b566eea  7.26T      -      -        -         -      -      -      -    ONLINE
  mirror-1                                      7.25T  56.3G  7.20T        -         -     0%  0.75%      -    ONLINE
    gptid/4a55d9e3-fbe4-11ed-ac09-ac1f6b566eea  7.26T      -      -        -         -      -      -      -    ONLINE
    gptid/4a63c5f1-fbe4-11ed-ac09-ac1f6b566eea  7.26T      -      -        -         -      -      -      -    ONLINE
  mirror-2                                      7.25T  56.3G  7.20T        -         -     0%  0.75%      -    ONLINE
    gptid/491da9f0-fbe4-11ed-ac09-ac1f6b566eea  7.26T      -      -        -         -      -      -      -    ONLINE
    gptid/4be8eb99-fbe4-11ed-ac09-ac1f6b566eea  7.26T      -      -        -         -      -      -      -    ONLINE
  mirror-3                                      7.25T  56.2G  7.20T        -         -     0%  0.75%      -    ONLINE
    gptid/4a9f9c8b-fbe4-11ed-ac09-ac1f6b566eea  7.26T      -      -        -         -      -      -      -    ONLINE
    gptid/4b9cf351-fbe4-11ed-ac09-ac1f6b566eea  7.26T      -      -        -         -      -      -      -    ONLINE
  mirror-4                                      7.25T  56.1G  7.20T        -         -     0%  0.75%      -    ONLINE
    gptid/4b7518a5-fbe4-11ed-ac09-ac1f6b566eea  7.26T      -      -        -         -      -      -      -    ONLINE
    gptid/4b5e6c08-fbe4-11ed-ac09-ac1f6b566eea  7.26T      -      -        -         -      -      -      -    ONLINE
  mirror-5                                      7.25T  56.2G  7.20T        -         -     0%  0.75%      -    ONLINE
    gptid/4904fd2e-fbe4-11ed-ac09-ac1f6b566eea  7.26T      -      -        -         -      -      -      -    ONLINE
    gptid/4ac53ef7-fbe4-11ed-ac09-ac1f6b566eea  7.26T      -      -        -         -      -      -      -    ONLINE
  mirror-6                                      7.25T  56.4G  7.19T        -         -     0%  0.76%      -    ONLINE
    gptid/495734c2-fbe4-11ed-ac09-ac1f6b566eea  7.26T      -      -        -         -      -      -      -    ONLINE
    gptid/4bb00352-fbe4-11ed-ac09-ac1f6b566eea  7.26T      -      -        -         -      -      -      -    ONLINE
  mirror-7                                      7.25T  56.4G  7.19T        -         -     0%  0.75%      -    ONLINE
    gptid/4b6e4069-fbe4-11ed-ac09-ac1f6b566eea  7.26T      -      -        -         -      -      -      -    ONLINE
    gptid/4b3d7786-fbe4-11ed-ac09-ac1f6b566eea  7.26T      -      -        -         -      -      -      -    ONLINE
  mirror-8                                      7.25T  56.4G  7.19T        -         -     0%  0.76%      -    ONLINE
    gptid/4e436d52-fbe4-11ed-ac09-ac1f6b566eea  7.26T      -      -        -         -      -      -      -    ONLINE
    gptid/4f69d0d5-fbe4-11ed-ac09-ac1f6b566eea  7.26T      -      -        -         -      -      -      -    ONLINE
  mirror-9                                      7.25T  56.4G  7.19T        -         -     0%  0.75%      -    ONLINE
    gptid/4e68a61a-fbe4-11ed-ac09-ac1f6b566eea  7.26T      -      -        -         -      -      -      -    ONLINE
    gptid/4f1cf63b-fbe4-11ed-ac09-ac1f6b566eea  7.26T      -      -        -         -      -      -      -    ONLINE
  mirror-10                                     7.25T  56.9G  7.19T        -         -     0%  0.76%      -    ONLINE
    gptid/4f63b958-fbe4-11ed-ac09-ac1f6b566eea  7.26T      -      -        -         -      -      -      -    ONLINE
    gptid/4f801d45-fbe4-11ed-ac09-ac1f6b566eea  7.26T      -      -        -         -      -      -      -    ONLINE
  mirror-11                                     7.25T  56.5G  7.19T        -         -     0%  0.76%      -    ONLINE
    gptid/4f8ed2aa-fbe4-11ed-ac09-ac1f6b566eea  7.26T      -      -        -         -      -      -      -    ONLINE
    gptid/4fe8597c-fbe4-11ed-ac09-ac1f6b566eea  7.26T      -      -        -         -      -      -      -    ONLINE
logs                                                -      -      -        -         -      -      -      -  -
  gptid/091e0445-fe98-11ed-8574-ac1f6b566eea    3.91G    32K  3.75G        -         -     0%  0.00%      -    ONLINE
  gptid/264b9b6e-fe9f-11ed-9cd5-ac1f6b566eea    3.91G      0  3.75G        -         -     0%  0.00%      -    ONLINE
  gptid/2677ee8c-fe9f-11ed-9cd5-ac1f6b566eea    3.91G      0  3.75G        -         -     0%  0.00%      -    ONLINE
  gptid/26795cef-fe9f-11ed-9cd5-ac1f6b566eea    3.91G      0  3.75G        -         -     0%  0.00%      -    ONLINE
  gptid/267ac10d-fe9f-11ed-9cd5-ac1f6b566eea    3.91G      0  3.75G        -         -     0%  0.00%      -    ONLINE
  gptid/267e3efb-fe9f-11ed-9cd5-ac1f6b566eea    3.91G      0  3.75G        -         -     0%  0.00%      -    ONLINE
root@freenas[~]#


1685419933072.png


I think we'll call this margin of error and that the performance does not appear to scale, positively or negatively, if you add a second physical device. In previous testing above, Random 4k write performance at Queue Depth 32 (and QD1) (which is probably what matters here), did scale.

Obviously this test itself probably doesn't "mean" anything. However, this testing may prove that further investigation is likely worth it. Does adding additional NVME namespaces to a single NVME device increase performance?

EDIT: I just noticed that only 6 devices are showing up. This last test is invalid. I'll investigate further tomorrow.
 
Last edited:
NickF

NickF

Guru
Joined
Jun 12, 2014
Messages
763
FWIW this is the same test with the PMEM/NVDIMM module
Code:
root@freenas[~]# zpool list -v
NAME                                             SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
freenas-boot                                     111G  3.50G   108G        -         -     0%     3%  1.00x    ONLINE  -
  da24p2                                         112G  3.50G   108G        -         -     0%  3.14%      -    ONLINE
test                                              87T   676G  86.3T        -         -     0%     0%  1.00x    ONLINE  /mnt
  mirror-0                                      7.25T  56.2G  7.20T        -         -     0%  0.75%      -    ONLINE
    gptid/4b89ccb5-fbe4-11ed-ac09-ac1f6b566eea  7.26T      -      -        -         -      -      -      -    ONLINE
    gptid/4bf9b5a4-fbe4-11ed-ac09-ac1f6b566eea  7.26T      -      -        -         -      -      -      -    ONLINE
  mirror-1                                      7.25T  56.3G  7.20T        -         -     0%  0.75%      -    ONLINE
    gptid/4a55d9e3-fbe4-11ed-ac09-ac1f6b566eea  7.26T      -      -        -         -      -      -      -    ONLINE
    gptid/4a63c5f1-fbe4-11ed-ac09-ac1f6b566eea  7.26T      -      -        -         -      -      -      -    ONLINE
  mirror-2                                      7.25T  56.3G  7.20T        -         -     0%  0.75%      -    ONLINE
    gptid/491da9f0-fbe4-11ed-ac09-ac1f6b566eea  7.26T      -      -        -         -      -      -      -    ONLINE
    gptid/4be8eb99-fbe4-11ed-ac09-ac1f6b566eea  7.26T      -      -        -         -      -      -      -    ONLINE
  mirror-3                                      7.25T  56.2G  7.20T        -         -     0%  0.75%      -    ONLINE
    gptid/4a9f9c8b-fbe4-11ed-ac09-ac1f6b566eea  7.26T      -      -        -         -      -      -      -    ONLINE
    gptid/4b9cf351-fbe4-11ed-ac09-ac1f6b566eea  7.26T      -      -        -         -      -      -      -    ONLINE
  mirror-4                                      7.25T  56.1G  7.20T        -         -     0%  0.75%      -    ONLINE
    gptid/4b7518a5-fbe4-11ed-ac09-ac1f6b566eea  7.26T      -      -        -         -      -      -      -    ONLINE
    gptid/4b5e6c08-fbe4-11ed-ac09-ac1f6b566eea  7.26T      -      -        -         -      -      -      -    ONLINE
  mirror-5                                      7.25T  56.2G  7.20T        -         -     0%  0.75%      -    ONLINE
    gptid/4904fd2e-fbe4-11ed-ac09-ac1f6b566eea  7.26T      -      -        -         -      -      -      -    ONLINE
    gptid/4ac53ef7-fbe4-11ed-ac09-ac1f6b566eea  7.26T      -      -        -         -      -      -      -    ONLINE
  mirror-6                                      7.25T  56.4G  7.19T        -         -     0%  0.76%      -    ONLINE
    gptid/495734c2-fbe4-11ed-ac09-ac1f6b566eea  7.26T      -      -        -         -      -      -      -    ONLINE
    gptid/4bb00352-fbe4-11ed-ac09-ac1f6b566eea  7.26T      -      -        -         -      -      -      -    ONLINE
  mirror-7                                      7.25T  56.4G  7.19T        -         -     0%  0.75%      -    ONLINE
    gptid/4b6e4069-fbe4-11ed-ac09-ac1f6b566eea  7.26T      -      -        -         -      -      -      -    ONLINE
    gptid/4b3d7786-fbe4-11ed-ac09-ac1f6b566eea  7.26T      -      -        -         -      -      -      -    ONLINE
  mirror-8                                      7.25T  56.4G  7.19T        -         -     0%  0.76%      -    ONLINE
    gptid/4e436d52-fbe4-11ed-ac09-ac1f6b566eea  7.26T      -      -        -         -      -      -      -    ONLINE
    gptid/4f69d0d5-fbe4-11ed-ac09-ac1f6b566eea  7.26T      -      -        -         -      -      -      -    ONLINE
  mirror-9                                      7.25T  56.4G  7.19T        -         -     0%  0.75%      -    ONLINE
    gptid/4e68a61a-fbe4-11ed-ac09-ac1f6b566eea  7.26T      -      -        -         -      -      -      -    ONLINE
    gptid/4f1cf63b-fbe4-11ed-ac09-ac1f6b566eea  7.26T      -      -        -         -      -      -      -    ONLINE
  mirror-10                                     7.25T  56.9G  7.19T        -         -     0%  0.76%      -    ONLINE
    gptid/4f63b958-fbe4-11ed-ac09-ac1f6b566eea  7.26T      -      -        -         -      -      -      -    ONLINE
    gptid/4f801d45-fbe4-11ed-ac09-ac1f6b566eea  7.26T      -      -        -         -      -      -      -    ONLINE
  mirror-11                                     7.25T  56.5G  7.19T        -         -     0%  0.76%      -    ONLINE
    gptid/4f8ed2aa-fbe4-11ed-ac09-ac1f6b566eea  7.26T      -      -        -         -      -      -      -    ONLINE
    gptid/4fe8597c-fbe4-11ed-ac09-ac1f6b566eea  7.26T      -      -        -         -      -      -      -    ONLINE
logs                                                -      -      -        -         -      -      -      -  -
  gptid/091e0445-fe98-11ed-8574-ac1f6b566eea    3.91G    32K  3.75G        -         -     0%  0.00%      -    ONLINE
  gptid/264b9b6e-fe9f-11ed-9cd5-ac1f6b566eea    3.91G      0  3.75G        -         -     0%  0.00%      -    ONLINE
  gptid/2677ee8c-fe9f-11ed-9cd5-ac1f6b566eea    3.91G      0  3.75G        -         -     0%  0.00%      -    ONLINE
  gptid/26795cef-fe9f-11ed-9cd5-ac1f6b566eea    3.91G      0  3.75G        -         -     0%  0.00%      -    ONLINE
  gptid/267ac10d-fe9f-11ed-9cd5-ac1f6b566eea    3.91G      0  3.75G        -         -     0%  0.00%      -    ONLINE
  gptid/267e3efb-fe9f-11ed-9cd5-ac1f6b566eea    3.91G      0  3.75G        -         -     0%  0.00%      -    ONLINE
root@freenas[~]#
root@freenas[~]# zpool list -v
NAME                                             SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
freenas-boot                                     111G  3.50G   108G        -         -     0%     3%  1.00x    ONLINE  -
  da24p2                                         112G  3.50G   108G        -         -     0%  3.14%      -    ONLINE
test                                              87T   677G  86.3T        -         -     0%     0%  1.00x    ONLINE  /mnt
  mirror-0                                      7.25T  56.3G  7.20T        -         -     0%  0.75%      -    ONLINE
    gptid/4b89ccb5-fbe4-11ed-ac09-ac1f6b566eea  7.26T      -      -        -         -      -      -      -    ONLINE
    gptid/4bf9b5a4-fbe4-11ed-ac09-ac1f6b566eea  7.26T      -      -        -         -      -      -      -    ONLINE
  mirror-1                                      7.25T  56.4G  7.19T        -         -     0%  0.75%      -    ONLINE
    gptid/4a55d9e3-fbe4-11ed-ac09-ac1f6b566eea  7.26T      -      -        -         -      -      -      -    ONLINE
    gptid/4a63c5f1-fbe4-11ed-ac09-ac1f6b566eea  7.26T      -      -        -         -      -      -      -    ONLINE
  mirror-2                                      7.25T  56.3G  7.19T        -         -     0%  0.75%      -    ONLINE
    gptid/491da9f0-fbe4-11ed-ac09-ac1f6b566eea  7.26T      -      -        -         -      -      -      -    ONLINE
    gptid/4be8eb99-fbe4-11ed-ac09-ac1f6b566eea  7.26T      -      -        -         -      -      -      -    ONLINE
  mirror-3                                      7.25T  56.2G  7.20T        -         -     0%  0.75%      -    ONLINE
    gptid/4a9f9c8b-fbe4-11ed-ac09-ac1f6b566eea  7.26T      -      -        -         -      -      -      -    ONLINE
    gptid/4b9cf351-fbe4-11ed-ac09-ac1f6b566eea  7.26T      -      -        -         -      -      -      -    ONLINE
  mirror-4                                      7.25T  56.2G  7.20T        -         -     0%  0.75%      -    ONLINE
    gptid/4b7518a5-fbe4-11ed-ac09-ac1f6b566eea  7.26T      -      -        -         -      -      -      -    ONLINE
    gptid/4b5e6c08-fbe4-11ed-ac09-ac1f6b566eea  7.26T      -      -        -         -      -      -      -    ONLINE
  mirror-5                                      7.25T  56.3G  7.20T        -         -     0%  0.75%      -    ONLINE
    gptid/4904fd2e-fbe4-11ed-ac09-ac1f6b566eea  7.26T      -      -        -         -      -      -      -    ONLINE
    gptid/4ac53ef7-fbe4-11ed-ac09-ac1f6b566eea  7.26T      -      -        -         -      -      -      -    ONLINE
  mirror-6                                      7.25T  56.5G  7.19T        -         -     0%  0.76%      -    ONLINE
    gptid/495734c2-fbe4-11ed-ac09-ac1f6b566eea  7.26T      -      -        -         -      -      -      -    ONLINE
    gptid/4bb00352-fbe4-11ed-ac09-ac1f6b566eea  7.26T      -      -        -         -      -      -      -    ONLINE
  mirror-7                                      7.25T  56.4G  7.19T        -         -     0%  0.76%      -    ONLINE
    gptid/4b6e4069-fbe4-11ed-ac09-ac1f6b566eea  7.26T      -      -        -         -      -      -      -    ONLINE
    gptid/4b3d7786-fbe4-11ed-ac09-ac1f6b566eea  7.26T      -      -        -         -      -      -      -    ONLINE
  mirror-8                                      7.25T  56.5G  7.19T        -         -     0%  0.76%      -    ONLINE
    gptid/4e436d52-fbe4-11ed-ac09-ac1f6b566eea  7.26T      -      -        -         -      -      -      -    ONLINE
    gptid/4f69d0d5-fbe4-11ed-ac09-ac1f6b566eea  7.26T      -      -        -         -      -      -      -    ONLINE
  mirror-9                                      7.25T  56.5G  7.19T        -         -     0%  0.76%      -    ONLINE
    gptid/4e68a61a-fbe4-11ed-ac09-ac1f6b566eea  7.26T      -      -        -         -      -      -      -    ONLINE
    gptid/4f1cf63b-fbe4-11ed-ac09-ac1f6b566eea  7.26T      -      -        -         -      -      -      -    ONLINE
  mirror-10                                     7.25T  57.0G  7.19T        -         -     0%  0.76%      -    ONLINE
    gptid/4f63b958-fbe4-11ed-ac09-ac1f6b566eea  7.26T      -      -        -         -      -      -      -    ONLINE
    gptid/4f801d45-fbe4-11ed-ac09-ac1f6b566eea  7.26T      -      -        -         -      -      -      -    ONLINE
  mirror-11                                     7.25T  56.5G  7.19T        -         -     0%  0.76%      -    ONLINE
    gptid/4f8ed2aa-fbe4-11ed-ac09-ac1f6b566eea  7.26T      -      -        -         -      -      -      -    ONLINE
    gptid/4fe8597c-fbe4-11ed-ac09-ac1f6b566eea  7.26T      -      -        -         -      -      -      -    ONLINE
logs                                                -      -      -        -         -      -      -      -  -
  gptid/80fa8a67-feef-11ed-9cd5-ac1f6b566eea    16.0G  1.13G  14.4G        -         -     0%  7.25%      -    ONLINE

1685454324037.png
 
NickF

NickF

Guru
Joined
Jun 12, 2014
Messages
763
Having troubles re-doing this testing today.


May 30 10:48:24 freenas.fusco.me nvme3: RESERVATION REPORT sqid:3 cid:127 nsid:1
May 30 10:48:24 freenas.fusco.me nvme3: INVALID OPCODE (00/01) sqid:3 cid:127 cdw0:0
May 30 10:48:24 freenas.fusco.me nvme0: RESERVATION REPORT sqid:4 cid:127 nsid:1
May 30 10:48:24 freenas.fusco.me nvme0: INVALID OPCODE (00/01) sqid:4 cid:127 cdw0:0
Click to expand...

Anyone have some suggestions?

I can see all 4 namespaces in nvmecontrol


root@freenas[~]# nvmecontrol devlist
nvme1: SAMSUNG MZWLL1T6HEHP-00003
nvme1ns1 (4000MB)
nvme1ns2 (4000MB)
nvme1ns3 (4000MB)
nvme1ns4 (4000MB)
Click to expand...

(don’t ask me why 8192000= 4 Gig, I have no idea, I was just plugging numbers…)

Devices show up in geom, I think this may be a similar problem @HoneyBadger was looking into in regards to dual actuator drives. Notice how the LUN ID is different by the identity is the same. What I can't explain is why it worked at all yesterday. Now only one shows up in the UI.

Geom name: nvd1
Providers:
1. Name: nvd1
Mediasize: 4195328000 (3.9G)
Sectorsize: 512
Mode: r0w0e0
descr: SAMSUNG MZWLL1T6HEHP-00003
lunid: 334844304bb012480025385800000001
ident: S3HDNX0KB01248
rotationrate: 0
fwsectors: 0
fwheads: 0

Geom name: nvd2
Providers:
1. Name: nvd2
Mediasize: 4195328000 (3.9G)
Sectorsize: 512
Mode: r0w0e0
descr: SAMSUNG MZWLL1T6HEHP-00003
lunid: 334844304bb012480025385800000002
ident: S3HDNX0KB01248
rotationrate: 0
fwsectors: 0
fwheads: 0
Click to expand...
1685471270396.png
 
HoneyBadger

HoneyBadger

actually does care
Administrator
Moderator
iXsystems
Joined
Feb 6, 2014
Messages
5,112
NickF said:
(don’t ask me why 8192000= 4 Gig, I have no idea, I was just plugging numbers…)
Click to expand...
8192000 sectors x 512 bytes per sector = 4G. :wink:

Not sure if your controller needs the reset command between namespace creation/deletion similar to how you can only do one SATA HPA SET per power cycle, but nvmecontrol reset nvme0 might be worth a shot.
 
NickF

NickF

Guru
Joined
Jun 12, 2014
Messages
763
HoneyBadger said:
8192000 sectors x 512 bytes per sector = 4G. :wink:

Not sure if your controller needs the reset command between namespace creation/deletion similar to how you can only do one SATA HPA SET per power cycle, but nvmecontrol reset nvme0 might be worth a shot.
Click to expand...
I've tried that and I've rebooted the whole server several times, unfortunately. :(

Also thanks for MATHing for me :)
 
D

DemohFoxfire

Dabbler
Joined
May 2, 2023
Messages
11
I didnt see any Micron 9300 Pro / 9300 Max series drives and have a few MTFDHAL6T4TDR drives I am about to repurpose. Its essentially the same as the MTFDHAL7T6TDP but is provisioned for higher endurance.

Code:
root@truenas[~]#
root@truenas[~]# smartctl -a /dev/nvd2
smartctl 7.2 2021-09-14 r5236 [FreeBSD 13.1-RELEASE-p7 amd64] (local build)
Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org

/dev/nvd2: To monitor NVMe disks use /dev/nvme* device names
Please specify device type with the -d option.

Use smartctl -h to get a usage summary

root@truenas[~]# smartctl -a /dev/nvme2
smartctl 7.2 2021-09-14 r5236 [FreeBSD 13.1-RELEASE-p7 amd64] (local build)
Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Number:                       Micron_9300_MTFDHAL6T4TDR
Serial Number:                      2013277FACBE
Firmware Version:                   11300DY0
PCI Vendor/Subsystem ID:            0x1344
IEEE OUI Identifier:                0x00a075
Total NVM Capacity:                 6,401,252,745,216 [6.40 TB]
Unallocated NVM Capacity:           0
Controller ID:                      1
NVMe Version:                       1.2
Number of Namespaces:               32
Local Time is:                      Thu Jun  1 05:43:16 2023 PDT
Firmware Updates (0x17):            3 Slots, Slot 1 R/O, no Reset required
Optional Admin Commands (0x000e):   Format Frmw_DL NS_Mngmt
Optional NVM Commands (0x0014):     DS_Mngmt Sav/Sel_Feat
Log Page Attributes (0x02):         Cmd_Eff_Lg
Maximum Data Transfer Size:         32 Pages
Warning  Comp. Temp. Threshold:     75 Celsius
Critical Comp. Temp. Threshold:     80 Celsius

Supported Power States
St Op     Max   Active     Idle   RL RT WL WT  Ent_Lat  Ex_Lat
 0 +    25.00W       -        -    0  0  0  0      100     100
 1 +    24.00W       -        -    1  1  1  1      115     115
 2 +    23.00W       -        -    2  2  2  2      130     130
 3 +    22.00W       -        -    3  3  3  3      145     145
 4 +    21.00W       -        -    4  4  4  4      160     160
 5 +    20.00W       -        -    5  5  5  5      175     175
 6 +    19.00W       -        -    6  6  6  6      190     190
 7 +    18.00W       -        -    7  7  7  7      205     205
 8 +    17.00W       -        -    8  8  8  8      220     220
 9 +    16.00W       -        -    9  9  9  9      235     235
10 +    15.00W       -        -   10 10 10 10      250     250
11 +    14.00W       -        -   11 11 11 11      265     265
12 +    13.00W       -        -   12 12 12 12      280     280
13 +    12.00W       -        -   13 13 13 13      295     295
14 +    11.00W       -        -   14 14 14 14      310     310
15 +    10.00W       -        -   15 15 15 15      325     325

=== START OF SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

SMART/Health Information (NVMe Log 0x02)
Critical Warning:                   0x00
Temperature:                        44 Celsius
Available Spare:                    100%
Available Spare Threshold:          5%
Percentage Used:                    0%
Data Units Read:                    274,678,191 [140 TB]
Data Units Written:                 91,151,165 [46.6 TB]
Host Read Commands:                 31,634,136,869
Host Write Commands:                8,651,359,543
Controller Busy Time:               27,255
Power Cycles:                       42
Power On Hours:                     3,088
Unsafe Shutdowns:                   27
Media and Data Integrity Errors:    0
Error Information Log Entries:      0
Warning  Comp. Temperature Time:    0
Critical Comp. Temperature Time:    0
Temperature Sensor 1:               49 Celsius
Temperature Sensor 2:               44 Celsius
Temperature Sensor 3:               40 Celsius
Temperature Sensor 4:               40 Celsius

Error Information (NVMe Log 0x01, 16 of 63 entries)
No Errors Logged



Code:
root@truenas[~]# diskinfo -wS /dev/nvd2
/dev/nvd2
        4096            # sectorsize
        6401252745216   # mediasize in bytes (5.8T)
        1562805846      # mediasize in sectors
        0               # stripesize
        0               # stripeoffset
        Micron_9300_MTFDHAL6T4TDR       # Disk descr.
        2013277FACBE    # Disk ident.
        nvme2           # Attachment
        Yes             # TRIM/UNMAP support
        0               # Rotation rate in RPM

Synchronous random writes:
           4 kbytes:     15.8 usec/IO =    247.4 Mbytes/s
           8 kbytes:     22.8 usec/IO =    342.0 Mbytes/s
          16 kbytes:     28.5 usec/IO =    548.1 Mbytes/s
          32 kbytes:     32.8 usec/IO =    953.5 Mbytes/s
          64 kbytes:     39.0 usec/IO =   1603.6 Mbytes/s
         128 kbytes:     64.2 usec/IO =   1947.5 Mbytes/s
         256 kbytes:    143.5 usec/IO =   1742.6 Mbytes/s
         512 kbytes:    193.9 usec/IO =   2578.0 Mbytes/s
        1024 kbytes:    334.3 usec/IO =   2991.0 Mbytes/s
        2048 kbytes:    631.4 usec/IO =   3167.4 Mbytes/s
        4096 kbytes:   1235.9 usec/IO =   3236.5 Mbytes/s
        8192 kbytes:   2399.7 usec/IO =   3333.8 Mbytes/s
root@truenas[~]#



I am highly contemplating using a pair of these for SLOG, either grabbing some MTFDHAL3T2TDR / MTFDHAL3T8TDP as for some cost savings but the 9300s are coming down in price. MTFDHAL15T3TDP / MTFDHAL12T8TDR are also.

If we wanted to split this thing up with additional namespaces like Nick was doing above could I still mirror that across 2 physical drives? Not that I would want to but I kind of want to try it if possible.
 
HoneyBadger

HoneyBadger

actually does care
Administrator
Moderator
iXsystems
Joined
Feb 6, 2014
Messages
5,112
DemohFoxfire said:
If we wanted to split this thing up with additional namespaces like Nick was doing above could I still mirror that across 2 physical drives? Not that I would want to but I kind of want to try it if possible.
Click to expand...
@NickF hit some odd snags with regards to the visibility of devices through the UI, likely due to the identifier fields from the device being the same, so it would be a CLI-configured option (and thus technically "unsupported"). You'd also have to be cautious about ensuring that your mirrors are set up to go across devices eg: (nvme1ns1 + nvme2ns1) and not (nvme1ns1 + nvme1ns2) otherwise a single device failure still takes you out.

Those look like some powerful devices; wonder how the performance scales down to the smaller sizes?
 
Etorix

Etorix

Wizard
Joined
Dec 30, 2020
Messages
2,134
DemohFoxfire said:
I am highly contemplating using a pair of these for SLOG, either grabbing some MTFDHAL3T2TDR / MTFDHAL3T8TDP as for some cost savings but the 9300s are coming down in price. MTFDHAL15T3TDP / MTFDHAL12T8TDR are also.
Click to expand...
Thanks for the numbers.
You realise that a SLOG only needs to be large enough to hold 10 seconds worth of transactions (two transaction groups), right?
And that a SLOG need not be mirrored, except for the most paranoid users or if servicing a failed SLOG device would be problematic.
 
You must log in or register to reply here.

Similar threads

E
  • Locked
Tuning BIOS towards low latency and improve SLOG/sync write performance
Replies
3
Views
3K
Ender117
E
S
Mirroed SLOG - 2x optane p900 OR p900 and p3700 ?
Replies
7
Views
3K
HoneyBadger
HoneyBadger
E
Block storage limits?
Replies
12
Views
2K
ehsab
E
Yves_
diskinfo submit error
Replies
12
Views
3K
wl714
W
J
24x FullFlash but bad Performance
Replies
9
Views
2K
fcojavrui
F
Top