SLOG benchmarking and finding the best SLOG

[DemohFoxfire]

@NickF hit some odd snags with regards to the visibility of devices through the UI, likely due to the identifier fields from the device being the same, so it would be a CLI-configured option (and thus technically "unsupported"). You'd also have to be cautious about ensuring that your mirrors are set up to go across devices eg: (nvme1ns1 + nvme2ns1) and not (nvme1ns1 + nvme1ns2) otherwise a single device failure still takes you out.

Those look like some powerful devices; wonder how the performance scales down to the smaller sizes?

Wouldnt theoretically using the micron tools to "reprovision" the drive in a different system then plopping it into the truenas server be "supported"? I read somewhere (I believe in Micron's own publication but dont quote me) that the Pro and Max drives are hardware identical and if you wanted the higher endurance but only purchased a pro you could reduce the provisioned size from the Pro size to the Max size... and vice versa.

On paper the drives perform very similar across their product line.

So you are saying you could mirror + stripe using namespaces for the slog?

Thanks for the numbers.
You realise that a SLOG only needs to be large enough to hold 10 seconds worth of transactions (two transaction groups), right?
And that a SLOG need not be mirrored, except for the most paranoid users or if servicing a failed SLOG device would be problematic.

Nope, I havent gotten that far into truenas and have been running a lot of different hardware / configs through its paces while I figure it all out. Hence most of my idiotic questions. So is this a cap of 2 transaction groups or is this something that is just 'normal'?

What I mean is (assuming round numbers for easy math): If I have a potato pool which is massive, say 100tb, but writes at 100MB/s or just under gigabit line speed and we have 25gbe available which the slog would have no problem keeping up with. With sync=always would the zil essentially cache a large transfer while the potato catches up even past 2 transaction groups? A transfer of 100GB @ potato speeds without a slog would take 16m40s @ 100MB/s. Say we get 2500MB/s with the SLOG would the network portion of the transfer finish at 40s where the remaining 16 minutes the potato is just writing the data from memory (ram) as what is in the SLOG gets abandoned as writes complete?

I ask because right now (or really next week) Ill finally have 36 potato drives in my lab and I am curious on how this reacts with the different write loads ill be throwing at it.

 

[Etorix]

SLOG is NOT a cache, it is a LOG. ZFS will keep up to two transaction groups cached in RAM (+ logged to ZIL/SLOG) but will then commit everything to potato-safe storage before accepting more data. So your 100 GB sync transfer will essentially be capped at the ingest speed of "potato".

2 tg @ 25 GbE = 10 s * 25 Gb/s = 250 Gb = 31 GB with default values (not advised to change unless you know what you're messing with)
Your SLOG need not be larger than 32 GB to cope. If you do need a SLOG (= sync writes), get an Optane DC drive while they are still around and keep these nice 6.4 TB Micron drives for uses which actually need their capacity.

 

[NickF]

Wouldnt theoretically using the micron tools to "reprovision" the drive in a different system then plopping it into the truenas server be "supported"? I read somewhere (I believe in Micron's own publication but dont quote me) that the Pro and Max drives are hardware identical and if you wanted the higher endurance but only purchased a pro you could reduce the provisioned size from the Pro size to the Max size... and vice versa.

On paper the drives perform very similar across their product line.

So you are saying you could mirror + stripe using namespaces for the slog?

AFAIK Micron's tool wouldn't do anything more than nvmecontrol in a pretty UI. The problem isn't the drives themselves, or even ZFS. It just that the TrueNAS UI doesn't know what to do with drives that have the same identity but a different LUN id. You can manually do it in the CLI, but YMMV.

: If I have a potato pool which is massive

What is a potato pool?

36 potato drives

What is a potato drive?

get an Optane DC drive while they are still around and keep these nice 6.4 TB Micron drives for uses which actually need their capacity.

I don't even think that's strictly necessary. His drive got:

4 kbytes: 15.8 usec/IO = 247.4 Mbytes/s
8 kbytes: 22.8 usec/IO = 342.0 Mbytes/s
16 kbytes: 28.5 usec/IO = 548.1 Mbytes/s
32 kbytes: 32.8 usec/IO = 953.5 Mbytes/s
64 kbytes: 39.0 usec/IO = 1603.6 Mbytes/s
128 kbytes: 64.2 usec/IO = 1947.5 Mbytes/s
256 kbytes: 143.5 usec/IO = 1742.6 Mbytes/s

My 900P 280GB got:

4 kbytes: 19.7 usec/IO = 198.5 Mbytes/s
8 kbytes: 22.8 usec/IO = 343.3 Mbytes/s
16 kbytes: 30.1 usec/IO = 518.5 Mbytes/s
32 kbytes: 45.1 usec/IO = 693.6 Mbytes/s
64 kbytes: 86.4 usec/IO = 723.2 Mbytes/s
128 kbytes: 153.9 usec/IO = 812.2 Mbytes/s
256 kbytes: 259.2 usec/IO = 964.4 Mbytes/s

So his drive is significantly faster than a smaller Gen3 Optane drive in this particular benchmark. So, realistically, his drive "overprivisioned" down to whatever he wants (lets say 32GB) or 8x4GB with namespaces as being discussed now, is faster than previous generation Optane. A P5800X on the other hand, is a different story. Those are not on sale and are harder to find for a good price.

 

[HoneyBadger]

What I mean is (assuming round numbers for easy math): If I have a potato pool which is massive, say 100tb, but writes at 100MB/s or just under gigabit line speed and we have 25gbe available which the slog would have no problem keeping up with. With sync=always would the zil essentially cache a large transfer while the potato catches up even past 2 transaction groups? A transfer of 100GB @ potato speeds without a slog would take 16m40s @ 100MB/s. Say we get 2500MB/s with the SLOG would the network portion of the transfer finish at 40s where the remaining 16 minutes the potato is just writing the data from memory (ram) as what is in the SLOG gets abandoned as writes complete

Not quite. Have a look at this old post of mine where I hopefully can shed some light on how the transaction groups and write throttle in ZFS works:

Some insights into SLOG/ZIL with ZFS on FreeNAS

they (all of a sudden) care more about the "speed" (exactly latency) than about the (small) possible data loss You can work with this. The lack of power loss prevention (PLP) on the drive is (or rather will be) directly related to the poor SLOG performance; most drives are very slow if you...

www.truenas.com

What is a potato

Boil em, mash em, stick em in a stew?

I assume it's the usual self-deprecating term about a piece of hardware with roughly the computing power of the tuber in question.

 

[NickF]

I assume it's the usual self-deprecating term about a piece of hardware with roughly the computing power of the tuber in question.

I always preferred calling them "Fisher Price"

Like Fisher Price Modem

Urban Dictionary: fisher-price modem

An internet connection that is very laggy or even non-responsive. The phrase is usually used in Real-Time Strategy or MMORPG games online. The phrase was first used in a 2v2 match of Stracraft when the game was close to lagging out or disconnecting on a host game. When the game dragged, the host...

www.urbandictionary.com

Man, I miss StarCraft. It's hilarious the Urban Dictionary even references it. I'm getting old.

 

[DemohFoxfire]

I always preferred calling them "Fisher Price"

Like Fisher Price Modem

Urban Dictionary: fisher-price modem

An internet connection that is very laggy or even non-responsive. The phrase is usually used in Real-Time Strategy or MMORPG games online. The phrase was first used in a 2v2 match of Stracraft when the game was close to lagging out or disconnecting on a host game. When the game dragged, the host...

www.urbandictionary.com

Man, I miss StarCraft. It's hilarious the Urban Dictionary even references it. I'm getting old.

Ahh yes, I still remember my CD key. but yes, the potato reference is to the processing power that GLaDOS had when powered via potato.

some more benchmarks, ICDPC5ED2ORA6.4T / SSDPEDME064T4S which is the Oracle flash accelerator F640. They are configured as dual independent 3.2 on a pcie 8x slot.

Code:

root@truenas[~]# diskinfo -wS /dev/nvd0
/dev/nvd0
        512             # sectorsize
        3200631791616   # mediasize in bytes (2.9T)
        6251233968      # mediasize in sectors
        131072          # stripesize
        0               # stripeoffset
        7335943:ICDPC5ED2ORA6.4T        # Disk descr.
        PHLE805500EB6P4BGN-1    # Disk ident.
        nvme0           # Attachment
        Yes             # TRIM/UNMAP support
        0               # Rotation rate in RPM

Synchronous random writes:
         0.5 kbytes:     15.3 usec/IO =     31.8 Mbytes/s
           1 kbytes:     14.4 usec/IO =     68.0 Mbytes/s
           2 kbytes:     14.2 usec/IO =    137.8 Mbytes/s
           4 kbytes:     14.0 usec/IO =    278.0 Mbytes/s
           8 kbytes:     16.4 usec/IO =    477.0 Mbytes/s
          16 kbytes:     25.4 usec/IO =    615.6 Mbytes/s
          32 kbytes:     38.6 usec/IO =    809.2 Mbytes/s
          64 kbytes:     78.3 usec/IO =    798.6 Mbytes/s
         128 kbytes:    154.8 usec/IO =    807.3 Mbytes/s
         256 kbytes:    262.4 usec/IO =    952.9 Mbytes/s
         512 kbytes:    514.3 usec/IO =    972.1 Mbytes/s
        1024 kbytes:   1010.3 usec/IO =    989.8 Mbytes/s
        2048 kbytes:   2062.4 usec/IO =    969.7 Mbytes/s
        4096 kbytes:   4116.3 usec/IO =    971.7 Mbytes/s
        8192 kbytes:   6011.3 usec/IO =   1330.8 Mbytes/s
root@truenas[~]# diskinfo -wS /dev/nvd1
/dev/nvd1
        512             # sectorsize
        3200631791616   # mediasize in bytes (2.9T)
        6251233968      # mediasize in sectors
        131072          # stripesize
        0               # stripeoffset
        7335943:ICDPC5ED2ORA6.4T        # Disk descr.
        PHLE805500EB6P4BGN-2    # Disk ident.
        nvme1           # Attachment
        Yes             # TRIM/UNMAP support
        0               # Rotation rate in RPM

Synchronous random writes:
         0.5 kbytes:     15.9 usec/IO =     30.8 Mbytes/s
           1 kbytes:     15.1 usec/IO =     64.9 Mbytes/s
           2 kbytes:     15.1 usec/IO =    129.5 Mbytes/s
           4 kbytes:     14.6 usec/IO =    266.8 Mbytes/s
           8 kbytes:     17.8 usec/IO =    438.0 Mbytes/s
          16 kbytes:     26.3 usec/IO =    593.4 Mbytes/s
          32 kbytes:     43.1 usec/IO =    724.4 Mbytes/s
          64 kbytes:     61.6 usec/IO =   1014.0 Mbytes/s
         128 kbytes:    115.7 usec/IO =   1080.8 Mbytes/s
         256 kbytes:    215.1 usec/IO =   1162.0 Mbytes/s
         512 kbytes:    426.2 usec/IO =   1173.3 Mbytes/s
        1024 kbytes:    855.6 usec/IO =   1168.8 Mbytes/s
        2048 kbytes:   1530.5 usec/IO =   1306.8 Mbytes/s
        4096 kbytes:   3286.0 usec/IO =   1217.3 Mbytes/s
        8192 kbytes:   6673.9 usec/IO =   1198.7 Mbytes/s
root@truenas[~]#

Code:

root@truenas[~]# smartctl -a /dev/nvd0
smartctl 7.2 2021-09-14 r5236 [FreeBSD 13.1-RELEASE-p7 amd64] (local build)
Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org

/dev/nvd0: To monitor NVMe disks use /dev/nvme* device names
Please specify device type with the -d option.

Use smartctl -h to get a usage summary

root@truenas[~]# smartctl -a /dev/nvme0
smartctl 7.2 2021-09-14 r5236 [FreeBSD 13.1-RELEASE-p7 amd64] (local build)
Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Number:                       7335943:ICDPC5ED2ORA6.4T
Serial Number:                      PHLE805500EB6P4BGN-1
Firmware Version:                   QDV1RF34
PCI Vendor ID:                      0x8086
PCI Vendor Subsystem ID:            0x108e
IEEE OUI Identifier:                0x5cd2e4
Total NVM Capacity:                 3,200,631,791,616 [3.20 TB]
Unallocated NVM Capacity:           0
Controller ID:                      0
NVMe Version:                       1.2
Number of Namespaces:               1
Namespace 1 Size/Capacity:          3,200,631,791,616 [3.20 TB]
Namespace 1 Formatted LBA Size:     512
Namespace 1 IEEE EUI-64:            5cd2e4 52a8150000
Local Time is:                      Tue Jun 20 06:44:39 2023 PDT
Firmware Updates (0x02):            1 Slot
Optional Admin Commands (0x0006):   Format Frmw_DL
Optional NVM Commands (0x0006):     Wr_Unc DS_Mngmt
Log Page Attributes (0x02):         Cmd_Eff_Lg
Maximum Data Transfer Size:         32 Pages
Warning  Comp. Temp. Threshold:     70 Celsius
Critical Comp. Temp. Threshold:     75 Celsius

Supported Power States
St Op     Max   Active     Idle   RL RT WL WT  Ent_Lat  Ex_Lat
 0 +    13.50W       -        -    0  0  0  0        0       0

Supported LBA Sizes (NSID 0x1)
Id Fmt  Data  Metadt  Rel_Perf
 0 +     512       0         2
 1 -    4096       0         0

=== START OF SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

SMART/Health Information (NVMe Log 0x02)
Critical Warning:                   0x00
Temperature:                        41 Celsius
Available Spare:                    99%
Available Spare Threshold:          10%
Percentage Used:                    4%
Data Units Read:                    979,438,983 [501 TB]
Data Units Written:                 1,394,603,406 [714 TB]
Host Read Commands:                 17,066,130,557
Host Write Commands:                22,411,399,614
Controller Busy Time:               6,318
Power Cycles:                       76
Power On Hours:                     30,564
Unsafe Shutdowns:                   42
Media and Data Integrity Errors:    0
Error Information Log Entries:      0
Warning  Comp. Temperature Time:    0
Critical Comp. Temperature Time:    0

Error Information (NVMe Log 0x01, 16 of 64 entries)
No Errors Logged

root@truenas[~]# smartctl -a /dev/nvme1
smartctl 7.2 2021-09-14 r5236 [FreeBSD 13.1-RELEASE-p7 amd64] (local build)
Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Number:                       7335943:ICDPC5ED2ORA6.4T
Serial Number:                      PHLE805500EB6P4BGN-2
Firmware Version:                   QDV1RF34
PCI Vendor ID:                      0x8086
PCI Vendor Subsystem ID:            0x108e
IEEE OUI Identifier:                0x5cd2e4
Total NVM Capacity:                 3,200,631,791,616 [3.20 TB]
Unallocated NVM Capacity:           0
Controller ID:                      0
NVMe Version:                       1.2
Number of Namespaces:               1
Namespace 1 Size/Capacity:          3,200,631,791,616 [3.20 TB]
Namespace 1 Formatted LBA Size:     512
Namespace 1 IEEE EUI-64:            5cd2e4 52a8150000
Local Time is:                      Tue Jun 20 06:44:42 2023 PDT
Firmware Updates (0x02):            1 Slot
Optional Admin Commands (0x0006):   Format Frmw_DL
Optional NVM Commands (0x0006):     Wr_Unc DS_Mngmt
Log Page Attributes (0x02):         Cmd_Eff_Lg
Maximum Data Transfer Size:         32 Pages
Warning  Comp. Temp. Threshold:     70 Celsius
Critical Comp. Temp. Threshold:     75 Celsius

Supported Power States
St Op     Max   Active     Idle   RL RT WL WT  Ent_Lat  Ex_Lat
 0 +    13.50W       -        -    0  0  0  0        0       0

Supported LBA Sizes (NSID 0x1)
Id Fmt  Data  Metadt  Rel_Perf
 0 +     512       0         2
 1 -    4096       0         0

=== START OF SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

SMART/Health Information (NVMe Log 0x02)
Critical Warning:                   0x00
Temperature:                        38 Celsius
Available Spare:                    99%
Available Spare Threshold:          10%
Percentage Used:                    3%
Data Units Read:                    976,970,539 [500 TB]
Data Units Written:                 1,075,053,924 [550 TB]
Host Read Commands:                 16,990,823,671
Host Write Commands:                16,499,677,002
Controller Busy Time:               5,040
Power Cycles:                       76
Power On Hours:                     30,565
Unsafe Shutdowns:                   42
Media and Data Integrity Errors:    0
Error Information Log Entries:      0
Warning  Comp. Temperature Time:    0
Critical Comp. Temperature Time:    0

Error Information (NVMe Log 0x01, 16 of 64 entries)
No Errors Logged

root@truenas[~]#

Now ill go ahead patiently wait for a pair of rms-200 cards to come in while I perform drive replacements on 4 of the 7 freenas boxes I was able to get into (of I think 11?,) 1 has permanent errors detected, 2 have degraded volumes. yay fun. At least all volumes are online. I have 1 server with a 12 wide z2 where all except 2 drives have crc errors reporting..... absolutely cant wait until I can migrate / consolidate to the new truenas hardware I am building.

 

[helloadam]

Here is our results for three devices we have already seen in this thread but I thought I would share our results and experience.

We tested the following cards on a lab bench machine. Our lab bench machine is by no means the best but it allows us to quickly test things without pushing hardware into production. I mention this because the results below should be compared against each other vs perhaps others in this thread for the same hardware.

1. Used - qty 2 - EMC's NVRAM-1616 16GB pulled from Isilon/PowerScale
2. Used - qty 2 - Radian Memory Systems RSM-300 8GB pulled from NetApp HCI cluster
3. New - qty 2 - Intel Optane SSD DC P5800X (400GB) from Supermicro

Our summary:

• In our test environment, we saw that the Intel Optane P5800X had the best performance overall. This card has an extremely high endurance rating (100 DWPD) did extremely well for a SLOG device.
• We could not figure out how to format the NVRAM-1616 from EMC and RMS-300 from NetApp to be native 4K sectors but the Intel Optane 5800X did even better under 4k sector sizes for larger writes
  • From our understanding this does not make a difference for a SLOG device as the transaction group writes tends to be in the 8-16 kbytes range.
• If endurance is a concern for you, the NVRAM-1616 or the RMS-300 are your best bet as they have unlimited endurance.
• Make sure you have a battery pack for your NVRAM-1616. The intended way to mount the battery pack is with double sided sticky tape to the inside chassis. This might make it complicated for some builds if you have a riser card or if the battery cable is not long enough. The RMS-300 has a built in capacity and does not require any external power for power loss protection. Intel Optane doesn't require anything speical and already has built in power loss protection.

• If your network / zfs pool is 1-2.5Gbps or less in terms of throughput, the SLOG device of NVRAM-1616 or the RMS-300 will be excellent as their 8-16 kbyte test will should not bottleneck your write performance.
• If you network / zfs pool is between 2.5Gbps - 7 Gbps in terms of throughput, the SLOG device of Intel Optane P5800X will be excellent as their 8-16 kbyte test will should not bottleneck your write performance.

Happy to hear if others agree or not with our summary

Raw results below.
-Adam


NVRAM-1616-16GB

Code:

nomad@NomadBSD$ sudo smartctl -a /dev/nvme2
Password:
smartctl 7.3 2022-02-28 r5338 [FreeBSD 13.1-RELEASE-p5 amd64] (local build)
Copyright (C) 2002-22, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Number:                       MTR_MLC_TS_16GB
Serial Number:                      REMOVED
Firmware Version:                   2.6.06.0
PCI Vendor/Subsystem ID:            0x11f8
IEEE OUI Identifier:                0x00e004
Controller ID:                      0
NVMe Version:                       <1.2
Number of Namespaces:               1
Namespace 1 Size/Capacity:          16,493,051,904 [16.4 GB]
Namespace 1 Formatted LBA Size:     512
Namespace 1 IEEE EUI-64:            04e000 018861ae01
Local Time is:                      Wed Aug  9 19:07:27 2023 PDT
Firmware Updates (0x07):            3 Slots, Slot 1 R/O
Optional Admin Commands (0x0006):   Format Frmw_DL
Log Page Attributes (0x01):         S/H_per_NS

Supported Power States
St Op     Max   Active     Idle   RL RT WL WT  Ent_Lat  Ex_Lat
 0 +    25.00W       -        -    0  0  0  0      100     100

Supported LBA Sizes (NSID 0x1)
Id Fmt  Data  Metadt  Rel_Perf
 0 +     512       0         3
 1 -     512       0         3
 2 -    4096       0         0
 3 -    4096       0         0

=== START OF SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

SMART/Health Information (NVMe Log 0x02)
Critical Warning:                   0x00
Temperature:                        51 Celsius
Available Spare:                    0%
Available Spare Threshold:          0%
Percentage Used:                    0%
Data Units Read:                    2 [1.02 MB]
Data Units Written:                 0
Host Read Commands:                 80
Host Write Commands:                0
Controller Busy Time:               0
Power Cycles:                       0
Power On Hours:                     0
Unsafe Shutdowns:                   0
Media and Data Integrity Errors:    0
Error Information Log Entries:      0

Error Information (NVMe Log 0x01, 16 of 63 entries)
No Errors Logged

nomad@NomadBSD$

nomad@NomadBSD$ sudo diskinfo -wS /dev/nvd2
/dev/nvd2
    512             # sectorsize
    16493051904     # mediasize in bytes (15G)
    32212992        # mediasize in sectors
    0               # stripesize
    0               # stripeoffset
    MTR_MLC_TS_16GB    # Disk descr.
    REMOVED    # Disk ident.
    nvme2           # Attachment
    No              # TRIM/UNMAP support
    0               # Rotation rate in RPM

Synchronous random writes:
     0.5 kbytes:     35.5 usec/IO =     13.8 Mbytes/s
       1 kbytes:     35.9 usec/IO =     27.2 Mbytes/s
       2 kbytes:     36.3 usec/IO =     53.8 Mbytes/s
       4 kbytes:     37.1 usec/IO =    105.2 Mbytes/s
       8 kbytes:     37.8 usec/IO =    206.6 Mbytes/s
      16 kbytes:     45.0 usec/IO =    347.4 Mbytes/s
      32 kbytes:     52.3 usec/IO =    597.7 Mbytes/s
      64 kbytes:     67.3 usec/IO =    929.2 Mbytes/s
     128 kbytes:    100.2 usec/IO =   1247.6 Mbytes/s
     256 kbytes:    135.7 usec/IO =   1842.4 Mbytes/s
     512 kbytes:    206.5 usec/IO =   2421.1 Mbytes/s
    1024 kbytes:    352.6 usec/IO =   2836.1 Mbytes/s
    2048 kbytes:    646.3 usec/IO =   3094.7 Mbytes/s
    4096 kbytes:   1258.8 usec/IO =   3177.7 Mbytes/s
    8192 kbytes:   2475.4 usec/IO =   3231.7 Mbytes/s
nomad@NomadBSD$ 

RMS-300-8GB

Code:

nomad@NomadBSD$ sudo smartctl -a /dev/nvme2
Password:
smartctl 7.3 2022-02-28 r5338 [FreeBSD 13.1-RELEASE-p5 amd64] (local build)
Copyright (C) 2002-22, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Number:                       RMS-300
Serial Number:                      REMOVED
Firmware Version:                   7d8422bc
PCI Vendor/Subsystem ID:            0x1cc7
IEEE OUI Identifier:                0x00e004
Controller ID:                      0
NVMe Version:                       1.2
Number of Namespaces:               1
Namespace 1 Size/Capacity:          8,549,695,488 [8.54 GB]
Namespace 1 Utilization:            0
Namespace 1 Formatted LBA Size:     512
Local Time is:                      Wed Aug  9 18:48:15 2023 PDT
Firmware Updates (0x09):            4 Slots, Slot 1 R/O
Optional Admin Commands (0x0004):   Frmw_DL
Optional NVM Commands (0x0004):     DS_Mngmt
Log Page Attributes (0x01):         S/H_per_NS

Supported Power States
St Op     Max   Active     Idle   RL RT WL WT  Ent_Lat  Ex_Lat
 0 +    25.00W       -        -    0  0  0  0      100     100

Supported LBA Sizes (NSID 0x1)
Id Fmt  Data  Metadt  Rel_Perf
 0 -     512       0         3
 1 -     512       8         3
 2 -    4096       0         0
 3 -    4096       8         0

=== START OF SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

SMART/Health Information (NVMe Log 0x02)
Critical Warning:                   0x00
Temperature:                        37 Celsius
Available Spare:                    0%
Available Spare Threshold:          0%
Percentage Used:                    0%
Data Units Read:                    87,546,214,105 [44.8 PB]
Data Units Written:                 87,294,580,693 [44.6 PB]
Host Read Commands:                 904,397,051
Host Write Commands:                882,487,308
Controller Busy Time:               263,649
Power Cycles:                       21
Power On Hours:                     52
Unsafe Shutdowns:                   11
Media and Data Integrity Errors:    0
Error Information Log Entries:      0

Error Information (NVMe Log 0x01, 16 of 63 entries)
No Errors Logged


nomad@NomadBSD$ sudo diskinfo -wS /dev/nvd2
/dev/nvd2
    512             # sectorsize
    8549695488      # mediasize in bytes (8.0G)
    16698624        # mediasize in sectors
    0               # stripesize
    0               # stripeoffset
    RMS-300         # Disk descr.
    REMOVED         # Disk ident.
    nvme2           # Attachment
    Yes             # TRIM/UNMAP support
    0               # Rotation rate in RPM

Synchronous random writes:
     0.5 kbytes:     29.1 usec/IO =     16.8 Mbytes/s
       1 kbytes:     30.8 usec/IO =     31.7 Mbytes/s
       2 kbytes:     33.9 usec/IO =     57.6 Mbytes/s
       4 kbytes:     41.7 usec/IO =     93.6 Mbytes/s
       8 kbytes:     40.5 usec/IO =    192.9 Mbytes/s
      16 kbytes:     46.6 usec/IO =    335.5 Mbytes/s
      32 kbytes:     51.7 usec/IO =    604.9 Mbytes/s
      64 kbytes:     70.7 usec/IO =    884.0 Mbytes/s
     128 kbytes:    109.3 usec/IO =   1143.4 Mbytes/s
     256 kbytes:    161.6 usec/IO =   1547.1 Mbytes/s
     512 kbytes:    268.3 usec/IO =   1863.6 Mbytes/s
    1024 kbytes:    482.2 usec/IO =   2073.8 Mbytes/s
    2048 kbytes:    929.6 usec/IO =   2151.4 Mbytes/s
    4096 kbytes:   1834.7 usec/IO =   2180.2 Mbytes/s
    8192 kbytes:   3623.7 usec/IO =   2207.7 Mbytes/s
nomad@NomadBSD$ 

Intel Optane SSD DC P5800X (400GB) - 512b sector

Code:

nomad@NomadBSD$ sudo smartctl -a /dev/nvme2
Password:
smartctl 7.3 2022-02-28 r5338 [FreeBSD 13.1-RELEASE-p5 amd64] (local build)
Copyright (C) 2002-22, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Number:                       INTEL SSDPF21Q400GB
Serial Number:                      REMOVED
Firmware Version:                   L0310600
PCI Vendor/Subsystem ID:            0x8086
IEEE OUI Identifier:                0x5cd2e4
Total NVM Capacity:                 400,088,457,216 [400 GB]
Unallocated NVM Capacity:           0
Controller ID:                      0
NVMe Version:                       1.4
Number of Namespaces:               128
Local Time is:                      Thu Aug 10 17:29:06 2023 PDT
Firmware Updates (0x18):            4 Slots, no Reset required
Optional Admin Commands (0x001e):   Format Frmw_DL NS_Mngmt Self_Test
Optional NVM Commands (0x005e):     Wr_Unc DS_Mngmt Wr_Zero Sav/Sel_Feat Timestmp
Log Page Attributes (0x1e):         Cmd_Eff_Lg Ext_Get_Lg Telmtry_Lg Pers_Ev_Lg
Maximum Data Transfer Size:         32 Pages
Warning  Comp. Temp. Threshold:     70 Celsius
Critical Comp. Temp. Threshold:     80 Celsius

Supported Power States
St Op     Max   Active     Idle   RL RT WL WT  Ent_Lat  Ex_Lat
 0 +    13.00W       -        -    0  0  0  0        0       0
 1 +    11.00W       -        -    0  1  0  1        0       0
 2 +     9.00W       -        -    0  2  0  2        0       0

=== START OF SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

SMART/Health Information (NVMe Log 0x02)
Critical Warning:                   0x00
Temperature:                        30 Celsius
Available Spare:                    100%
Available Spare Threshold:          0%
Percentage Used:                    0%
Data Units Read:                    1 [512 KB]
Data Units Written:                 0
Host Read Commands:                 139
Host Write Commands:                0
Controller Busy Time:               0
Power Cycles:                       2
Power On Hours:                     0
Unsafe Shutdowns:                   0
Media and Data Integrity Errors:    0
Error Information Log Entries:      0
Warning  Comp. Temperature Time:    0
Critical Comp. Temperature Time:    0

Error Information (NVMe Log 0x01, 16 of 256 entries)
No Errors Logged

nomad@NomadBSD$

nomad@NomadBSD$ sudo diskinfo -wS /dev/nvd2
/dev/nvd2
    512             # sectorsize
    400088457216    # mediasize in bytes (373G)
    781422768       # mediasize in sectors
    4096            # stripesize
    0               # stripeoffset
    INTEL SSDPF21Q400GB    # Disk descr.
    REMOVED    # Disk ident.
    nvme2           # Attachment
    Yes             # TRIM/UNMAP support
    0               # Rotation rate in RPM

Synchronous random writes:
     0.5 kbytes:     13.9 usec/IO =     35.0 Mbytes/s
       1 kbytes:     13.9 usec/IO =     70.3 Mbytes/s
       2 kbytes:     13.5 usec/IO =    144.9 Mbytes/s
       4 kbytes:     12.6 usec/IO =    310.1 Mbytes/s
       8 kbytes:     14.6 usec/IO =    534.6 Mbytes/s
      16 kbytes:     19.9 usec/IO =    784.8 Mbytes/s
      32 kbytes:     23.6 usec/IO =   1326.4 Mbytes/s
      64 kbytes:     31.5 usec/IO =   1984.3 Mbytes/s
     128 kbytes:     48.3 usec/IO =   2589.0 Mbytes/s
     256 kbytes:     77.2 usec/IO =   3239.7 Mbytes/s
     512 kbytes:    129.1 usec/IO =   3871.8 Mbytes/s
    1024 kbytes:    234.0 usec/IO =   4272.9 Mbytes/s
    2048 kbytes:    444.5 usec/IO =   4499.6 Mbytes/s
    4096 kbytes:    866.7 usec/IO =   4615.4 Mbytes/s
    8192 kbytes:   1710.0 usec/IO =   4678.3 Mbytes/s
nomad@NomadBSD$

Intel Optane SSD DC P5800X (400GB) - 4096b sector

Code:

nomad@NomadBSD$ sudo smartctl -a /dev/nvme2
smartctl 7.3 2022-02-28 r5338 [FreeBSD 13.1-RELEASE-p5 amd64] (local build)
Copyright (C) 2002-22, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Number:                       INTEL SSDPF21Q400GB
Serial Number:                      REMOVED
Firmware Version:                   L0310600
PCI Vendor/Subsystem ID:            0x8086
IEEE OUI Identifier:                0x5cd2e4
Total NVM Capacity:                 400,088,457,216 [400 GB]
Unallocated NVM Capacity:           0
Controller ID:                      0
NVMe Version:                       1.4
Number of Namespaces:               128
Local Time is:                      Thu Aug 10 17:45:29 2023 PDT
Firmware Updates (0x18):            4 Slots, no Reset required
Optional Admin Commands (0x001e):   Format Frmw_DL NS_Mngmt Self_Test
Optional NVM Commands (0x005e):     Wr_Unc DS_Mngmt Wr_Zero Sav/Sel_Feat Timestmp
Log Page Attributes (0x1e):         Cmd_Eff_Lg Ext_Get_Lg Telmtry_Lg Pers_Ev_Lg
Maximum Data Transfer Size:         32 Pages
Warning  Comp. Temp. Threshold:     70 Celsius
Critical Comp. Temp. Threshold:     80 Celsius

Supported Power States
St Op     Max   Active     Idle   RL RT WL WT  Ent_Lat  Ex_Lat
 0 +    13.00W       -        -    0  0  0  0        0       0
 1 +    11.00W       -        -    0  1  0  1        0       0
 2 +     9.00W       -        -    0  2  0  2        0       0

=== START OF SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

SMART/Health Information (NVMe Log 0x02)
Critical Warning:                   0x00
Temperature:                        38 Celsius
Available Spare:                    100%
Available Spare Threshold:          0%
Percentage Used:                    0%
Data Units Read:                    20 [10.2 MB]
Data Units Written:                 71,501 [36.6 GB]
Host Read Commands:                 621
Host Write Commands:                728,250
Controller Busy Time:               0
Power Cycles:                       3
Power On Hours:                     0
Unsafe Shutdowns:                   0
Media and Data Integrity Errors:    0
Error Information Log Entries:      0
Warning  Comp. Temperature Time:    0
Critical Comp. Temperature Time:    0

Error Information (NVMe Log 0x01, 16 of 256 entries)
No Errors Logged

nomad@NomadBSD$


nomad@NomadBSD$ sudo diskinfo -wS /dev/nvd2
/dev/nvd2
    4096            # sectorsize
    400088457216    # mediasize in bytes (373G)
    97677846        # mediasize in sectors
    0               # stripesize
    0               # stripeoffset
    INTEL SSDPF21Q400GB    # Disk descr.
    REMOVED    # Disk ident.
    nvme2           # Attachment
    Yes             # TRIM/UNMAP support
    0               # Rotation rate in RPM

Synchronous random writes:
       4 kbytes:     12.5 usec/IO =    311.8 Mbytes/s
       8 kbytes:     14.6 usec/IO =    536.5 Mbytes/s
      16 kbytes:     19.8 usec/IO =    789.2 Mbytes/s
      32 kbytes:     23.5 usec/IO =   1331.0 Mbytes/s
      64 kbytes:     31.5 usec/IO =   1985.4 Mbytes/s
     128 kbytes:     48.0 usec/IO =   2602.6 Mbytes/s
     256 kbytes:     76.4 usec/IO =   3273.7 Mbytes/s
     512 kbytes:    128.1 usec/IO =   3902.4 Mbytes/s
    1024 kbytes:    233.1 usec/IO =   4289.5 Mbytes/s
    2048 kbytes:    444.1 usec/IO =   4503.8 Mbytes/s
    4096 kbytes:    865.8 usec/IO =   4620.2 Mbytes/s
    8192 kbytes:   1709.1 usec/IO =   4680.9 Mbytes/s
nomad@NomadBSD$ 

 

[Elliot Dierksen]

I am in the process of upgrading my home lab, so I have some parts I am testing. The DC P4800X is a little faster than my original 900P.

280GB Intel Optane SSD 900P Series

Code:

root@freenas2:/ # smartctl -x /dev/nvme1
smartctl 7.2 2021-09-14 r5236 [FreeBSD 13.1-RELEASE-p7 amd64] (local build)
Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Number:                       INTEL SSDPED1D280GA
Serial Number:                      PHMB742401A6280CGN
Firmware Version:                   E2010603
PCI Vendor/Subsystem ID:            0x8086
IEEE OUI Identifier:                0x5cd2e4
Controller ID:                      0
NVMe Version:                       <1.2
Number of Namespaces:               1
Namespace 1 Size/Capacity:          280,065,171,456 [280 GB]
Namespace 1 Formatted LBA Size:     512
Local Time is:                      Tue Nov 14 08:19:38 2023 EST
Firmware Updates (0x02):            1 Slot
Optional Admin Commands (0x0007):   Security Format Frmw_DL
Optional NVM Commands (0x0006):     Wr_Unc DS_Mngmt
Log Page Attributes (0x0a):         Cmd_Eff_Lg Telmtry_Lg
Maximum Data Transfer Size:         32 Pages

Supported Power States
St Op     Max   Active     Idle   RL RT WL WT  Ent_Lat  Ex_Lat
 0 +    18.00W       -        -    0  0  0  0        0       0

Supported LBA Sizes (NSID 0x1)
Id Fmt  Data  Metadt  Rel_Perf
 0 +     512       0         2

=== START OF SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

SMART/Health Information (NVMe Log 0x02)
Critical Warning:                   0x00
Temperature:                        31 Celsius
Available Spare:                    100%
Available Spare Threshold:          0%
Percentage Used:                    0%
Data Units Read:                    1,035,634 [530 GB]
Data Units Written:                 234,033,091 [119 TB]
Host Read Commands:                 4,575,271
Host Write Commands:                4,809,883,438
Controller Busy Time:               1,153
Power Cycles:                       173
Power On Hours:                     36,966
Unsafe Shutdowns:                   22
Media and Data Integrity Errors:    0
Error Information Log Entries:      0

Error Information (NVMe Log 0x01, 16 of 64 entries)
No Errors Logged

root@freenas2:/ #
root@freenas2:/ # diskinfo -wS /dev/nvd1
/dev/nvd1
        512             # sectorsize
        280065171456    # mediasize in bytes (261G)
        547002288       # mediasize in sectors
        0               # stripesize
        0               # stripeoffset
        INTEL SSDPED1D280GA     # Disk descr.
        PHMB742401A6280CGN      # Disk ident.
        nvme1           # Attachment
        Yes             # TRIM/UNMAP support
        0               # Rotation rate in RPM

Synchronous random writes:
         0.5 kbytes:     14.2 usec/IO =     34.5 Mbytes/s
           1 kbytes:     14.2 usec/IO =     68.7 Mbytes/s
           2 kbytes:     14.7 usec/IO =    133.1 Mbytes/s
           4 kbytes:     11.8 usec/IO =    330.0 Mbytes/s
           8 kbytes:     14.1 usec/IO =    552.9 Mbytes/s
          16 kbytes:     19.4 usec/IO =    806.8 Mbytes/s
          32 kbytes:     29.2 usec/IO =   1069.0 Mbytes/s
          64 kbytes:     56.1 usec/IO =   1114.2 Mbytes/s
         128 kbytes:     95.2 usec/IO =   1313.0 Mbytes/s
         256 kbytes:    169.9 usec/IO =   1471.3 Mbytes/s
         512 kbytes:    314.2 usec/IO =   1591.5 Mbytes/s
        1024 kbytes:    603.7 usec/IO =   1656.4 Mbytes/s
        2048 kbytes:   1175.7 usec/IO =   1701.1 Mbytes/s
        4096 kbytes:   2334.3 usec/IO =   1713.5 Mbytes/s
        8192 kbytes:   4714.9 usec/IO =   1696.8 Mbytes/s

375GB Intel SSD DC PCIE P4800X

Code:

root@freenas2:/ # smartctl -x /dev/nvme0
smartctl 7.2 2021-09-14 r5236 [FreeBSD 13.1-RELEASE-p7 amd64] (local build)
Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Number:                       INTEL MDTPED1K375GA
Serial Number:                      PHKS912500HK375AGN
Firmware Version:                   E2010603
PCI Vendor/Subsystem ID:            0x8086
IEEE OUI Identifier:                0x5cd2e4
Controller ID:                      0
NVMe Version:                       <1.2
Number of Namespaces:               1
Namespace 1 Size/Capacity:          375,083,606,016 [375 GB]
Namespace 1 Formatted LBA Size:     512
Namespace 1 IEEE EUI-64:            5cd2e4 3c44ce0100
Local Time is:                      Tue Nov 14 08:19:02 2023 EST
Firmware Updates (0x02):            1 Slot
Optional Admin Commands (0x0007):   Security Format Frmw_DL
Optional NVM Commands (0x0006):     Wr_Unc DS_Mngmt
Log Page Attributes (0x0a):         Cmd_Eff_Lg Telmtry_Lg
Maximum Data Transfer Size:         32 Pages

Supported Power States
St Op     Max   Active     Idle   RL RT WL WT  Ent_Lat  Ex_Lat
 0 +    18.00W       -        -    0  0  0  0        0       0

Supported LBA Sizes (NSID 0x1)
Id Fmt  Data  Metadt  Rel_Perf
 0 +     512       0         2
 1 -     512       8         2
 2 -     512      16         2
 3 -    4096       0         0
 4 -    4096       8         0
 5 -    4096      64         0
 6 -    4096     128         0

=== START OF SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

SMART/Health Information (NVMe Log 0x02)
Critical Warning:                   0x00
Temperature:                        38 Celsius
Available Spare:                    100%
Available Spare Threshold:          0%
Percentage Used:                    0%
Data Units Read:                    1,322 [676 MB]
Data Units Written:                 10,899,461 [5.58 TB]
Host Read Commands:                 20,932
Host Write Commands:                70,010,727
Controller Busy Time:               41
Power Cycles:                       93
Power On Hours:                     22,212
Unsafe Shutdowns:                   37
Media and Data Integrity Errors:    0
Error Information Log Entries:      0

Error Information (NVMe Log 0x01, 16 of 64 entries)
No Errors Logged

root@freenas2:/ #
root@freenas2:/ # diskinfo -wS /dev/nvd0
/dev/nvd0
        512             # sectorsize
        375083606016    # mediasize in bytes (349G)
        732585168       # mediasize in sectors
        0               # stripesize
        0               # stripeoffset
        INTEL MDTPED1K375GA     # Disk descr.
        PHKS912500HK375AGN      # Disk ident.
        nvme0           # Attachment
        Yes             # TRIM/UNMAP support
        0               # Rotation rate in RPM

Synchronous random writes:
         0.5 kbytes:     13.5 usec/IO =     36.2 Mbytes/s
           1 kbytes:     13.6 usec/IO =     72.0 Mbytes/s
           2 kbytes:     13.9 usec/IO =    141.0 Mbytes/s
           4 kbytes:     11.4 usec/IO =    343.6 Mbytes/s
           8 kbytes:     13.2 usec/IO =    593.4 Mbytes/s
          16 kbytes:     17.4 usec/IO =    897.8 Mbytes/s
          32 kbytes:     25.5 usec/IO =   1223.5 Mbytes/s
          64 kbytes:     46.3 usec/IO =   1351.3 Mbytes/s
         128 kbytes:     80.0 usec/IO =   1561.8 Mbytes/s
         256 kbytes:    139.0 usec/IO =   1799.1 Mbytes/s
         512 kbytes:    252.8 usec/IO =   1978.1 Mbytes/s
        1024 kbytes:    479.6 usec/IO =   2085.0 Mbytes/s
        2048 kbytes:    933.6 usec/IO =   2142.3 Mbytes/s
        4096 kbytes:   1838.2 usec/IO =   2176.0 Mbytes/s
        8192 kbytes:   3671.7 usec/IO =   2178.8 Mbytes/s

I have some other newer ones that should be showing up soon, so I'll post results on them when they arrive. I am excited to see how the P4800X and changing from spinning rust to WD RED NAS SSD's affect my performance.

 

[Elliot Dierksen]

WD (HGST) Ultrastar SN200 1.6T

Code:

root@freenas:/ # smartctl -x /dev/nvme0
smartctl 7.2 2021-09-14 r5236 [FreeBSD 13.1-RELEASE-p7 amd64] (local build)
Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Number:                       UCSC-NVMEHW-H1600
Serial Number:                      SDM00006923F
Firmware Version:                   KNCCD101
PCI Vendor ID:                      0x1c58
PCI Vendor Subsystem ID:            0x1137
IEEE OUI Identifier:                0x000cca
Total NVM Capacity:                 1,602,022,801,408 [1.60 TB]
Unallocated NVM Capacity:           0
Controller ID:                      566
NVMe Version:                       1.2.1
Number of Namespaces:               1
Namespace 1 Size/Capacity:          1,600,321,314,816 [1.60 TB]
Namespace 1 Formatted LBA Size:     512
Namespace 1 IEEE EUI-64:            000cca 0c025a7c80
Local Time is:                      Fri Nov 17 14:52:54 2023 EST
Firmware Updates (0x0b):            5 Slots, Slot 1 R/O
Optional Admin Commands (0x000e):   Format Frmw_DL NS_Mngmt
Optional NVM Commands (0x003f):     Comp Wr_Unc DS_Mngmt Wr_Zero Sav/Sel_Feat Resv
Log Page Attributes (0x03):         S/H_per_NS Cmd_Eff_Lg
Warning  Comp. Temp. Threshold:     70 Celsius
Critical Comp. Temp. Threshold:     75 Celsius

Supported Power States
St Op     Max   Active     Idle   RL RT WL WT  Ent_Lat  Ex_Lat
 0 +    25.00W       -        -    0  0  0  0    15000   15000
 1 +    24.00W       -        -    1  1  1  1    15000   15000
 2 +    23.00W       -        -    2  2  2  2    15000   15000
 3 +    22.00W       -        -    3  3  3  3    15000   15000
 4 +    21.00W       -        -    4  4  4  4    15000   15000
 5 +    20.00W       -        -    5  5  5  5    15000   15000
 6 +    19.00W       -        -    6  6  6  6    15000   15000
 7 +    18.00W       -        -    7  7  7  7    15000   15000
 8 +    17.00W       -        -    8  8  8  8    15000   15000
 9 +    16.00W       -        -    9  9  9  9    15000   15000
10 +    15.00W       -        -   10 10 10 10    15000   15000
11 +    14.00W       -        -   11 11 11 11    15000   15000
12 +    13.00W       -        -   12 12 12 12    15000   15000
13 +    12.00W       -        -   13 13 13 13    15000   15000
14 +    11.00W       -        -   14 14 14 14    15000   15000
15 -    10.00W       -        -   15 15 15 15    15000   15000

Supported LBA Sizes (NSID 0x1)
Id Fmt  Data  Metadt  Rel_Perf
 0 +     512       0         0
 1 -     512       8         2
 2 -    4096       0         0
 3 -    4096       8         1

=== START OF SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

SMART/Health Information (NVMe Log 0x02)
Critical Warning:                   0x00
Temperature:                        49 Celsius
Available Spare:                    100%
Available Spare Threshold:          10%
Percentage Used:                    0%
Data Units Read:                    5,069,341 [2.59 TB]
Data Units Written:                 24,764,362 [12.6 TB]
Host Read Commands:                 207,895,165
Host Write Commands:                1,472,104,715
Controller Busy Time:               124,930
Power Cycles:                       59
Power On Hours:                     10,125
Unsafe Shutdowns:                   56
Media and Data Integrity Errors:    0
Error Information Log Entries:      0
Warning  Comp. Temperature Time:    0
Critical Comp. Temperature Time:    0
Temperature Sensor 1:               49 Celsius
Temperature Sensor 2:               47 Celsius
Temperature Sensor 3:               40 Celsius

Error Information (NVMe Log 0x01, 16 of 256 entries)
No Errors Logged

root@freenas:/ #
root@freenas:/ # diskinfo -wS /dev/nvd0
/dev/nvd0
        512             # sectorsize
        1600321314816   # mediasize in bytes (1.5T)
        3125627568      # mediasize in sectors
        0               # stripesize
        0               # stripeoffset
        UCSC-NVMEHW-H1600       # Disk descr.
        SDM00006923F    # Disk ident.
        nvme0           # Attachment
        Yes             # TRIM/UNMAP support
        0               # Rotation rate in RPM

Synchronous random writes:
         0.5 kbytes:     17.7 usec/IO =     27.6 Mbytes/s
           1 kbytes:     18.6 usec/IO =     52.5 Mbytes/s
           2 kbytes:     20.2 usec/IO =     96.6 Mbytes/s
           4 kbytes:     24.3 usec/IO =    161.0 Mbytes/s
           8 kbytes:     27.7 usec/IO =    282.0 Mbytes/s
          16 kbytes:     35.5 usec/IO =    439.8 Mbytes/s
          32 kbytes:     54.6 usec/IO =    572.1 Mbytes/s
          64 kbytes:     85.8 usec/IO =    728.8 Mbytes/s
         128 kbytes:    145.9 usec/IO =    856.7 Mbytes/s
         256 kbytes:    151.6 usec/IO =   1649.6 Mbytes/s
         512 kbytes:    239.4 usec/IO =   2088.6 Mbytes/s
        1024 kbytes:    479.8 usec/IO =   2084.2 Mbytes/s
        2048 kbytes:   1044.3 usec/IO =   1915.2 Mbytes/s
        4096 kbytes:   2578.4 usec/IO =   1551.4 Mbytes/s
        8192 kbytes:   5058.5 usec/IO =   1581.5 Mbytes/s

Micro 7450 Max 800G

Code:

root@freenas:/ # smartctl -x /dev/nvme1
smartctl 7.2 2021-09-14 r5236 [FreeBSD 13.1-RELEASE-p7 amd64] (local build)
Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Number:                       Micron_7450_MTFDKCC800TFS
Serial Number:                      233643282FD9
Firmware Version:                   E2MU200
PCI Vendor/Subsystem ID:            0x1344
IEEE OUI Identifier:                0x00a075
Total NVM Capacity:                 800,166,076,416 [800 GB]
Unallocated NVM Capacity:           0
Controller ID:                      0
NVMe Version:                       1.4
Number of Namespaces:               132
Local Time is:                      Fri Nov 17 14:53:26 2023 EST
Firmware Updates (0x17):            3 Slots, Slot 1 R/O, no Reset required
Optional Admin Commands (0x005e):   Format Frmw_DL NS_Mngmt Self_Test MI_Snd/Rec
Optional NVM Commands (0x00df):     Comp Wr_Unc DS_Mngmt Wr_Zero Sav/Sel_Feat Timestmp Verify
Log Page Attributes (0x1e):         Cmd_Eff_Lg Ext_Get_Lg Telmtry_Lg Pers_Ev_Lg
Maximum Data Transfer Size:         1024 Pages
Warning  Comp. Temp. Threshold:     77 Celsius
Critical Comp. Temp. Threshold:     85 Celsius

Supported Power States
St Op     Max   Active     Idle   RL RT WL WT  Ent_Lat  Ex_Lat
 0 +    25.00W       -        -    0  0  0  0        0       0
 1 +    20.00W       -        -    1  1  1  1        0       0
 2 +    18.00W       -        -    2  2  2  2        0       0
 3 +    16.00W       -        -    3  3  3  3        0       0
 4 +    14.00W       -        -    4  4  4  4        0       0
 5 +    12.00W       -        -    5  5  5  5        0       0
 6 +    10.00W       -        -    6  6  6  6        0       0
 7 +     8.25W       -        -    7  7  7  7        0       0

=== START OF SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

SMART/Health Information (NVMe Log 0x02)
Critical Warning:                   0x00
Temperature:                        35 Celsius
Available Spare:                    100%
Available Spare Threshold:          10%
Percentage Used:                    0%
Data Units Read:                    4 [2.04 MB]
Data Units Written:                 0
Host Read Commands:                 32
Host Write Commands:                0
Controller Busy Time:               0
Power Cycles:                       2
Power On Hours:                     0
Unsafe Shutdowns:                   1
Media and Data Integrity Errors:    0
Error Information Log Entries:      0
Warning  Comp. Temperature Time:    0
Critical Comp. Temperature Time:    0
Temperature Sensor 1:               41 Celsius
Temperature Sensor 2:               37 Celsius
Temperature Sensor 3:               35 Celsius

Error Information (NVMe Log 0x01, 16 of 256 entries)
No Errors Logged

root@freenas:/ #
root@freenas:/ # diskinfo -wS /dev/nvd1
/dev/nvd1
        512             # sectorsize
        800166076416    # mediasize in bytes (745G)
        1562824368      # mediasize in sectors
        4096            # stripesize
        0               # stripeoffset
        Micron_7450_MTFDKCC800TFS       # Disk descr.
        233643282FD9    # Disk ident.
        nvme1           # Attachment
        Yes             # TRIM/UNMAP support
        0               # Rotation rate in RPM

Synchronous random writes:
         0.5 kbytes:     11.8 usec/IO =     41.5 Mbytes/s
           1 kbytes:     11.9 usec/IO =     82.1 Mbytes/s
           2 kbytes:     12.0 usec/IO =    162.3 Mbytes/s
           4 kbytes:     10.9 usec/IO =    359.3 Mbytes/s
           8 kbytes:     12.8 usec/IO =    609.9 Mbytes/s
          16 kbytes:     15.8 usec/IO =    988.1 Mbytes/s
          32 kbytes:     22.4 usec/IO =   1393.1 Mbytes/s
          64 kbytes:     42.9 usec/IO =   1457.6 Mbytes/s
         128 kbytes:     91.4 usec/IO =   1367.4 Mbytes/s
         256 kbytes:    172.8 usec/IO =   1446.6 Mbytes/s
         512 kbytes:    352.5 usec/IO =   1418.3 Mbytes/s
        1024 kbytes:    691.4 usec/IO =   1446.3 Mbytes/s
        2048 kbytes:   1425.4 usec/IO =   1403.1 Mbytes/s
        4096 kbytes:   2742.5 usec/IO =   1458.5 Mbytes/s
        8192 kbytes:   5603.9 usec/IO =   1427.6 Mbytes/s

 

[pawelr]

One more Radian RMS-200. This is 4GB version.

=== START OF INFORMATION SECTION ===
Model Number: RMS-200
Serial Number: 0000956
Firmware Version: 35ac92cc
PCI Vendor/Subsystem ID: 0x1cc7
IEEE OUI Identifier: 0x00e0cf
Controller ID: 0
NVMe Version: <1.2
Number of Namespaces: 1
Namespace 1 Size/Capacity: 4,286,578,688 [4.28 GB]
Namespace 1 Utilization: 0
Namespace 1 Formatted LBA Size: 512
Local Time is: Fri Dec 22 00:58:27 2023 CET
Firmware Updates (0x09): 4 Slots, Slot 1 R/O
Optional Admin Commands (0x0006): Format Frmw_DL
Optional NVM Commands (0x0006): Wr_Unc DS_Mngmt
Log Page Attributes (0x01): S/H_per_NS

Supported Power States
St Op Max Active Idle RL RT WL WT Ent_Lat Ex_Lat
0 + 25.00W - - 0 0 0 0 100 100

Supported LBA Sizes (NSID 0x1)
Id Fmt Data Metadt Rel_Perf
0 - 512 0 3
1 - 512 8 3
2 - 4096 0 0
3 - 4096 8 0

=== START OF SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

SMART/Health Information (NVMe Log 0x02)
Critical Warning: 0x00
Temperature: 43 Celsius
Available Spare: 0%
Available Spare Threshold: 0%
Percentage Used: 0%
Data Units Read: 90,432,076,629 [46.3 PB]
Data Units Written: 90,599,869,799 [46.3 PB]
Host Read Commands: 380,309,164
Host Write Commands: 385,411,210
Controller Busy Time: 0
Power Cycles: 957
Power On Hours: 10,301
Unsafe Shutdowns: 953
Media and Data Integrity Errors: 0
Error Information Log Entries: 94

=== START OF INFORMATION SECTION ===
Model Number: RMS-200
Serial Number: 0000956
Firmware Version: 35ac92cc
PCI Vendor/Subsystem ID: 0x1cc7
IEEE OUI Identifier: 0x00e0cf
Controller ID: 0
NVMe Version: <1.2
Number of Namespaces: 1
Namespace 1 Size/Capacity: 4,286,578,688 [4.28 GB]
Namespace 1 Utilization: 0
Namespace 1 Formatted LBA Size: 512
Local Time is: Fri Dec 22 00:58:27 2023 CET
Firmware Updates (0x09): 4 Slots, Slot 1 R/O
Optional Admin Commands (0x0006): Format Frmw_DL
Optional NVM Commands (0x0006): Wr_Unc DS_Mngmt
Log Page Attributes (0x01): S/H_per_NS

Supported Power States
St Op Max Active Idle RL RT WL WT Ent_Lat Ex_Lat
0 + 25.00W - - 0 0 0 0 100 100

Supported LBA Sizes (NSID 0x1)
Id Fmt Data Metadt Rel_Perf
0 - 512 0 3
1 - 512 8 3
2 - 4096 0 0
3 - 4096 8 0

=== START OF SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

SMART/Health Information (NVMe Log 0x02)
Critical Warning: 0x00
Temperature: 43 Celsius
Available Spare: 0%
Available Spare Threshold: 0%
Percentage Used: 0%
Data Units Read: 90,432,076,629 [46.3 PB]
Data Units Written: 90,599,869,799 [46.3 PB]
Host Read Commands: 380,309,164
Host Write Commands: 385,411,210
Controller Busy Time: 0
Power Cycles: 957
Power On Hours: 10,301
Unsafe Shutdowns: 953
Media and Data Integrity Errors: 0
Error Information Log Entries: 94

 

[s00hr7]

I recently got a cheap RMS-200 from ebay. This one has already seen the occasional Petabyte :)

Code:

Data Units Read:                    82,290,983,918 [42.1 PB]
Data Units Written:                 3,746,666,139,387 [1918 PB]
Host Read Commands:                 411,692,593
Host Write Commands:                379,509,339,451
Controller Busy Time:               397
Power Cycles:                       19
Power On Hours:                     35,975
Unsafe Shutdowns:                   12

I formatted it to 4k and it seems to perfrom extremely well:

Code:

diskinfo -wS /dev/nvd0
/dev/nvd0
        4096            # sectorsize
        8581545984      # mediasize in bytes (8.0G)
        2095104         # mediasize in sectors
        0               # stripesize
        0               # stripeoffset
        RMS-200         # Disk descr.
        0091713         # Disk ident.
        nvme0           # Attachment
        Yes             # TRIM/UNMAP support
        0               # Rotation rate in RPM

Synchronous random writes:
           4 kbytes:     19.1 usec/IO =    205.0 Mbytes/s
           8 kbytes:     19.9 usec/IO =    391.9 Mbytes/s
          16 kbytes:     25.8 usec/IO =    605.1 Mbytes/s
          32 kbytes:     33.0 usec/IO =    947.3 Mbytes/s
          64 kbytes:     48.8 usec/IO =   1280.8 Mbytes/s
         128 kbytes:     89.7 usec/IO =   1393.3 Mbytes/s
         256 kbytes:     93.4 usec/IO =   2675.4 Mbytes/s
         512 kbytes:    146.4 usec/IO =   3414.6 Mbytes/s
        1024 kbytes:    241.9 usec/IO =   4134.0 Mbytes/s
        2048 kbytes:    465.3 usec/IO =   4298.6 Mbytes/s
        4096 kbytes:    906.8 usec/IO =   4411.3 Mbytes/s
        8192 kbytes:   1727.9 usec/IO =   4629.8 Mbytes/s

I am surprised that this seems to be significantly faster than the result helloadam has been posting for his RMS-300. Could this just be the different LBA size?

I am actually baffled by the improvement in write speed for sync writes. I used my TrueNAS for the last couple of years mainly as an archive storage. Recently I started using it for VM data storage and started doing some benchmarks. As expected, the sync write performance was abysmal (4x 870 EVOs in 2x striped mirrors):

Code:

$ fio --name=seq-write --ioengine=posixaio --rw=randwrite --bs=128k --size=128m --runtime=120 --time_based   
[...]
WRITE: bw=42.1MiB/s (44.1MB/s), 42.1MiB/s-42.1MiB/s (44.1MB/s-44.1MB/s), io=5052MiB (5297MB), run=120002-120002msec

Adding the RMS-200 improved the performance slightly :)

Code:

$ zpool add SSD log /dev/nvd0
$ fio --name=seq-write --ioengine=posixaio --rw=randwrite --bs=128k --size=128m --runtime=120 --time_based   
[...]
WRITE: bw=1308MiB/s (1372MB/s), 1308MiB/s-1308MiB/s (1372MB/s-1372MB/s), io=153GiB (165GB), run=120001-120001msec

The only thing that annoyed me about the card was the fan. I have my rack right next to my working desk and this fan was by far the loudest noise in the entire rack. I replaced it with a NF-A4x20 which is not only significantly quiter but also reduced the overall temperature by ~10°C. Note that the original fan had a height of 7mm and this one is 20mm. By placing it in the lowest PCIe slot, this wasn't an issue.

 

[HoneyBadger]

Code:

Data Units Written: 3,746,666,139,387 [1918 PB]

Almost 2 EB - that card's got some mileage on it - but it's still trucking along!

I am surprised that this seems to be significantly faster than the result @helloadam has been posting for his RMS-300. Could this just be the different LBA size?

I believe that @helloadam mentioned a disclaimer around their testing machine that likely explains this:

Our lab bench machine is by no means the best but it allows us to quickly test things without pushing hardware into production. I mention this because the results below should be compared against each other vs perhaps others in this thread for the same hardware.

NVMe devices, especially as they get faster, are very sensitive to things like CPU performance, power savings settings, lane availability/generation and slot placement.

I am actually baffled by the improvement in write speed for sync writes.

The secret sauce is the ability of the RMS cards (and other PLP-equipped devices) to treat all writes as having landed on non-volatile storage, thus letting them immediately respond to SCSI commands to flush their volatile cache. It simply says "already did it, return success" when asked. That's why we point out the difference between the PLP-equipped SSDs here and how they differ from even a "performance consumer" SSD like your EVOs (although the SATA bus is a pretty significant bottleneck in 2024)