Resource icon

SLOG benchmarking and finding the best SLOG

dak180

Patron
Joined
Nov 22, 2017
Messages
307
I would say that it likely would not matter with 1G max clients. With 10G connections to FreeNAS, it definitely matters!
If I had 10G it would not have even been a question; but my best guess is that I am unlikely get a 10G link to my freenas for at least 5 years, so not something I am going to worry about now.
 

Constantin

Vampire Pig
Joined
May 19, 2017
Messages
1,828
With a single gigabit connection, maximum Ethernet speed will be about 125MB/s. I don’t see how ultra-fast SLOGs could make a significant difference - even a standard SSD on a SATA3 connection is likely “fast” enough.
 

HoneyBadger

actually does care
Administrator
Moderator
iXsystems
Joined
Feb 6, 2014
Messages
5,110
With a single gigabit connection, maximum Ethernet speed will be about 125MB/s. I don’t see how ultra-fast SLOGs could make a significant difference - even a standard SSD on a SATA3 connection is likely “fast” enough.
The key is the record size in question. It's much easier to hit line speed at a large 128K record on a file share versus smaller ones like 4K for VM disks, for example. At 16K there's a number of SATA SSDs that can do it, but go down to 8K or 4K and you'll have to go with pretty high end SATA to get there.
 

Constantin

Vampire Pig
Joined
May 19, 2017
Messages
1,828
My (hopefully!) next SLOG just arrived. Still waiting on the motherboard and RAM though! Can't wait to test! 100GB Intel Optane 4801x...

P4801x.jpg
 

HoneyBadger

actually does care
Administrator
Moderator
iXsystems
Joined
Feb 6, 2014
Messages
5,110
My (hopefully!) next SLOG just arrived. Still waiting on the motherboard and RAM though! Can't wait to test! 100GB Intel Optane 4801x...

View attachment 29070
Seriously excited about the potential of these. It's significantly less expensive than the P4800X but still has that stamp of "Enterprise Approved, Not Consumer Grade" that makes it oh so tasty. (Although if you're going to build an enterprise solution, put a U.2 model in a hotswap-friendly bay. No one wants downtime for a broken SLOG.)

Give it some good cooling and let's see those numbers (at both 512e and 4Kn sectors, maybe?)
 

Constantin

Vampire Pig
Joined
May 19, 2017
Messages
1,828
Here you go. Intel Optane 100GB P4801x as delivered, apparently with 512 sector sizes. Should I reformat?

Code:
/dev/nvd0
        512             # sectorsize
        100030242816    # mediasize in bytes (93G)
        195371568       # mediasize in sectors
        0               # stripesize
        0               # stripeoffset
        INTEL SSDPEL1K100GA     # Disk descr.
        PHKM90150015100D        # Disk ident.
        Yes             # TRIM/UNMAP support
        0               # Rotation rate in RPM

Synchronous random writes:
         0.5 kbytes:     28.0 usec/IO =     17.4 Mbytes/s
           1 kbytes:     27.0 usec/IO =     36.1 Mbytes/s
           2 kbytes:     27.5 usec/IO =     71.1 Mbytes/s
           4 kbytes:     23.4 usec/IO =    167.1 Mbytes/s
           8 kbytes:     30.4 usec/IO =    257.0 Mbytes/s
          16 kbytes:     44.1 usec/IO =    354.3 Mbytes/s
          32 kbytes:     64.6 usec/IO =    483.5 Mbytes/s
          64 kbytes:    103.7 usec/IO =    602.7 Mbytes/s
         128 kbytes:    161.1 usec/IO =    776.1 Mbytes/s
         256 kbytes:    285.8 usec/IO =    874.6 Mbytes/s
         512 kbytes:    527.5 usec/IO =    947.9 Mbytes/s
        1024 kbytes:    988.2 usec/IO =   1012.0 Mbytes/s
        2048 kbytes:   1905.6 usec/IO =   1049.5 Mbytes/s
        4096 kbytes:   3730.2 usec/IO =   1072.3 Mbytes/s
        8192 kbytes:   7398.6 usec/IO =   1081.3 Mbytes/s


Anyhow, the biggest delta vs. my older SLOGs (200GB Intel S3710 SATA3 SSDs) is at the low end - 8x faster - while with large writes the speed difference is "only" 3x. When I manage to get my old SLOGs removed from the pool (that's another story) I might be able to see the benefits...
 
Last edited:

rshakin

Dabbler
Joined
Apr 2, 2019
Messages
24
Here's a fun one, something you don't see very often a PMC-1604NVRAM

Code:
=== START OF INFORMATION SECTION ===
Model Number:                       MTR_MLC_TS_4GB
Serial Number:                      04000017A3ED
Firmware Version:                   2.6.06.1
PCI Vendor/Subsystem ID:            0x11f8
IEEE OUI Identifier:                0x00e004
Controller ID:                      0
Number of Namespaces:               1
Namespace 1 Size/Capacity:          4,255,121,408 [4.25 GB]
Namespace 1 Formatted LBA Size:     512
Namespace 1 IEEE EUI-64:            04e000 017ef75c01
Local Time is:                      Tue Apr  2 15:42:18 2019 PDT
Firmware Updates (0x07):            3 Slots, Slot 1 R/O
Optional Admin Commands (0x0006):   Format Frmw_DL

Supported Power States
St Op     Max   Active     Idle   RL RT WL WT  Ent_Lat  Ex_Lat
 0 +    25.00W       -        -    0  0  0  0      100     100

Supported LBA Sizes (NSID 0x1)
Id Fmt  Data  Metadt  Rel_Perf
 0 +     512       0         3
 1 -     512       0         3
 2 -    4096       0         0
 3 -    4096       0         0

=== START OF SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

SMART/Health Information (NVMe Log 0x02, NSID 0xffffffff)
Critical Warning:                   0x00
Temperature:                        50 Celsius
Available Spare:                    0%
Available Spare Threshold:          0%
Percentage Used:                    0%
Data Units Read:                    522 [267 MB]
Data Units Written:                 55,657 [28.4 GB]
Host Read Commands:                 3,266
Host Write Commands:                366,349
Controller Busy Time:               0
Power Cycles:                       0
Power On Hours:                     74
Unsafe Shutdowns:                   3
Media and Data Integrity Errors:    0
Error Information Log Entries:      0

Error Information (NVMe Log 0x01, max 63 entries)
No Errors Logged


Code:
root@freenas[~]# diskinfo -wS /dev/nvd0
/dev/nvd0
        512             # sectorsize
        4255121408      # mediasize in bytes (4.0G)
        8310784         # mediasize in sectors
        0               # stripesize
        0               # stripeoffset
        MTR_MLC_TS_4GB  # Disk descr.
        04000017A3ED    # Disk ident.
        No              # TRIM/UNMAP support
        0               # Rotation rate in RPM

Synchronous random writes:
         0.5 kbytes:     47.4 usec/IO =     10.3 Mbytes/s
           1 kbytes:     48.4 usec/IO =     20.2 Mbytes/s
           2 kbytes:     48.1 usec/IO =     40.6 Mbytes/s
           4 kbytes:     50.1 usec/IO =     78.0 Mbytes/s
           8 kbytes:     51.5 usec/IO =    151.7 Mbytes/s
          16 kbytes:     60.5 usec/IO =    258.3 Mbytes/s
          32 kbytes:     67.9 usec/IO =    460.3 Mbytes/s
          64 kbytes:     83.6 usec/IO =    747.4 Mbytes/s
         128 kbytes:    126.0 usec/IO =    991.7 Mbytes/s
         256 kbytes:    143.9 usec/IO =   1737.5 Mbytes/s
         512 kbytes:    198.3 usec/IO =   2520.9 Mbytes/s
        1024 kbytes:    312.4 usec/IO =   3201.2 Mbytes/s
        2048 kbytes:    553.5 usec/IO =   3613.7 Mbytes/s
        4096 kbytes:    953.2 usec/IO =   4196.5 Mbytes/s
        8192 kbytes:   1800.1 usec/IO =   4444.1 Mbytes/s
root@freenas[~]#

 

HoneyBadger

actually does care
Administrator
Moderator
iXsystems
Joined
Feb 6, 2014
Messages
5,110
For what's essentially RAM on an NVMe bus, those small recordsize numbers are surprisingly low. I'd have expected it to perform a lot better than that; as it sits it's getting outpaced by the Optane cards down at the 4/8/16K.

It absolutely blows the doors off of everything else at the high end though. 4GB/s? Gonna have to find one of those.
 

Constantin

Vampire Pig
Joined
May 19, 2017
Messages
1,828
Gonna have to find one of those.
Ebay has a bunch on sale for about 1/2 the cost of the Optane P4801x. Seems like a good deal unless you need the speed at the small record sizes. Only downside is the higher power consumption (is it really 25W?- If so, wow!) and the loss of a PCIe slot. I'm surprised that this module isn't listed in ServeTheHome as one of the alternatives to ZeusRAM and so on.

Anyhow, if the 25W power consumption is true a mirrored SLOG would consume 50W and 2 PCIe slots. In my rig, that would eliminate all PCIe slots and use more power than the entire rest of the motherboard. :oops:
 

HoneyBadger

actually does care
Administrator
Moderator
iXsystems
Joined
Feb 6, 2014
Messages
5,110
Ebay has a bunch on sale for about 1/2 the cost of the Optane P4801x
Most of the ones I can find seem to be lacking the supercapacitor/battery backup module (which is kind of important for an SLOG device) but I'll keep an alert up for them and see if I can strike a deal.

Speaking of power consumption numbers, @rshakin did you make sure that any PCIe power-saving or sleep states were disabled? Or maybe it just requires a vendor-specific NVMe driver (which is virtually guaranteed not to be available on FreeBSD)
 

rshakin

Dabbler
Joined
Apr 2, 2019
Messages
24
For what's essentially RAM on an NVMe bus, those small recordsize numbers are surprisingly low. I'd have expected it to perform a lot better than that; as it sits it's getting outpaced by the Optane cards down at the 4/8/16K.

It absolutely blows the doors off of everything else at the high end though. 4GB/s? Gonna have to find one of those.

Yeah I am not sure why but this is their low end model, it's almost impossible to get those hi-end 16gb versions that support dma access.... now thats a beast that I would love to catch. Also this is 2014 tech... so who knows still does very well for slog in my box, I have 2 of these ;)
Most of the ones I can find seem to be lacking the supercapacitor/battery backup module (which is kind of important for an SLOG device) but I'll keep an alert up for them and see if I can strike a deal.

Speaking of power consumption numbers, @rshakin did you make sure that any PCIe power-saving or sleep states were disabled? Or maybe it just requires a vendor-specific NVMe driver (which is virtually guaranteed not to be available on FreeBSD)
Most of the ones I can find seem to be lacking the supercapacitor/battery backup module (which is kind of important for an SLOG device) but I'll keep an alert up for them and see if I can strike a deal.

Speaking of power consumption numbers, @rshakin did you make sure that any PCIe power-saving or sleep states were disabled? Or maybe it just requires a vendor-specific NVMe driver (which is virtually guaranteed not to be available on FreeBSD)

Well I can live with a little bit of that, but no nothing was done to the pcie power, one problem that I have is that even if I do a safe shutdown on the smart data it shows as unsafe shutdown... As far as everything else seems to work.
 

HoneyBadger

actually does care
Administrator
Moderator
iXsystems
Joined
Feb 6, 2014
Messages
5,110
I've got 2005 tech as an SLOG in a box right now, kind of like the great-grandfather to the PMC card. ;)

I'll keep my eyes peeled for one of these, since if it can make the numbers on the datasheet it would be a hell of an SLOG device, basically equivalent to running sync=disabled in performance. But I'll also try to chase down an Optane P4801X as well. So many toys, so little time.
 

rshakin

Dabbler
Joined
Apr 2, 2019
Messages
24
I've got 2005 tech as an SLOG in a box right now, kind of like the great-grandfather to the PMC card. ;)

I'll keep my eyes peeled for one of these, since if it can make the numbers on the datasheet it would be a hell of an SLOG device, basically equivalent to running sync=disabled in performance. But I'll also try to chase down an Optane P4801X as well. So many toys, so little time.
Yeah I still have to test actual nfs performance still saving money for some 10 gig cards and for ram to put in my nodes. This is still a very much build in progress.
 

HoneyBadger

actually does care
Administrator
Moderator
iXsystems
Joined
Feb 6, 2014
Messages
5,110
Yeah I still have to test actual nfs performance still saving money for some 10 gig cards and for ram to put in my nodes. This is still a very much build in progress.
Excited to see the end results. If I do manage to get my hands on one of these cards I'll see if I can get it to give better numbers at small recordsize.
 

rshakin

Dabbler
Joined
Apr 2, 2019
Messages
24
Yeah I am not quite sure where or whats the deal with that. Unfortunately there's really no documents available from microsemi what they actually used for ram chips on here. So don't know where to start.
 

HoneyBadger

actually does care
Administrator
Moderator
iXsystems
Joined
Feb 6, 2014
Messages
5,110
Yeah I am not quite sure where or whats the deal with that. Unfortunately there's really no documents available from microsemi what they actually used for ram chips on here. So don't know where to start.
I managed to get signed up for their site, and there's source for a driver that will, according to their instructions, supposedly compile on FreeBSD. So there's potential there. (Mind you, the driver itself is labeled as a Linux driver. So who knows?)

Are you just using the included NVMe driver in FreeNAS? Version? What's the rest of the hardware here?
 

HoneyBadger

actually does care
Administrator
Moderator
iXsystems
Joined
Feb 6, 2014
Messages
5,110
"Seek, and ye shall find."

Code:
smartctl 6.6 2017-11-05 r4594 [FreeBSD 11.1-STABLE amd64] (local build)
Copyright (C) 2002-17, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Number:                       MTR_MLC_TS_4GB
Serial Number:                      04000015A597
Firmware Version:                   2.6.06.1
PCI Vendor/Subsystem ID:            0x11f8
IEEE OUI Identifier:                0x00e004
Controller ID:                      0
Number of Namespaces:               1
Namespace 1 Size/Capacity:          4,255,121,408 [4.25 GB]
Namespace 1 Formatted LBA Size:     512
Namespace 1 IEEE EUI-64:            04e000 0172c55501
Local Time is:                      Wed Apr  3 20:28:18 2019 PDT
Firmware Updates (0x07):            3 Slots, Slot 1 R/O
Optional Admin Commands (0x0006):   Format Frmw_DL

Supported Power States
St Op     Max   Active     Idle   RL RT WL WT  Ent_Lat  Ex_Lat
 0 +    25.00W       -        -    0  0  0  0      100     100

Supported LBA Sizes (NSID 0x1)
Id Fmt  Data  Metadt  Rel_Perf
 0 +     512       0         3
 1 -     512       0         3
 2 -    4096       0         0
 3 -    4096       0         0

=== START OF SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

SMART/Health Information (NVMe Log 0x02, NSID 0xffffffff)
Critical Warning:                   0x00
Temperature:                        47 Celsius
Available Spare:                    0%
Available Spare Threshold:          0%
Percentage Used:                    0%
Data Units Read:                    6 [3.07 MB]
Data Units Written:                 54,852 [28.0 GB]
Host Read Commands:                 270
Host Write Commands:                247,065
Controller Busy Time:               0
Power Cycles:                       0
Power On Hours:                     0
Unsafe Shutdowns:                   0
Media and Data Integrity Errors:    0
Error Information Log Entries:      0

Error Information (NVMe Log 0x01, max 63 entries)
No Errors Logged


Spoiler alert - the NVMe driver in 11.1 really didn't like this thing.

Code:
/dev/nvd0
        512             # sectorsize
        4255121408      # mediasize in bytes (4.0G)
        8310784         # mediasize in sectors
        0               # stripesize
        0               # stripeoffset
        MTR_MLC_TS_4GB  # Disk descr.
        04000015A597    # Disk ident.

Synchronous random writes:
         0.5 kbytes:    999.4 usec/IO =      0.5 Mbytes/s
           1 kbytes:    999.0 usec/IO =      1.0 Mbytes/s
           2 kbytes:    999.2 usec/IO =      2.0 Mbytes/s
           4 kbytes:    999.4 usec/IO =      3.9 Mbytes/s
           8 kbytes:    999.0 usec/IO =      7.8 Mbytes/s
          16 kbytes:    999.4 usec/IO =     15.6 Mbytes/s
          32 kbytes:    999.1 usec/IO =     31.3 Mbytes/s
          64 kbytes:    999.4 usec/IO =     62.5 Mbytes/s
         128 kbytes:    998.7 usec/IO =    125.2 Mbytes/s
         256 kbytes:    999.0 usec/IO =    250.2 Mbytes/s
         512 kbytes:    999.1 usec/IO =    500.4 Mbytes/s
        1024 kbytes:    999.5 usec/IO =   1000.5 Mbytes/s
        2048 kbytes:    999.9 usec/IO =   2000.2 Mbytes/s
        4096 kbytes:   1999.6 usec/IO =   2000.4 Mbytes/s
        8192 kbytes:   3002.2 usec/IO =   2664.7 Mbytes/s

Under 11.2-U3 it's MUCH better although I'm being bottlenecked at the top end by either the CPU in this machine (E5620) or the slot I'm using being PCIe 2.0 vs the 3.0 on the card.

Code:
Synchronous random writes:
         0.5 kbytes:     31.3 usec/IO =     15.6 Mbytes/s
           1 kbytes:     31.5 usec/IO =     31.0 Mbytes/s
           2 kbytes:     31.9 usec/IO =     61.2 Mbytes/s
           4 kbytes:     33.1 usec/IO =    117.9 Mbytes/s
           8 kbytes:     34.4 usec/IO =    227.4 Mbytes/s
          16 kbytes:     40.9 usec/IO =    382.4 Mbytes/s
          32 kbytes:     48.5 usec/IO =    644.8 Mbytes/s
          64 kbytes:     63.3 usec/IO =    987.9 Mbytes/s
         128 kbytes:     94.2 usec/IO =   1327.1 Mbytes/s
         256 kbytes:    131.0 usec/IO =   1908.6 Mbytes/s
         512 kbytes:    204.4 usec/IO =   2445.9 Mbytes/s
        1024 kbytes:    355.5 usec/IO =   2812.6 Mbytes/s
        2048 kbytes:    660.2 usec/IO =   3029.3 Mbytes/s
        4096 kbytes:   1288.2 usec/IO =   3105.1 Mbytes/s
        8192 kbytes:   2528.8 usec/IO =   3163.6 Mbytes/s


I've also still got the "Activity LED" blinking, which according to the documents indicates that the firmware was loaded but the NVMe driver was not. I might have to boot up a penguin-based OS to try that.
 

HoneyBadger

actually does care
Administrator
Moderator
iXsystems
Joined
Feb 6, 2014
Messages
5,110
Better results with an E3-1220 behind it, but still only about on par with the P4801X at the small records. (And lower top-end on this machine.)

Need to see if I can get nvmecontrol format to switch the LBA modes to 4K emulation vs. 512b

Executive Summary - PMC NVRAM cards are neat, but relatively hard to get versus an Optane, and definitely no warranty. But if you're doing recordsizes beyond the typical 128K max, nothing else does Fast like this.

Code:
Synchronous random writes:
         0.5 kbytes:     26.9 usec/IO =     18.1 Mbytes/s
           1 kbytes:     27.2 usec/IO =     35.9 Mbytes/s
           2 kbytes:     27.7 usec/IO =     70.6 Mbytes/s
           4 kbytes:     28.6 usec/IO =    136.6 Mbytes/s
           8 kbytes:     30.1 usec/IO =    259.6 Mbytes/s
          16 kbytes:     36.3 usec/IO =    431.0 Mbytes/s
          32 kbytes:     44.1 usec/IO =    708.0 Mbytes/s
          64 kbytes:     59.4 usec/IO =   1052.2 Mbytes/s
         128 kbytes:     91.5 usec/IO =   1366.5 Mbytes/s
         256 kbytes:    129.8 usec/IO =   1925.6 Mbytes/s
         512 kbytes:    206.6 usec/IO =   2419.7 Mbytes/s
        1024 kbytes:    363.8 usec/IO =   2748.5 Mbytes/s
        2048 kbytes:    681.1 usec/IO =   2936.6 Mbytes/s
        4096 kbytes:   1323.2 usec/IO =   3022.9 Mbytes/s
        8192 kbytes:   2620.3 usec/IO =   3053.1 Mbytes/s
 
Last edited:

HoneyBadger

actually does care
Administrator
Moderator
iXsystems
Joined
Feb 6, 2014
Messages
5,110
Just to beat this dead horse a little more, the NV1604 is extremely NVMe-driver-dependent. With a PenguinOS (CentOS 7) I'm getting significantly better numbers out of fio testing:

Code:
fio --filename=/dev/nvme0n1 --direct=1 --rw=randrw --refill_buffers --norandommap --randrepeat=0 --ioengine=libaio --time_based --runtime=30 --group_reporting --name=latency --rwmixread=0 --rwmixwrite=100 --iodepth=1 --numjobs=1 --bs=4K

4K: IOPS=55.2k, BW=216MiB/s (226MB/s)(6472MiB/30000msec)
8K: IOPS=48.9k, BW=382MiB/s (400MB/s)(11.2GiB/30001msec)
16K: IOPS=35.6k, BW=556MiB/s (583MB/s)(16.3GiB/30001msec)

Not having any luck getting nvme-cli to make it switch formats to 4Kn, it keeps telling me that it's an invalid LBA format despite it showing available in the namespace.

Edit: Apparently the auto-backup to flash functionality is disabled by default, and needs to be re-enabled on every boot. Auto-restore is also disabled by default but at least persists. I've requested access to the code/firmware, might need to go under NDA which will complicate things.
 
Last edited:

Constantin

Vampire Pig
Joined
May 19, 2017
Messages
1,828
Makes you wonder why MB manufacturers didn't setting aside one DIMM slot with battery backup connector for precisely this function. Make it a $50 add-on sale for relatively little cost on their part (offer it for larger boards where there is abundant space, etc.)

Intel seems to be going that direction with the latest RAM it's offering - Optane in a DIMM stick format. The OS recognizes Optane for what it is, and if you have 8 DIMM slots, why not set aside 2 to make a nice mirrored SLOG?
 
Top