Resource icon

SLOG benchmarking and finding the best SLOG

Constantin

Vampire Pig
Joined
May 19, 2017
Messages
1,828
LOL. First, I'd have to either switch to SSD-only or add a lot more disks before I'd even get close to saturating 10GbE.

But, Optane did help speed things along for the HDDs and I hope that my special VDEV will help improve responsiveness with small files and the metadata when I switch to TN12. One of the more annoying "features" of modern OSX is that the OS no longer visually indicates to the user whether a network folder is still being updated or not. I hope that by moving all my metadata to SSD this issue will be mitigated somewhat.

I do find that 10GbE makes a big difference for rsync and like operations even if the total transfer speed is unimpressive. For example, the other day I did some rsync transfers of small files that came out to about 8-10MB/s with 1GbE (misconfigured LAGG). Once I fixed the LAGG issue, the transfer speed jumped to 6-80 MB/s which suggests that the 10GbE likely helped with lower latency, not raw transfer speed alone.
 

CrimsonMars

Dabbler
Joined
Aug 7, 2020
Messages
24
LOL. First, I'd have to either switch to SSD-only or add a lot more disks before I'd even get close to saturating 10GbE.

But, Optane did help speed things along for the HDDs and I hope that my special VDEV will help improve responsiveness with small files and the metadata when I switch to TN12. One of the more annoying "features" of modern OSX is that the OS no longer visually indicates to the user whether a network folder is still being updated or not. I hope that by moving all my metadata to SSD this issue will be mitigated somewhat.

I do find that 10GbE makes a big difference for rsync and like operations even if the total transfer speed is unimpressive. For example, the other day I did some rsync transfers of small files that came out to about 8-10MB/s with 1GbE (misconfigured LAGG). Once I fixed the LAGG issue, the transfer speed jumped to 6-80 MB/s which suggests that the 10GbE likely helped with lower latency, not raw transfer speed alone.
Hmmm something is strange, in my case I far exceed my 21G total link, only sync writes are held back by optane at roughly 160MB/s.
That is the optane limitation, getting a couple more to see agregated throughput, and in the future I thing will get 4x p800 to go close to my 20G link. Link agregaton does not work on my hybrid cards, an old ones are to hot for my taste so running multipath at iscsi level
 

CrimsonMars

Dabbler
Joined
Aug 7, 2020
Messages
24
Been playing around with the extra SSD´s that I have.
As I have 2 of them in mirror as a special vdv for one pool, the other 2 I have split up for SLOG and seen that there is some interesting thing going on on the ZFS side.
Dunno if this is the right place to post this, if so please advice and will move/delete this.

So here goes:

1. L2ARC does not spread over 160G roughly although I have like 800G allocated to it, so it sits at roughly 3xRAM, as recommended by ZFS, my guess is it´s limited, I was just monitoring to see if it overspreads and what the impact is if so.

2. My VM´s seem a lot more responsive with a L2ARC SSD.
Although I never passed the actual bandwidth on reads, and previous ARC hit ratio was above 90% initially, after adding L2ARC I can see an improvement wile logging in the VM´s and doing stuff, they are so much more responsive.

3. While never using more that 160G which is roughly speaking 3xARC size as my system has 64G of RAM and uses roughly 50-55G for ARC, it basically balances this amount of L2ARC across all pools dynamically (the ones that have L2ARC assigned).
 

CrimsonMars

Dabbler
Joined
Aug 7, 2020
Messages
24
16GB optane, for posteriority:


root@freenas[/mnt/SAS]# smartctl -a /dev/nvme0
smartctl 7.1 2019-12-30 r5022 [FreeBSD 12.1-STABLE amd64] (local build)
Copyright (C) 2002-19, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Number: INTEL MEMPEK1W016GA
Serial Number: PHBT813403S5016D
Firmware Version: K3110310
PCI Vendor/Subsystem ID: 0x8086
IEEE OUI Identifier: 0x5cd2e4
Controller ID: 0
Number of Namespaces: 1
Namespace 1 Size/Capacity: 14,403,239,936 [14.4 GB]
Namespace 1 Formatted LBA Size: 512
Local Time is: Fri Aug 28 10:35:37 2020 CEST
Firmware Updates (0x02): 1 Slot
Optional Admin Commands (0x0006): Format Frmw_DL
Optional NVM Commands (0x0046): Wr_Unc DS_Mngmt Timestmp
Maximum Data Transfer Size: 32 Pages

Supported Power States
St Op Max Active Idle RL RT WL WT Ent_Lat Ex_Lat
0 + 4.50W - - 0 0 0 0 0 0

Supported LBA Sizes (NSID 0x1)
Id Fmt Data Metadt Rel_Perf
0 + 512 0 2

=== START OF SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

SMART/Health Information (NVMe Log 0x02)
Critical Warning: 0x00
Temperature: 57 Celsius
Available Spare: 100%
Available Spare Threshold: 0%
Percentage Used: 0%
Data Units Read: 271 [138 MB]
Data Units Written: 2,919,304 [1.49 TB]
Host Read Commands: 6,427
Host Write Commands: 25,410,239
Controller Busy Time: 0
Power Cycles: 18
Power On Hours: 263
Unsafe Shutdowns: 1
Media and Data Integrity Errors: 0
Error Information Log Entries: 0

Error Information (NVMe Log 0x01, max 64 entries)
No Errors Logged

root@freenas[/mnt/SAS]# diskinfo -wS /dev/nvd0
/dev/nvd0
512 # sectorsize
14403239936 # mediasize in bytes (13G)
28131328 # mediasize in sectors
0 # stripesize
0 # stripeoffset
INTEL MEMPEK1W016GA # Disk descr.
PHBT813403S5016D # Disk ident.
Yes # TRIM/UNMAP support
0 # Rotation rate in RPM

Synchronous random writes:
0.5 kbytes: 13.7 usec/IO = 35.7 Mbytes/s
1 kbytes: 13.1 usec/IO = 74.4 Mbytes/s
2 kbytes: 19.2 usec/IO = 101.9 Mbytes/s
4 kbytes: 32.6 usec/IO = 120.0 Mbytes/s
8 kbytes: 59.7 usec/IO = 130.8 Mbytes/s
16 kbytes: 114.0 usec/IO = 137.0 Mbytes/s
32 kbytes: 224.6 usec/IO = 139.1 Mbytes/s
64 kbytes: 445.1 usec/IO = 140.4 Mbytes/s
128 kbytes: 897.9 usec/IO = 139.2 Mbytes/s
256 kbytes: 1770.4 usec/IO = 141.2 Mbytes/s
512 kbytes: 3513.3 usec/IO = 142.3 Mbytes/s
1024 kbytes: 6974.2 usec/IO = 143.4 Mbytes/s
2048 kbytes: 13872.6 usec/IO = 144.2 Mbytes/s
4096 kbytes: 27682.9 usec/IO = 144.5 Mbytes/s
8192 kbytes: 55192.2 usec/IO = 144.9 Mbytes/s


Optane connected in pcix 2.0 X1:
root@freenas[~]# diskinfo -wS /dev/nvd1
/dev/nvd1
512 # sectorsize
14403239936 # mediasize in bytes (13G)
28131328 # mediasize in sectors
0 # stripesize
0 # stripeoffset
INTEL MEMPEK1W016GA # Disk descr.
PHBT813403YH016D # Disk ident.
Yes # TRIM/UNMAP support
0 # Rotation rate in RPM

Synchronous random writes:
0.5 kbytes: 18.0 usec/IO = 27.1 Mbytes/s
1 kbytes: 17.9 usec/IO = 54.5 Mbytes/s
2 kbytes: 24.2 usec/IO = 80.8 Mbytes/s
4 kbytes: 38.2 usec/IO = 102.2 Mbytes/s
8 kbytes: 67.4 usec/IO = 115.8 Mbytes/s
16 kbytes: 127.6 usec/IO = 122.4 Mbytes/s
32 kbytes: 249.0 usec/IO = 125.5 Mbytes/s
64 kbytes: 496.9 usec/IO = 125.8 Mbytes/s
128 kbytes: 988.9 usec/IO = 126.4 Mbytes/s
256 kbytes: 1952.2 usec/IO = 128.1 Mbytes/s
512 kbytes: 3837.4 usec/IO = 130.3 Mbytes/s
1024 kbytes: 7572.2 usec/IO = 132.1 Mbytes/s
2048 kbytes: 15198.8 usec/IO = 131.6 Mbytes/s
4096 kbytes: 29953.4 usec/IO = 133.5 Mbytes/s
8192 kbytes: 59712.3 usec/IO = 134.0 Mbytes/s
root@freenas[~]# diskinfo -wS /dev/nvd2
/dev/nvd2
512 # sectorsize
14403239936 # mediasize in bytes (13G)
28131328 # mediasize in sectors
0 # stripesize
0 # stripeoffset
INTEL MEMPEK1W016GA # Disk descr.
PHBT813404B2016D # Disk ident.
Yes # TRIM/UNMAP support
0 # Rotation rate in RPM

Synchronous random writes:
0.5 kbytes: 17.2 usec/IO = 28.4 Mbytes/s
1 kbytes: 17.7 usec/IO = 55.1 Mbytes/s
2 kbytes: 24.0 usec/IO = 81.4 Mbytes/s
4 kbytes: 37.7 usec/IO = 103.6 Mbytes/s
8 kbytes: 66.3 usec/IO = 117.9 Mbytes/s
16 kbytes: 124.7 usec/IO = 125.3 Mbytes/s
32 kbytes: 243.5 usec/IO = 128.3 Mbytes/s
64 kbytes: 482.8 usec/IO = 129.4 Mbytes/s
128 kbytes: 963.2 usec/IO = 129.8 Mbytes/s
256 kbytes: 1868.3 usec/IO = 133.8 Mbytes/s
512 kbytes: 3721.1 usec/IO = 134.4 Mbytes/s
1024 kbytes: 7407.1 usec/IO = 135.0 Mbytes/s
2048 kbytes: 14707.9 usec/IO = 136.0 Mbytes/s
4096 kbytes: 29137.1 usec/IO = 137.3 Mbytes/s
8192 kbytes: 57974.8 usec/IO = 138.0 Mbytes/s
root@freenas[~]#


root@freenas[~]# smartctl -d nvme -a /dev/nvme1
smartctl 7.1 2019-12-30 r5022 [FreeBSD 12.1-STABLE amd64] (local build)
Copyright (C) 2002-19, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Number: INTEL MEMPEK1W016GA
Serial Number: PHBT813403YH016D
Firmware Version: K3110310
PCI Vendor/Subsystem ID: 0x8086
IEEE OUI Identifier: 0x5cd2e4
Controller ID: 0
Number of Namespaces: 1
Namespace 1 Size/Capacity: 14,403,239,936 [14.4 GB]
Namespace 1 Formatted LBA Size: 512
Local Time is: Fri Sep 11 14:50:40 2020 CEST
Firmware Updates (0x02): 1 Slot
Optional Admin Commands (0x0006): Format Frmw_DL
Optional NVM Commands (0x0046): Wr_Unc DS_Mngmt Timestmp
Maximum Data Transfer Size: 32 Pages

Supported Power States
St Op Max Active Idle RL RT WL WT Ent_Lat Ex_Lat
0 + 4.50W - - 0 0 0 0 0 0

Supported LBA Sizes (NSID 0x1)
Id Fmt Data Metadt Rel_Perf
0 + 512 0 2

=== START OF SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

SMART/Health Information (NVMe Log 0x02)
Critical Warning: 0x00
Temperature: 48 Celsius
Available Spare: 100%
Available Spare Threshold: 0%
Percentage Used: 0%
Data Units Read: 11 [5.63 MB]
Data Units Written: 0
Host Read Commands: 1,248
Host Write Commands: 0
Controller Busy Time: 0
Power Cycles: 14
Power On Hours: 0
Unsafe Shutdowns: 0
Media and Data Integrity Errors: 0
Error Information Log Entries: 0

Error Information (NVMe Log 0x01, max 64 entries)
No Errors Logged

4x16GB optane in raid 10 produce roghly 300MB/s of sync writes
Will soon buy some 112G ones to see the difference, guess 4x112G will reach roughly 1GB/s, so al little beyond what a 10G link can carry

This is what I get in a VM(it´s not the only VM active so real number are a little better, total bandwidth of VM is caped at roughly 800MB/s)
1599831386925.png
 
Last edited:

svtkobra7

Patron
Joined
Jan 12, 2017
Messages
202
Can a kind soul advise why
256 kbytes: diskinfo: AIO write submit error: Operation not supported
results when diskinfo -wS /dev/nvdx is run, please?
Last I used it, it went to 8192 kb ... Hopefully a workaround exists as I have 2x of 4 different M.2 drives to add to the sample set:

[EDIT: Issue confirmed to exist in FreeNAS-11.3-U4.1 by myself and @Elliot Dierksen. Issue does not exist in TrueNAS-12.0-RC1.]

Thx in advance.
  • Samsung SSD 970 EVO Plus 500GB
  • WD SN750 500 GB
  • INTEL SSDPEL1D380GA (905P 380GB)
  • P4801x 100GB
diskinfo -wS /dev/nvd3
/dev/nvd3
512 # sectorsize
...
Synchronous random writes:
0.5 kbytes: 42.6 usec/IO = 11.5 Mbytes/s
1 kbytes: 44.3 usec/IO = 22.0 Mbytes/s
....
128 kbytes: 127.1 usec/IO = 983.6 Mbytes/s
256 kbytes: diskinfo: AIO write submit error: Operation not supported
 
Last edited:

svtkobra7

Patron
Joined
Jan 12, 2017
Messages
202
Interesting ... This issue (prior post) is nonexistent in TrueNAS ...
Spun up / applied hw.pci.honor_msi_blacklist=0 (avoid nvme interrupt msg) / tested ...

Intel Optane SSD 905P Series | 380GB, M.2 110mm PCIe x4, 20nm, 3D XPoint
Connected via SMCI AOC-SLG3-2M2

I have a P4801x I'll test as well ... curious to throw a couple of these together in a mirror and toss a P4801x in as a SLOG to see the impact (just for fun).

Code:
=== START OF INFORMATION SECTION ===
Model Number:                       INTEL SSDPEL1D380GA
Serial Number:                      
Firmware Version:                   E2010480
PCI Vendor/Subsystem ID:            0x8086
IEEE OUI Identifier:                0x5cd2e4
Controller ID:                      0
Number of Namespaces:               1
Namespace 1 Size/Capacity:          380,084,576,256 [380 GB]
Namespace 1 Formatted LBA Size:     512
Local Time is:                      Fri Sep 25 19:50:26 2020 PDT

root@truenas[~]# diskinfo -wS /dev/nvd0
/dev/nvd0
        512             # sectorsize
        380084576256    # mediasize in bytes (354G)
        742352688       # mediasize in sectors
        0               # stripesize
        0               # stripeoffset
        INTEL SSDPEL1D380GA     # Disk descr.
        PHMC020200C6380A        # Disk ident.
        Yes             # TRIM/UNMAP support
        0               # Rotation rate in RPM

Synchronous random writes:
         0.5 kbytes:     21.0 usec/IO =     23.2 Mbytes/s
           1 kbytes:     23.8 usec/IO =     41.0 Mbytes/s
           2 kbytes:     22.4 usec/IO =     87.1 Mbytes/s
           4 kbytes:     21.4 usec/IO =    182.6 Mbytes/s
           8 kbytes:     22.9 usec/IO =    340.8 Mbytes/s
          16 kbytes:     25.7 usec/IO =    608.3 Mbytes/s
          32 kbytes:     33.4 usec/IO =    936.6 Mbytes/s
          64 kbytes:     47.7 usec/IO =   1309.5 Mbytes/s
         128 kbytes:     95.9 usec/IO =   1302.9 Mbytes/s
         256 kbytes:    147.0 usec/IO =   1700.8 Mbytes/s
         512 kbytes:    258.0 usec/IO =   1937.8 Mbytes/s
        1024 kbytes:    491.8 usec/IO =   2033.4 Mbytes/s
        2048 kbytes:    927.2 usec/IO =   2157.1 Mbytes/s
        4096 kbytes:   1807.4 usec/IO =   2213.2 Mbytes/s
        8192 kbytes:   3584.0 usec/IO =   2232.1 Mbytes/s
 
Joined
Dec 29, 2014
Messages
1,135
Interesting @svtkobra7 . Mine didn't do this before, but it looks like I am getting the same issue you have
Code:
root@freenas2:/ # diskinfo -wS /dev/nvd0
/dev/nvd0
        512             # sectorsize
        280065171456    # mediasize in bytes (261G)
        547002288       # mediasize in sectors
        0               # stripesize
        0               # stripeoffset
        INTEL SSDPED1D280GA     # Disk descr.
        PHMB742401A6280CGN      # Disk ident.
        Yes             # TRIM/UNMAP support
        0               # Rotation rate in RPM

Synchronous random writes:
         0.5 kbytes:     13.9 usec/IO =     35.1 Mbytes/s
           1 kbytes:     14.1 usec/IO =     69.4 Mbytes/s
           2 kbytes:     14.3 usec/IO =    136.8 Mbytes/s
           4 kbytes:     11.6 usec/IO =    336.8 Mbytes/s
           8 kbytes:     13.8 usec/IO =    566.9 Mbytes/s
          16 kbytes:     18.2 usec/IO =    859.9 Mbytes/s
          32 kbytes:     26.6 usec/IO =   1175.2 Mbytes/s
          64 kbytes:     44.2 usec/IO =   1414.9 Mbytes/s
         128 kbytes:     86.5 usec/IO =   1444.5 Mbytes/s
         256 kbytes: diskinfo: AIO write submit error: Operation not supported

I am running FreeNAS-11.3-U4.1 .
 
Joined
Dec 29, 2014
Messages
1,135
This error appears to have been around a while. See this thread. https://www.ixsystems.com/community/threads/diskinfo-submit-error.85409/
I don't routinely removed my SLOG and re-test it when I upgrade. I just did it a few moments ago after being asked to check it. I am not sure when I last updated the firmware on my Optane 900P (purchased in 2018), but I know I have done it since I bought them. Is there already a big filed for this? I not, perhaps I should submit one.
 

svtkobra7

Patron
Joined
Jan 12, 2017
Messages
202
I just did it a few moments ago after being asked to check it. I am not sure when I last updated the firmware on my Optane 900P (purchased in 2018), but I know I have done it since I bought them. Is there already a big filed for this? I not, perhaps I should submit one.
  • Current f/w for 900p is E2010480 (July 2020) and the only update since initial release, so you would have updated f/w after July 2020.
  • 11.3 updates in release train which align are constrained to 11.3-U4 and 11.3-U4.1.
  • Not sure if that provides you any relevant deets for your potential ticket.
References:
 
Joined
Dec 29, 2014
Messages
1,135
This is the update that I did on my 900P's.
1601134451398.png

I have not updated it since then.
 
Joined
Dec 29, 2014
Messages
1,135
Still getting the same error after upgrading firmware to this version.
1601138681055.png
 

svtkobra7

Patron
Joined
Jan 12, 2017
Messages
202
Still getting the same error after upgrading firmware to this version.

Its not the firmware of the drive that causes the 256 kbytes: diskinfo: AIO write submit error: Operation not supported error otherwise I wouldn't have seen it occur on two different optane models, one WD, and one Samsung. And to boot, none present the error in TrueNAS 12. Apparently this is the cause of the error:


diskinfo on FreeBSD 11.2 and 11.3 are identical except one line of comment. It yields that error message when the system call to aio_write() returns -1 (=>failed to schedule the async write operation). The file descriptor used to setup aio_write is the same one used to make sync pwrite call. The function that does these sync and async writes (for slogbench) here:
Code:
parwrite(int fd, size_t size, off_t off)
{
    struct aiocb aios[MAXIOS];
    off_t o;
    int n, error;
    struct aiocb *aiop;

    // if size > MAXIO, use AIO to write n - 1 pieces in parallel
    for (n = 0, o = 0; size > MAXIO; n++, size -= MAXIO, o += MAXIO) {
        aiop = &aios[n];
        bzero(aiop, sizeof(*aiop));
        aiop->aio_buf = &buf[o];
        aiop->aio_fildes = fd;
        aiop->aio_offset = off + o;
        aiop->aio_nbytes = MAXIO;
        error = aio_write(aiop);
        if (error != 0)
            err(EX_IOERR, "AIO write submit error");
    }
    // Use synchronous writes for the runt of size <= MAXIO
    error = pwrite(fd, &buf[o], size, off + o);
    if (error < 0)
        err(EX_IOERR, "Sync write error");
    for (; n > 0; n--) {
        error = aio_waitcomplete(&aiop, NULL);
        if (error < 0)
            err(EX_IOERR, "AIO write wait error");
    }
}


Anyone knows why aio_write() failed to schedule while pwrite runs without problem?
 

Constantin

Vampire Pig
Joined
May 19, 2017
Messages
1,828
FWIW, my optane speed went down after that firmware revision. Small file writes clocked in at 15.8MB/s vs. 17+ with the older firmware. Even more curious was that the S3610s did not show up at all in the updater.

My usual ISO go-to tool (BalenaEtcher) successfully made an un-bootable USB stick despite verifying the media as correct. Thanks to @Spearfoot (thank you!) I was pointed to Rufus, which happily produced a bootable USB stick for updating the p4801x firmware.

The hint should have been that OSX couldn't mount the BalenaEtcher-made drive (though I didn't know it was supposed to be a FAT32 volume). The Rufus volume mounted on my Mac as well as on my server. FWIW, Parallels v16 and Rufus worked perfectly together - boot into Windows on Parallels, assign the drive to the Windows OS, start Rufus, drag the ISO, burn, done.

Per the devs @ the BalenaEtcher project, their ISO-maker is designed to make byte-by-byte copies and it does not deal with whatever is necessary to make a drive bootable, unlike Rufus. I've requested they consider adding a check after writing and verifying that the media actually is bootable. If no file system presents itself, something likely went wrong and the user should know that. Given how eager the devs at the BalenaEtcher project were to close the ticket, I doubt it will happen, however.
 
Last edited:
Joined
Dec 29, 2014
Messages
1,135
My usual ISO go-to tool (BalenaEtcher) made a unusable USB stick despite verifying the media as correct. Thanks to @Spearfoot (thank you!) I was pointed to Rufus, which happily produced a perfect USB stick for updating the p4801x firmware.
Rufus is my go to tool based on things I had heard here. Certain things (Like Cisco server firmware update ISO's) need the 'support for older BIOS' box checked, but general Rufus is bim-bam-boom-done!
 

masru

Dabbler
Joined
Aug 9, 2020
Messages
26
I'm setting up my first home NAS, currently I test how to set up best for Time Machine backups. As suggested earlier in this thread adding a SLOG improves the backup times on a SMB share.

Adding a SLOG (Crucial MX500) to a pair of mirrored 72k HDDs with metadata-vdev was (almost exactly) as fast as doing the same backup on a pool with mirrored SSDs.
My test backup improved from 59Mbit/sec to 64Mbit/sec. That might not look very "cool", but it shaped off 30 minutes from a 6:30h backup task, which is quite welcome.
(I can add more info on what I tested, but I assume it would be a bit off topic here).

The SSD I had at hand was a Crucial MX500 1TB, which might be a little bit oversized for the task ;)

Code:
=== START OF INFORMATION SECTION ===
Model Family:     Crucial/Micron BX/MX1/2/3/500, M5/600, 1100 SSDs
Device Model:     CT1000MX500SSD1
Serial Number:    2025E2AE5095
LU WWN Device Id: 5 00a075 1e2ae5095
Firmware Version: M3CR032
User Capacity:    1,000,204,886,016 bytes [1.00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    Solid State Device
Form Factor:      2.5 inches
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ACS-3 T13/2161-D revision 5
SATA Version is:  SATA 3.3, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Fri Oct 23 16:19:43 2020 CEST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

Code:
diskinfo -wS /dev/ada6
/dev/ada6
        512             # sectorsize
        1000204886016   # mediasize in bytes (932G)
        1953525168      # mediasize in sectors
        4096            # stripesize
        0               # stripeoffset
        1938021         # Cylinders according to firmware.
        16              # Heads according to firmware.
        63              # Sectors according to firmware.
        CT1000MX500SSD1 # Disk descr.
        2025E2AE5095    # Disk ident.
        id1,enc@n3061686369656d31/type@0/slot@6/elmdesc@Slot_05 # Physical path
        Yes             # TRIM/UNMAP support
        0               # Rotation rate in RPM
        Not_Zoned       # Zone Mode

Synchronous random writes:
         0.5 kbytes:    508.1 usec/IO =      1.0 Mbytes/s
           1 kbytes:    536.5 usec/IO =      1.8 Mbytes/s
           2 kbytes:    505.6 usec/IO =      3.9 Mbytes/s
           4 kbytes:    444.5 usec/IO =      8.8 Mbytes/s
           8 kbytes:    454.1 usec/IO =     17.2 Mbytes/s
          16 kbytes:    473.9 usec/IO =     33.0 Mbytes/s
          32 kbytes:    503.4 usec/IO =     62.1 Mbytes/s
          64 kbytes:    612.7 usec/IO =    102.0 Mbytes/s
         128 kbytes:    821.5 usec/IO =    152.2 Mbytes/s
         256 kbytes:   1237.6 usec/IO =    202.0 Mbytes/s
         512 kbytes:   2118.3 usec/IO =    236.0 Mbytes/s
        1024 kbytes:   4351.6 usec/IO =    229.8 Mbytes/s
        2048 kbytes:   5618.9 usec/IO =    355.9 Mbytes/s
        4096 kbytes:   9607.8 usec/IO =    416.3 Mbytes/s
        8192 kbytes:  17929.1 usec/IO =    446.2 Mbytes/s


My motherboard Supermicro A2SDi-4C-HLN4F has one M.2 slot. Unfortunately it has only 2 lanes, but with the results I got earlier I will test a NVMe SSD I have at hand:

KIOXIA EXCERIA SSD (only two PCIe lanes used):
Code:
=== START OF INFORMATION SECTION ===
Model Number:                       KIOXIA-EXCERIA SSD
Serial Number:                      20GA8423K4T1
Firmware Version:                   ECFA12.7
PCI Vendor/Subsystem ID:            0x1e0f
IEEE OUI Identifier:                0x8ce38e
Total NVM Capacity:                 500,107,862,016 [500 GB]
Unallocated NVM Capacity:           0
Controller ID:                      1
Number of Namespaces:               1
Namespace 1 Size/Capacity:          500,107,862,016 [500 GB]
Namespace 1 Formatted LBA Size:     512
Namespace 1 IEEE EUI-64:            8ce38e 03002de535
Local Time is:                      Fri Oct 23 16:26:02 2020 CEST
Firmware Updates (0x12):            1 Slot, no Reset required
Optional Admin Commands (0x0017):   Security Format Frmw_DL Self_Test
Optional NVM Commands (0x005f):     Comp Wr_Unc DS_Mngmt Wr_Zero Sav/Sel_Feat Timestmp
Maximum Data Transfer Size:         512 Pages
Warning  Comp. Temp. Threshold:     72 Celsius
Critical Comp. Temp. Threshold:     90 Celsius

Supported Power States
St Op     Max   Active     Idle   RL RT WL WT  Ent_Lat  Ex_Lat
 0 +     7.69W       -        -    0  0  0  0        1       1
 1 +     6.18W       -        -    1  1  1  1        1       1
 2 +     5.42W       -        -    2  2  2  2        1       1
 3 -   0.0500W       -        -    3  3  3  3     7000    5000
 4 -   0.0050W       -        -    4  4  4  4    13000   36000

Supported LBA Sizes (NSID 0x1)
Id Fmt  Data  Metadt  Rel_Perf
 0 +     512       0         2
 1 -    4096       0         1

Code:
 diskinfo -wS /dev/nvd0
/dev/nvd0
        512             # sectorsize
        500107862016    # mediasize in bytes (466G)
        976773168       # mediasize in sectors
        0               # stripesize
        0               # stripeoffset
        KIOXIA-EXCERIA SSD      # Disk descr.
        20GA8423K4T1    # Disk ident.
        Yes             # TRIM/UNMAP support
        0               # Rotation rate in RPM

Synchronous random writes:
         0.5 kbytes:    691.9 usec/IO =      0.7 Mbytes/s
           1 kbytes:   2302.4 usec/IO =      0.4 Mbytes/s
           2 kbytes:    716.4 usec/IO =      2.7 Mbytes/s
           4 kbytes:    514.0 usec/IO =      7.6 Mbytes/s
           8 kbytes:    525.2 usec/IO =     14.9 Mbytes/s
          16 kbytes:    536.7 usec/IO =     29.1 Mbytes/s
          32 kbytes:    346.2 usec/IO =     90.3 Mbytes/s
          64 kbytes:    370.0 usec/IO =    168.9 Mbytes/s
         128 kbytes:    417.4 usec/IO =    299.5 Mbytes/s
         256 kbytes:    512.1 usec/IO =    488.1 Mbytes/s
         512 kbytes:    683.5 usec/IO =    731.5 Mbytes/s
        1024 kbytes:   1079.9 usec/IO =    926.0 Mbytes/s
        2048 kbytes:   1686.0 usec/IO =   1186.3 Mbytes/s
        4096 kbytes:   3021.6 usec/IO =   1323.8 Mbytes/s
        8192 kbytes:   5627.2 usec/IO =   1421.7 Mbytes/s


I have seen the data of the OPTANE 800P devices earlier in this thread. It would at least match the two lanes of my M.2 slot. Do you think that device would make a difference for my use case?
 

Constantin

Vampire Pig
Joined
May 19, 2017
Messages
1,828
I doubt it will make a big difference re time machine. Unless something has changed, Apple throttles time machine backups at the host computer level, even if the host computer has nothing to do (powered sleep, waking up, backing up, going back to sleep).

Plus, the first backup might be a multi hour affair but subsequently, TM only runs incremental backups, which tend to be much smaller.

I’m slowly gearing up for 12 - have to make two backups of everything before I nuke the pool, upgrade the machine and build a new pool with two special VDEVs - one for metadata, the other for small files.
 

dak180

Patron
Joined
Nov 22, 2017
Messages
308
Need to see if I can get nvmecontrol format to switch the LBA modes to 4K emulation vs. 512b
Were you ever able to do this? I am trying to format a new nvme drive and it keeps telling me nvmecontrol: format request returned error and the man page is not very helpful.
 

HoneyBadger

actually does care
Administrator
Moderator
iXsystems
Joined
Feb 6, 2014
Messages
5,110
Were you ever able to do this? I am trying to format a new nvme drive and it keeps telling me nvmecontrol: format request returned error and the man page is not very helpful.
Is this specifically for the PMC NV1604 you're trying to do this, or a different deviec?
 

dak180

Patron
Joined
Nov 22, 2017
Messages
308
Is this specifically for the PMC NV1604 you're trying to do this, or a different deviec?
A different device; mostly I am just trying to figure out how that command is supposed to work because the man page does not fully document it and your quick mention is one of the very few examples I found online of someone talking about using the nvmecontrol command, almost everything else uses the nvme tool on linux.
 
Top