Register for the iXsystems Community to get an ad-free experience and exclusive discounts in our eBay Store.
Resource icon

SLOG benchmarking and finding the best SLOG

Western Digital Drives - The Preferred Drives of FreeNAS and TrueNAS CORE

Kevin666

Newbie
Joined
Jul 23, 2021
Messages
2
Annybody tried a samsung PM9a3 ? they seem cheap and fast, although i cant find the latency in their documentation.
 

Kevin666

Newbie
Joined
Jul 23, 2021
Messages
2
I got these rapports over a 1GB intel raidcard just for giggles with intel SSD 540s
How do you guys get the info in that code - format ?
1627238367579.png
 

HoneyBadger

Mushroom! Mushroom!
Joined
Feb 6, 2014
Messages
3,339
(Interrupt throttling) happens on the smaller block sizes; anything up to 8-16KiB. I'm now thinking it is this throttling that might be hampering the performance with those small block sizes, as for the larger block sizes the performance is comparable to when power management is set to performance. Is there a way to circumvent the throttling other than changing the power mngmt setting to test this hypothesis? I'm hoping I'd be able to have power management set balanced and still have the higher performing log devices (total power draw goes up from ~140W to ~250W on idle)...
I don't believe so. The power management is constantly trying to get the CPU/PCIe link/everything it can to shift into a lower power state, and there's probably just enough time between disk writes (whether waiting for disk, waiting on PCIe link, or waiting on vmkernel) to have it constantly toggling between them and spewing interrupts all over VMware.

Full-throttle mode causes it to never go to sleep, but at the cost of significantly higher idle power consumption.

Sorry. Don't think there's an easy win here. Solar panels to offset the consumption maybe?
 

Stux

Wizened Sage
Joined
Jun 2, 2016
Messages
4,219
Another P4801X 100GB optane.

This time in a Supermicro X10SRi-F

smart results
Code:
root@titan:~ # smartctl -a /dev/nvme0
smartctl 7.2 2020-12-30 r5155 [FreeBSD 12.2-RELEASE-p6 amd64] (local build)
Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Number:                       INTEL SSDPEL1K100GA
Serial Number:                      PHKM034100FX100D
Firmware Version:                   E2010485
PCI Vendor/Subsystem ID:            0x8086
IEEE OUI Identifier:                0x5cd2e4
Controller ID:                      0
NVMe Version:                       <1.2
Number of Namespaces:               1
Namespace 1 Size/Capacity:          100,030,242,816 [100 GB]
Namespace 1 Formatted LBA Size:     512
Namespace 1 IEEE EUI-64:            5cd2e4 b6f07b0100
Local Time is:                      Mon Jul 26 16:14:19 2021 AEST
Firmware Updates (0x02):            1 Slot
Optional Admin Commands (0x0007):   Security Format Frmw_DL
Optional NVM Commands (0x0006):     Wr_Unc DS_Mngmt
Log Page Attributes (0x0a):         Cmd_Eff_Lg Telmtry_Lg
Maximum Data Transfer Size:         32 Pages

Supported Power States
St Op     Max   Active     Idle   RL RT WL WT  Ent_Lat  Ex_Lat
0 +    10.00W       -        -    0  0  0  0        0       0

Supported LBA Sizes (NSID 0x1)
Id Fmt  Data  Metadt  Rel_Perf
0 +     512       0         2
1 -     512       8         2
2 -     512      16         2
3 -    4096       0         0
4 -    4096       8         0
5 -    4096      64         0
6 -    4096     128         0

=== START OF SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

SMART/Health Information (NVMe Log 0x02)
Critical Warning:                   0x00
Temperature:                        43 Celsius
Available Spare:                    100%
Available Spare Threshold:          0%
Percentage Used:                    0%
Data Units Read:                    6 [3.07 MB]
Data Units Written:                 17,507 [8.96 GB]
Host Read Commands:                 280
Host Write Commands:                416,250
Controller Busy Time:               0
Power Cycles:                       4
Power On Hours:                     0
Unsafe Shutdowns:                   0
Media and Data Integrity Errors:    0
Error Information Log Entries:      0

Error Information (NVMe Log 0x01, 16 of 64 entries)
No Errors Logged


First results at 512k sector size
Code:
root@titan:~ # diskinfo -wS /dev/nvd0
/dev/nvd0
    512             # sectorsize
    100030242816    # mediasize in bytes (93G)
    195371568       # mediasize in sectors
    0               # stripesize
    0               # stripeoffset
    INTEL SSDPEL1K100GA    # Disk descr.
    PHKM034100FX100D    # Disk ident.
    Yes             # TRIM/UNMAP support
    0               # Rotation rate in RPM

Synchronous random writes:
     0.5 kbytes:     19.6 usec/IO =     24.9 Mbytes/s
       1 kbytes:     16.7 usec/IO =     58.6 Mbytes/s
       2 kbytes:     17.3 usec/IO =    112.7 Mbytes/s
       4 kbytes:     13.4 usec/IO =    290.5 Mbytes/s
       8 kbytes:     19.0 usec/IO =    411.8 Mbytes/s
      16 kbytes:     27.7 usec/IO =    564.5 Mbytes/s
      32 kbytes:     43.4 usec/IO =    720.1 Mbytes/s
      64 kbytes:     74.8 usec/IO =    835.3 Mbytes/s
     128 kbytes:    148.1 usec/IO =    844.0 Mbytes/s
     256 kbytes:    285.2 usec/IO =    876.6 Mbytes/s
     512 kbytes:    516.2 usec/IO =    968.7 Mbytes/s
    1024 kbytes:    971.7 usec/IO =   1029.1 Mbytes/s
    2048 kbytes:   1878.6 usec/IO =   1064.6 Mbytes/s
    4096 kbytes:   3702.5 usec/IO =   1080.4 Mbytes/s
    8192 kbytes:   7344.9 usec/IO =   1089.2 Mbytes/s


Wasn't 100% happy so tried to enter the bios. That's when I found that the bios setup was hanging.

This box is on 2.0 bios, so upgraded to 3.3 bios (latest)

Tried power fiddles before bios update (which invovled pulling the card a few times), but that made no difference.

The bios updated fixed the hang, and added PCIe Bifurcation settings, so that's a win :)

I split the x16 slot into x4x4x4x4 (not needed yet, but I plan to get a quad carrier card)

And then re-ran the bench.

Code:
Synchronous random writes:
     0.5 kbytes:     16.2 usec/IO =     30.2 Mbytes/s
       1 kbytes:     16.5 usec/IO =     59.3 Mbytes/s
       2 kbytes:     17.1 usec/IO =    114.5 Mbytes/s
       4 kbytes:     14.2 usec/IO =    275.3 Mbytes/s
       8 kbytes:     17.6 usec/IO =    442.7 Mbytes/s
      16 kbytes:     26.7 usec/IO =    585.7 Mbytes/s
      32 kbytes:     42.5 usec/IO =    735.5 Mbytes/s
      64 kbytes:     74.8 usec/IO =    836.1 Mbytes/s
     128 kbytes:    148.7 usec/IO =    840.6 Mbytes/s
     256 kbytes:    282.9 usec/IO =    883.8 Mbytes/s
     512 kbytes:    514.8 usec/IO =    971.3 Mbytes/s
    1024 kbytes:    967.6 usec/IO =   1033.4 Mbytes/s
    2048 kbytes:   1883.4 usec/IO =   1061.9 Mbytes/s
    4096 kbytes:   3703.2 usec/IO =   1080.2 Mbytes/s
    8192 kbytes:   7340.4 usec/IO =   1089.9 Mbytes/s


a measurable improvement. power management is "let the os do it" and putting it on bios + performance doesn't seem to make a difference.

Next I switched the sector size to 4KB. This means using Intel Memory And Storage tool. This is available for Linux, so I installed it in a VM, added the Optane to the PCI pass-thru list, reboot, add the PCI device to the VM. boot that...

then checked the firwmare was up2date... it was and, reformatted:

intelmas start -intelssd 0 -nvmeformat LBAformat = 3

shutdown vm, remove pci device, remove from pass-thru list, and reboot..

and then we can test with 4096B sectors.

Code:
root@titan:~ # diskinfo -wS /dev/nvd0
/dev/nvd0
    4096            # sectorsize
    100030242816    # mediasize in bytes (93G)
    24421446        # mediasize in sectors
    0               # stripesize
    0               # stripeoffset
    INTEL SSDPEL1K100GA    # Disk descr.
    PHKM034100FX100D    # Disk ident.
    Yes             # TRIM/UNMAP support
    0               # Rotation rate in RPM

Synchronous random writes:
       4 kbytes:     13.1 usec/IO =    298.6 Mbytes/s
       8 kbytes:     17.1 usec/IO =    456.2 Mbytes/s
      16 kbytes:     27.6 usec/IO =    565.7 Mbytes/s
      32 kbytes:     42.5 usec/IO =    735.9 Mbytes/s
      64 kbytes:     72.9 usec/IO =    856.9 Mbytes/s
     128 kbytes:    135.1 usec/IO =    925.3 Mbytes/s
     256 kbytes:    255.5 usec/IO =    978.5 Mbytes/s
     512 kbytes:    509.0 usec/IO =    982.4 Mbytes/s
    1024 kbytes:    972.2 usec/IO =   1028.6 Mbytes/s
    2048 kbytes:   1879.6 usec/IO =   1064.1 Mbytes/s
    4096 kbytes:   3693.3 usec/IO =   1083.0 Mbytes/s
    8192 kbytes:   7320.3 usec/IO =   1092.8 Mbytes/s


That'll do :)

bonus:

Repartitioning after removing manually after TrueNAS formatted

Code:
root@titan:~ # gpart show nvd0
=>       6  24421435  nvd0  GPT  (93G)
         6       122        - free -  (488K)
       128  24421313     1  freebsd-zfs  (93G)

root@titan:~ # gpart delete -i 1 nvd0
nvd0p1 deleted
root@titan:~ # gpart add -i 1 -t freebsd-zfs -a 512k -s 16G nvd0
nvd0p1 added
root@titan:~ # gpart show nvd0
=>       6  24421435  nvd0  GPT  (93G)
         6       122        - free -  (488K)
       128   4194304     1  freebsd-zfs  (16G)
   4194432  20227009        - free -  (77G)


Benched that a bit... saw it *almost* filling up... so repartitioned to 32G.

And i'm actually seeing it hit 1.08GB/s write with iostat, and some SMB 10gbe traffic.

Screen Shot 2021-07-26 at 11.17.23 pm.png


32G means there's plenty of space, but I can still create additional SLOGs for another pool or two if I wanted to.

PS: A drive was resilvering during this testing session, not sure how much that affected things... oh well. This is in service now.
 
Last edited:

QonoS

Member
Joined
Apr 1, 2021
Messages
49
Micron 5100 MAX 1.92TB SATA

Code:
root@truenas[~]# smartctl -a /dev/ada1
smartctl 7.2 2020-12-30 r5155 [FreeBSD 12.2-RELEASE-p6 amd64] (local build)
Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Micron 5100 Pro / 52x0 / 5300 SSDs
Device Model:     Micron_5100_MTFDDAK1T9TCC
Serial Number:    ---
LU WWN Device Id: ---
Firmware Version: D0MU075
User Capacity:    1,920,383,410,176 bytes [1.92 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    Solid State Device
Form Factor:      2.5 inches
TRIM Command:     Available, deterministic, zeroed
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ACS-3 T13/2161-D revision 5
SATA Version is:  SATA 3.2, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Tue Aug  3 15:17:26 2021 PDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x00) Offline data collection activity
                                        was never started.
                                        Auto Offline Data Collection: Disabled.
Self-test execution status:      (   0) The previous self-test routine completed
                                        without error or no self-test has ever
                                        been run.
Total time to complete Offline
data collection:                ( 6697) seconds.
Offline data collection
capabilities:                    (0x7b) SMART execute Offline immediate.
                                        Auto Offline data collection on/off support.
                                        Suspend Offline collection upon new
                                        command.
                                        Offline surface scan supported.
                                        Self-test supported.
                                        Conveyance Self-test supported.
                                        Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                                        power-saving mode.
                                        Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                                        General Purpose Logging supported.
Short self-test routine
recommended polling time:        (   2) minutes.
Extended self-test routine
recommended polling time:        (  14) minutes.
Conveyance self-test routine
recommended polling time:        (   3) minutes.
SCT capabilities:              (0x0035) SCT Status supported.
                                        SCT Feature Control supported.
                                        SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x002f   100   100   050    Pre-fail  Always       -       21
  5 Reallocated_Sector_Ct   0x0032   100   100   001    Old_age   Always       -       95
  9 Power_On_Hours          0x0032   100   100   001    Old_age   Always       -       29340
12 Power_Cycle_Count       0x0032   100   100   001    Old_age   Always       -       71
170 Reserved_Block_Pct      0x0033   100   100   010    Pre-fail  Always       -       95
171 Program_Fail_Count      0x0032   100   100   000    Old_age   Always       -       0
172 Erase_Fail_Count        0x0032   100   100   001    Old_age   Always       -       0
173 Avg_Block-Erase_Count   0x0032   099   099   000    Old_age   Always       -       83
174 Unexpect_Power_Loss_Ct  0x0032   100   100   000    Old_age   Always       -       55
183 SATA_Int_Downshift_Ct   0x0032   100   100   000    Old_age   Always       -       0
184 End-to-End_Error        0x0032   100   100   000    Old_age   Always       -       0
187 Reported_Uncorrect      0x0032   100   100   000    Old_age   Always       -       20
188 Command_Timeout         0x0032   100   100   000    Old_age   Always       -       123
194 Temperature_Celsius     0x0022   065   059   000    Old_age   Always       -       35 (Min/Max 17/41)
195 Hardware_ECC_Recovered  0x0032   100   100   000    Old_age   Always       -       1
196 Reallocated_Event_Count 0x0032   100   100   000    Old_age   Always       -       95
197 Current_Pending_Sector  0x0032   100   100   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0030   100   100   000    Old_age   Offline      -       1
199 UDMA_CRC_Error_Count    0x0032   100   100   000    Old_age   Always       -       0
202 Percent_Lifetime_Remain 0x0030   099   099   001    Old_age   Offline      -       1
206 Write_Error_Rate        0x000e   100   100   000    Old_age   Always       -       0
246 Total_LBAs_Written      0x0032   100   100   000    Old_age   Always       -       326665288896
247 Host_Program_Page_Count 0x0032   100   100   000    Old_age   Always       -       10195047722
248 Bckgnd_Program_Page_Cnt 0x0032   100   100   000    Old_age   Always       -       7222013067
180 Unused_Rsvd_Blk_Cnt_Tot 0x0033   000   000   000    Pre-fail  Always       -       55014
210 RAIN_Success_Recovered  0x0032   100   100   000    Old_age   Always       -       1468
211 Integ_Scan_Complete_Cnt 0x0032   100   100   000    Old_age   Always       -       4
212 Integ_Scan_Folding_Cnt  0x0032   100   100   000    Old_age   Always       -       2


Code:
root@truenas[~]# diskinfo -wS /dev/ada1
/dev/ada1
        512             # sectorsize
        1920383410176   # mediasize in bytes (1.7T)
        3750748848      # mediasize in sectors
        4096            # stripesize
        0               # stripeoffset
        3720981         # Cylinders according to firmware.
        16              # Heads according to firmware.
        63              # Sectors according to firmware.
        Micron 5100 MTFDDAK1T9TCC       # Disk descr.
        ---    # Disk ident.
        Yes             # TRIM/UNMAP support
        0               # Rotation rate in RPM
        Not_Zoned       # Zone Mode

Synchronous random writes:
         0.5 kbytes:     63.2 usec/IO =      7.7 Mbytes/s
           1 kbytes:     63.1 usec/IO =     15.5 Mbytes/s
           2 kbytes:     65.5 usec/IO =     29.8 Mbytes/s
           4 kbytes:     65.2 usec/IO =     59.9 Mbytes/s
           8 kbytes:     72.5 usec/IO =    107.7 Mbytes/s
          16 kbytes:     89.3 usec/IO =    175.0 Mbytes/s
          32 kbytes:    121.5 usec/IO =    257.2 Mbytes/s
          64 kbytes:    185.8 usec/IO =    336.4 Mbytes/s
         128 kbytes:    316.6 usec/IO =    394.9 Mbytes/s
         256 kbytes:    566.5 usec/IO =    441.3 Mbytes/s
         512 kbytes:   1070.2 usec/IO =    467.2 Mbytes/s
        1024 kbytes:   2085.4 usec/IO =    479.5 Mbytes/s
        2048 kbytes:   4078.0 usec/IO =    490.4 Mbytes/s
        4096 kbytes:   8079.1 usec/IO =    495.1 Mbytes/s
        8192 kbytes:  15873.4 usec/IO =    504.0 Mbytes/s
 
Last edited:

Leo Fender

Newbie
Joined
Jan 15, 2021
Messages
1
I don't remember seeing any results of this drive yet, so here is my fresh out of the box Samsung 883 DCT 480GB:

Code:
root@truenas[/]# smartctl -a /dev/ada4
smartctl 7.2 2020-12-30 r5155 [FreeBSD 12.2-RELEASE-p9 amd64] (local build)
Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Samsung based SSDs
Device Model:     Samsung SSD 883 DCT 480GB
Serial Number:    --REDACTED--
LU WWN Device Id: --REDACTED--
Firmware Version: HXT7904Q
User Capacity:    480,103,981,056 bytes [480 GB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    Solid State Device
Form Factor:      2.5 inches
TRIM Command:     Available, deterministic, zeroed
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ACS-4 T13/BSR INCITS 529 revision 5
SATA Version is:  SATA 3.2, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Thu Aug 19 18:57:47 2021 AEST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x00)    Offline data collection activity
                    was never started.
                    Auto Offline Data Collection: Disabled.
Self-test execution status:      (   0)    The previous self-test routine completed
                    without error or no self-test has ever
                    been run.
Total time to complete Offline
data collection:         (    0) seconds.
Offline data collection
capabilities:              (0x53) SMART execute Offline immediate.
                    Auto Offline data collection on/off support.
                    Suspend Offline collection upon new
                    command.
                    No Offline surface scan supported.
                    Self-test supported.
                    No Conveyance Self-test supported.
                    Selective Self-test supported.
SMART capabilities:            (0x0003)    Saves SMART data before entering
                    power-saving mode.
                    Supports SMART auto save timer.
Error logging capability:        (0x01)    Error logging supported.
                    General Purpose Logging supported.
Short self-test routine
recommended polling time:      (   2) minutes.
Extended self-test routine
recommended polling time:      (  35) minutes.
SCT capabilities:            (0x003d)    SCT Status supported.
                    SCT Error Recovery Control supported.
                    SCT Feature Control supported.
                    SCT Data Table supported.

SMART Attributes Data Structure revision number: 1
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  5 Reallocated_Sector_Ct   0x0033   100   100   010    Pre-fail  Always       -       0
  9 Power_On_Hours          0x0032   100   100   000    Old_age   Always       -       0
 12 Power_Cycle_Count       0x0032   099   099   000    Old_age   Always       -       2
177 Wear_Leveling_Count     0x0013   100   100   005    Pre-fail  Always       -       0
179 Used_Rsvd_Blk_Cnt_Tot   0x0013   100   100   010    Pre-fail  Always       -       0
180 Unused_Rsvd_Blk_Cnt_Tot 0x0013   100   100   010    Pre-fail  Always       -       1432
181 Program_Fail_Cnt_Total  0x0032   100   100   010    Old_age   Always       -       0
182 Erase_Fail_Count_Total  0x0032   100   100   010    Old_age   Always       -       0
183 Runtime_Bad_Block       0x0013   100   100   010    Pre-fail  Always       -       0
184 End-to-End_Error        0x0033   100   100   097    Pre-fail  Always       -       0
187 Uncorrectable_Error_Cnt 0x0032   100   100   000    Old_age   Always       -       0
190 Airflow_Temperature_Cel 0x0032   073   070   000    Old_age   Always       -       27
194 Temperature_Celsius     0x0022   073   060   000    Old_age   Always       -       27 (Min/Max 19/30)
195 ECC_Error_Rate          0x001a   200   200   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0032   100   100   000    Old_age   Always       -       0
199 CRC_Error_Count         0x003e   100   100   000    Old_age   Always       -       0
202 Exception_Mode_Status   0x0033   100   100   010    Pre-fail  Always       -       0
235 POR_Recovery_Count      0x0012   100   100   000    Old_age   Always       -       0
241 Total_LBAs_Written      0x0032   100   100   000    Old_age   Always       -       0
242 Total_LBAs_Read         0x0032   099   099   000    Old_age   Always       -       6301
243 SATA_Downshift_Ct       0x0032   100   100   000    Old_age   Always       -       0
244 Thermal_Throttle_St     0x0032   100   100   000    Old_age   Always       -       0
245 Timed_Workld_Media_Wear 0x0032   100   100   000    Old_age   Always       -       65535
246 Timed_Workld_RdWr_Ratio 0x0032   100   100   000    Old_age   Always       -       65535
247 Timed_Workld_Timer      0x0032   100   100   000    Old_age   Always       -       65535
251 NAND_Writes             0x0032   100   100   000    Old_age   Always       -       0

Code:
root@truenas[/]# diskinfo -wS /dev/ada4
/dev/ada4
    512             # sectorsize
    480103981056    # mediasize in bytes (447G)
    937703088       # mediasize in sectors
    4096            # stripesize
    0               # stripeoffset
    930261          # Cylinders according to firmware.
    16              # Heads according to firmware.
    63              # Sectors according to firmware.
    Samsung SSD 883 DCT 480GB    # Disk descr.
    --REDACTED--    # Disk ident.
    Yes             # TRIM/UNMAP support
    0               # Rotation rate in RPM
    Not_Zoned       # Zone Mode

Synchronous random writes:
     0.5 kbytes:     41.7 usec/IO =     11.7 Mbytes/s
       1 kbytes:     42.5 usec/IO =     23.0 Mbytes/s
       2 kbytes:     44.2 usec/IO =     44.2 Mbytes/s
       4 kbytes:     47.4 usec/IO =     82.4 Mbytes/s
       8 kbytes:     54.4 usec/IO =    143.6 Mbytes/s
      16 kbytes:     75.7 usec/IO =    206.3 Mbytes/s
      32 kbytes:    108.1 usec/IO =    289.0 Mbytes/s
      64 kbytes:    173.0 usec/IO =    361.2 Mbytes/s
     128 kbytes:    302.8 usec/IO =    412.8 Mbytes/s
     256 kbytes:    571.9 usec/IO =    437.1 Mbytes/s
     512 kbytes:   1136.0 usec/IO =    440.2 Mbytes/s
    1024 kbytes:   2220.9 usec/IO =    450.3 Mbytes/s
    2048 kbytes:   4390.1 usec/IO =    455.6 Mbytes/s
    4096 kbytes:   8701.8 usec/IO =    459.7 Mbytes/s
    8192 kbytes:  17365.9 usec/IO =    460.7 Mbytes/s

For a SATA drive, I'm quite happy with these numbers.

Also, just for fun, here is a not so fresh Samsung 860 EVO 1TB:

Code:
root@truenas[/]# smartctl -a /dev/ada4
smartctl 7.2 2020-12-30 r5155 [FreeBSD 12.2-RELEASE-p9 amd64] (local build)
Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Samsung based SSDs
Device Model:     Samsung SSD 860 EVO 1TB
Serial Number:    --REDACTED--
LU WWN Device Id: --REDACTED--
Firmware Version: RVT04B6Q
User Capacity:    1,000,204,886,016 bytes [1.00 TB]
Sector Size:      512 bytes logical/physical
Rotation Rate:    Solid State Device
Form Factor:      2.5 inches
TRIM Command:     Available, deterministic, zeroed
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ACS-4 T13/BSR INCITS 529 revision 5
SATA Version is:  SATA 3.2, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Thu Aug 19 18:02:03 2021 AEST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x00)    Offline data collection activity
                    was never started.
                    Auto Offline Data Collection: Disabled.
Self-test execution status:      (   0)    The previous self-test routine completed
                    without error or no self-test has ever
                    been run.
Total time to complete Offline
data collection:         (    0) seconds.
Offline data collection
capabilities:              (0x53) SMART execute Offline immediate.
                    Auto Offline data collection on/off support.
                    Suspend Offline collection upon new
                    command.
                    No Offline surface scan supported.
                    Self-test supported.
                    No Conveyance Self-test supported.
                    Selective Self-test supported.
SMART capabilities:            (0x0003)    Saves SMART data before entering
                    power-saving mode.
                    Supports SMART auto save timer.
Error logging capability:        (0x01)    Error logging supported.
                    General Purpose Logging supported.
Short self-test routine
recommended polling time:      (   2) minutes.
Extended self-test routine
recommended polling time:      (  85) minutes.
SCT capabilities:            (0x003d)    SCT Status supported.
                    SCT Error Recovery Control supported.
                    SCT Feature Control supported.
                    SCT Data Table supported.

SMART Attributes Data Structure revision number: 1
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  5 Reallocated_Sector_Ct   0x0033   100   100   010    Pre-fail  Always       -       0
  9 Power_On_Hours          0x0032   099   099   000    Old_age   Always       -       4168
 12 Power_Cycle_Count       0x0032   099   099   000    Old_age   Always       -       85
177 Wear_Leveling_Count     0x0013   099   099   000    Pre-fail  Always       -       5
179 Used_Rsvd_Blk_Cnt_Tot   0x0013   100   100   010    Pre-fail  Always       -       0
181 Program_Fail_Cnt_Total  0x0032   100   100   010    Old_age   Always       -       0
182 Erase_Fail_Count_Total  0x0032   100   100   010    Old_age   Always       -       0
183 Runtime_Bad_Block       0x0013   100   100   010    Pre-fail  Always       -       0
187 Uncorrectable_Error_Cnt 0x0032   100   100   000    Old_age   Always       -       0
190 Airflow_Temperature_Cel 0x0032   076   057   000    Old_age   Always       -       24
195 ECC_Error_Rate          0x001a   200   200   000    Old_age   Always       -       0
199 CRC_Error_Count         0x003e   100   100   000    Old_age   Always       -       0
235 POR_Recovery_Count      0x0012   099   099   000    Old_age   Always       -       25
241 Total_LBAs_Written      0x0032   099   099   000    Old_age   Always       -       8780283195

Code:
root@truenas[/]# diskinfo -wS /dev/ada4
/dev/ada4
    512             # sectorsize
    1000204886016    # mediasize in bytes (932G)
    1953525168      # mediasize in sectors
    0               # stripesize
    0               # stripeoffset
    1938021         # Cylinders according to firmware.
    16              # Heads according to firmware.
    63              # Sectors according to firmware.
    Samsung SSD 860 EVO 1TB    # Disk descr.
    --REDACTED--    # Disk ident.
    Yes             # TRIM/UNMAP support
    0               # Rotation rate in RPM
    Not_Zoned       # Zone Mode

Synchronous random writes:
     0.5 kbytes:    669.2 usec/IO =      0.7 Mbytes/s
       1 kbytes:    705.6 usec/IO =      1.4 Mbytes/s
       2 kbytes:    672.4 usec/IO =      2.9 Mbytes/s
       4 kbytes:    670.0 usec/IO =      5.8 Mbytes/s
       8 kbytes:    649.8 usec/IO =     12.0 Mbytes/s
      16 kbytes:    666.7 usec/IO =     23.4 Mbytes/s
      32 kbytes:    736.1 usec/IO =     42.5 Mbytes/s
      64 kbytes:    847.5 usec/IO =     73.8 Mbytes/s
     128 kbytes:   1098.1 usec/IO =    113.8 Mbytes/s
     256 kbytes:   1530.7 usec/IO =    163.3 Mbytes/s
     512 kbytes:   1721.0 usec/IO =    290.5 Mbytes/s
    1024 kbytes:   2836.5 usec/IO =    352.6 Mbytes/s
    2048 kbytes:   5078.4 usec/IO =    393.8 Mbytes/s
    4096 kbytes:   9496.4 usec/IO =    421.2 Mbytes/s
    8192 kbytes:  17990.0 usec/IO =    444.7 Mbytes/s
 

rshakin

Junior Member
Joined
Apr 2, 2019
Messages
20
RMS-200 Quick Update, Was going to see if I could be there for when it hit 500pb but missed it by a little

Code:
root@freenas[~]# smartctl -a /dev/nvme0
smartctl 7.2 2020-12-30 r5155 [FreeBSD 12.2-RELEASE-p6 amd64] (local build)
Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Number:                       RMS-200
Serial Number:                      0085217
Firmware Version:                   ae34b8cc
PCI Vendor/Subsystem ID:            0x1cc7
IEEE OUI Identifier:                0x00e0cf
Controller ID:                      0
NVMe Version:                       <1.2
Number of Namespaces:               1
Namespace 1 Size/Capacity:          8,581,545,984 [8.58 GB]
Namespace 1 Utilization:            0
Namespace 1 Formatted LBA Size:     512
Local Time is:                      Thu Aug 26 21:58:21 2021 PDT
Firmware Updates (0x09):            4 Slots, Slot 1 R/O
Optional Admin Commands (0x0006):   Format Frmw_DL
Optional NVM Commands (0x0006):     Wr_Unc DS_Mngmt
Log Page Attributes (0x01):         S/H_per_NS

Supported Power States
St Op     Max   Active     Idle   RL RT WL WT  Ent_Lat  Ex_Lat
 0 +    25.00W       -        -    0  0  0  0      100     100

Supported LBA Sizes (NSID 0x1)
Id Fmt  Data  Metadt  Rel_Perf
 0 -     512       0         3
 1 -     512       8         3
 2 -    4096       0         0
 3 -    4096       8         0

=== START OF SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

SMART/Health Information (NVMe Log 0x02)
Critical Warning:                   0x00
Temperature:                        52 Celsius
Available Spare:                    0%
Available Spare Threshold:          0%
Percentage Used:                    0%
Data Units Read:                    95,516,794,919 [48.9 PB]
Data Units Written:                 1,006,800,245,478 [515 PB]
Host Read Commands:                 1,148,560,059
Host Write Commands:                9,453,074,925
Controller Busy Time:               931,385
Power Cycles:                       42
Power On Hours:                     22,542
Unsafe Shutdowns:                   36
Media and Data Integrity Errors:    0
Error Information Log Entries:      1

Error Information (NVMe Log 0x01, 16 of 63 entries)
No Errors Logged

root@freenas[~]#
 

NugentS

Senior Member
Joined
Apr 16, 2020
Messages
429
Wow. I notice that techbuyer have one listed on Ebay UK, as a graphics card!!

Now gone
:smile:
 
Last edited:

HoneyBadger

Mushroom! Mushroom!
Joined
Feb 6, 2014
Messages
3,339
RMS-200 Quick Update, Was going to see if I could be there for when it hit 500pb but missed it by a little
Even Optane would be burning out at that kind of write load I have to imagine.

Side note - did yours come with the FH bracket or is it not needed for your setup?
 

rshakin

Junior Member
Joined
Apr 2, 2019
Messages
20
I am pretty sure it's half bracket stock one. I am hoping it wont get taken out of commission before we hit 1000PB that would be cool, but we will see if that will happen. We had to replace some things in that system already so might be getting close to that time of getting scrapped, but maybe if it runs for another year or so we shall hit the 1000 and that would be just cool.
 

Kailee71

Member
Joined
Jul 8, 2018
Messages
37
And another RMS-200... Not sure whether the total written is a concern (>800PB:eek:)... but got it cheap just to try out. Nothing tuned at all, just threw it into my trusty old HPE Microserver G8;

Code:
root@waldorf[~]# smartctl -a /dev/nvme0 |less
smartctl 7.2 2020-12-30 r5155 [FreeBSD 12.2-RELEASE-p9 amd64] (local build)
Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Number:                       RMS-200
Serial Number:                      0031867
Firmware Version:                   ae34b8cc
PCI Vendor/Subsystem ID:            0x1cc7
IEEE OUI Identifier:                0x00e0cf
Controller ID:                      0
NVMe Version:                       <1.2
Number of Namespaces:               1
Namespace 1 Size/Capacity:          8,581,545,984 [8.58 GB]
Namespace 1 Utilization:            0
Namespace 1 Formatted LBA Size:     512
Local Time is:                      Wed Sep  1 12:27:39 2021 BST
Firmware Updates (0x09):            4 Slots, Slot 1 R/O
Optional Admin Commands (0x0006):   Format Frmw_DL
Optional NVM Commands (0x0006):     Wr_Unc DS_Mngmt
Log Page Attributes (0x01):         S/H_per_NS

Supported Power States
St Op     Max   Active     Idle   RL RT WL WT  Ent_Lat  Ex_Lat
 0 +    25.00W       -        -    0  0  0  0      100     100

Supported LBA Sizes (NSID 0x1)
Id Fmt  Data  Metadt  Rel_Perf
 0 -     512       0         3
 1 -     512       8         3
 2 -    4096       0         0
 3 -    4096       8         0

=== START OF SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

SMART/Health Information (NVMe Log 0x02)
Critical Warning:                   0x00
Temperature:                        36 Celsius
Available Spare:                    0%
Available Spare Threshold:          0%
Percentage Used:                    0%
Data Units Read:                    99,211,764,012 [50.7 PB]
Data Units Written:                 1,620,540,716,700 [829 PB]
Host Read Commands:                 455,532,805
Host Write Commands:                309,357,943,886
Controller Busy Time:               377
Power Cycles:                       19
Power On Hours:                     29,197
Unsafe Shutdowns:                   9
Media and Data Integrity Errors:    0
Error Information Log Entries:      0

Error Information (NVMe Log 0x01, 16 of 63 entries)
No Errors Logged


and

Code:
root@waldorf[~]# diskinfo -wS /dev/nvd0
/dev/nvd0
        512             # sectorsize
        8581545984      # mediasize in bytes (8.0G)
        16760832        # mediasize in sectors
        0               # stripesize
        0               # stripeoffset
        RMS-200         # Disk descr.
        0031867         # Disk ident.
        Yes             # TRIM/UNMAP support
        0               # Rotation rate in RPM

Synchronous random writes:
         0.5 kbytes:     23.1 usec/IO =     21.1 Mbytes/s
           1 kbytes:     23.7 usec/IO =     41.3 Mbytes/s
           2 kbytes:     24.2 usec/IO =     80.6 Mbytes/s
           4 kbytes:     24.6 usec/IO =    159.0 Mbytes/s
           8 kbytes:     26.0 usec/IO =    300.8 Mbytes/s
          16 kbytes:     32.4 usec/IO =    481.8 Mbytes/s
          32 kbytes:     40.2 usec/IO =    777.7 Mbytes/s
          64 kbytes:     62.5 usec/IO =    999.4 Mbytes/s
         128 kbytes:     86.3 usec/IO =   1448.9 Mbytes/s
         256 kbytes:    117.4 usec/IO =   2130.0 Mbytes/s
         512 kbytes:    197.1 usec/IO =   2536.4 Mbytes/s
        1024 kbytes:    345.4 usec/IO =   2895.1 Mbytes/s
        2048 kbytes:    652.4 usec/IO =   3065.5 Mbytes/s
        4096 kbytes:   1225.1 usec/IO =   3265.1 Mbytes/s
        8192 kbytes:   2399.6 usec/IO =   3333.8 Mbytes/s


So up to 256kbytes ahead of @rshakin, thereafter I'm guessing there would be some BIOS tweaking needed. Leaving it alone as it will go into my DL380p next week.
 

Kailee71

Member
Joined
Jul 8, 2018
Messages
37
Wow. I notice that techbuyer have one listed on Ebay UK, as a graphics card!!

Now gone
:smile:
... two left now ... ;-)

BTW I can fully recommend TechBuyer. Not affiliated in any way but have only fully positive experiences with them.
 

erthink

Newbie
Joined
Sep 2, 2021
Messages
1
Apparently the auto-backup to flash functionality is disabled by default, and needs to be re-enabled on every boot. Auto-restore is also disabled by default but at least persists. I've requested access to the code/firmware, might need to go under NDA which will complicate things.
Hi,

Could you help me with some cli-utilities and/or linux drivers for NV16xx?
Unfortunately, I couldn't find anything usable and downloads from PMC's site are no longer availale (for instance, from the "documentation" tab at http://216.241.235.166/products/storage/flashtec_nvram_drives/nv1616/).
I would be grateful for any information, especially for a source code of utilities and/or drivers (for any OS), etc.
 

rshakin

Junior Member
Joined
Apr 2, 2019
Messages
20
Hi,

Could you help me with some cli-utilities and/or linux drivers for NV16xx?
Unfortunately, I couldn't find anything usable and downloads from PMC's site are no longer availale (for instance, from the "documentation" tab at http://216.241.235.166/products/storage/flashtec_nvram_drives/nv1616/).
I would be grateful for any information, especially for a source code of utilities and/or drivers (for any OS), etc.

Trust Me Comrad I tried getting that stuff from them even got a hold of the main boys in development, no go, and everything is locked tight
 

rshakin

Junior Member
Joined
Apr 2, 2019
Messages
20
And another RMS-200... Not sure whether the total written is a concern (>800PB:eek:)... but got it cheap just to try out. Nothing tuned at all, just threw it into my trusty old HPE Microserver G8;

Code:
root@waldorf[~]# smartctl -a /dev/nvme0 |less
smartctl 7.2 2020-12-30 r5155 [FreeBSD 12.2-RELEASE-p9 amd64] (local build)
Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Number:                       RMS-200
Serial Number:                      0031867
Firmware Version:                   ae34b8cc
PCI Vendor/Subsystem ID:            0x1cc7
IEEE OUI Identifier:                0x00e0cf
Controller ID:                      0
NVMe Version:                       <1.2
Number of Namespaces:               1
Namespace 1 Size/Capacity:          8,581,545,984 [8.58 GB]
Namespace 1 Utilization:            0
Namespace 1 Formatted LBA Size:     512
Local Time is:                      Wed Sep  1 12:27:39 2021 BST
Firmware Updates (0x09):            4 Slots, Slot 1 R/O
Optional Admin Commands (0x0006):   Format Frmw_DL
Optional NVM Commands (0x0006):     Wr_Unc DS_Mngmt
Log Page Attributes (0x01):         S/H_per_NS

Supported Power States
St Op     Max   Active     Idle   RL RT WL WT  Ent_Lat  Ex_Lat
0 +    25.00W       -        -    0  0  0  0      100     100

Supported LBA Sizes (NSID 0x1)
Id Fmt  Data  Metadt  Rel_Perf
0 -     512       0         3
1 -     512       8         3
2 -    4096       0         0
3 -    4096       8         0

=== START OF SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

SMART/Health Information (NVMe Log 0x02)
Critical Warning:                   0x00
Temperature:                        36 Celsius
Available Spare:                    0%
Available Spare Threshold:          0%
Percentage Used:                    0%
Data Units Read:                    99,211,764,012 [50.7 PB]
Data Units Written:                 1,620,540,716,700 [829 PB]
Host Read Commands:                 455,532,805
Host Write Commands:                309,357,943,886
Controller Busy Time:               377
Power Cycles:                       19
Power On Hours:                     29,197
Unsafe Shutdowns:                   9
Media and Data Integrity Errors:    0
Error Information Log Entries:      0

Error Information (NVMe Log 0x01, 16 of 63 entries)
No Errors Logged


and

Code:
root@waldorf[~]# diskinfo -wS /dev/nvd0
/dev/nvd0
        512             # sectorsize
        8581545984      # mediasize in bytes (8.0G)
        16760832        # mediasize in sectors
        0               # stripesize
        0               # stripeoffset
        RMS-200         # Disk descr.
        0031867         # Disk ident.
        Yes             # TRIM/UNMAP support
        0               # Rotation rate in RPM

Synchronous random writes:
         0.5 kbytes:     23.1 usec/IO =     21.1 Mbytes/s
           1 kbytes:     23.7 usec/IO =     41.3 Mbytes/s
           2 kbytes:     24.2 usec/IO =     80.6 Mbytes/s
           4 kbytes:     24.6 usec/IO =    159.0 Mbytes/s
           8 kbytes:     26.0 usec/IO =    300.8 Mbytes/s
          16 kbytes:     32.4 usec/IO =    481.8 Mbytes/s
          32 kbytes:     40.2 usec/IO =    777.7 Mbytes/s
          64 kbytes:     62.5 usec/IO =    999.4 Mbytes/s
         128 kbytes:     86.3 usec/IO =   1448.9 Mbytes/s
         256 kbytes:    117.4 usec/IO =   2130.0 Mbytes/s
         512 kbytes:    197.1 usec/IO =   2536.4 Mbytes/s
        1024 kbytes:    345.4 usec/IO =   2895.1 Mbytes/s
        2048 kbytes:    652.4 usec/IO =   3065.5 Mbytes/s
        4096 kbytes:   1225.1 usec/IO =   3265.1 Mbytes/s
        8192 kbytes:   2399.6 usec/IO =   3333.8 Mbytes/s


So up to 256kbytes ahead of @rshakin, thereafter I'm guessing there would be some BIOS tweaking needed. Leaving it alone as it will go into my DL380p next week.
I am wondering how you got it to that speed, but I think my system is set up to handle large files or maybe there is some kind of a tweak in there. What system did you test it on ?
 

Kailee71

Member
Joined
Jul 8, 2018
Messages
37
I am wondering how you got it to that speed, but I think my system is set up to handle large files or maybe there is some kind of a tweak in there. What system did you test it on ?
Sorry for the delayed reply. System is a box-standard HPE Microserver G8 with the "standard" G1610T Celeron, 16Gb ECC, 2 Hitachi 2Tib drives, 2 WD 4Tib Reds as striped mirrors, and temporarily the RMS-200.

TrueNAS-12.0-U5.1 with no tunables. Nothing special *at all*. Just threw in the RMS and did the diskinfo. Let me know if you need any details, and how to get them. This system has no 10GbE so can't do any testing that means anything but I'm back home next week where the RMS will replace 2 32Gib Optanes in my production system... More testing will follow then.
 

Kailee71

Member
Joined
Jul 8, 2018
Messages
37
And just for completions' sake, here is that same RMS-200 in my DL380p G8;
Code:
truenas# smartctl -a /dev/nvme0
smartctl 7.2 2020-12-30 r5155 [FreeBSD 12.2-RELEASE-p9 amd64] (local build)
Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Number:                       RMS-200
Serial Number:                      0031867
Firmware Version:                   ae34b8cc
PCI Vendor/Subsystem ID:            0x1cc7
IEEE OUI Identifier:                0x00e0cf
Controller ID:                      0
NVMe Version:                       <1.2
Number of Namespaces:               1
Namespace 1 Size/Capacity:          8,581,545,984 [8.58 GB]
Namespace 1 Utilization:            0
Namespace 1 Formatted LBA Size:     512
Local Time is:                      Wed Sep 22 22:40:19 2021 CEST
Firmware Updates (0x09):            4 Slots, Slot 1 R/O
Optional Admin Commands (0x0006):   Format Frmw_DL
Optional NVM Commands (0x0006):     Wr_Unc DS_Mngmt
Log Page Attributes (0x01):         S/H_per_NS

Supported Power States
St Op     Max   Active     Idle   RL RT WL WT  Ent_Lat  Ex_Lat
0 +    25.00W       -        -    0  0  0  0      100     100

Supported LBA Sizes (NSID 0x1)
Id Fmt  Data  Metadt  Rel_Perf
0 -     512       0         3
1 -     512       8         3
2 -    4096       0         0
3 -    4096       8         0

=== START OF SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

SMART/Health Information (NVMe Log 0x02)
Critical Warning:                   0x00
Temperature:                        45 Celsius
Available Spare:                    0%
Available Spare Threshold:          0%
Percentage Used:                    0%
Data Units Read:                    99,211,766,494 [50.7 PB]
Data Units Written:                 1,622,149,164,390 [830 PB]
Host Read Commands:                 455,533,395
Host Write Commands:                309,372,726,826
Controller Busy Time:               386
Power Cycles:                       21
Power On Hours:                     29,198
Unsafe Shutdowns:                   10
Media and Data Integrity Errors:    0
Error Information Log Entries:      0

Error Information (NVMe Log 0x01, 16 of 63 entries)
No Errors Logged


and

Code:
truenas#  diskinfo -wS /dev/nvd0
/dev/nvd0
        512             # sectorsize
        8581545984      # mediasize in bytes (8.0G)
        16760832        # mediasize in sectors
        0               # stripesize
        0               # stripeoffset
        RMS-200         # Disk descr.
        0031867         # Disk ident.
        Yes             # TRIM/UNMAP support
        0               # Rotation rate in RPM

Synchronous random writes:
         0.5 kbytes:     45.3 usec/IO =     10.8 Mbytes/s
           1 kbytes:     45.4 usec/IO =     21.5 Mbytes/s
           2 kbytes:     49.2 usec/IO =     39.7 Mbytes/s
           4 kbytes:     48.3 usec/IO =     80.9 Mbytes/s
           8 kbytes:     52.0 usec/IO =    150.2 Mbytes/s
          16 kbytes:     60.0 usec/IO =    260.4 Mbytes/s
          32 kbytes:     69.9 usec/IO =    447.0 Mbytes/s
          64 kbytes:     82.2 usec/IO =    760.0 Mbytes/s
         128 kbytes:    151.3 usec/IO =    826.2 Mbytes/s
         256 kbytes:    203.2 usec/IO =   1230.5 Mbytes/s
         512 kbytes:    239.7 usec/IO =   2086.0 Mbytes/s
        1024 kbytes:    358.0 usec/IO =   2792.9 Mbytes/s
        2048 kbytes:    589.1 usec/IO =   3395.0 Mbytes/s
        4096 kbytes:   1060.2 usec/IO =   3772.9 Mbytes/s
        8192 kbytes:   1921.4 usec/IO =   4163.6 Mbytes/s
truenas#


Hm.... Instrumenting, as my little girl used to say. Will need to check out why small I/O breaks down so badly. It's not until1024kbytes that it catches up. I sense some BIOS tweaking coming. Anybody have an idea before I reboot a few dozen times?

In the mean time; a quick comparison chart:
1632348449232.png
 
Last edited:
Top