How do I change a hard drive's logical sector size from 512 bytes to 4096 bytes?

[thalf]

So I've gone and purchased a pair of WD Red 20TB disks, and am about to do burn-in testing on them.

To start with I'm doing a long S.M.A.R.T. test, which will take almost 35 hours.

Then I thought I'd run jgreco's solnet-array-test-v2.sh script for a week or two (instead of using badblocks which I've done previously with other disks), and finish with another long S.M.A.R.T. test.

But I noticed this in the smartctl -a output:
Sector Sizes: 512 bytes logical, 4096 bytes physical

From what I understand, it's not a bad idea to change the drives' logical sector sizes to 4k as well. (Or is that just for SSDs?) And if I change it, I though it'd be a good idea to do it before running jgreco's script, so the burn-in is working on the drives as they will be configured when in use.

In Linux, logical sector size can be set using hdparm (according to stackexchange):
hdparm --set-sector-size 4096 --please-destroy-my-drive /dev/sdX

Is there a way to change a drive's logical sector size in TrueNAS?

Would sg_format --format --size=4096 -v /dev/ada2 do what I want? Have anyone here done this before and can comment or give advice? sg_format was used to change sector size in the "Replacing 4Kn disks" thread, with mixed results.

 

[NugentS]

Based on my experience where I did just this and then trashed the array and put them back to 512e

Don't bother.
1. There is little (if any) performace benefit
2. I have some spares that despite saying 4096 - I was not able to format to 4096 which would have made replacing a drive with one of my spares difficult

 

[markanderson]

You can't change the sector size of a hard drive, the sectors are created when the hard drive is manufactured and can't be changed. You can change the NTFS cluster size (allocation unit size} of the file system using Disk Management when you format the disk.

 

[HoneyBadger]

You can't change the sector size of a hard drive, the sectors are created when the hard drive is manufactured and can't be changed. You can change the NTFS cluster size (allocation unit size} of the file system using Disk Management when you format the disk.

@thalf is asking about the logical sector size - it is already 4K physically, but the theory behind the conversion is that it will remove whatever drive firmware overhead is in place that causes it to be broken into eight 512-byte sectors.

The default TrueNAS configuration will never use an ashift value lower than 12 on data vdevs, meaning the smallest write to disk that TrueNAS will ever make is 4K - so the read-modify-write from 512e isn't a risk here, but the thought process is "why go from 4K down to 8x512b back to 4K, and potentially introduce some edge-case failure?"

It doesn't seem to provide much in the way of a performance benefit as mentioned by @NugentS - I've seen some benefit on SSDs anecdotally, but it seems almost margin-of-error and likely varies between SSD model/firmware. If I ever find the time, I'll do some repeated, controlled testing and collect hard data on at least the models I do have.

hdparm --set-sector-size 4096 --please-destroy-my-drive

I quite enjoy when parameters like this are required in order to confirm a destructive action. Removes any ambiguity and gives me a chuckle at the same time.

 

[ChrisRJ]

I quite enjoy when parameters like this are required in order to confirm a destructive action. Removes any ambiguity and gives me a chuckle at the same time.

Closely followed by the com.acme.app.CommitSuicideException and Do_not_use_in_production.exe

 

[HoneyBadger]

Closely followed by the com.acme.app.CommitSuicideException and Do_not_use_in_production.exe

hdparm also requires --yes-i-know-what-i-am-doing to set the Host Protected Area, I believe, for similar "will probably destroy your existing data" reasons.

 

[thalf]

I'm using ZFS, not NTFS (of course, since this is a TrueNAS forum).

I've started burn-in tests using jgreco's solnet-array-test-v2.sh script. The "initial parallel array read" phase on two identical 20TB WD Red drives took 24h 13m 31s (i.e. 1453 minutes, while the script estimated 1187 minutes; ~22% slower than estimated), and I am currently waiting for the "initial parallel seek-stress array read" phase to finish. It should've finished 2 hours ago according to the script's estimates, but I'm in no hurry. If this phase is also ~22% slower than estimated, it'll be done in about 3 hours.

After the script finishes, I plan to switch to 4k logical sectors using sg_format --format --size=4096 -v /dev/ada2 (unless someone warns me that that command won't work as expected or will destroy the drive etc.), and then rerun the script to see if there's any noticeable difference. I'll report back after that.

 

[Daisuke]

I was not able to format to 4096 which would have made replacing a drive with one of my spares difficult

I had the same experience, I could not format a 512e to 4Kn. From my understanding, if a vdev of 6 drives has a mix of 512e and 4Kn drives, they should be replaced to have the same logical sector size, either 512e or 4Kn. Please let me know if I'm correct. If both types can co-exist in a vdev, what would be the impact of have them mixed? I think one performance factor is if your drive is a common 512-byte physical sector and the vdev has an ashift=12:

Code:

# zpool status software
  pool: software
 state: ONLINE
  scan: scrub repaired 0B in 00:01:15 with 0 errors on Sun Jan  1 00:01:18 2023
config:

    NAME                                      STATE     READ WRITE CKSUM
    software                                  ONLINE       0     0     0
      mirror-0                                ONLINE       0     0     0
        a089bcf4-3dc1-42d7-af8c-01122fcfba2a  ONLINE       0     0     0
        257bfd2f-b400-4aa5-b3e6-dbde1f479f2f  ONLINE       0     0     0

errors: No known data errors

# lsblk -o NAME,PARTUUID,PATH,FSTYPE /dev/sdj
NAME        PARTUUID                             PATH              FSTYPE
sdj                                              /dev/sdj
|-sdj1      53555172-7cf5-4ae1-97e9-e760b337c21a /dev/sdj1         linux_raid_member
| `-md127                                        /dev/md127
|   `-md127                                      /dev/mapper/md127 swap
`-sdj2      257bfd2f-b400-4aa5-b3e6-dbde1f479f2f /dev/sdj2         zfs_member

# fdisk -l /dev/sdj
Disk /dev/sdj: 465.76 GiB, 500107862016 bytes, 976773168 sectors
Disk model: Samsung SSD 870
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 2207AFEE-22F9-4DBE-94BC-3283F16901F1

# zpool get ashift software
NAME      PROPERTY  VALUE   SOURCE
software  ashift    12      local

the thought process is "why go from 4K down to 8x512b back to 4K, and potentially introduce some edge-case failure?"

Can you please detail more on this? I will update my guide with this useful information, take a look at it and let me know if everything is accurate into ZFS and Advanced Format Hard Drives section.

 

[artlessknave]

4kn and 512e should generally work fine together. like using 5900 RPM and 5700 RPM drives. (unless they are SMR - which WD RED drives are)

that makes 2 posts I have seen today with people trying to "format" sector sizes. me confused.

 

[winnielinnie]

From my understanding, if a vdev of 6 drives has a mix of 512e and 4Kn drives, they should be replaced to have the same logical sector size, either 512e or 4Kn. Please let me know if I'm correct. If both types can co-exist in a vdev, what would be the impact of have them mixed? I think one performance factor is if your drive is a common 512-byte physical sector and the vdev has an ashift=12

A vdev with an ashift=12 will use 4 KiB units as the smallest write operations. Whether your drive is "4Kn" or "4K/512e", they will be treated the same as part of the same ZFS vdev/pool. These 4 KiB writes are compatible (with essentially the same performance) in either of those drives of the same vdev.

Had you created a vdev with "4K/512e" drives (or even older "512-byte physical sector" drives) using ashift=9, then you could not add "4Kn" drives to this vdev, nor replace any failed drives with "4Kn" drives.

But, per the sane defaults of TrueNAS, all newly created pools use ashift=12 (4 KiB) for their vdevs, which means "4Kn" and "4K/512e" drives are good to go, even in harmony.

Any stories you hear of people who say "I successfully added a 4K-sector drive to my vdev with ashift=9", are actually referring to "4K/512e" drives, not "4Kn".

 

[artlessknave]

ashift=9, then you could not add "4Kn" drives to this vdev

I think you can actually add the drives. the pool will just perform like expensive garbage.

 

[winnielinnie]

I thought the firmware of 4Kn drives would refuse to even try to write a 512-byte block? What would such a drive do with such a request from the filesystem software? Accept the 512-byte I/Os from the filesystem software, and always pad an extra 3,584 bytes of zeros to align it to its own logical/physical 4 KiB sectors?

I may have misinterpreted how an ashift value smaller than 4 KiB can still in fact be accepted by 4Kn drives' firmware. (I understand that a 4K/512e drive can "handle" it, since it basically "lies" to the filesystem. But a 4Kn drive does not "lie" nor try to "emulate" anything.)

 

[artlessknave]

I thought the firmware of 4Kn drives would refuse to even try to write a 512-byte block?

I believe this is part of why the performance absolutely tanks. think like the pool gets 5MB/s speeds slow. possibly not unlike how badly things go with SMR drives. doesn't actually kill it...but it's not usable either. it has been awhile since I read the chatter about ashift, so I could be mis-remembering.

 

[HoneyBadger]

Can you please detail more on this? I will update my guide with this useful information, take a look at it and let me know if everything is accurate into ZFS and Advanced Format Hard Drives section.

Hard drive firmware is a "black box" in that we aren't really party to what's happening internally, in that transition from a 4K write from ZFS (ashift=12) down to eight logical 512-byte sectors (still at the OS/storage driver) and then back to a 4K write at the drive firmware. The thought here would be that if ZFS isn't ever going to write anything smaller than 4K, we might as well not have that middle step of splitting up the single 4K I/O request into eight 512-byte ones, and then have the drive have to glom them back together into a 4K. Give the "black box" fewer chances to potentially have a bug crop up, as well as reduce any chances of misaligned partitions (although TrueNAS aims to avoid that during setup as well.)

I thought the firmware of 4Kn drives would refuse to even try to write a 512-byte block? What would such a drive do with such a request from the filesystem software? Accept the 512-byte I/Os from the filesystem software, and always pad an extra 3,584 bytes of zeros to align it to its own logical/physical 4 KiB sectors?

A truly "512-byte write" should be rejected by the OS/storage driver as an invalid write to a 4Kn device - it would get padded out to 4K at the ZFS/OS level, and then written into a 4K sector on its own with a whole lot of empty space. But a true 512-byte write is fairly rare (lock files on VMware being one of the more common modern examples) so it doesn't happen often enough to really be a pressing concern.

 

[Daisuke]

if ZFS isn't ever going to write anything smaller than 4K, we might as well not have that middle step of splitting up the single 4K I/O request into eight 512-byte ones

That was my reasoning also, if I understand correctly, is preferable to have disks formatted to 4Kn logical volumes. However, based on my tests, confirmed by @NugentS, I was not able to format several 512e disks to 4Kn.

 

[jgreco]

hdparm also requires --yes-i-know-what-i-am-doing to set the Host Protected Area, I believe, for similar "will probably destroy your existing data" reasons.

Our shop bench PXE environment has a menu full of disk stuff. "DBAN" is protected by a dialog that requires you to type "avada kedavra".

 

[artlessknave]

"avada kedavra".

ah. a man of culture.

 

[HoneyBadger]

That was my reasoning also, if I understand correctly, is preferable to have disks formatted to 4Kn logical volumes. However, based on my tests, confirmed by @NugentS, I was not able to format several 512e disks to 4Kn.

The disk's firmware has to explicitly support 4Kn sectors - this is common in "enterprise" or "professional" drives, but might be absent in a "consumer" drive. Where each manufacturer decides to drive that line between their products is often unclear, or changes over time.

 

[Daisuke]

The disk's firmware has to explicitly support 4Kn sectors

I use HGST helium enterprise drives used in datacenters, disk is 4Kn:

I think the problem is related to ancient version of hdparm 9.60 used in Debian Bullseye. WD added some fixes starting with 9.61. I tried downloading the 9.65 version from Bookworm but it requires a different Glib library version:

Code:

# /usr/local/bin/hdparm -V
/usr/local/bin/hdparm: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.33' not found (required by /usr/local/bin/hdparm)
/usr/local/bin/hdparm: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.34' not found (required by /usr/local/bin/hdparm)

Output of current disk with 9.60, I don't see anywhere the supported format options:

Code:

# hdparm -I /dev/sda
/dev/sda:

ATA device, with non-removable media
    Model Number:       HUH728080ALE601
    Firmware Revision:  A4GL0003
    Transport:          Serial, ATA8-AST, SATA 1.0a, SATA II Extensions, SATA Rev 2.5, SATA Rev 2.6, SATA Rev 3.0; Revision: ATA8-AST T13 Project D1697 Revision 0b
Standards:
    Used: unknown (minor revision code 0x0029)
    Supported: 9 8 7 6 5
    Likely used: 9
Configuration:
    Logical        max    current
    cylinders    16383    16383
    heads        16    16
    sectors/track    63    63
    --
    CHS current addressable sectors:    16514064
    LBA    user addressable sectors:   268435455
    LBA48  user addressable sectors: 15628053168
    Logical  Sector size:                   512 bytes
    Physical Sector size:                  4096 bytes
    Logical Sector-0 offset:                  0 bytes
    device size with M = 1024*1024:     7630885 MBytes
    device size with M = 1000*1000:     8001563 MBytes (8001 GB)
    cache/buffer size  = unknown
    Form Factor: 3.5 inch
    Nominal Media Rotation Rate: 7200
Capabilities:
    LBA, IORDY(can be disabled)
    Queue depth: 32
    Standby timer values: spec'd by Standard, no device specific minimum
    R/W multiple sector transfer: Max = 16    Current = 0
    Advanced power management level: disabled
    DMA: mdma0 mdma1 mdma2 udma0 udma1 udma2 udma3 udma4 udma5 *udma6
         Cycle time: min=120ns recommended=120ns
    PIO: pio0 pio1 pio2 pio3 pio4
         Cycle time: no flow control=120ns  IORDY flow control=120ns
Commands/features:
    Enabled    Supported:
       *    SMART feature set
            Security Mode feature set
       *    Power Management feature set
       *    Write cache
       *    Look-ahead
       *    Host Protected Area feature set
       *    WRITE_BUFFER command
       *    READ_BUFFER command
       *    NOP cmd
       *    DOWNLOAD_MICROCODE
            Advanced Power Management feature set
            Power-Up In Standby feature set
       *    SET_FEATURES required to spinup after power up
            SET_MAX security extension
       *    48-bit Address feature set
       *    Device Configuration Overlay feature set
       *    Mandatory FLUSH_CACHE
       *    FLUSH_CACHE_EXT
       *    SMART error logging
       *    SMART self-test
       *    Media Card Pass-Through
       *    General Purpose Logging feature set
       *    WRITE_{DMA|MULTIPLE}_FUA_EXT
       *    64-bit World wide name
       *    URG for READ_STREAM[_DMA]_EXT
       *    URG for WRITE_STREAM[_DMA]_EXT
       *    WRITE_UNCORRECTABLE_EXT command
       *    {READ,WRITE}_DMA_EXT_GPL commands
       *    Segmented DOWNLOAD_MICROCODE
            unknown 119[6]
            unknown 119[7]
       *    Gen1 signaling speed (1.5Gb/s)
       *    Gen2 signaling speed (3.0Gb/s)
       *    Gen3 signaling speed (6.0Gb/s)
       *    Native Command Queueing (NCQ)
       *    Host-initiated interface power management
       *    Phy event counters
       *    NCQ priority information
       *    READ_LOG_DMA_EXT equivalent to READ_LOG_EXT
            Non-Zero buffer offsets in DMA Setup FIS
       *    DMA Setup Auto-Activate optimization
            Device-initiated interface power management
            In-order data delivery
       *    Software settings preservation
            unknown 78[7]
            unknown 78[10]
            unknown 78[11]
       *    SMART Command Transport (SCT) feature set
       *    SCT Write Same (AC2)
       *    SCT Error Recovery Control (AC3)
       *    SCT Features Control (AC4)
       *    SCT Data Tables (AC5)
       *    SANITIZE feature set
       *    CRYPTO_SCRAMBLE_EXT command
       *    OVERWRITE_EXT command
       *    Extended number of user addressable sectors
       *    Device encrypts all user data
       *    WRITE BUFFER DMA command
       *    READ BUFFER DMA command
Security:
    Master password revision code = 65533
        supported
    not    enabled
    not    locked
    not    frozen
    not    expired: security count
        supported: enhanced erase
    986min for SECURITY ERASE UNIT. 2min for ENHANCED SECURITY ERASE UNIT.
Logical Unit WWN Device Identifier: 5000cca254d9cd21
    NAA        : 5
    IEEE OUI    : 000cca
    Unique ID    : 254d9cd21
Checksum: correct

I never used Ubuntu, they don't provide a true Live CD so I'll be stuck downloading the packages when I mount their ISO on a Virtual CD. I also tried the proprietary SeaChest and openSeaChest, but they don't support format options for SATA, unless I'm missing something? sg_format does not have any love either.

I think the only solution is to use the Debian Bookworm alpha ISO and see if I can get lucky, I'm open to suggestions.

 

[winnielinnie]

I think the problem is related to ancient version of hdparm 9.60 used in Debian Bullseye. WD added some fixes starting with 9.61. I tried downloading the 9.65 version from Bookworm but it requires a different Glib library version:

You can use the latest stable live ISO of SystemRescue, which comes pre-shipped with hdparm 9.65 "ready to go" without needing to do anything extra.

It's like "Swiss army knife" live Linux ISO. I've used it many times.

Just boot into the default session from the menu selection, and you're good to go.

The same disclaimer about "be careful not to destroy anything" applies.