Daisuke
Contributor
- Joined
- Jun 23, 2011
- Messages
- 1,041
Synopsis
In today's world, reliable data storage is the most valuable asset, treat it accordingly and always purchase your CMR hard drives from reputable sellers. If you cannot successfully perform the formatting procedure detailed below, it usually means your hard drive has custom firmware applied or is just defective. You are welcome to share your found fixes in this thread, I will add them as reference for other users.
This guide is part of my Bluefin Recommended Settings and Optimizations guide, I recommend you go through all sections to validate if you don't experience other issues.
ZFS and Advanced Format Hard Drives
I'm going to look at the differences from a ZFS usage perspective, since we are dealing with Scale. Each hard drive has two type of sectors:
Known Issues
Some users get this warning message, after Bluefin upgrade:
Another case is disks are not being seen by pool, yet the pool reports no errors. You might deal with one or a combination of both issues listed below. Bluefin's newer kernel/ZFS version makes available these feature checks and throws a warning, which explains why you did not see it before.
T10 Protection Information
T10-PI is an extension of the existing T10 SCSI Block Commands specification, covering communication between SCSI controllers and storage devices, Protection Information (PI) adds an extra 8 bytes of information to the 512-byte sectors typical of enterprise hard drives.
Branded Disks
Linux does not support 520-byte sectors, unless the drive is formatted with DIF and installed into a DIF-capable HBA. This explains why OEM branded HDDs or SSDs from major storage vendors are "enhanced" with 520-byte sectors. In rare cases, you might also find disk sectors extended to 528-byte, with custom proprietary firmware.
Formatting Procedure
HGST Ultrastar Firmware (for Advanced Users)
HGST uses proprietary firmware on their Ultrastar models and Western Digital provides upon request a proprietary tool used for performing low-level maintenance on compatible disk drives, such as conversion to 4K native sectoring. I created a resource with the software I used to format my HGST Helium hard drives from 512e to 4Kn logical sectors.
It is strongly recommended to contact Western Digital and provide your hard drive serial number, to make sure you obtain a compatible low-level maintenance tool. Using the wrong tool could permanently brick your device. The steps listed below will apply to any of their proprietary formatting tools, always check the documentation to make sure the Hugo related commands are still valid.
Since my disks are OEM, WD did not provided any support, so I had to find a solution by myself.
Firmware Update
Newer HGST firmware might be available at HDDGuru, search on page for your disk model number (i.e.
I will update the flashing procedure with Hugo, when I have some time.
In today's world, reliable data storage is the most valuable asset, treat it accordingly and always purchase your CMR hard drives from reputable sellers. If you cannot successfully perform the formatting procedure detailed below, it usually means your hard drive has custom firmware applied or is just defective. You are welcome to share your found fixes in this thread, I will add them as reference for other users.
This guide is part of my Bluefin Recommended Settings and Optimizations guide, I recommend you go through all sections to validate if you don't experience other issues.
ZFS and Advanced Format Hard Drives
I'm going to look at the differences from a ZFS usage perspective, since we are dealing with Scale. Each hard drive has two type of sectors:
- physical - 512, 520, or 528-byte for common drives or 4096, 4112, 4160, and 4224-byte for Advanced Format drives
- logical - 512, 520, or 528-byte for common drives or 512 emulation (512e) and 4K native (4Kn) for Advanced Format drives
Most hardware and software components adopt hard drives configured around the 512-byte sector. To solve this problem, a 4K physical sectors hard drive is coupled with 512-byte conversion firmware emulation, so that the 4K physical sectors used in Advanced Format are translated into 8 traditional 512-byte logical sectors compatible with operating systems. In other words, the hard drive must perform extra mechanical steps in the form of reading a 4K sector, modifying the contents and then writing the data. This process is called a read-modify-write (RMW) cycle. The 4K native hard drive configured with 4K physical sectors and 4K logical sectors, there is no emulation layer in place.
Example of logical and physical sectors present in an AF 512 emulation (512e) hard drive:
TrueNAS Scale uses by default the
Based on my own tests, confirmed also by @NugentS, there are no real performance benefits formatting the 512e logical sectors of a hard drive to 4Kn, but as @HoneyBadger mentioned in this post, is preferred to avoid the read-modify-write cycle, if your disk can be formatted to 4Kn logical sectors. Both 512e and 4Kn logical sector hard drives can co-exist in a pool vdev.
Example of logical and physical sectors present in an AF 512 emulation (512e) hard drive:
Code:
# fdisk -l /dev/sda Disk /dev/sda: 7.28 TiB, 8001563222016 bytes, 15628053168 sectors Disk model: HUH728080ALE601 Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 4096 bytes I/O size (minimum/optimal): 4096 bytes / 4096 bytes
TrueNAS Scale uses by default the
ashift=12 flag to define the 4096-byte sectors, when a new pool vdev is created. Basically, ashift tells ZFS what's the underlying physical block size used by your hard drives. It is measured in bits:ashift=9- 512B sectors, used by common drivesashift=12- 4K sectors, used by AF hard drivesashift=13- 8K sectors used by some modern SSDs
ashift value set on default pool:Code:
# zpool get ashift default NAME PROPERTY VALUE SOURCE default ashift 12 local
Based on my own tests, confirmed also by @NugentS, there are no real performance benefits formatting the 512e logical sectors of a hard drive to 4Kn, but as @HoneyBadger mentioned in this post, is preferred to avoid the read-modify-write cycle, if your disk can be formatted to 4Kn logical sectors. Both 512e and 4Kn logical sector hard drives can co-exist in a pool vdev.
Known Issues
Some users get this warning message, after Bluefin upgrade:
Code:
Disk(s): sda ... are formatted with Data Integrity Feature (DIF) which is unsupported.
Another case is disks are not being seen by pool, yet the pool reports no errors. You might deal with one or a combination of both issues listed below. Bluefin's newer kernel/ZFS version makes available these feature checks and throws a warning, which explains why you did not see it before.
T10 Protection Information
T10-PI is an extension of the existing T10 SCSI Block Commands specification, covering communication between SCSI controllers and storage devices, Protection Information (PI) adds an extra 8 bytes of information to the 512-byte sectors typical of enterprise hard drives.
Example of T10-PI Type 1 protected disk:
Example of T10-PI Type 2 protected disk:
Example of non T10-PI protected disk:
Code:
# sg_readcap -l /dev/sda Read Capacity results: Protection: prot_en=1, p_type=0, p_i_exponent=0 [type 1 protection]
Example of T10-PI Type 2 protected disk:
Code:
# sg_readcap -l /dev/sda Read Capacity results: Protection: prot_en=1, p_type=1, p_i_exponent=0 [type 2 protection]
Example of non T10-PI protected disk:
Code:
# sg_scan -i /dev/sda
/dev/sda: scsi0 channel=0 id=0 lun=0 [em]
ATA HUH728080ALE601 0003 [rmb=0 cmdq=1 pqual=0 pdev=0x0]
Branded Disks
Linux does not support 520-byte sectors, unless the drive is formatted with DIF and installed into a DIF-capable HBA. This explains why OEM branded HDDs or SSDs from major storage vendors are "enhanced" with 520-byte sectors. In rare cases, you might also find disk sectors extended to 528-byte, with custom proprietary firmware.
Example of 520-byte sectors disk:
Code:
# sg_format -v /dev/sda
NETAPP X287_S15K5288A15 3P01 peripheral_type: disk [0x0]
PROTECT=0
Unit serial number: 00028BF3
LU name: 5000cca254d9cd21
mode sense(10) cdb: [5a 00 01 00 00 00 00 00 fc 00]
Mode Sense (block descriptor) data, prior to changes:
Number of blocks=0 [0x0]
Block size=520 [0x208]
Formatting Procedure
- User is responsible for any destructive actions, if you're not sure of something, ask before you execute any formatting commands
- All commands must be executed as
rootuser - For related inquiries or questions, please post the
sg_formatoutput, using[CODE]command[/CODE]
Disk is Part of a Pool
In UI, go to Storage > Pool Name > Manage Devices and take the disk offline:
The safest way is to run the
Initial Troubleshooting
Run the
T10 Protection Information
To remove the T10-PI protection, run the
Branded Disks
To format a 520-byte sectors disk, run the
Advanced Format Disks
To format a 512e to 4Kn sectors disk, run the
T10 Protection Information and Branded Disks
If
No, you cannot combine all flags into one command,
Disk Replacement
Back in UI, forcibly Replace the disk:
The replacement will trigger an automatic resilvering, the process will take several hours.
In UI, go to Storage > Pool Name > Manage Devices and take the disk offline:
The safest way is to run the
sg_format commands directly from Scale server console, but you can also use the tmux procedure detailed below, to avoid losing connectivity to server.Start a
Print the current format status:
To attach to
To detach from current session, press
tmux session and execute your format command, Enter is part of command, do not remove it:Code:
# tmux new-session -ds format # tmux send-keys -t format 'time sg_format -v [your format flag] /dev/sda' Enter
Print the current format status:
Code:
# tmux capture-pane -pt format
To attach to
format session, run:Code:
# tmux attach -t format
To detach from current session, press
control+b then d.Initial Troubleshooting
Run the
sg_format command without formatting options, to troubleshoot the T10-PI and 520-byte sectors details:Code:
# sg_format -v /dev/sda
T10 Protection Information
To remove the T10-PI protection, run the
sg_format command with following formatting options, time measures the command execution time:Code:
# time sg_format -v -F /dev/sda
Branded Disks
To format a 520-byte sectors disk, run the
sg_format command with following formatting options:Code:
# time sg_format -v -F -s 512 /dev/sda
Advanced Format Disks
To format a 512e to 4Kn sectors disk, run the
sg_format command with following formatting options:Code:
# time sg_format -v -F -s 4096 /dev/sda
T10 Protection Information and Branded Disks
If
sg_format reports you have both 520-byte sectors and the T10-PI protection enabled, you need to disable first the T10-PI protection with sg_format -v -F, then run sg_format -v -F -s 512 to format the sectors. Example where both issues are present:Code:
# sg_format -v /dev/sda
NETAPP X287_S15K5288A15 3P01 peripheral_type: disk [0x0]
PROTECT=2
Unit serial number: 00028BF3
LU name: 5000cca254d9cd21
mode sense(10) cdb: [5a 00 01 00 00 00 00 00 fc 00]
Mode Sense (block descriptor) data, prior to changes:
Number of blocks=0 [0x0]
Block size=520 [0x208]
# time sg_format -v -F /dev/sda
# time sg_format -v -F -s 512 /dev/sda
sg_format will prohibit this operation.Disk Replacement
Back in UI, forcibly Replace the disk:
The replacement will trigger an automatic resilvering, the process will take several hours.
HGST Ultrastar Firmware (for Advanced Users)
HGST uses proprietary firmware on their Ultrastar models and Western Digital provides upon request a proprietary tool used for performing low-level maintenance on compatible disk drives, such as conversion to 4K native sectoring. I created a resource with the software I used to format my HGST Helium hard drives from 512e to 4Kn logical sectors.
It is strongly recommended to contact Western Digital and provide your hard drive serial number, to make sure you obtain a compatible low-level maintenance tool. Using the wrong tool could permanently brick your device. The steps listed below will apply to any of their proprietary formatting tools, always check the documentation to make sure the Hugo related commands are still valid.
Since my disks are OEM, WD did not provided any support, so I had to find a solution by myself.
With the procedure listed below, I successfully performed a format on one disk at the time on all devices, directly on my Scale server.
Refer to included PDF file or run
➊ Once Hugo is installed as detailed into resource, take the disk offline into UI.
➋ Wipe the hard drive signatures, the process is instant:
➌ Start a
Using the
➍ Verify the 4Kn logical sectors with
➎ Back in UI, forcibly Replace the disk.
The replacement will trigger an automatic resilvering, the process will take several hours.
Refer to included PDF file or run
hugo format -h for all command flags, for some reason man hugo is missing some of them:Code:
USAGE:
format {-m <model number> ... |-s <serial number> ... |-g <device path>
... } [--danger-zone] [--simple-progress] [--hide-progress] [-p
<protection type>] [-c] [--fastformat] [--merge] [-n <number of
blocks>] [-b <block size>] [--no-sg] [--no-ad] [--no-mr]
[--no-serial] [-h]
Where:
-m <model number>, --model <model number> (accepted multiple times)
(OR required) Format all devices of specified model number
-- OR --
-s <serial number>, --serial <serial number> (accepted multiple times)
(OR required) Format device specified by serial number
-- OR --
-g <device path>, --target <device path> (accepted multiple times)
(OR required) Operate on targets with device handles specified by
this option. See README for more detail.
--danger-zone
Flag tells the application that you know you are going to destroy your
data with this command and will not prompt the user.
--simple-progress
Prevent the display of the progress bar screen, useful when running
commands from a script. (same as hide-progress flag)
--hide-progress
Prevent the display of the progress bar screen, useful when running
commands from a script.
-p <protection type>, --protection <protection type>
Specify a type of Protection Information (0,1,2,3)
-c, --media-compatibility-check
Perform media compatibility check
--fastformat
Set Fast Format
--merge
Merge G-List and P-List
-n <number of blocks>, --numblocks <number of blocks>
Specify number of blocks to Format. Default: Current size. Specifying
'max' will format to maximum number of blocks supported by the
device.
-b <block size>, --blocksize <block size>
Format the device to a specified Block size (512, 4096). Additional
special case sizes for SAS drives only include: 520, 528, 4112, 4160,
and 4224. Special cases may not be supported by your specific
firmware
--no-sg
Do not use the SG driver
--no-ad
Do not use the AD driver
--no-mr
Do not use the MR driver
--no-serial
Do not use the Serial driver
-h, --help
Displays usage information and exits.
➊ Once Hugo is installed as detailed into resource, take the disk offline into UI.
➋ Wipe the hard drive signatures, the process is instant:
Code:
# wipefs -af /dev/sda
➌ Start a
tmux session and format the hard drive with Hugo, the process takes many hours:Code:
# tmux new-session -ds sda # tmux send-keys -t sda 'hugo format --danger-zone --simple-progress -b 4096 -p 0 -g /dev/sda' Enter # tmux capture-pane -pt sda
Using the
--fastformat option will result in corrupted sectors which make surface little after the disk is back online. The safest way is to proceed with the standard long format. Once the format finished, kill the tmux session:Code:
# tmux kill-session -t sda
➍ Verify the 4Kn logical sectors with
fdisk:Code:
# fdisk -l /dev/sda Disk /dev/sdc: 7.28 TiB, 8001563222016 bytes, 1953506646 sectors Disk model: HUH728080ALE601 Units: sectors of 1 * 4096 = 4096 bytes Sector size (logical/physical): 4096 bytes / 4096 bytes I/O size (minimum/optimal): 4096 bytes / 4096 bytes
➎ Back in UI, forcibly Replace the disk.
The replacement will trigger an automatic resilvering, the process will take several hours.
Firmware Update
Newer HGST firmware might be available at HDDGuru, search on page for your disk model number (i.e.
HUH728080ALE601). Files are zipped with an incompatible Linux format, so I created a simple Python script allowing you to download and unzip the firmware on your TrueNAS server.Paste the following code into your terminal and press Enter, it will create the
Execute the following commands to download and extract the firmware (
/tmp/extract.py file:Code:
cat > /tmp/extract.py <<EOF
import urllib.request as request
import shutil
import sys
import zipfile
zip_file = sys.argv[1]
request_url = '/'.join(['https://files.hddguru.com/download/Firmware%20updates/Hitachi', zip_file])
print(' '.join(['Downloading', zip_file, '...']))
try:
with request.urlopen(request_url) as response, open(zip_file, 'wb') as file:
shutil.copyfileobj(response, file)
except:
print('Download error')
else:
print('OK')
print(' '.join(['Extracting', zip_file, '...']))
try:
with zipfile.ZipFile(zip_file, 'r') as zip_ref:
zip_ref.extractall('.')
except:
print('File extraction error')
else:
print('OK')
EOF
Execute the following commands to download and extract the firmware (
A4GNVD05.zip used as example for HUH728080ALE601 model):Code:
# cd /tmp # python extract.py A4GNVD05.zip Downloading A4GNVD05.zip ... OK Extracting A4GNVD05.zip ... OK # ls -lah A4GNVD05* -rw-r--r-- 1 root root 1.9M Dec 13 03:09 A4GNVD05.bin -rw-r--r-- 1 root root 1.1M Dec 13 03:09 A4GNVD05.zip
Last edited:
