Drives with error "Data Integrity Feature"

[sdooweloc]

TrueNAS-SCALE-22.12.0 virtualized within ProxMox

Good day all. I got a group of refurb SAS 14TB drives that I was hoping to add into TrueNAS as a larger pool and migrate my data to it. However I am getting errors when I try to create a pool of

Code:

sdn are formatted with Data Integrity Feature (DIF) which is unsupported.

I tried following @Daisuke guide for solving this problem however I am running into issues I cant figure out. https://www.truenas.com/community/threads/troubleshooting-disk-format-warnings-in-bluefin.106051/

When running the

Code:

time sg_format -vFf 0 -s 512 /dev/sdh

command I get the below output:

Code:

root@truenas[~]# time sg_format -vFf 0 -s 512 /dev/sdh
    IBM-ESXS  ST14000NM0288 E   ECH8   peripheral_type: disk [0x0]
      PROTECT=1
      << supports protection information>>
      Unit serial number: ZHZ129DT0000C8510JBN
      LU name: 5000c500a74fabcb
    mode sense(10) cdb: [5a 00 01 00 00 00 00 00 fc 00]
Mode Sense (block descriptor) data, prior to changes:
number of blocks will max out, set <<longlba>>
    mode sense(10) cdb: [5a 10 01 00 00 00 00 00 fc 00]
  <<< longlba flag set (64 bit lba) >>>
  Number of blocks=3394240512 [0xca500000]
  Block size=4096 [0x1000]
    mode select(10) cdb: [55 11 00 00 00 00 00 00 24 00]
mode select(10):
Descriptor format, current; Sense key: Illegal Request
Additional sense: Invalid field in parameter list
  Descriptor type: Sense key specific: Field pointer:
        Error in Data parameters: byte 21 bit 7
  Descriptor type: Field replaceable unit code: 0x11
  Descriptor type: Vendor specific [0x80]
    00 00 05 26 00 11 ff ff ff ff ff ff 00 00
Try MODE SELECT again with SP=0 this time
    mode select(10) cdb: [55 10 00 00 00 00 00 00 24 00]
mode select(10):
Descriptor format, current; Sense key: Illegal Request
Additional sense: Invalid field in parameter list
  Descriptor type: Sense key specific: Field pointer:
        Error in Data parameters: byte 21 bit 7
  Descriptor type: Field replaceable unit code: 0x11
  Descriptor type: Vendor specific [0x80]
    00 00 05 26 00 11 ff ff ff ff ff ff 00 00
MODE SELECT command: Illegal request, type: sense key, apart from Invalid opcode
sg_format -vFf 0 -s 512 /dev/sdh  0.00s user 0.00s system 5% cpu 0.045 total

Similar results with

Code:

time sg_format -vFs 520 /dev/sdh

Code:

root@truenas[~]# time sg_format -vFs 520 /dev/sdh
    IBM-ESXS  ST14000NM0288 E   ECH8   peripheral_type: disk [0x0]
      PROTECT=1
      << supports protection information>>
      Unit serial number: ZHZ129DT0000C8510JBN
      LU name: 5000c500a74fabcb
    mode sense(10) cdb: [5a 00 01 00 00 00 00 00 fc 00]
Mode Sense (block descriptor) data, prior to changes:
number of blocks will max out, set <<longlba>>
    mode sense(10) cdb: [5a 10 01 00 00 00 00 00 fc 00]
  <<< longlba flag set (64 bit lba) >>>
  Number of blocks=3394240512 [0xca500000]
  Block size=4096 [0x1000]
    mode select(10) cdb: [55 11 00 00 00 00 00 00 24 00]
mode select(10):
Descriptor format, current; Sense key: Illegal Request
Additional sense: Invalid field in parameter list
  Descriptor type: Sense key specific: Field pointer:
        Error in Data parameters: byte 21 bit 7
  Descriptor type: Field replaceable unit code: 0x11
  Descriptor type: Vendor specific [0x80]
    00 00 05 26 00 11 ff ff ff ff ff ff 00 00
Try MODE SELECT again with SP=0 this time
    mode select(10) cdb: [55 10 00 00 00 00 00 00 24 00]
mode select(10):
Descriptor format, current; Sense key: Illegal Request
Additional sense: Invalid field in parameter list
  Descriptor type: Sense key specific: Field pointer:
        Error in Data parameters: byte 21 bit 7
  Descriptor type: Field replaceable unit code: 0x11
  Descriptor type: Vendor specific [0x80]
    00 00 05 26 00 11 ff ff ff ff ff ff 00 00
MODE SELECT command: Illegal request, type: sense key, apart from Invalid opcode
sg_format -vFs 520 /dev/sdh  0.00s user 0.00s system 3% cpu 0.068 total

On page 4 of the above linked thread I saw someone running into the same issue with suggestions to run the below commands but as you can see from the provided outputs these commands also did not work.

Code:

root@truenas[~]# openSeaChest_FormatUnit -d /dev/sdh --stopOnListError --protectionType 0 --formatUnit 512 --poll
==========================================================================================
 openSeaChest_Format - openSeaChest drive utilities - NVMe Enabled
 Copyright (c) 2014-2022 Seagate Technology LLC and/or its Affiliates, All Rights Reserved
 openSeaChest_Format Version: 2.2.1-2_2_1 X86_64
 Build Date: Dec 13 2022
 Today: Thu Jan  5 15:34:58 2023    User: root
==========================================================================================

/dev/sg8 - ST14000NM0288 E - ZHZ129DT0000C8510JBN - SCSI
Format Unit

You must add the flag:
"this-will-erase-data"
to the command line arguments to run a format unit.

e.g.: openSeaChest_Format -d /dev/sg? --formatUnit current --confirm this-will-erase-data

Code:

root@truenas[~]# openSeaChest_FormatUnit -d /dev/sdh --stopOnListError --protectionType 0 --formatUnit 512 --poll --confirm this-will-erase-data
==========================================================================================
 openSeaChest_Format - openSeaChest drive utilities - NVMe Enabled
 Copyright (c) 2014-2022 Seagate Technology LLC and/or its Affiliates, All Rights Reserved
 openSeaChest_Format Version: 2.2.1-2_2_1 X86_64
 Build Date: Dec 13 2022
 Today: Thu Jan  5 15:35:15 2023    User: root
==========================================================================================

/dev/sg8 - ST14000NM0288 E - ZHZ129DT0000C8510JBN - SCSI
Format Unit
Format Unit Failed!

Code:

root@truenas[~]# openSeaChest_FormatUnit -d /dev/sdh --protectionType 0 --formatUnit 512 --poll --confirm this-will-erase-data
==========================================================================================
 openSeaChest_Format - openSeaChest drive utilities - NVMe Enabled
 Copyright (c) 2014-2022 Seagate Technology LLC and/or its Affiliates, All Rights Reserved
 openSeaChest_Format Version: 2.2.1-2_2_1 X86_64
 Build Date: Dec 13 2022
 Today: Thu Jan  5 15:35:40 2023    User: root
==========================================================================================

/dev/sg8 - ST14000NM0288 E - ZHZ129DT0000C8510JBN - SCSI
Format Unit
Format Unit Failed!

Code:

root@truenas[~]# openSeaChest_Erase -d /dev/sdh --sanitize info
==========================================================================================
 openSeaChest_Erase - openSeaChest drive utilities - NVMe Enabled
 Copyright (c) 2014-2022 Seagate Technology LLC and/or its Affiliates, All Rights Reserved
 openSeaChest_Erase Version: 3.0.3-2_2_1 X86_64
 Build Date: Dec 13 2022
 Today: Thu Jan  5 15:36:06 2023    User: root
==========================================================================================

/dev/sg8 - ST14000NM0288 E - ZHZ129DT0000C8510JBN - SCSI

Sanitize
    The following sanitize commands are supported:
        Crypto Erase    --sanitize cryptoerase
        Overwrite    --sanitize overwrite

I have tried with the drive plugged into my JBOD which is in turn plugged into an IT Mode flashed Dell H200e which is plugged into my Dell R720 II and passed through directly to my TrueNAS VM.
I have also tried plugging directly into the R720's front bays with the exact same results. (The R720's backplane is also in IT mode and passed fully to the TrueNAS VM)

Please if I have ommitted needed information let me know and I will work to remedy the missing information but I am honestly at a loss of what to do to get these drives to work for what I want. Any and all help is appreciated.

 

[Daisuke]

The Dell R720 is perfect for testing this issue, what PERC you have installed? I have the H710 mini, flashed to IT mode. Looking forward to see what other experienced users say.

 

[artlessknave]

I have tried with the drive plugged into my JBOD which is in turn plugged into an IT Mode flashed Dell H200e

what does "JBOD" indicate in this context? JBOD isn't something you can "plug" into, it just means a bunch of disks connected.

 

[sdooweloc]

The Dell R720 is perfect for testing this issue, what PERC you have installed? I have the H710 mini, flashed to IT mode. Looking forward to see what other experienced users say.

I have an H710 in IT mode in mine as well iirc with the whole PERC passed to my TrueNAS VM.
This is how it shows up in Proxmox's terminal LSPCI

Code:

03:00.0 Serial Attached SCSI controller: Broadcom / LSI SAS2308 PCI-Express Fusion-MPT SAS-2 (rev 05)

what does "JBOD" indicate in this context? JBOD isn't something you can "plug" into, it just means a bunch of disks connected.

Sorry I have been shorthanding in conversations with friends. I have a EMC KTN-STL3 drive shelf.

 

[artlessknave]

EMC KTN-STL3

ahh. that makes much more sense. things like this are why the forum rules say to list your hardware. "R720" gives us some idea but these servers can come in many different configurations, and if you are using external expansion that is also very important to know.
unfortunately, other than getting the hardware info into your thread, I have no ideas on your actual issue.

 

[sdooweloc]

Okay so I have a test drive going now with some information that a buddy from Discord found me.
He managed to track down a blogger who had a similar issue with being unable to use Protection type 2 devices for storage (not in truenas but should be similar enough) http://talesinit.blogspot.com/2015/11/formatted-with-type-2-protection-huh.html
Basically boils down to running

Code:

sg_format --format --fmtpinfo=0 /dev/sd*

I have 1 drive running now but being 14TB it is going to take a while. Will update here when it finishes as this should clear out the protection and make the drive useable.
@Daisuke @BulgarianHealer

 

[Daisuke]

Basically boils down to running

Your command:

Code:

# sg_format --format --fmtpinfo=0 /dev/sda

Is the same as:

Code:

# sg_format -v -F /dev/sda

-v is for verbose, you get detailed info, see man sg_format. You can run sg_format -v -F /dev/sda, as default value for fmtpinfo is 0. You are telling us that running the additional -s 512 512-byte sector format complains with that error?

Edit: I looked into manual and is stated: Some SCSI disks have 512 byte logical blocks by default and allow an alternate logical block size of 4096 bytes. If the given size in unacceptable to the disk, most likely an "Invalid field in parameter list" message will appear in sense data. That explains why you experienced the issue.

I've split the commands into guide, this should address all issues. Thank you for improving the guide with your shared experience.

 

[BulgarianHealer]

-v is for verbose, you get detailed info, see man sg_format. You can even run sg_format -vF /dev/sda, as default value for fmtpinfo is 0. You are telling us that running the additional -s 512 512-byte sector format complains with that error?

I could format it with -s 512, but I keep it 4k(its already formated). Today i will put the HDD to resilver later, and while I m at work I was able to run the same command on two more HDDs(so 3 hdd formatting on raidz3, not a problem for now) on the main server where the disk originally belongs, I don't know why "sg_format -vFf 0 /dev/sda" doesn't work, strange. For me it's gonna be fun to resilver so many TB's of data.

 

[Daisuke]

@BulgarianHealer just to be clear, you ran this exact command and it failed with an error?

Code:

sg_format -v -F -f 0 /dev/sda

Please provide the disk info with this command:

Code:

sg_format /dev/sda

What happens when you run the flags separate, protection is set to 0 by default:

Code:

sg_format -v -F /dev/sda

 

[sdooweloc]

Your command:

Code:

# sg_format --format --fmtpinfo=0 /dev/sda

Is the same as:

Code:

# sg_format -vFf 0 /dev/sda

-v is for verbose, you get detailed info, see man sg_format. You can even run sg_format -vF /dev/sda, as default value for fmtpinfo is 0. You are telling us that running the additional -s 512 512-byte sector format complains with that error?

Edit: I looked into manual and is stated: Some SCSI disks have 512 byte logical blocks by default and allow an alternate logical block size of 4096 bytes. If the given size in unacceptable to the disk, most likely an "Invalid field in parameter list" message will appear in sense data. That explains why you experienced the issue.

I've split the commands into guide, this should address all issues. Thank you for improving the guide with your shared experience.

Ah that definitely explains what I was experiencing. I am glad to have the additional explanation. Thank you so much for your guide otherwise I would have been starting from 0 and likely would not have had a good time.

 

[Daisuke]

@BulgarianHealer @sdooweloc I can confirm sg_format does not supports combined flags. The only command that has the same design is sed.

Code:

# sg_format -vF /dev/sda
    ATA       HUH728080ALE601   0003   peripheral_type: disk [0x0]
    mode sense(10) cdb: [5a 00 01 00 00 00 00 00 fc 00]
Mode Sense (block descriptor) data, prior to changes:
  Number of blocks=0 [0x0]
  Block size=512 [0x200]
Format unit:
Fixed format, current; Sense key: Illegal Request
Additional sense: Invalid field in cdb
Format unit command: Illegal request, type: sense key, apart from Invalid opcode

Example of combined flags for other command:

Code:

# wipefs -af /dev/sda

I updated the guide with the correct commands.

 

[BulgarianHealer]

@BulgarianHealer just to be clear, you ran this exact command and it failed with an error?

Yes

The normal command behaviour in Linux is to combine the flags together. Please provide the disk info with this command:

Maybe on Sunday when it's ready with the resilvering(22 hours 49 minutes 21 seconds remaining, its going 0.5/1.2 GiB/s read) i will try

 

[Daisuke]

Maybe on Sunday when it's ready with the resilvering(22 hours 49 minutes 21 seconds remaining, its going 0.5/1.2 GiB/s read) i will try

Thanks, appreciated. Please post the command outputs listed below in this thread.

Print disk details, it will show the block size and T10 protection (I need this info):

Code:

# sg_format /dev/sda

Remove the protection from disk:

Code:

# time sg_format -v -F /dev/sda

Here it is my sg_format output:

Code:

# sg_format /dev/sda
    ATA       HUH728080ALE601   0003   peripheral_type: disk [0x0]
      LU name: 5000cca254d9cd21
Mode Sense (block descriptor) data, prior to changes:
  Number of blocks=0 [0x0]
  Block size=512 [0x200]
Read Capacity (16) results:
   Protection: prot_en=0, p_type=0, p_i_exponent=0
   Logical block provisioning: lbpme=0, lbprz=0
   Logical blocks per physical block exponent=3
   Lowest aligned logical block address=0
   Number of logical blocks=15628053168
   Logical block size=512 bytes

Is important to note that the correct block size for sectors is 512-byte. The -s 512 format will set the sector block size, nothing to do with the 4096 physical size:

Code:

# fdisk -l /dev/sda
Disk /dev/sda: 7.28 TiB, 8001563222016 bytes, 15628053168 sectors
Disk model: HUH728080ALE601
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: 976A87DC-A64F-43F9-97E3-4A02BA95231F

Device       Start         End     Sectors  Size Type
/dev/sda1      128     4194304     4194177    2G Linux swap
/dev/sda2  4194432 15628053134 15623858703  7.3T Solaris /usr & Apple ZFS

 

[sdooweloc]

How vital is the sector block size being 512?
I am not exactly thrilled about the thought of doing 2 formats but if it is a must then I guess I must. My first drive completed and this is my current output from it.

Code:

root@truenas[~]# sg_format /dev/sdh
    IBM-ESXS  ST14000NM0288 E   ECH8   peripheral_type: disk [0x0]
      << supports protection information>>
      Unit serial number: ZHZ129DT0000C8510JBN
      LU name: 5000c500a74fabcb
Mode Sense (block descriptor) data, prior to changes:
  Number of blocks=3394240512 [0xca500000]
  Block size=4096 [0x1000]
Read Capacity (10) results:
   Number of logical blocks=3394240512
   Logical block size=4096 bytes
No changes made. To format use '--format'. To resize use '--resize'


root@truenas[~]# fdisk -l /dev/sdh
Disk /dev/sdh: 12.64 TiB, 13902809137152 bytes, 3394240512 sectors
Disk model: ST14000NM0288 E 
Units: sectors of 1 * 4096 = 4096 bytes
Sector size (logical/physical): 4096 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes

 

[Daisuke]

How vital is the sector block size being 512?

You want to be aligned with the standards applied to all disks you purchase. You definitely do not want to end-up with a mix of 512 and 4096-byte sectors in your server. I'm sure other smart people who know more about this, will chime in.

I am not exactly thrilled about the thought of doing 2 formats

Technically, you can run this command and see if it will fix both problems with a single format:

Code:

# time sg_format -v -F -s 512 /dev/sda

From the manual, in case you deal with the "Invalid field in parameter list" message:

The default value is whatever is currently reported by the block descriptor in a MODE SENSE command. If the block size given by this option is different from the current value then a MODE SELECT command is used to change it prior to the FORMAT UNIT command being started as recommended in the SBC standards). Some SCSI disks have 512 byte logical blocks by default and allow an alternate logical block size of 4096 bytes (your case @sdooweloc). If the given size in unacceptable to the disk, most likely an "Invalid field in parameter list" message will appear in sense data (requires the use of '-v' to decode sense data).

Another note from manual:

Note that formatting a disk to add or remove protection information is not regarded as a change to its logical block size so this option should not be used.

In other words, if you combine the two command options like above, you might get warnings. However, when I look how sg_format works behind the scenes, it will set the sector size to 512-byte and also remove the protection for you (default option). Like explained into manual, your disk might just report 4096-byte but it is actually a 512-byte, like the rest of disks planet.

Can you try the combined command on a disk and report the results? If you get warnings, you have no choice but to run first the sg_format -v -F command, then the sg_format -v -F -s 512 command. In your case, you already did the first step, time to change the sectors? Looking forward to hear from you.

Edit: your disk does not show the protection information, this is the related line (look at my command output):

Code:

Protection: prot_en=0, p_type=0, p_i_exponent=0

Do you have another protected disk with 4096-byte sectors, so you could test the combined command?

 

[sdooweloc]

Unfortunately I cannot run the combined command on a protected drive because at the moment the remaining 6 have all been started and I dont want to start a format part way through unless I am grossly overestimating the risk there.

I will however run the 512 format on the completed unprotected disk.

Edit: your disk does not show the protection information, this is the related line (look at my command output):

Code:

Protection: prot_en=0, p_type=0, p_i_exponent=0

I did figure this was the relevant line but the confirmation is very relaxing.

 

[Daisuke]

the remaining 6 have all been started

All these disks are part of an active pool with data stored on it? You did not offline one disk at the time? If that's the case, your data is gone.

 

[sdooweloc]

All these disks are part of an active pool with data stored on it? You did not offline one disk at the time? If that's the case, your data is gone.

no no no these are all brand new disks not yet in pools.

 

[sdooweloc]

You want to be aligned with the standards applied to all disks you purchase. You definitely do not want to end-up with a mix of 512 and 4096-byte sectors in your server. I'm sure other smart people who know more about this, will chime in.

Technically, you can run this command and see if it will fix both problems with a single format:

Code:

# time sg_format -v -F -s 512 /dev/sda

From the manual, in case you deal with the "Invalid field in parameter list" message:

Another note from manual:

In other words, if you combine the two command options like above, you might get warnings. However, when I look how sg_format works behind the scenes, it will set the sector size to 512-byte and also remove the protection for you (default option). Like explained into manual, your disk might just report 4096-byte but it is actually a 512-byte, like the rest of disks planet.

Can you try the combined command on a disk and report the results? If you get warnings, you have no choice but to run first the sg_format -v -F command, then the sg_format -v -F -s 512 command. In your case, you already did the first step, time to change the sectors? Looking forward to hear from you.

Edit: your disk does not show the protection information, this is the related line (look at my command output):

Code:

Protection: prot_en=0, p_type=0, p_i_exponent=0

Do you have another protected disk with 4096-byte sectors, so you could test the combined command?

So trying to run the 512 sizing on my drive that I have already removed the protection on I get the below. This seems to be the issue that you pulled from the man pages but I am not sure what to really do with it from there.

Code:

root@truenas[~]# time sg_format -v -F -s 512 /dev/sdh
    IBM-ESXS  ST14000NM0288 E   ECH8   peripheral_type: disk [0x0]
      PROTECT=1
      << supports protection information>>
      Unit serial number: ZHZ129DT0000C8510JBN
      LU name: 5000c500a74fabcb
    mode sense(10) cdb: [5a 00 01 00 00 00 00 00 fc 00]
Mode Sense (block descriptor) data, prior to changes:
number of blocks will max out, set <<longlba>>
    mode sense(10) cdb: [5a 10 01 00 00 00 00 00 fc 00]
  <<< longlba flag set (64 bit lba) >>>
  Number of blocks=3394240512 [0xca500000]
  Block size=4096 [0x1000]
    mode select(10) cdb: [55 11 00 00 00 00 00 00 24 00]
mode select(10):
Descriptor format, current; Sense key: Illegal Request
Additional sense: Invalid field in parameter list
  Descriptor type: Sense key specific: Field pointer:
        Error in Data parameters: byte 21 bit 7
  Descriptor type: Field replaceable unit code: 0x11
  Descriptor type: Vendor specific [0x80]
    00 00 05 26 00 11 ff ff ff ff ff ff 00 00 
Try MODE SELECT again with SP=0 this time
    mode select(10) cdb: [55 10 00 00 00 00 00 00 24 00]
mode select(10):
Descriptor format, current; Sense key: Illegal Request
Additional sense: Invalid field in parameter list
  Descriptor type: Sense key specific: Field pointer:
        Error in Data parameters: byte 21 bit 7
  Descriptor type: Field replaceable unit code: 0x11
  Descriptor type: Vendor specific [0x80]
    00 00 05 26 00 11 ff ff ff ff ff ff 00 00 
MODE SELECT command: Illegal request, type: sense key, apart from Invalid opcode
sg_format -v -F -s 512 /dev/sdh  0.00s user 0.00s system 0% cpu 0.266 total

 

[Daisuke]

This seems to be the issue that you pulled from the man

The thing I don’t understand is why your disk was protected, without initially showing the protection info? If sg_format would see the protection set to 0, it would not complain when you try to set the sectors to a different format.

Edit: never mind, it shows you have protection type 1 enabled on your disk. You already ran the sg_format -v -F on it? This will set the protection type to 0.