Drives with error "Data Integrity Feature"

[sdooweloc]

The thing I don’t understand is why your disk was protected, without initially showing the protection info? If sg_format would see the protection set to 0, it would not complain when you try to set the sectors to a different format.

Edit: never mind, it shows you have protection type 1 enabled on your disk. You already ran the sg_format -v -F on it? This will set the protection type to 0.

I ran the

Code:

 sg_format --format --fmtpinfo=0

on this one.

 

[Daisuke]

Look at the top of your sg_format output, it says PROTECT=1, protection was not removed.

How is your disk connected to Scale server? I would not specify the format value and let sg_format do it by itself. The correct way to do it is sg_format -v -F. Or the long commands, verbose and format. But you need both, forget the fmtpinfo. sg_format have a special relation with the verbose mode, is not just extra noise printed on your terminal. Your disk is an IBM branded disk with a totally different sg_format output.

 

[sdooweloc]

Look at the top of your sg_format output, it says PROTECT=1, protection was not removed.

How is your disk connected to Scale server? I would not specify the format value and let sg_format do it by itself. The correct way to do it is sg_format -v -F. Or the long commands, verbose and format. But you need both, forget the fmtpinfo. Your disk is an IBM branded disk with a totally different sg_format output.

This one is the one that is plugged directly into the front of my r720 whos backplane HBA I have in IT mode and have it passed through to the Scale VM.
I can rerun with the

Code:

sg_format -v -F

just means I wont be able to really test anything else on this drive till late Sat or Sun because it takes like 17+ hours to format and I wont be home most of tomorrow

 

[Daisuke]

From a quick search, IBM-ESXS are IBM enclosure specific drives.

 

[sdooweloc]

Well sg_format still says protect=1 but all the other commands display no protection and trueNAS supports 4k blocks and I was able to create a pool so I am counting this as a success.

 

[Daisuke]

Well sg_format still says protect=1

That means your format did not worked.

 

[sdooweloc]

Well it lets me add the drives to a pool now so the format maybe changed it from protect 2 to protect 1 which somehow isnt a problem for TrueNAS. Idk.

That means your format did not worked.

 

[BulgarianHealer]

Print disk details, it will show the block size and T10 protection (I need this info):
Code:
# sg_format /dev/sda

Output:

Code:

root@truenas[~]# sg_format /dev/sdj
    IBM-D050  ST6000NM0014      BC7B   peripheral_type: disk [0x0]
      << supports protection information>>
      Unit serial number: S4D092V10000K6030MA2
      LU name: 5000c5008e4eff7b
Mode Sense (block descriptor) data, prior to changes:
  Number of blocks=1465130646 [0x57541e96]
  Block size=4096 [0x1000]
Read Capacity (10) results:
   Number of logical blocks=1465130646
   Logical block size=4096 bytes
No changes made. To format use '--format'. To resize use '--resize'

Remove the protection from disk:
Code:
# time sg_format -v -F /dev/sda

Output:

Code:

root@truenas[~]# time sg_format -v -F /dev/sdj
    IBM-D050  ST6000NM0014      BC7B   peripheral_type: disk [0x0]
      PROTECT=1
      << supports protection information>>
      Unit serial number: S4D092V10000K6030MA2
      LU name: 5000c5008e4eff7b
    mode sense(10) cdb: [5a 00 01 00 00 00 00 00 fc 00]
Mode Sense (block descriptor) data, prior to changes:
  Number of blocks=1465130646 [0x57541e96]
  Block size=4096 [0x1000]

A FORMAT UNIT will commence in 15 seconds
    ALL data on /dev/sdj will be DESTROYED
        Press control-C to abort

From 12 disks, 9 are 4k others are 512, so after i finish removing protection and resilvering will make the other 3 disks 4k size

 

[Daisuke]

will make the other 3 disks 4k size

You cannot change the logical sector size to 4096 with sg_format or any tool, unless the disk firmware supports that size. Let me know if you can.

 

[BulgarianHealer]

You cannot change the logical sector size to 4096 with sg_format or any tool, unless the disk firmware supports that size. Let me know if you can.

All disks have the same firmware, so it must support. i will give try with sg_format (https://www.truenas.com/community/t...tor-size-from-512-bytes-to-4096-bytes.103556/ and https://manpages.ubuntu.com/manpages/focal/en/man8/sg_format.8.html seems like maybe support).

 

[Daisuke]

I say this because I tried to format a 512-byte disk to 4096 and it failed. Please let me know.

 

[bods77]

Guess it's my turn now

I have runned the format commands and removed protection but I still get

Replacing Disk​

Error: [EFAULT] Disk: 'sdi' is incorrectly formatted with Data Integrity Feature (DIF).

When I try to replace the disk

sg_format /dev/sdi
HGST H7240AS60SUN4.0T A3A0 peripheral_type: disk [0x0]
<< supports protection information>>
Unit serial number: 001515E7E7TX PEG7E7TX
LU name: 5000cca0730d815c
Mode Sense (block descriptor) data, prior to changes:
<<< longlba flag set (64 bit lba) >>>
Number of blocks=7814037168 [0x1d1c0beb0]
Block size=512 [0x200]
Read Capacity (16) results:
Protection: prot_en=0, p_type=0, p_i_exponent=0
Logical block provisioning: lbpme=0, lbprz=0
Logical blocks per physical block exponent=0
Lowest aligned logical block address=0
Number of logical blocks=7814037168
Logical block size=512 bytes
No changes made. To format use '--format'. To resize use '--resize'

What am I doing wrong......

Setup is R720 with h710 in IT mode.

 

[bods77]

Guess it's my turn now

I have runned the format commands and removed protection but I still get



When I try to replace the disk



What am I doing wrong......

Setup is R720 with h710 in IT mode.

Well looks like all it needed was a reboot. It is currently resilvering.

 

[LucVL]

I have the same "Data Integrity Feature" issue with an IBM SAS disk when failing to create a pool under Truenas SCALE. Creating a pool with Truenas Core works fine. Thereafter, the pool is recognized ok by Scale, with a mere warning.

 

[averyfreeman]

Thanks to everyone who participated in this thread, it saved my bacon: I had 10 drives with prot_en=1, had no idea it was even an issue.

I had been using the drives with zfs on another distro, but repurposed them for a friend's restaurant, where I wanted them to have something more user-friendly and comprehensive, so TrueNAS Scale seems like a great fit.

Big picture question: Why is TrueNAS Scale concerned about DIF protection bits being set on the drives now? I'm sure I've used these same drives with Core a few years ago and it never complained - is this specific to TrueNAS, or newer versions of zfs? (feel free to reply with any reference I overlooked).

I suppose I should share my experience with sg_format -vF $DEV_PATH in case it helps someone. Overall it was very positive, but all I was doing was removing the prot_en=1 flag. Each took an excrutiating 7 1/2 hours per 4TB drive, but formatting all 10 in parallel had max 10-15 min deviation in period (aka =< %0.03), so must not be all that CPU or IO heavy.

I was learning it as I went, so I ended up doing it over 10 ssh sessions via wifi: It was pins and needles worrying my laptop would get booted from the network and kill the process right at 99% - recommend other people ignore my poor example and connect via ethernet. Thankfully, It all worked out.

According to sg_format -v, all that changed was this:

Code:

$ diff -u dev-sdj-start.txt dev-sdj-end.txt
--- dev-sdj-start.txt    2024-01-14 14:46:56.047681276 -0800
+++ dev-sdj-end.txt    2024-01-14 15:06:31.642292021 -0800
@@ -1,4 +1,4 @@
-sg_format -v /dev/sdj 
+sg_format -v /dev/sdj
     HGST      H7240AS60SUN4.0T  A2D2   peripheral_type: disk [0x0]
       PROTECT=1
       << supports protection information>>
@@ -13,7 +13,7 @@
   Block size=512 [0x200]
     read capacity(16) cdb: [9e 10 00 00 00 00 00 00 00 00 00 00 00 20 00 00]
 Read Capacity (16) results:
-   Protection: prot_en=1, p_type=0, p_i_exponent=0
+   Protection: prot_en=0, p_type=0, p_i_exponent=0
    Logical block provisioning: lbpme=0, lbprz=0
    Logical blocks per physical block exponent=0
    Lowest aligned logical block address=0

That was taken before and after the format. TrueNAS web UI let me make a pool after format, while before the process halted and exited due to DIF.

Here's a full output of sg_format -v on one of the drives after format:

Code:

# sg_format -v /dev/sda
    HGST      H7240AS60SUN4.0T  A1B2   peripheral_type: disk [0x0]
      PROTECT=1
      << supports protection information>>
      Unit serial number: 001342E4THXX        PBG4THXX
      LU name: 5000cca03b08b604
    mode sense(10) cdb: [5a 00 01 00 00 00 00 00 fc 00]
Mode Sense (block descriptor) data, prior to changes:
block count maxed out, set <<longlba>>
    mode sense(10) cdb: [5a 10 01 00 00 00 00 00 fc 00]
  <<< longlba flag set (64 bit lba) >>>
  Number of blocks=7814037168 [0x1d1c0beb0]
  Block size=512 [0x200]
    read capacity(16) cdb: [9e 10 00 00 00 00 00 00 00 00 00 00 00 20 00 00]
Read Capacity (16) results:
   Protection: prot_en=0, p_type=0, p_i_exponent=0
   Logical block provisioning: lbpme=0, lbprz=0
   Logical blocks per physical block exponent=0
   Lowest aligned logical block address=0
   Number of logical blocks=7814037168
   Logical block size=512 bytes
No changes made. To format use '--format'. To resize use '--resize

I wasn't really sure what I was looking for in the above, but the man page for sg_format helped narrow it down. There's more people's experiences on servethehome forums if someone's still struggling by the end of this thread.

Tangentially, I learned bash can iterate alphabetically in for loops, so that is awesome, e.g serial and lun printing one-liner.

Code:

# for d in {a..z}; do echo $d; sg_format -v /dev/sd$d; grep -E 'serial|lun'; echo ' '; done >> luns-and-serials.txt 

Aside: Anyone who's better at bash than I am know if the above could exit each sg_format invocation with & so a single script could be used to run command in parallel? e.g. ... sg_format -vF /dev/sd$d &; ... <next disk>?

Thanks again!

 

[fwright]

I am attaching a response to this because I am having a similar problem and need help.

My TrueNAS Server is a Dell R720 with the RAID Controller flashed to IT Mode There are 8 8TB HGST SAS Drives connected to it.
I installed TrueNAS Core 13.0-U6.1 and I created a single RaidZ2 ZFS pool

Running sg_format /dev/daX from the TrueNAS command shell reported 6 drives with protect = 1 and 2 with protect = 0

I have an additional 8TB HGST SAS Drive with protect = 0.
My plan is to do the following with the 6 disks

1. Remove disk 1 from the RaidZ2 pool and replace it with this disk
2. Run sg_format -vF on the removed disk to hopefully setting the protect status to 0 (using a separate computer)
3. Remove disk 2 from the RaidZ2 pool and replace it with the disk just formatted
4. Reboot the TrueNAS server
5. << repeat for all remaining disks with protect = 1>>

This seems inefficient but I have the impression that this is safer. (I have data on the pool and it would be inconvenient to delete it)

Two questions

1. Is this the best/safest way to do this?
2. Do I really need to use a separate computer to do the formatting?

My reason for all of this is because

1. I am getting the impression that I might have problems if I leave the disks with protect = 1
2. I want to change from TrueNAS Core to TrueNAS Scale.

(TrueNAS Scale shows a warning message on the pool when I upgraded from TrueNAS Core. so I reinstalled the server and reverted to TrueNAS Core but as I stated I want to use Scale)

I hope my question is clear and does not violate any rules.
Thank You,

 

[fwright]

One more thing. What exactly is the command that should do this removal of Protect = 1?

I tried sg_format -v -F /dev/daX seems to do nothing

Protect still = 1 when put back into TrueNAS

What am I doing wrong?

 

[fwright]

One more thing. What exactly is the command that should do this removal of Protect = 1?

I tried sg_format -v -F /dev/daX seems to do nothing

Protect still = 1 when put back into TrueNAS

What am I doing wrong?

Could it be that I need to do the format on a computer that has the RAID Controller flashed to IT Mode?

 

[fwright]

An update:

I noticed that the 8 disks do not always appear in the same order after I reboot.
The effect of that is that when I check a disk after the reboot the disk I am checking is not the sone I think I am checking because /dev/sdb is now /dev/sdc and the one that was /dev/sdc now might be /dev/sdg.

I decided to enable ssh and run the format of each disk each in a separate ssh session all at once.
After a 14 hour wait and a reboot I will check if all this worked.

Good Luck to me

 

[fwright]

Well it all worked so just a quick summary of my process for anyone else who is as inexperienced as me and needed a summarized set of instructions

1. Flash my Perc Raid Controller to IT Mode
2. Install TrueNAS Scale (on a separate tiny disk
3. Enable SSH
4. Launch an SSH Session (1 for each disk I plan to format)
5. In each session run the command "sudo sg_format --format --size=512 /dev/sdX (each of my disks is identified as /dev/sda to /dev/sdh)
6. Wait 14 hours -- at the end exit each ssh session, then reboot TrueNAS
7. Once the server reboots, launch an SSH Session and check for protect removal by running sudo sg_readcap /dev/sdX

Please note: look at the list of disks in the TrueNAS Web UI to determine the name of each of the disks because as I mentioned the order of the disk detection is not consistent. You might end up with your boot disk showing up in the middle of the list of disks or at the end or at the beginning

Hope this helps
Thanks