Troubleshooting disk format warnings in TrueNAS SCALE 7.4.5

[Daisuke]

I updated the guide with the correct commands, sg_format does not support combined flags.

 

[The_Real_Fayz]

I am hoping you might be able to help me. I was trying to get another app running, and was advised on the Truecharts Discord to update. After the update I got the error described here for all of the drives in my pool so I was not able to apply the fix. I have rolled back and I am able to get access to my pool again, but now I have no apps as apparently you cannot roll back per TC support. I am new to Truenas and Linux so I am sorry if I have any dumb questions. I have historically run Hyper-V but was convinced to give this a shot. I have sda-sdh and all are affected, but none of them appear to be 512 520 or 528, they look to be 4096.

Code:

# sg_format -v /dev/sdh
    SEAGATE   ST8000NM0065      K003   peripheral_type: disk [0x0]
      PROTECT=1
      << supports protection information>>
      Unit serial number: ZA1BPFWD0000C843FLA6
      LU name: 5000c500a64058fb
    mode sense(10) cdb: [5a 00 01 00 00 00 00 00 fc 00]
Mode Sense (block descriptor) data, prior to changes:
  Number of blocks=1953506646 [0x74702556]
  Block size=4096 [0x1000]
    read capacity(10) cdb: [25 00 00 00 00 00 00 00 00 00]
Read Capacity (10) results:
   Number of logical blocks=1953506646
   Logical block size=4096 bytes
No changes made. To format use '--format'. To resize use '--resize'

 

[Daisuke]

@The_Real_Fayz you have one issue:

• PROTECT=1, protection needs to be set to 0

Follow the guide and let me know if you have any questions.

 

[Alphagocc]

I updated from Angelfish recently. Yesterday I bought a new HDD to replace my failed drive. And then I met the DIF warnings so I followed the guide in the forum. Today I checked other drives in my zpool, but found some of them have PROTECT=1 and some of them have PROTECT=0. Strangely, Bluefin doesn't complain about these things. Notice that turning off protection will destory all the data on the drive and takes very loooooong time, I'm wondering if I need to format drives with protection?

 

[Daisuke]

The guide tells you what to do, if up to you if you want to keep mixed formats in your pool. I personally don't.

 

[The_Real_Fayz]

@Alphagocc I had a similar issue, and after a lot of research, a lot of help from Daisuke, and some help from some Seagate staff I have discovered that that PROTECT=O AND PROTECT=1 do not relate directly to a protection mode, but rather whether the drive supports DIF or not. I had to use openSeaChest in my case to format my drives and turn it off. Try # openSeaChest_Info -d /dev/sg0 -i. If it works it should show Features Supported as shown below. as Protection Type 0 is the lack of protection it is not going to be listed, but if anything is enabled it will show as Protection Type 1 [Enabled] for instance.

Off

Code:

Features Supported:
                Protection Type 1
                Protection Type 2
                Application Client Logging
                Self Test
                Automatic Write Reassignment [Enabled]
                Automatic Read Reassignment [Enabled]
                EPC
                Informational Exceptions [Mode 6]
                Translate Address
                Rebuild Assist
                Seagate In Drive Diagnostics (IDD)
                Format Unit
                Sanitize

On

Code:

Features Supported:
                Protection Type 1
                Protection Type 2 [Enabled]
                Application Client Logging
                Self Test
                Automatic Write Reassignment [Enabled]
                Automatic Read Reassignment [Enabled]
                EPC
                Informational Exceptions [Mode 6]
                Translate Address
                Rebuild Assist
                Seagate In Drive Diagnostics (IDD)
                Format Unit
                Sanitize

 

[littleNewton]

I am using two Dell OEC 8TB HDDs which were produced by TOSHIBA.

Code:

 ⚡ root@epyc-truenas  ~  sg_format /dev/sdq
    TOSHIBA   MG06SCA800EY      EH06   peripheral_type: disk [0x0]
      << supports protection information>>
      Unit serial number: 1990A0TXF1GF
      LU name: 500003992823697d
Mode Sense (block descriptor) data, prior to changes:
  <<< longlba flag set (64 bit lba) >>>
  Number of blocks=15628053168 [0x3a3812ab0]
  Block size=512 [0x200]
Read Capacity (16) results:
   Protection: prot_en=1, p_type=1, p_i_exponent=0
   Logical block provisioning: lbpme=0, lbprz=0
   Logical blocks per physical block exponent=3
   Lowest aligned logical block address=0
   Number of logical blocks=15628053168
   Logical block size=512 bytes
No changes made. To format use '--format'. To resize use '--resize'
 ⚡ root@epyc-truenas  ~  sg_format /dev/sdr
    TOSHIBA   MG06SCA800EY      EH06   peripheral_type: disk [0x0]
      << supports protection information>>
      Unit serial number: 1990A06WF1GF
      LU name: 500003992822e315
Mode Sense (block descriptor) data, prior to changes:
  <<< longlba flag set (64 bit lba) >>>
  Number of blocks=15628053168 [0x3a3812ab0]
  Block size=512 [0x200]
Read Capacity (16) results:
   Protection: prot_en=1, p_type=1, p_i_exponent=0
   Logical block provisioning: lbpme=0, lbprz=0
   Logical blocks per physical block exponent=3
   Lowest aligned logical block address=0
   Number of logical blocks=15628053168
   Logical block size=512 bytes
No changes made. To format use '--format'. To resize use '--resize'

 

[littleNewton]

I am re-formatting those two HDDs with sg_format tools. This progress will take a very long time especially for a big one.

I will report everythings I find about DIF in this forum.

 

[Daisuke]

I am re-formatting those two HDDs with sg_format tools.

If you run the sg_format -v /dev/sdq, should tell you have PROTECT=2 enabled. Best way to check for the protection type is with the sg_readcap command listed into troubleshooting.

 

[radier]

Hello,

A similar issue has haunted me with DIF type 2.

I have Dell Poweredge R740xd with PERC H740P mini in eHBA mode. This server was used a few years ago with RHEL and Perc in RAID mode.
I am trying to repurpose it to TrueNAS Scale with the intended use for NAS and VMs.

So after changing mode to eHBA my disks are visible for Truenas. Installation went fine but as soon I logged into WebUI I saw this message:

Code:

Disk(s): sdc, sdj, sdk, sdm, sdo, sdq, sdp, sda, sdb, sdd, sde, sdf, sdg, sdi, sdh are formatted with Data Integrity Feature (DIF) which is unsupported.

I cannot create any pool. I am pretty sure my block size is 512. I have installed Windows just to test a few things, wipe the disks, etc.
My disk are Dell branded Exos 7EB, model ST8000NM0185 SAS 12Gbps, 8TB 512e, RPM 7.2K.

I have used time sg_format -v -F /dev/sda on one of my drives but I have encountered a few errors during this process, Ctrl+C, and now I killed one of my 16 drives. It shows 0GB capacity. But leave it for now.

I have also tested TrueNAS Core and it worked flawlessly. Pools were created in no time but I believe for Virtualisation purposes it is better to use the Scale version.

So the first question does my drive are not 512 ?
My only solution is to run time sg_format -v -F /dev/xxx on every drive and wait forever? 15*8TB. Is there a way to make it all at once?
But really don't want to kill any more drives.

BR,

 

[radier]

In iDRAC 9 in Controller status I can see this:

T10 PI Capability

Not Capable

 

[jgreco]

I have Dell Poweredge R740xd with PERC H740P mini in eHBA mode.

This is a RAID controller, whether or not it is in some lobotomized "eHBA mode", and is unsuitable for use with TrueNAS.

What's all the noise about HBAs, and why can't I use a RAID controller?

1) An HBA is a Host Bus Adapter. This is a controller that allows SAS and SATA devices to be attached to, and communicate directly with, a server. RAID controllers typically aggregate several disks into a Virtual Disk abstraction of some sort...

www.truenas.com

Please swap it out for an actual LSI HBA.

 

[radier]

This is a RAID controller, whether or not it is in some lobotomized "eHBA mode", and is unsuitable for use with TrueNAS.

What's all the noise about HBAs, and why can't I use a RAID controller?

1) An HBA is a Host Bus Adapter. This is a controller that allows SAS and SATA devices to be attached to, and communicate directly with, a server. RAID controllers typically aggregate several disks into a Virtual Disk abstraction of some sort...

www.truenas.com

Please swap it out for an actual LSI HBA.

But it works in Trunas Core. I could easily create pools. Then I have installed Scale version and imported those pools.

 

[Ericloewe]

These specific models often work, but they end up seriously sucking performance-wise. You can also just replace them with HBA330 minis, so there's very little reason to go through the pain.

My only solution is to run time sg_format -v -F /dev/xxx on every drive and wait forever? 15*8TB. Is there a way to make it all at once?

Tack an ampersand at the end of the command? Multiple terminal emulators/SSH sessions? Multiple windows in a tmux session? There's a bunch of options for running them in parallel. I'm probably missing a few good ones.

 

[radier]

Swapping controller will solve my problem with DIF?

 

[Ericloewe]

Not super likely.

 

[jgreco]

But it works in Trunas Core. I could easily create pools. Then I have installed Scale version and imported those pools.

Just because something seems to work doesn't mean it's an acceptable choice.

 

[radier]

I might replace the controller but it won't be solution to my primary problem, right?

 

[jgreco]

It seems likely to be involved in it. We know that the mrsas based cards are focused on the task of creating virtual disk devices and that this can cause problems with ZFS. mrsas supplanted the mfi driver some time ago as the driver for the high end LSI cards; mfi was absolutely known to be unusable in the long run for ZFS. Some people have said "mrsas is much better" which may have some truth in it, but I can't say with any confidence that it is "100% compatible" with ZFS because there's really no evidence to support such an outlandish and unsubstantiated claim.

 

[filllip]

New Dell T150 server here with 8TB Toshiba drives on the HBA355 controller. Got security disabled, but still after running

Code:

time sg_format -v -F -s 4096 /dev/sdb

the device still reports

Code:

Logical block size:   528 bytes
Physical block size:  4224 bytes

Here are some details to see if anyone has any tips

Show

Code:

root@truenas[/home/admin]# sg_readcap -l /dev/sdb
Read Capacity results:
   Protection: prot_en=0, p_type=0, p_i_exponent=0
   Logical block provisioning: lbpme=0, lbprz=0
   Last LBA=15154475799 (0x38746f317), Number of logical blocks=15154475800
   Logical block length=528 bytes
   Logical blocks per physical block exponent=3 [so physical block length=4224 bytes]
   Lowest aligned LBA=0
Hence:
   Device size: 8001563222400 bytes, 7630885.3 MiB, 8001.56 GB, 8.00 TB
root@truenas[/home/admin]# smartctl -i /dev/sdb
smartctl 7.2 2020-12-30 r5155 [x86_64-linux-5.15.79+truenas] (local build)
Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Vendor:               TOSHIBA
Product:              MG08SDA800EY
Revision:             EL01
Compliance:           SPC-4
User Capacity:        8,001,563,222,400 bytes [8.00 TB]
Logical block size:   528 bytes
Physical block size:  4224 bytes
Rotation Rate:        7200 rpm
Form Factor:          3.5 inches
Logical Unit id:      0x5000039bc82290e1
Serial number:        72K0A0E3FGAH
Device type:          disk
Transport protocol:   SAS (SPL-3)
Local Time is:        Mon Feb 20 08:22:51 2023 CET
SMART support is:     Available - device has SMART capability.
SMART support is:     Enabled
Temperature Warning:  Disabled or Not Supported

root@truenas[/home/admin]# time sg_format -v /dev/sdb
    TOSHIBA   MG08SDA800EY      EL01   peripheral_type: disk [0x0]
      PROTECT=1
      << supports protection information>>
      Unit serial number: 72K0A0E3FGAH
      LU name: 5000039bc82290e1
    mode sense(10) cdb: [5a 00 01 00 00 00 00 00 fc 00]
Mode Sense (block descriptor) data, prior to changes:
block count maxed out, set <<longlba>>
    mode sense(10) cdb: [5a 10 01 00 00 00 00 00 fc 00]
  <<< longlba flag set (64 bit lba) >>>
  Number of blocks=15154475800 [0x38746f318]
  Block size=528 [0x210]
    read capacity(16) cdb: [9e 10 00 00 00 00 00 00 00 00 00 00 00 20 00 00]
Read Capacity (16) results:
   Protection: prot_en=0, p_type=0, p_i_exponent=0
   Logical block provisioning: lbpme=0, lbprz=0
   Logical blocks per physical block exponent=3
   Lowest aligned logical block address=0
   Number of logical blocks=15154475800
   Logical block size=528 bytes
No changes made. To format use '--format'. To resize use '--resize'
sg_format -v /dev/sdb  0.00s user 0.00s system 56% cpu 0.006 total

A format only takes 22 hours, so I'm kind of pounding my head in the wall right now, wishing I stuck with my gut and purchased disks separately...