Troubleshooting disk format warnings in TrueNAS SCALE 7.4.5

[MR.B]

I'm on holiday. I will try this when I'm back. We have around 110 discs. If I want format and replace the discs one by one I have many ours of work. Any other solution?

 

[Daisuke]

If I want format and replace the discs one by one I have many ours of work.

Are the disks in a pool? If yes, then you can backup your data, destroy pool and format as many disks as you can at once. There is no way around this. If these disks are inside a server, they will get super hot if you format all of them at once. To a point where your server might shutdown without proper cooling.

 

[João Fernandes]

I'm trying to add 6 disks to my pool (seagate savvio 10k 900GB SAS drives) and got a message as "Device not ready" when running any sg_ commands. Any hint on how to troubleshoot this? I'm on a DL380Gen8 with P420i in HBA mode. The original disks have no issues at all.

 

[Daisuke]

got a message as "Device not ready" when running any sg_ commands

Not related to this thread topic, you already have a thread open for your issue. As you said, your disks are fine.

 

[João Fernandes]

Not related to this thread topic, you already have a thread open for your issue. As you said, your disks are fine.

I was following your troubleshooting steps that's why I commented on this thread. My original disks are fine, the new ones are not since I'm unable to move past "device not ready" message BTW, what do you mean I already have a thread open?

 

[Daisuke]

BTW, what do you mean I already have a thread open?

I thought you have already one open, my bad. "device not ready" means the disk is defective, or there is an issue with the way you connect your disk, or with your controller. Time to open your own thread for that issue, this thread is related to badly formatted disks fixes only.

 

[jgreco]

got a message as "Device not ready" when running any sg_ commands. Any hint on how to troubleshoot this? I'm on a DL380Gen8 with P420i in HBA mode.

The P420i RAID controller is most likely interfering. There is no such thing as "HBA mode", regardless of what HP believes. It is just a lobotomized RAID controller, and that's not acceptable for TrueNAS, which needs direct unfettered disk access.

What's all the noise about HBAs, and why can't I use a RAID controller?

1) An HBA is a Host Bus Adapter. This is a controller that allows SAS and SATA devices to be attached to, and communicate directly with, a server. RAID controllers typically aggregate several disks into a Virtual Disk abstraction of some sort...

www.truenas.com

 

[BulgarianHealer]

Hi, after update from core to scale (Bluefin) i have the same problem. I have read all the above about 512(+8 bytes) + T10, but in my scenario, i think it's a little different maybe, not sure. Can somebody help?

sda, sdh, sdg, sdl, sdx, sdu, sdf, sdk are formatted with Data Integrity Feature (DIF) which is unsupported.​

​

So 8/12 (this is one whole pool in raidz3) HDD have DIF, example output from one of the HDD's, they are all the same model, same logical/physical sector size, same:

smartctl:
root@truenas[~]# smartctl -i /dev/sdk

Code:

smartctl 7.2 2020-12-30 r5155 [x86_64-linux-5.15.79+truenas] (local build)
Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Vendor:               IBM-D050
Product:              ST6000NM0014
Revision:             BC7B
Compliance:           SPC-4
User Capacity:        6,001,175,126,016 bytes [6.00 TB]
Logical block size:   4096 bytes
Formatted with type 2 protection
8 bytes of protection information per logical block
LU is fully provisioned
Rotation Rate:        7200 rpm
Form Factor:          3.5 inches
Logical Unit id:      0x5000c5008e590dc7
Serial number:        S4D090T60000K603GMT4
Device type:          disk
Transport protocol:   SAS (SPL-3)
Local Time is:        Wed Jan  4 22:47:58 2023 EET
SMART support is:     Available - device has SMART capability.
SMART support is:     Enabled
Temperature Warning:  Enabled

sg_readcap:

Code:

root@truenas[~]# sg_readcap -l /dev/sdk
Read Capacity results:
   Protection: prot_en=1, p_type=1, p_i_exponent=0 [type 2 protection]
   Logical block provisioning: lbpme=0, lbprz=0
   Last LBA=1465130645 (0x57541e95), Number of logical blocks=1465130646
   Logical block length=4096 bytes
   Logical blocks per physical block exponent=0
   Lowest aligned LBA=0
Hence:
   Device size: 6001175126016 bytes, 5723166.6 MiB, 6001.18 GB, 6.00 TB

The other 4 drives are 512 bytes. So I can format these (the bad ones) to 512 or 4kn?

 

[Daisuke]

Hi, after update from core to scale (Bluefin) i have the same problem.

Instructions are provided into OP, how to fix the issue. Taking a disk offline and fixing it will not break anything in your raidz3 pool.

 

[BulgarianHealer]

Instructions are provided into OP, how to fix the issue. Taking a disk offline and fixing it will not break anything in your raidz3 pool.

And i got some strange error. any idea? i have tried with --six(from another forum suggestion, didn't help)

Code:

root@truenas[~]# time sg_format -vFf 0 -s 512 /dev/sda     
    IBM-D050  ST6000NM0014      BC7B   peripheral_type: disk [0x0]
      PROTECT=1
      << supports protection information>>
      Unit serial number: S4D090ZD0000K601HQC0
      LU name: 5000c5008e58e757
    mode sense(10) cdb: [5a 00 01 00 00 00 00 00 fc 00]
Mode Sense (block descriptor) data, prior to changes:
number of blocks will max out, set <<longlba>>
    mode sense(10) cdb: [5a 10 01 00 00 00 00 00 fc 00]
  <<< longlba flag set (64 bit lba) >>>
  Number of blocks=1465130646 [0x57541e96]
  Block size=4096 [0x1000]
    mode select(10) cdb: [55 11 00 00 00 00 00 00 24 00]
mode select(10):
Fixed format, current; Sense key: Illegal Request
Additional sense: Invalid field in parameter list
  Field replaceable unit code: 17
  Sense Key Specific: Error in Data parameters: byte 21 bit 7
Try MODE SELECT again with SP=0 this time
    mode select(10) cdb: [55 10 00 00 00 00 00 00 24 00]
mode select(10):
Fixed format, current; Sense key: Illegal Request
Additional sense: Invalid field in parameter list
  Field replaceable unit code: 17
  Sense Key Specific: Error in Data parameters: byte 21 bit 7
MODE SELECT command: Illegal request, type: sense key, apart from Invalid opcode
sg_format -vFf 0 -s 512 /dev/sda  0.00s user 0.00s system 1% cpu 0.139 total

 

[Daisuke]

What's the output for:

Code:

# openSeaChest_Info -id /dev/sda

 

[BulgarianHealer]

What's the output for:

Code:

# openSeaChest_Info -id /dev/sda

The output:

Code:

root@truenas[~]# openSeaChest_Info -id /dev/sda
==========================================================================================
 openSeaChest_Info - openSeaChest drive utilities - NVMe Enabled
 Copyright (c) 2014-2022 Seagate Technology LLC and/or its Affiliates, All Rights Reserved
 openSeaChest_Info Version: 2.0.0-2_2_1 X86_64
 Build Date: Dec 13 2022
 Today: Thu Jan  5 00:38:31 2023        User: root
==========================================================================================

/dev/sg2 - ST6000NM0014 - S4D090ZD0000K601HQC0 - SCSI
        Vendor ID: IBM-D050
        Model Number: ST6000NM0014   
        Serial Number: S4D090ZD0000K601HQC0
        Firmware Revision: BC7B
        World Wide Name: 5000C5008E58E757
        Drive Capacity (TB/TiB): 6.00/5.46
        Temperature Data:
                Current Temperature (C): 37
                Highest Temperature (C): Not Reported
                Lowest Temperature (C): Not Reported
        Power On Time:  5 years 19 days 8 hours 36 minutes
        Power On Hours: 44264.60
        MaxLBA: 1465130645
        Native MaxLBA: Not Reported
        Logical Sector Size (B): 4096
        Physical Sector Size (B): 4096
        Sector Alignment: 0
        Rotation Rate (RPM): 7200
        Form Factor: 3.5"
        Last DST information:
                Time since last DST (hours): 0.60
                DST Status/Result: 0x0
                DST Test run: 0x1
        Long Drive Self Test Time:  10 hours 43 minutes
        Interface speed:
                Port 0 (Current Port)
                        Max Speed (GB/s): 12.0
                        Negotiated Speed (Gb/s): 6.0
                Port 1
                        Max Speed (GB/s): 12.0
                        Negotiated Speed (Gb/s): Not Reported
        Annualized Workload Rate (TB/yr): 82.63
        Total Bytes Read (TB): 311.36
        Total Bytes Written (TB): 106.15
        Encryption Support: Not Supported
        Cache Size (MiB): Not Reported
        Read Look-Ahead: Not Supported
        Write Cache: Disabled
        SMART Status: Good
        ATA Security Information: Not Supported
        Firmware Download Support: Full, Segmented, Deferred
        Number of Logical Units: 1
        Specifications Supported:
                SPC-4
                SAM-5
                SAS-3
                SPL-3
                SPC-4
                SBC-3
        Features Supported:
                Protection Type 1
                Protection Type 2 [Enabled]
                Application Client Logging
                Self Test
                Automatic Write Reassignment [Enabled]
                Automatic Read Reassignment [Enabled]
                EPC
                Informational Exceptions [Mode 4]
                Translate Address
                Format Unit
                Sanitize
        Adapter Information:
                Vendor ID: 1000h
                Product ID: 0087h
                Revision: 0005h

 

[Daisuke]

@BulgarianHealer, your SAS drive does not support SBC4, so you cannot use a fast format. Let's try this command, specific for SAS drives:

Code:

# openSeaChest_FormatUnit -d /dev/sda --stopOnListError --protectionType 0 --formatUnit 512 --poll

If you get an error, post the output.

 

[BulgarianHealer]

@BulgarianHealer, your SAS drive does not support SBC4, so you cannot use a fast format. Let's try this command, specific for SAS drives:

Code:

# openSeaChest_FormatUnit -d /dev/sda --stopOnListError --protectionType 0 --formatUnit 512 --poll

If you get an error, post the output.

First i got

Code:

root@truenas[~]# openSeaChest_FormatUnit -d /dev/sda --stopOnListError --protectionType 0 --formatUnit 512 --poll
==========================================================================================
 openSeaChest_Format - openSeaChest drive utilities - NVMe Enabled
 Copyright (c) 2014-2022 Seagate Technology LLC and/or its Affiliates, All Rights Reserved
 openSeaChest_Format Version: 2.2.1-2_2_1 X86_64
 Build Date: Dec 13 2022
 Today: Thu Jan  5 01:21:15 2023        User: root
==========================================================================================

/dev/sg2 - ST6000NM0014 - S4D090ZD0000K601HQC0 - SCSI
Format Unit

You must add the flag:
"this-will-erase-data" 
to the command line arguments to run a format unit.

e.g.: openSeaChest_Format -d /dev/sg? --formatUnit current --confirm this-will-erase-data

I have added the flag

Code:

root@truenas[~]# openSeaChest_FormatUnit -d /dev/sda --stopOnListError --protectionType 0 --formatUnit 512 --poll --confirm this-will-erase-data
==========================================================================================
 openSeaChest_Format - openSeaChest drive utilities - NVMe Enabled
 Copyright (c) 2014-2022 Seagate Technology LLC and/or its Affiliates, All Rights Reserved
 openSeaChest_Format Version: 2.2.1-2_2_1 X86_64
 Build Date: Dec 13 2022
 Today: Thu Jan  5 01:22:34 2023        User: root
==========================================================================================

/dev/sg2 - ST6000NM0014 - S4D090ZD0000K601HQC0 - SCSI
Format Unit
Format Unit Failed!

 

[Daisuke]

@BulgarianHealer how is the disk attached? --stopOnListError sets the stop format bit in a format unit. If the device cannot locate or access an existing primary or grown defect list, the format will stop and return with an error. Can you try it without it?

Code:

# openSeaChest_FormatUnit -d /dev/sda --protectionType 0 --formatUnit 512 --poll --confirm this-will-erase-data

 

[BulgarianHealer]

How is the disk attached? --stopOnListError sets the stop format bit in a format unit. If the device cannot locate or access an existing primary or grown defect list, the format will stop and return with an error. Can you try it without it?

Code:

# openSeaChest_FormatUnit -d /dev/sda --protectionType 0 --formatUnit 512 --poll --confirm this-will-erase-data

The disk is attached to the backplane(BPN-SAS826A SAS2), the backplane is connected to the expander(Intel RES2SV240) and the expander is connected to the LSI 9217-8I.

PS:

I got the same error with

Code:

root@truenas[~]# openSeaChest_FormatUnit -d /dev/sda --protectionType 0 --formatUnit 512 --poll --confirm this-will-erase-data
==========================================================================================
 openSeaChest_Format - openSeaChest drive utilities - NVMe Enabled
 Copyright (c) 2014-2022 Seagate Technology LLC and/or its Affiliates, All Rights Reserved
 openSeaChest_Format Version: 2.2.1-2_2_1 X86_64
 Build Date: Dec 13 2022
 Today: Thu Jan  5 01:51:39 2023        User: root
==========================================================================================

/dev/sg2 - ST6000NM0014 - S4D090ZD0000K601HQC0 - SCSI
Format Unit
Format Unit Failed!

 

[Daisuke]

The disk is attached to the backplane

Open a new thread, there are a lot of smart people here on forums who have experience with backplanes and your LSI card. Once you get an answer if Scale kernel behaves with your hardware, try to get an answer on STH or r/homelab and post all troubleshooting commands you ran, then update us in this thread with the fix for your formatting issue. I'll add it to the guide.

I'm curious what's the output for:

Code:

# openSeaChest_Erase -d /dev/sda --sanitize info

 

[jgreco]

Open a new thread, there are a lot of smart people here on forums who have experience with backplanes and your LSI card.

My finger was already hovering over the reply button a few messages back.

There was something that set off in my head when I saw the format problem juxtaposed with the NM model class and an SAS expander. This may mean that I've seen this combination as problematic in the past. We've had a bunch of threads about people coming to FreeNAS with 520-byte sector sizes (similar to this problem). It might be worth it to connect the drives directly to the HBA (this should be fairly easy on an 826A backplane since it's all SFF8087 cabling) and then try the format. I may just be wasting your time, but on the other hand, maybe not.

 

[BulgarianHealer]

Open a new thread, there are a lot of smart people here on forums who have experience with backplanes and your LSI card. Once you get an answer if Scale kernel behaves with your hardware, try to get an answer on STH or r/homelab and update us with the fix for your formatting issue. I'll add it to the guide.

I'm curious what's the output for:

Code:

# openSeaChest_Erase -d /dev/sda --sanitize info

Okay, i will open a new thread tomorrow because it's getting too late for me. :)
Thank you for trying to help me, i will keep you updated.


The output:

Code:

root@truenas[~]# openSeaChest_Erase -d /dev/sda --sanitize info
==========================================================================================
 openSeaChest_Erase - openSeaChest drive utilities - NVMe Enabled
 Copyright (c) 2014-2022 Seagate Technology LLC and/or its Affiliates, All Rights Reserved
 openSeaChest_Erase Version: 3.0.3-2_2_1 X86_64
 Build Date: Dec 13 2022
 Today: Thu Jan  5 02:09:38 2023        User: root
==========================================================================================

/dev/sg2 - ST6000NM0014 - S4D090ZD0000K601HQC0 - SCSI

Sanitize
        The following sanitize commands are supported:
                Overwrite       --sanitize overwrite

 

[Daisuke]

Okay, i will open a new thread tomorrow

Ping me and @jgreco in that thread, let's do what he suggested, connect the problem drive directly.