Troubleshooting disk format warnings in TrueNAS SCALE 7.4.5

[Daisuke]

Descriptor type: Sense key specific: Progress indication is increasing the % completed

That's a Seagate disk, TrueNAS includes the openSeaChest utilities. Use them instead.

 

[mcblacksmith]

@Daisuke can you please help me out? I don't have a strong handle of BSD or Linux cmds and I'm running into what I think is the same issue.

When trying to create a new pool I get this error.

Code:

Error: Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/middlewared/job.py", line 426, in run
    await self.future
  File "/usr/lib/python3/dist-packages/middlewared/job.py", line 461, in __run_body
    rv = await self.method(*([self] + args))
  File "/usr/lib/python3/dist-packages/middlewared/schema.py", line 1186, in nf
    res = await f(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/middlewared/schema.py", line 1318, in nf
    return await func(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/middlewared/plugins/pool.py", line 765, in do_create
    await self.middleware.call('pool.format_disks', job, disks)
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1386, in call
    return await self._call(
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1335, in _call
    return await methodobj(*prepared_call.args)
  File "/usr/lib/python3/dist-packages/middlewared/plugins/pool_/format_disks.py", line 33, in format_disks
    await asyncio_map(format_disk, disks.items(), limit=16)
  File "/usr/lib/python3/dist-packages/middlewared/utils/asyncio_.py", line 16, in asyncio_map
    return await asyncio.gather(*futures)
  File "/usr/lib/python3/dist-packages/middlewared/utils/asyncio_.py", line 13, in func
    return await real_func(arg)
  File "/usr/lib/python3/dist-packages/middlewared/plugins/pool_/format_disks.py", line 25, in format_disk
    await self.middleware.call('disk.format', disk, config.get('min_size'), swap_size, False)
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1386, in call
    return await self._call(
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1346, in _call
    return await self.run_in_executor(prepared_call.executor, methodobj, *prepared_call.args)
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1249, in run_in_executor
    return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs))
  File "/usr/lib/python3.9/concurrent/futures/thread.py", line 52, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/lib/python3/dist-packages/middlewared/plugins/disk_/format.py", line 56, in format
    swapsize, alignment = self.validate_disk(disk, min_size, swapgb)
  File "/usr/lib/python3/dist-packages/middlewared/plugins/disk_/format.py", line 20, in validate_disk
    raise CallError(f'Disk: {disk!r} is incorrectly formatted with Data Integrity Feature (DIF).')
middlewared.service_exception.CallError: [EFAULT] Disk: 'sdf' is incorrectly formatted with Data Integrity Feature (DIF).

TrueNAS-SCALE-22.12.2

After reading through these post I thought I had a decent grasp of what I needed to do but I'm pretty sure I bricked 5 10TB SAS disks
The drives in question are Seagate EXOS Model ST1000NM0226. I have 10 in total with 5 now not showing up outside of BIOS and while in BIOS they're formatted 0KB (512B) instead of 10TB (4Kn)

I want to try to get the other 5 working but not sure where to go from here.

Thanks in advance for any help you can provide
Mike

 

[Daisuke]

The drives in question are Seagate EXOS Model ST1000NM0226.

TrueNAS includes the openSeaChest utilities for Seagate disks, use them instead. Manual for openSeaChest_Format.

Example for checking the disk supported formats:

Code:

sudo openSeaChest_Format --device /dev/sg1 --showSupportedSectorSizes

Use sg_map to get the correct device name.

 

[mcblacksmith]

TrueNAS includes the openSeaChest utilities for Seagate disks, use them instead. Manual for openSeaChest_Format.

Example for checking the disk supported formats:

Code:

sudo openSeaChest_Format --device /dev/sg1 --showSupportedSectorSizes

Use sg_map to get the correct device name.

Thanks for the quick reply. I was able to use sg_map to find the correct drives but when I try any of the openSeaChest cmds i get this

Code:

root@TNS02[~]# sudo openSeaChest_Format --device /dev/sg5 --showSupportedSectorSizes
sudo: openSeaChest_Format: command not found
root@TNS02[~]#

I tried it with and without sudo in both the web shell as well as and SSH session.

 

[Daisuke]

I tried it with and without sudo

You are root at prompt root@TNS02[~]# (note the #), so there is no need to use sudo, read the Linux Terminal Conventions section in this thread. To see all available commands, type:

Code:

openSeaChest_

and press tab, it will show you all available commands. In your case, use openSeaChest_FormatUnit. I don't know why they changed the binary name, but if you run openSeaChest_FormatUnit --help, it will still refer into their help menu as openSeaChest_Format.

 

[mcblacksmith]

You are root at prompt root@TNS02[~]# (note the #), so there is no need to use sudo, read the Linux Terminal Conventions section in this thread. To see all available commands, type:

Code:

openSeaChest_

and press tab, it will show you all available commands. In your case, use openSeaChest_FormatUnit. I don't know why they changed the binary name, but if you run openSeaChest_FormatUnit --help, it will still refer into their help menu as openSeaChest_Format.

Thanks again for the reply.
I spoke with TrueNAS support and it seems this was going to be an ongoing issue for me which I would tackle if it was business related but since it is personal storage, I reached out to the retailer (techmikeny) and they're going to exchange the drives for non-self encrypting drives. A truly wonderful team of people.

Thanks again for being ready and willing to help.

 

[inolen]

@Daisuke just updated and saw this warning due to the T10 protection that's set for every disk in my RaidZ1 pool. Is the idea to take each drive offline, format, and then resilver one by on by one?

 

[inolen]

I tried taking one of my drives reporting this offline, ran sg_format on it but for some reason that seemed to anger the gods. It got to around .8% and then printed out -

Code:

truenas md/raid1:md126: Fisk failure on sdd1, disabling device

and then stopped printing anything more. Tried running sg_requests / sd_readcaps on this device a few hours in and both seemed to hang (couldn't kill -9 them) so I eventually rebooted the machine.

Now, I the disk shows as unavailable in the UI and doesn't seem to be mounted. Any suggestions on how I can get it mounted so I can try formatting it again?

Code:

root@truenas[~]# zpool status

....

          raidz1-0                                DEGRADED     0     0     0
            c9ea2698-d131-469b-8a6c-16a17cad50c3  ONLINE       0     0     0
            abfafe60-0eab-430f-93eb-7ae8521da47d  ONLINE       0     0     0
            9c9b9ff7-9ac5-430d-a7fa-2b61631e4c4d  ONLINE       0     0     0
            b96e0fea-2e1a-413e-84f7-b850bf2bc5ac  ONLINE       0     0     0
            37117f26-4445-4015-949a-9ec59c87d939  ONLINE       0     0     0
            fc902c9e-9e69-4375-9316-4e31e05a4df4  ONLINE       0     0     0
            94e8ae54-dd75-48e9-b4d3-90d18d5907f9  ONLINE       0     0     0
            d529ada8-04cd-43a2-88f2-3da24dd9fc7e  ONLINE       0     0     0
            7871222667381338731                   UNAVAIL      0     0     0  was /dev/disk/by-partuuid/f4a65604-9107-4a14-89f7-899932228d8b

root@truenas[~]# sg_map                       
/dev/sg0
/dev/sg1  /dev/sdb
/dev/sg2  /dev/sda
/dev/sg3  /dev/sdc
/dev/sg4  /dev/sdd
/dev/sg5  /dev/sde
/dev/sg6  /dev/sdf
/dev/sg7  /dev/sdg
/dev/sg8  /dev/sdh
/dev/sg9  /dev/sdi
/dev/sg10  /dev/sdj
/dev/sg11  /dev/sdk
/dev/sg12

root@truenas[~]# sg_readcap -l /dev/sg12       
read capacity(16): transport: Host_status=0x07 [DID_ERROR]
Driver_status=0x08 [DRIVER_SENSE]

READ CAPACITY (16) failed: Illegal request, Invalid opcode
sg_readcap failed: Illegal request, Invalid opcode

 

[Daisuke]

Try wiping the disk with wipefs -af /dev/sg12 or sgdisk -Z /dev/sg12 and format it again.

 

[jzilla]

So my server crashed while formatting 5 different drives. 3 HGST and 2 Seagate Constellation. I can no longer "see" any of them. I have tried pulling a drive and placing in another system as well. Does anyone have a way to recover these?

 

[Sawtaytoes]

How are you finding the correct HGST firmware?

I just bought a bunch of `H7210A520SUN010T` which are clearly Sun-branded running A680. They're 512e drives that I'd like to change to 4Kn. They're not in an array thankfully, so I can fix all the issues offline.

Another server has `HUH721010AL42C0` drives with the A3Z4 firmware, and I think those drives have a 4Kn label on top. They also have 9.1TiB whereas these news ones have 8.91TiB. Weird...

For those new Sun-branded drives, I've run the sg format utility on them in regular mode, and that fixed the T10 type 1 issue. What about the 4Kn thing? Do I just format it to 4096 bytes or do I upgrade the firmware or use HUGO?

 

[Ericloewe]

Sun-branded? When were they manufactured?

 

[Sawtaytoes]

Sun-branded? When were they manufactured?

2018. They came in drive carriages with Sun branding including the S-cube logo. I think it says "Oracle Sun" though.

 

[Tolete]

Rough start here with Truenas seeing the 'Data Integrity Feature' error while trying to create pool.

'sg_map' shows sg0-67 62/63=boot
-someone please tell me I don't have to remove the protection one disk at a time and then format one disk at a time.

 

[Tolete]

i was able to use

Code:

for disk in sda sdd sdk sdq sdz sdb sdf sdg sdj sdl sdi sds; do (time sg_format -v -F /dev/$disk &); done

tor un on multiple disks at the same time.

 

[jgreco]

New to managing low level disk features huh. :-/ Happy to buy you a beer to cry over though.

 

[CoolWolf]

How are you finding the correct HGST firmware?

I just bought a bunch of `H7210A520SUN010T` which are clearly Sun-branded running A680. They're 512e drives that I'd like to change to 4Kn. They're not in an array thankfully, so I can fix all the issues offline.

Another server has `HUH721010AL42C0` drives with the A3Z4 firmware, and I think those drives have a 4Kn label on top. They also have 9.1TiB whereas these news ones have 8.91TiB. Weird...

For those new Sun-branded drives, I've run the sg format utility on them in regular mode, and that fixed the T10 type 1 issue. What about the 4Kn thing? Do I just format it to 4096 bytes or do I upgrade the firmware or use HUGO?

I used HUGO to remove the T10 type 1 protection and format to 4Kn in a single command:

sudo hugo format --danger-zone --simple-progress -b 4096 -p 0 -g /dev/sdq

The -q option allows to specify multiple devices so you can format multiple drives at the same time.

 

[Sawtaytoes]

i was able to use

Code:

for disk in sda sdd sdk sdq sdz sdb sdf sdg sdj sdl sdi sds; do (time sg_format -v -F /dev/$disk &); done

tor un on multiple disks at the same time.

That's better than what I did!

I've formatted 74 drives already and did them in bulk by using my text editor to duplicate the `tmux` script on each line, then I pasted that in TrueNAS's shell.

It's the brute-force way of doing it, but it also works :p.

 

[Sawtaytoes]

I used HUGO to remove the T10 type 1 protection and format to 4Kn in a single command:

sudo hugo format --danger-zone --simple-progress -b 4096 -p 0 -g /dev/sdq

The -q option allows to specify multiple devices so you can format multiple drives at the same time.

Did you do that to a 512e drive? All of mine are that way, but I'd like to make them 4Kn if possible.

 

[CoolWolf]

Did you do that to a 512e drive? All of mine are that way, but I'd like to make them 4Kn if possible.

Yes. All of my 6 disks (2x HUH728080AL5200 and 4x HUH728080AL5204) were originally 512e formatted:

Hard disk drive specifications - HGST Ultrastar He8​

Code:

sudo fdisk -l /dev/sdh   

Disk /dev/sdh: 7.28 TiB, 8001563222016 bytes, 1953506646 sectors
Disk model: HUH728080AL5204
Units: sectors of 1 * 4096 = 4096 bytes
Sector size (logical/physical): 4096 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes

Good luck!