pool create failed due device or resoucre busy

[ascii-]

hello togehter,

i'm just the process of setting up my first truenas build.
i can't create my data pool. turenas scale always says device or resource busy,
it is supposted to be an array of 6 disks.

once i confirm to wipe the disc i get

Error: Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/middlewared/job.py", line 411, in run
await self.future
File "/usr/lib/python3/dist-packages/middlewared/job.py", line 446, in __run_body
rv = await self.method(*([self] + args))
File "/usr/lib/python3/dist-packages/middlewared/schema.py", line 1140, in nf
res = await f(*args, **kwargs)
File "/usr/lib/python3/dist-packages/middlewared/schema.py", line 1272, in nf
return await func(*args, **kwargs)
File "/usr/lib/python3/dist-packages/middlewared/plugins/pool.py", line 740, in do_create
await self.middleware.call('pool.format_disks', job, disks)
File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1344, in call
return await self._call(
File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1293, in _call
return await methodobj(*prepared_call.args)
File "/usr/lib/python3/dist-packages/middlewared/plugins/pool_/format_disks.py", line 56, in format_disks
await asyncio_map(format_disk, disks.items(), limit=16)
File "/usr/lib/python3/dist-packages/middlewared/utils/asyncio_.py", line 16, in asyncio_map
return await asyncio.gather(*futures)
File "/usr/lib/python3/dist-packages/middlewared/utils/asyncio_.py", line 13, in func
return await real_func(arg)
File "/usr/lib/python3/dist-packages/middlewared/plugins/pool_/format_disks.py", line 29, in format_disk
await self.middleware.call(
File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1344, in call
return await self._call(
File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1304, in _call
return await self.run_in_executor(prepared_call.executor, methodobj, *prepared_call.args)
File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1205, in run_in_executor
return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs))
File "/usr/lib/python3.9/concurrent/futures/thread.py", line 52, in run
result = self.fn(*self.args, **self.kwargs)
File "/usr/lib/python3/dist-packages/middlewared/plugins/disk_/format.py", line 26, in format
raise CallError(f'Failed to wipe disk {disk}: {job.error}')
middlewared.service_exception.CallError: [EFAULT] Failed to wipe disk sdd: [Errno 16] Device or resource busy: '/dev/sdd'

when i remove the sdd from the pool it's just another drive that is busy.

i have rebootet several times and tried.

i have a 9305-16i in the system
i tried updateing firmware on the disks. nothing helps

root@nas[~]# sas3flash -listall
Avago Technologies SAS3 Flash Utility
Version 16.00.00.00 (2017.05.02)
Copyright 2008-2017 Avago Technologies. All rights reserved.

Adapter Selected is a Avago SAS: SAS3224(A1)

Num Ctlr FW Ver NVDATA x86-BIOS PCI Addr
----------------------------------------------------------------------------

0 SAS3224(A1) 16.00.11.00 10.00.00.05 08.33.00.00 00:01:00:00

Finished Processing Commands Successfully.
Exiting SAS3Flash.

root@nas[~]# lsscsi -sig
[0:0:0:0] disk HGST HUH721010AL5200 A9G0 /dev/sda 35000cca26b9312b0 /dev/sg1 10.0TB
[0:0:1:0] disk HGST HUH721010AL5200 A9G0 /dev/sdb 35000cca26a279a80 /dev/sg2 10.0TB
[0:0:3:0] disk HGST HUH721010AL5200 A21D /dev/sdd 35000cca26b583bb0 /dev/sg4 10.0TB
[0:0:5:0] disk HGST HUH721010AL5200 A9G0 /dev/sdf 35000cca26b935384 /dev/sg6 10.0TB
[0:0:6:0] disk HGST HUH721010AL5200 A9G0 /dev/sdg 35000cca26a295fd0 /dev/sg7 10.0TB
[0:0:7:0] disk HGST HUH721010AL5200 A9G0 /dev/sdj 35000cca26b584948 /dev/sg5 10.0TB
[5:0:0:0] disk ATA SAMSUNG MZ7LM240 0F3Q /dev/sdi - /dev/sg8 240GB
[6:0:0:0] disk ATA SAMSUNG MZ7LM240 0F3Q /dev/sdh - /dev/sg9 240GB
[7:0:0:0] enclosu AHCI SGPIO Enclosure 2.00 - - /dev/sg0 -
root@nas[~]#

root@nas[~]# smartctl -a /dev/sg1
smartctl 7.2 2020-12-30 r5155 [x86_64-linux-5.10.131+truenas] (local build)
Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Vendor: HGST
Product: HUH721010AL5200
Revision: A9G0
Compliance: SPC-4
User Capacity: 10,000,831,348,736 bytes [10.0 TB]
Logical block size: 512 bytes
Physical block size: 4096 bytes
LU is fully provisioned
Rotation Rate: 7200 rpm
Form Factor: 3.5 inches
Logical Unit id: 0x5000cca26b9312b0
Serial number: 1SJLW000
Device type: disk
Transport protocol: SAS (SPL-3)
Local Time is: Sat Sep 3 17:41:35 2022 CEST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
Temperature Warning: Enabled

=== START OF READ SMART DATA SECTION ===
SMART Health Status: OK

Grown defects during certification <not available>
Total blocks reassigned during format <not available>
Total new blocks reassigned <not available>
Power on minutes since format <not available>
Current Drive Temperature: 35 C
Drive Trip Temperature: 85 C

Accumulated power on time, hours:minutes 36223:09
Manufactured in week 01 of year 2018
Specified cycle count over device lifetime: 50000
Accumulated start-stop cycles: 30
Specified load-unload count over device lifetime: 600000
Accumulated load-unload cycles: 1536
Elements in grown defect list: 0

Vendor (Seagate Cache) information
Blocks sent to initiator = 26147385440731136

Error counter log:
Errors Corrected by Total Correction Gigabytes Total
ECC rereads/ errors algorithm processed uncorrected
fast | delayed rewrites corrected invocations [10^9 bytes] errors
read: 0 3 0 3 994431 16134.711 0
write: 0 0 0 0 79731 100434.581 0
verify: 0 610 0 610 38481 1970163.849 0

Non-medium error count: 106

No Self-tests have been logged

root@nas[~]# smartctl -a /dev/sg2
smartctl 7.2 2020-12-30 r5155 [x86_64-linux-5.10.131+truenas] (local build)
Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Vendor: HGST
Product: HUH721010AL5200
Revision: A9G0
Compliance: SPC-4
User Capacity: 10,000,831,348,736 bytes [10.0 TB]
Logical block size: 512 bytes
Physical block size: 4096 bytes
LU is fully provisioned
Rotation Rate: 7200 rpm
Form Factor: 3.5 inches
Logical Unit id: 0x5000cca26a279a80
Serial number: 2TGP000
Device type: disk
Transport protocol: SAS (SPL-3)
Local Time is: Sat Sep 3 17:41:38 2022 CEST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
Temperature Warning: Enabled

=== START OF READ SMART DATA SECTION ===
SMART Health Status: OK

Grown defects during certification <not available>
Total blocks reassigned during format <not available>
Total new blocks reassigned <not available>
Power on minutes since format <not available>
Current Drive Temperature: 35 C
Drive Trip Temperature: 85 C

Accumulated power on time, hours:minutes 36223:39
Manufactured in week 03 of year 2018
Specified cycle count over device lifetime: 50000
Accumulated start-stop cycles: 312
Specified load-unload count over device lifetime: 600000
Accumulated load-unload cycles: 452
Elements in grown defect list: 0

Vendor (Seagate Cache) information
Blocks sent to initiator = 3843022651392

Error counter log:
Errors Corrected by Total Correction Gigabytes Total
ECC rereads/ errors algorithm processed uncorrected
fast | delayed rewrites corrected invocations [10^9 bytes] errors
read: 0 0 0 0 25 0.117 0
write: 0 0 0 0 24 18.562 0
verify: 0 0 0 0 10411 1970163.836 0

Non-medium error count: 197

SMART Self-test log
Num Test Status segment LifeTime LBA_first_err [SK ASC ASQ]
Description number (hours)
# 1 Background short Completed - 36222 - [- - -]

Long (extended) Self-test duration: 65535 seconds [1092.2 minutes]

root@nas[~]# smartctl -a /dev/sg4
smartctl 7.2 2020-12-30 r5155 [x86_64-linux-5.10.131+truenas] (local build)
Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Vendor: HGST
Product: HUH721010AL5200
Revision: A21D
Compliance: SPC-4
User Capacity: 10,000,831,348,736 bytes [10.0 TB]
Logical block size: 512 bytes
Physical block size: 4096 bytes
LU is fully provisioned
Rotation Rate: 7200 rpm
Form Factor: 3.5 inches
Logical Unit id: 0x5000cca26b583bb0
Serial number: 1SHKJ000
Device type: disk
Transport protocol: SAS (SPL-3)
Local Time is: Sat Sep 3 17:41:47 2022 CEST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
Temperature Warning: Enabled

=== START OF READ SMART DATA SECTION ===
SMART Health Status: OK

Current Drive Temperature: 38 C
Drive Trip Temperature: 85 C

Accumulated power on time, hours:minutes 33919:08
Manufactured in week 45 of year 2017
Specified cycle count over device lifetime: 50000
Accumulated start-stop cycles: 10
Specified load-unload count over device lifetime: 600000
Accumulated load-unload cycles: 1421
Elements in grown defect list: 0

Vendor (Seagate Cache) information
Blocks sent to initiator = 8608886957277184

Error counter log:
Errors Corrected by Total Correction Gigabytes Total
ECC rereads/ errors algorithm processed uncorrected
fast | delayed rewrites corrected invocations [10^9 bytes] errors
read: 0 0 0 0 423591 13444.862 0
write: 0 0 0 0 98919 32690.463 0
verify: 0 4 0 4 41672 1749975.368 0

Non-medium error count: 13

No Self-tests have been logged

root@nas[~]#

i have no idea what else i can try.

i could use cfdisk to create a parition on the drive. so it is responding

 

[Samuel Tai]

Where is your boot pool installed?

 

[ascii-]

it' on 2 sperate mirrored SSD with 256GB.
i Split the SSD in 2 partions. 16GB for Boot and rest for Apps cache/data

 

[ChrisRJ]

Please describe your system in detail, as per the forum rules. That will make it easier to help you.

 

[ascii-]

Sorry I totally missed that
Mainboard: Supermicro X11SCL-LN4F
CPU: E-2176G
RAM: 4x 16GB DDR4 ECC UDIMM
HDD: 6x10TB HGST Enterprise SAS slow / archiv
HDD: 8x960GB 2.5" Enterprise SAS-SSDs fast / temp storage
HDD: 2x256GB 2.5" Enterprise SATA-SSDs mirror to boot
HBA: LSI 9305-16i (FW: 16.0.11.0)
Truenas Scale installed 3 days ago.

 

[Samuel Tai]

Are all the HGST pool drives fresh from the factory, or do they have partitions already? It may be worth manually wiping them before pool creation.

 

[ascii-]

All drives are used.
I pulled them 2 different Servers from 2 locations.
I tried to wipe them from the gui. that doesn't work.
I could delete the Partition from cfdisk. tested it on 1 drive.
is there a right way of wipping the drives via cli?
I might try to build the Pool via cli. I just need to look up the commands that I need to use.

 

[Samuel Tai]

Hmm. If they're used drives, they may have had their firmware locked as a security measure, which would prevent repartitioning. Try using hdparm to wipe these disks.

Advanced: Erasing SATA Drives by using the Linux hdparm Utility - GROK Knowledge Base

GROK Knowledgebase is Louisiana State University's online support environment.

grok.lsu.edu

 

[ascii-]

the link says it's for SATA drives only. Mine are als SAS drives so hdparm will not work.
I doubt that they are locked. I could put on a New Partition via cfdisk and also Update the Firmware.

 

[Samuel Tai]

OK, try to get rid of all partitions, and put a GPT partition table on them, or no partition table if possible. They need to look as pristine as if they came straight from the factory.

 

[ascii-]

well i tried a lot of things the last days.
it seems to be a problem with the spinners.
i wiped the partition and gpt with

sgdisk -Z /dev/sdXYZ

the disk are blank. also did it with 2 SAS SSD connected to the same controller.
with blank SSDs i can create a pool. with the HDDs it is not possible.

both spinners are blank

root@nas[~]# fdisk -l /dev/sdb
Disk /dev/sdb: 9.1 TiB, 10000831348736 bytes, 19532873728 sectors
Disk model: HUH721010AL5200
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
root@nas[~]# fdisk -l /dev/sda
Disk /dev/sda: 9.1 TiB, 10000831348736 bytes, 19532873728 sectors
Disk model: HUH721010AL5200
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes

both SSDs are blank

root@nas[~]# fdisk -l /dev/sdf
Disk /dev/sdf: 894.25 GiB, 960197124096 bytes, 1875385008 sectors
Disk model: VO000960JWTBK
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
root@nas[~]# fdisk -l /dev/sdg
Disk /dev/sdg: 894.25 GiB, 960197124096 bytes, 1875385008 sectors
Disk model: VO000960JWTBK
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes

creating a pool with the SSD works

root@nas[~]# zpool create -f fast mirror /dev/sdf /dev/sdg

creating a pool with the spinners

root@nas[~]# zpool create -f slow mirror /dev/sda /dev/sdb
/dev/sdb is in use and contains a unknown filesystem.

filesystem after creating of the spinners

root@nas[~]# fdisk -l /dev/sda
Disk /dev/sda: 9.1 TiB, 10000831348736 bytes, 19532873728 sectors
Disk model: HUH721010AL5200
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
root@nas[~]# fdisk -l /dev/sdb
Disk /dev/sdb: 9.1 TiB, 10000831348736 bytes, 19532873728 sectors
Disk model: HUH721010AL5200
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes

filesystem after cearting the pool of the SSDs

root@nas[~]# fdisk -l /dev/sdf
Disk /dev/sdf: 894.25 GiB, 960197124096 bytes, 1875385008 sectors
Disk model: VO000960JWTBK
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: 26EBBE20-B3EF-3F4B-AF3F-D7AFE3ACF1B4

Device Start End Sectors Size Type
/dev/sdf1 2048 1875367935 1875365888 894.2G Solaris /usr & Apple ZFS
/dev/sdf9 1875367936 1875384319 16384 8M Solaris reserved 1
root@nas[~]# fdisk -l /dev/sdg
Disk /dev/sdg: 894.25 GiB, 960197124096 bytes, 1875385008 sectors
Disk model: VO000960JWTBK
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: 4F4D45B1-4039-E849-8D1C-846B2BE18626

Device Start End Sectors Size Type
/dev/sdg1 2048 1875367935 1875365888 894.2G Solaris /usr & Apple ZFS
/dev/sdg9 1875367936 1875384319 16384 8M Solaris reserved 1
root@nas[~]#

i also stopped the smart service to be sure it is not accessing the drive.

so it looks like samething is accessing the spinners for what ever reason and i need to find out what it is and stop ist.

 

[ascii-]

so after the i don't know 100th reboot i could create the pool.
i have no idea why. nothing has changed

 

[mmaura]

Hello, thank this post resolve me.

I have the same problem, with a disk coming from the raid of my Netgear.

i use sgdisk -Z /dev/sdXYZ on all disk (except system), and reboot quickly. After that i can create my pool without problem.

Thank you.

 

[Benji99]

Hi guys, just wanted to chime in that I ran into this issue as well trying to re-use 4x 8TB drives that I had pulled from my soon-to-be-decommed QNAP NAS. the command sgdisk -Z /dev/sdXYZ and a reboot did the trick as well!