Missing disks on new build

[RedMercury]

I have just installed TrueNAS-SCALE-22.12.2 on a new machine. I have a brand new QNAP enclosure (QNAP TR-004U) configured in individual disk mode with brand new 6TB SATA drives installed. The enclosure is connected to the TrueNAS machine using USB3.

When I go to create a new pool using the new disks, only one disk appears, and the rest do not. The disk is the size of the individual HDD and I've double checked that the enclosure is configured in individual disk mode.

dmesg log confirms that the individual disks are known to the kernel:

[ 3.929666] sd 3:0:0:0: [sdb] Very big device. Trying to use READ CAPACITY(16).
[ 3.929933] sd 3:0:0:1: [sdc] Very big device. Trying to use READ CAPACITY(16).
[ 3.930052] sd 3:0:0:0: [sdb] 11721045168 512-byte logical blocks: (6.00 TB/5.46 TiB)
[ 3.930057] sd 3:0:0:0: [sdb] 4096-byte physical blocks
[ 3.930204] sd 3:0:0:1: [sdc] 11721045168 512-byte logical blocks: (6.00 TB/5.46 TiB)
[ 3.930208] sd 3:0:0:1: [sdc] 4096-byte physical blocks
[ 3.930491] sd 3:0:0:0: [sdb] Write Protect is off
[ 3.930493] sd 3:0:0:0: [sdb] Mode Sense: 47 00 00 08
[ 3.930792] sd 3:0:0:0: [sdb] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[ 3.931150] sd 3:0:0:1: [sdc] Write Protect is off
[ 3.931153] sd 3:0:0:1: [sdc] Mode Sense: 47 00 00 08
[ 3.931327] sd 3:0:0:2: [sdd] Very big device. Trying to use READ CAPACITY(16).
[ 3.931448] sd 3:0:0:2: [sdd] 11721045168 512-byte logical blocks: (6.00 TB/5.46 TiB)
[ 3.931450] sd 3:0:0:2: [sdd] 4096-byte physical blocks
[ 3.931738] sd 3:0:0:2: [sdd] Write Protect is off
[ 3.931740] sd 3:0:0:2: [sdd] Mode Sense: 47 00 00 08
[ 3.932027] sd 3:0:0:2: [sdd] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[ 3.933057] sd 3:0:0:3: [sde] Very big device. Trying to use READ CAPACITY(16).
[ 3.933346] sd 3:0:0:1: [sdc] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[ 3.935224] sd 3:0:0:3: [sde] 11721045168 512-byte logical blocks: (6.00 TB/5.46 TiB)
[ 3.935228] sd 3:0:0:3: [sde] 4096-byte physical blocks
[ 3.935518] sd 3:0:0:3: [sde] Write Protect is off
[ 3.935520] sd 3:0:0:3: [sde] Mode Sense: 47 00 00 08
[ 3.935807] sd 3:0:0:3: [sde] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[ 3.943207] sdd:
[ 3.944326] sd 3:0:0:2: [sdd] Attached SCSI disk
[ 3.944764] sdb:
[ 3.946318] sd 3:0:0:0: [sdb] Attached SCSI disk
[ 3.947606] sdc:
[ 3.947859] sde:
[ 3.948886] sd 3:0:0:1: [sdc] Attached SCSI disk
[ 3.950032] sd 3:0:0:3: [sde] Attached SCSI disk
[ 5.386077] Installing knfsd (copyright (C) 1996 okir@monad.swb.de).
[ 5.515423] sd 2:0:0:0: Attached scsi generic sg0 type 0
[ 5.515486] sd 3:0:0:0: Attached scsi generic sg1 type 0
[ 5.515528] sd 3:0:0:1: Attached scsi generic sg2 type 0
[ 5.515582] sd 3:0:0:2: Attached scsi generic sg3 type 0
[ 5.515642] sd 3:0:0:3: Attached scsi generic sg4 type 0

Any ideas why this would be the case and how I can fix it? I've attempted to manually install a new GPT on each drive from the shell, but that didn't fix things.

Thanks!

 

[RedMercury]

I managed to get it working with zpool create on the command line. This looks like a UI bug.

 

[RedMercury]

Ahh, but creating it on the command line means that the new zpool is persona non gratia to the UI. Back to square one.

 

[jgreco]

Doing that is not supported; USB attached disks have a variety of issues which may or may not apply to your specific case. See the excellent resource by @Arwen

Why you should avoid USB attached drives for data pool disks

This subject has been coming up quite often in the last year or 2. Perhaps because of TrueNAS SCALE has brought forth some attention to TrueNAS in general and ZFS in specific. Please note that this is about USB attached storage for ZFS data...

www.truenas.com

 

[RedMercury]

Thanks for the info, although some of those don't apply in this case. USB3 is event-driven instead of the polling mess of USB2 so not quite as ill-performant, and I'm just a homelab tinkerer so I'm not too worried about latches, etc. The enclosure is rackmount and cooled. I get that in some (most?) it's a bad idea, but does 'not supported' mean 'won't work' ?

Each drive has a unique serial number according to hdparm.

 

[jgreco]

but does 'not supported' mean 'won't work' ?

Means "not expected to work well or maybe at all". We're not really able to offer technical support for this in the forums. If you are intent on moving forward with a configuration that is not supported, you're still welcome to post here and see what comments people make, but be advised that we're not going to try to fix this for you. TrueNAS and ZFS both have limited support for arbitrary hardware choices and you generally need to understand that there is no magic wand to make stuff that is not up to snuff magically work.

Common signs are stuff like "disks don't appear" and "pool won't appear/import". You seem to have both these issues.

 

[RedMercury]

I get that, and I'm really grateful that such a powerful piece of software is available for free (and open source!). I'll spelunk through the code and logs to see if it's an easy fix. Thanks for your help!

 

[RedMercury]

You were correct, it was caused by a duplicate ID_SERIAL_SHORT, as udev appears to be taking that from the USB serial instead of the disk. However, ID_SERIAL is unique, suffixed by the disk number in the array.

Locally changing these statements so that ID_SERIAL is used preferentially to ID_SERIAL_SHORT led to success and I'm now able to view individual disks, create a pool from them, etc. I know I'm in very unsupported territory here, perhaps adding a USB udev rule to force ID_SERIAL_SHORT to ID_SERIAL would have been better.

Is this the kind of change accepted as a tinkerer? Happy to send a PR if you want.

 

[jgreco]

You can try submitting a PR. However, there's a certain amount of hazard in tinkering with this stuff. There's apparently active work going on with the multipath stuff, which might be affected by such a change, and iX has been resistant to changes in the fundamentals for years now, because they want stability rather than unexpected problems. Also, the people who implemented the original device detection system for FreeBSD at iX have, I believe, moved on, so there may not be anyone left willing to tinker. This is my way of saying "don't hold your breath".

 

[prototype]

RedMercury I'm basically in the same position as you how exactly did you manage to make it work ? I'm not very tech savvy so would appresiate if you can explain it a bit simpler .

 

[Monstieur]

I have the same issue with NVMe namespaces. I have 2 namespaces nvd3 & nvd4 on a single disk. TrueNAS shows only nvd4 in the GUI. This used to work a few weeks ago when I originally added each namespace to a different VDEV. Today I removed the disk to resize the namespaces and now one of the namespaces doesn't show up in the GUI.