New drives not showing in one TrueNAS Core box

[jbond]

I just purchased a few 4TB HGST NetApp branded drives to replace three identical drives (HITACHI HUS724040ALE64DB).

When I plug them in I get this dmesg

Code:

(da2:mps0:0:10:1): SERVICE ACTION IN(16). CDB: 9e 10 00 00 00 00 00 00 00 00 00 00 00 20 00 00
(da2:mps0:0:10:1): SCSI sense: ABORTED COMMAND asc:44,0 (Internal target failure)
(da2:mps0:0:10:1): Field Replaceable Unit: 61
(da2:mps0:0:10:1): fatal error, failed to attach to device

for all three new drives.

I already tried rebooting with drives already plugged, changing bays and trays and nothing changes, exgluding issues with the disk shelf, controller, HBA and cables.

I also added them to another truenas box and there they work perfectly fine, they correctly mounted and I was even able to reformat the block size from 520 to 512.

The other truenas box has identical software (Both on TrueNAS-13.0-U3.1), both mostly fresh installs with no specific modification.

Here a brief summary of what could be wrong and why I excluded it:

• Drives: they work correctly on the second box
• Shelf: every other disk work, even using different bays and trays makes no difference
• Cables: reseated cables, issue still here and only for these specific drives independent of bay or tray
• HBA: every other disk work correctly
• Drive firmware: drives work correctly on the second box

What remains is software on the first box.

I tried searching online for this issue but it doesn't look like it happened to anyone else, I can't get SMART data on this box since the drives don't even get a da[0-99] number and drives don't show up anywhere.

Here is the output of camcontrol devlist in case it might be useful

Code:

<HITACHI HUS724040ALE64DB 4321>    at scbus33 target 9 lun 1 (pass3,da1)
<HITACHI HUS724040ALE64DB 4321>    at scbus33 target 10 lun 1 (pass5) //this is one of the problem drives, reformatted to 512 block size
<HITACHI HUS724040ALE64DB 4321>    at scbus33 target 11 lun 1 (pass7,da3)
<HITACHI HUS724040ALE64DB NA01>    at scbus33 target 12 lun 1 (pass9) //this is one of the problem drives, formatted to 520 block size
<HITACHI HUS724040ALE64DB 4321>    at scbus33 target 13 lun 1 (pass11,da5)
<HITACHI HUS724040ALE64DB 4321>    at scbus33 target 14 lun 1 (pass13,da6)
<HITACHI HUS724040ALE64DB 4321>    at scbus33 target 15 lun 1 (pass15,da7)
<HITACHI HUS724040ALE64DB 4321>    at scbus33 target 16 lun 1 (pass17,da8)
<HITACHI HUS724040ALE64DB 4321>    at scbus33 target 17 lun 1 (pass19,da9)
<HITACHI HUS724040ALE64DB 4321>    at scbus33 target 18 lun 1 (pass21,da10)
<HITACHI HUS724040ALE64DB 4321>    at scbus33 target 19 lun 1 (pass23,da11)

the last drive doesnt show at all here but only in dmesg

 

[jbond]

Anybody has any idea?