TrueNAS 12.0-U1.1 LSI 9400-16i not detecting drives

[cyrus104]

Updated: Turns out the SFF-8643 to SATA breakout cables were trash. Once those were changed out everything works as expected.

I have an Broadcom LSI 9400-16i that I flashed with the latest P17 firmware and Legacy SAS25BIOS to be used only for SATA drives (spinners and ssds), I do not need mixed mode.

I was able to use storcli in TrueNAS to download the new files and can confirm that the versions were updated.

None of the 6 drives currently connected are enumerating and I am struggling to figure out why. I have got through the forums here and at STH and have no luck.

I also see that JBOD is enabled but not supported, which I thought was only an issue with the 9460 model.

Any thoughts would be great, I'm pretty new to the HBA world.

Code:

root@truenas[~]# ./storcli64 /c0 show all
CLI Version = 007.1613.0000.0000 Oct 29, 2020
Operating system = FreeBSD 12.2-RELEASE-p2
Controller = 0
Status = Success
Description = None


Basics :
======
Controller = 0
Adapter Type =   SAS3416(B0)
Model = HBA 9400-16i
Serial Number = XW843100FA
Current System Date/time = 02/08/2021 09:03:34
Concurrent commands supported = 6656
SAS Address =  500605b00e0196fd
PCI Address = 00:81:00:00
Mfg Date = 00/00/00
Rework Date = 00/00/00
Revision No = N/A


Version :
=======
Firmware Package Build = 17.00.00.00
Firmware Version = 17.00.00.00
Bios Version = 09.33.00.00_17.00.00.00
NVDATA Version = 17.01.00.22
PSOC Version = 05689001
Driver Name = mpr
Driver Version = 23.00.00.00


PCI Version :
===========
Vendor Id = 0x1000
Device Id = 0xAC
SubVendor Id = 0x1000
SubDevice Id = 0x3000
Host Interface = PCIE
Device Interface = SAS-12G
Bus Number = 129
Device Number = 0
Function Number = 0
Domain ID = 0


Pending Images in Flash :
=======================
Image name = No pending images


Status :
======
Controller Status = OK
Memory Correctable Errors = 0
Memory Uncorrectable Errors = 0
Bios was not detected during boot = No
Controller has booted into safe mode = No
Controller has booted into certificate provision mode = No
Package Stamp Mismatch = No


Supported Adapter Operations :
============================
Alarm Control = No
Cluster Support = No
Self Diagnostic = No
Deny SCSI Passthrough = No
Deny SMP Passthrough = No
Deny STP Passthrough = No
Support more than 8 Phys = Yes
FW and Event Time in GMT = No
Support Enclosure Enumeration = Yes
Support Allowed Operations = Yes
Support Multipath = Yes
Support Security = No
Support Config Page Model = No
Support the OCE without adding drives = No
support EKM = No
Snapshot Enabled = No
Support PFK = No
Support PI = No
Support Shield State = No
Support Set Link Speed = No
Support JBOD = No
Disable Online PFK Change = No
Real Time Scheduler = No
Support Reset Now = No
Support Emulated Drives = No
Support Secure Boot = No
Support Platform Security = No
Support Package Stamp Mismatch Reporting = Yes


HwCfg :
=====
ChipRevision =  B0
BatteryFRU = N/A
Front End Port Count = 1
Backend Port Count = 21
Serial Debugger = Absent
NVRAM Size = 0KB
Flash Size = 16MB
On Board Memory Size = 0MB
On Board Expander = Absent
Temperature Sensor for ROC = Present
Temperature Sensor for Controller = Absent
Current Size of CacheCade (GB) = 0
Current Size of FW Cache (MB) = 0
ROC temperature(Degree Celcius) = 66


Policies :
========

Policies Table :
==============

------------------------------------------------
Policy                          Current Default
------------------------------------------------
Predictive Fail Poll Interval   0 sec
Interrupt Throttle Active Count 0
Interrupt Throttle Completion   0 us
Rebuild Rate                    0 %     30%
PR Rate                         0 %     30%
BGI Rate                        0 %     30%
Check Consistency Rate          0 %     30%
Reconstruction Rate             0 %     30%
Cache Flush Interval            0s
------------------------------------------------

Flush Time(Default) = 4s
Drive Coercion Mode = none
Auto Rebuild = Off
Battery Warning = Off
ECC Bucket Size = 0
ECC Bucket Leak Rate (hrs) = 0
Restore HotSpare on Insertion = Off
Expose Enclosure Devices = Off
Maintain PD Fail History = Off
Reorder Host Requests = On
Auto detect BackPlane = SGPIO/i2c SEP
Load Balance Mode = None
Security Key Assigned = Off
Disable Online Controller Reset = Off
Use drive activity for locate = Off


Boot :
====
Max Drives to Spinup at One Time = 0
Maximum number of direct attached drives to spin up in 1 min = 0
Delay Among Spinup Groups (sec) = 0
Allow Boot with Preserved Cache = On


Defaults :
========
Phy Polarity = 0
Phy PolaritySplit = 0
Cached IO = Off
Default spin down time (mins) = 0
Coercion Mode = None
ZCR Config = Unknown
Max Chained Enclosures = 0
Direct PD Mapping = No
Restore Hot Spare on Insertion = No
Expose Enclosure Devices = No
Maintain PD Fail History = No
Zero Based Enclosure Enumeration = No
Disable Puncturing = No
Un-Certified Hard Disk Drives = Block
SMART Mode = Mode 6
Enable LED Header = No
LED Show Drive Activity = No
Dirty LED Shows Drive Activity = No
EnableCrashDump = No
Disable Online Controller Reset = No
Treat Single span R1E as R10 = No
Power Saving option = Enable
TTY Log In Flash = No
Auto Enhanced Import = No
Enable Shield State = No
Time taken to detect CME = 60 sec


Capabilities :
============
Supported Drives = SAS, SATA
Enable JBOD = Yes
Max Parallel Commands = 6656
Max SGE Count = 128
Max Data Transfer Size = 32 sectors
Max Strips PerIO = 0
Max Configurable CacheCade Size = 0
Min Strip Size = 512Bytes
Max Strip Size = 512Bytes

Scheduled Tasks = NA

Security Protocol properties :
============================
Security Protocol = None

System:
CPU: Epyc 7282
MOBO: ASRock Rack Romed8-2t
RAM: 128GB
HBA: LSI 9400-16i

 

[Samuel Tai]

That ASRockRack board boots via UEFI. Did you flash the SAS35BIOS_Rel or the UEFI_BSD_HII_SAS3.5_IT_X64 firmware? The latter will put the HBA into IT mode required for JBOD access.

 

[cyrus104]

@Samuel Tai, Thanks for the quick response.

I have tried both, the output above was with the Rel but I just reflashed the UEFI.

Code:

root@truenas[..._BIOS_UEFI/UEFI_BSD_HII_SAS3.5_IT_X64]# ~/storcli64 /c0 download efibios file= mpt35sas_x64.rom
Downloading image.Please wait...

CLI Version = 007.1613.0000.0000 Oct 29, 2020
Operating system = FreeBSD 12.2-RELEASE-p2
Controller = 0
Status = Success
Description = EFI Bios Flash Successful

 

[cyrus104]

I'm still not seeing the drives and not sure how to check the drive status.

I'm still getting:
Support JBOD = No
Supported Drives = SAS, SATA
Enable JBOD = Yes

 

[cyrus104]

I made some progress and can now see the LSI card in the bios but there are no options.

 

[Samuel Tai]

Explore all the BIOS options. Make sure the HBA itself sees all the drives, and is operating in IT mode. It may be in RAID mode, and expecting you to assign drives to a RAID pool. Don't operate it in RAID mode.

 

[cyrus104]

There are no other options in the BIOS for the drives besides the ones that I sent and these have no options to choose from.

Not sure how to check the mode. When using storcli /c0 show jbod, profile, personality all show failure and un-supported command.

 

[Samuel Tai]

If the HBA BIOS doesn't see the drives, then it's likely a cable or backplane problem. What cables are you using?

 

[cyrus104]

I'm using, CableCreation cables that had pretty high reviews on amazon.

 

[Samuel Tai]

No, I mean the specific model of cables.

 

[jgreco]

That raises my eyebrows. As an obscure product for the technical market, I wouldn't expect there to be many (probably any) legitimate reviews, because those of us who buy such things tend to be professionals and I know I sure as heck don't write reviews for Amazon's benefit -- my customers don't pay me to do that.

 

[cyrus104]

Internal HD Mini SAS Cable, #CS0034

CableCreation Focus to the Ideal Design of All of Our Quality Cables CableCreation is not Only a Brand But Also a Creative Team With Own Factory manufacturing Side 1: Mini SAS (SFF-8643) Male with Latch Side 2: 4x SATA Female Port Cable: 30AWG Cable Length:0.5M Cable Color: Black+Blue SFF-8643 C...

www.cablecreation.com

 

[Samuel Tai]

Those appear to be the correct cables. Which 2 ports on the HBA are you using?

 

[Samuel Tai]

Also, are your drives getting power?

 

[cyrus104]

Drives are getting power for sure.

I am using the very top port and the very bottom port. I have the other 2 ports wired up with the same cables but no drives plugged in yet. Eventually I'll have 4-4-4-2 plugged in when the drives comes in.

 

[Samuel Tai]

Are you connecting through a backplane or to the drives directly?

 

[cyrus104]

I'm connecting to the drives directly. Currently 4 are SSDs and 2 are HDD, I can feel the HDDs spinning.

 

[Samuel Tai]

Do the drives appear if you're connecting through the motherboard's SATA ports?

 

[Samuel Tai]

This may help too:

https://docs.broadcom.com/doc/12352476

According to that manual for storcli, you may need to manually set all the drives to JBOD mode.

 

[cyrus104]

I'm starting to think it's a faulty batch of cables. I tested 2 of them on the motherboard directly and one of the shows 3 drives and the other shows 0. I moved the drive order around and can get all of the drives to enumerate on the motherboard connector but not at the same time. I'm still not sure why they aren't showing up at all on the LSI card but I'll start with baby steps and get a cable that works well enough to show all drives.