No controller found?

[sunshine931]

I'm building a lab with an IBM x3650 M4. It's got an IBM M5110 RAID card (used only for boot), and an Intel RSP3GD016J (a rebranded LSI 9400-16e HBA). It's in IT mode, and has been flashed with the latest 24.00.01.01 Intel-provided firmware. I have two IBM SAS disk expansion trays attached to the 9400-16e, populated with 24x 8TB NLSAS drives.

TrueNAS Scale 22.12.3.

I can access the drives just fine (and am in the middle of removing Type 2 protection from them), but I can't seem to use sas3ircu. Should I be exclusively using storcli for the 9400-16e?

With sas3ircu I get:

Code:

root@truenas[~/util]# sas3ircu list
Avago Technologies SAS3 IR Configuration Utility.
Version 16.00.00.00 (2017.04.26)
Copyright (c) 2009-2017 Avago Technologies. All rights reserved.

SAS3IRCU: MPTLib2 Error 1
root@truenas[~/util]# sas3ircu 0 display
Avago Technologies SAS3 IR Configuration Utility.
Version 16.00.00.00 (2017.04.26)
Copyright (c) 2009-2017 Avago Technologies. All rights reserved.

SAS3IRCU: No Controller Found at index 0.
root@truenas[~/util]# sas3ircu 1 display
Avago Technologies SAS3 IR Configuration Utility.
Version 16.00.00.00 (2017.04.26)
Copyright (c) 2009-2017 Avago Technologies. All rights reserved.

SAS3IRCU: No Controller Found at index 1.
root@truenas[~/util]#

dmesg output:

Code:

root@truenas[~/util]# dmesg| grep sas
[    6.542193] megasas: 07.717.02.00-rc1
[    6.549234] megaraid_sas 0000:1b:00.0: BAR:0x1  BAR's base_addr(phys):0x00000000bc4bc000  mapped virt_addr:0x000000000ae3a0ce
[    6.549243] megaraid_sas 0000:1b:00.0: FW now in Ready state
[    6.555491] megaraid_sas 0000:1b:00.0: 63 bit DMA mask and 32 bit consistent mask
[    6.602398] mpt3sas version 39.100.00.00 loaded
[    6.724079] mpt3sas_cm0: 63 BIT PCI BUS DMA ADDRESSING SUPPORTED, total mem (329990536 kB)
[    6.859014] megaraid_sas 0000:1b:00.0: firmware supports msix        : (16)
[    6.873333] megaraid_sas 0000:1b:00.0: requested/available msix 16/16 poll_queue 0
[    6.880522] megaraid_sas 0000:1b:00.0: current msix/online cpus      : (16/32)
[    6.887671] megaraid_sas 0000:1b:00.0: RDPQ mode     : (disabled)
[    6.894745] megaraid_sas 0000:1b:00.0: Current firmware supports maximum commands: 1008       LDIO threshold: 0
[    7.690339] mpt3sas_cm0: MSI-X vectors supported: 128
[    7.703815] mpt3sas_cm0:  0 32 32
[    7.718695] mpt3sas_cm0: High IOPs queues : disabled
[    7.727773] mpt3sas0-msix0: PCI-MSI-X enabled: IRQ 93
[    7.734238] mpt3sas0-msix1: PCI-MSI-X enabled: IRQ 94
[    7.740639] mpt3sas0-msix2: PCI-MSI-X enabled: IRQ 95
[    7.746919] mpt3sas0-msix3: PCI-MSI-X enabled: IRQ 96
[    7.753117] mpt3sas0-msix4: PCI-MSI-X enabled: IRQ 97
[    7.759224] mpt3sas0-msix5: PCI-MSI-X enabled: IRQ 98
[    7.765271] mpt3sas0-msix6: PCI-MSI-X enabled: IRQ 99
[    7.771166] mpt3sas0-msix7: PCI-MSI-X enabled: IRQ 100
[    7.776882] mpt3sas0-msix8: PCI-MSI-X enabled: IRQ 101
[    7.782445] mpt3sas0-msix9: PCI-MSI-X enabled: IRQ 102
[    7.787867] mpt3sas0-msix10: PCI-MSI-X enabled: IRQ 103
[    7.793269] mpt3sas0-msix11: PCI-MSI-X enabled: IRQ 104
[    7.798545] mpt3sas0-msix12: PCI-MSI-X enabled: IRQ 105
[    7.803672] mpt3sas0-msix13: PCI-MSI-X enabled: IRQ 106
[    7.808643] mpt3sas0-msix14: PCI-MSI-X enabled: IRQ 107
[    7.813511] mpt3sas0-msix15: PCI-MSI-X enabled: IRQ 108
[    7.818254] mpt3sas0-msix16: PCI-MSI-X enabled: IRQ 109
[    7.822889] mpt3sas0-msix17: PCI-MSI-X enabled: IRQ 110
[    7.827414] mpt3sas0-msix18: PCI-MSI-X enabled: IRQ 111
[    7.831776] mpt3sas0-msix19: PCI-MSI-X enabled: IRQ 112
[    7.836094] mpt3sas0-msix20: PCI-MSI-X enabled: IRQ 113
[    7.840334] mpt3sas0-msix21: PCI-MSI-X enabled: IRQ 114
[    7.844440] mpt3sas0-msix22: PCI-MSI-X enabled: IRQ 115
[    7.848414] mpt3sas0-msix23: PCI-MSI-X enabled: IRQ 116
[    7.852382] mpt3sas0-msix24: PCI-MSI-X enabled: IRQ 117
[    7.856226] mpt3sas0-msix25: PCI-MSI-X enabled: IRQ 118
[    7.859985] mpt3sas0-msix26: PCI-MSI-X enabled: IRQ 119
[    7.863615] mpt3sas0-msix27: PCI-MSI-X enabled: IRQ 120
[    7.867132] mpt3sas0-msix28: PCI-MSI-X enabled: IRQ 121
[    7.870519] mpt3sas0-msix29: PCI-MSI-X enabled: IRQ 122
[    7.873787] mpt3sas0-msix30: PCI-MSI-X enabled: IRQ 123
[    7.876895] mpt3sas0-msix31: PCI-MSI-X enabled: IRQ 124
[    7.879861] mpt3sas_cm0: iomem(0x00000000be200000), mapped(0x00000000110a6533), size(1048576)
[    7.883127] mpt3sas_cm0: ioport(0x0000000000002f00), size(256)
[    8.079851] megaraid_sas 0000:1b:00.0: Performance mode :Latency (latency index = 1)
[    8.083914] megaraid_sas 0000:1b:00.0: FW supports sync cache        : No
[    8.088048] megaraid_sas 0000:1b:00.0: megasas_disable_intr_fusion is called outbound_intr_mask:0x40000009
[    8.693102] megaraid_sas 0000:1b:00.0: FW provided supportMaxExtLDs: 1       max_lds: 64
[    8.698294] megaraid_sas 0000:1b:00.0: controller type       : MR(1024MB)
[    8.703549] megaraid_sas 0000:1b:00.0: Online Controller Reset(OCR)  : Enabled
[    8.708832] megaraid_sas 0000:1b:00.0: Secure JBOD support   : No
[    8.714204] megaraid_sas 0000:1b:00.0: NVMe passthru support : No
[    8.719631] megaraid_sas 0000:1b:00.0: FW provided TM TaskAbort/Reset timeout        : 0 secs/0 secs
[    8.725260] megaraid_sas 0000:1b:00.0: JBOD sequence map support     : No
[    8.730966] megaraid_sas 0000:1b:00.0: PCI Lane Margining support    : No
[    9.732478] megaraid_sas 0000:1b:00.0: megasas_enable_intr_fusion is called outbound_intr_mask:0x40000000
[    9.739964] megaraid_sas 0000:1b:00.0: INIT adapter done
[    9.747501] megaraid_sas 0000:1b:00.0: JBOD sequence map is disabled megasas_setup_jbod_map 5804
[    9.763736] megaraid_sas 0000:1b:00.0: pci id                : (0x1000)/(0x005b)/(0x1014)/(0x040b)
[    9.779569] megaraid_sas 0000:1b:00.0: unevenspan support    : no
[    9.787503] megaraid_sas 0000:1b:00.0: firmware crash dump   : no
[    9.795360] megaraid_sas 0000:1b:00.0: JBOD sequence map     : disabled
[    9.811175] megaraid_sas 0000:1b:00.0: Max firmware commands: 1007 shared with default hw_queues = 15 poll_queues 0
[    9.838809] mpt3sas_cm0: sending message unit reset !!
[    9.848315] mpt3sas_cm0: message unit reset: SUCCESS
[    9.882433] mpt3sas_cm0: scatter gather: sge_in_main_msg(1), sge_per_chain(7), sge_per_io(128), chains_per_io(19)
[    9.899017] mpt3sas_cm0: request pool(0x00000000cb66956b) - dma(0x102100000): depth(6756), frame_size(128), pool_size(844 kB)
[   10.083137] mpt3sas_cm0: sense pool(0x00000000c9602676) - dma(0x102600000): depth(6551), element_size(96), pool_size (614 kB)
[   10.092196] mpt3sas_cm0: reply pool(0x000000007748d33c) - dma(0x102700000): depth(6820), frame_size(128), pool_size(852 kB)
[   10.101005] mpt3sas_cm0: config page(0x00000000b8abcbc2) - dma(0x12393d000): size(512)
[   10.109580] mpt3sas_cm0: Allocated physical memory: size(31938 kB)
[   10.118059] mpt3sas_cm0: Current Controller Queue Depth(6548),Max Controller Queue Depth(6656)
[   10.126680] mpt3sas_cm0: Scatter Gather Elements per IO(128)
[   10.179459] mpt3sas_cm0: _base_display_fwpkg_version: complete
[   10.188071] mpt3sas_cm0: FW Package Ver(24.00.01.01)
[   10.196902] mpt3sas_cm0: TimeSync Interval in Manuf page-11 is not enabled. Periodic Time-Sync will be disabled
[   10.206044] mpt3sas_cm0: SAS3416: FWVersion(24.00.01.00), ChipRevision(0x01), BiosVersion(00.00.00.00)
[   10.214935] mpt3sas_cm0: Intel(R) Controller: Subsystem ID: 0x3020
[   10.223858] mpt3sas_cm0: Protocol=(Initiator,Target), Capabilities=(TLR,EEDP,Diag Trace Buffer,Task Set Full,NCQ)
[   10.241910] mpt3sas 0000:0c:00.0: Max SCSIIO MPT commands: 6548 shared with nr_hw_queues = 32
[   10.327046] mpt3sas_cm0: sending port enable !!
[   10.336753] mpt3sas_cm0: hba_port entry: 0000000061776388, port: 0 is added to hba_port list
[   10.347289] mpt3sas_cm0: hba_port entry: 000000001c2d8f01, port: 8 is added to hba_port list
[   10.366706] mpt3sas_cm0: host_add: handle(0x0001), sas_addr(0x500605b00dd93380), phys(21)
[   10.377658] mpt3sas_cm0: expander_add: handle(0x0033), parent(0x0001), sas_addr(0x500507681105b4bf), phys(37)
[   10.387759]  expander-1:0: add: handle(0x0033), sas_addr(0x500507681105b4bf)
[   10.402591] mpt3sas_cm0: handle(0x34) sas_address(0x5000c500a6cc1de9) port_type(0x1)
[   10.428023] scsi 1:0:0:0: SSP: handle(0x0034), sas_addr(0x5000c500a6cc1de9), phy(0), device_name(0x5000c500a6cc1de8)
[   10.450956]  end_device-1:0:0: add: handle(0x0034), sas_addr(0x5000c500a6cc1de9)
[   10.461018] mpt3sas_cm0: handle(0x35) sas_address(0x5000c500a6cc2c41) port_type(0x1)
[   10.516021] scsi 1:0:1:0: SSP: handle(0x0035), sas_addr(0x5000c500a6cc2c41), phy(1), device_name(0x5000c500a6cc2c40)
[   10.568872]  end_device-1:0:1: add: handle(0x0035), sas_addr(0x5000c500a6cc2c41)
[   10.586704] mpt3sas_cm0: handle(0x36) sas_address(0x5000c500a6cd4209) port_type(0x1)
[   10.612444] scsi 1:0:2:0: SSP: handle(0x0036), sas_addr(0x5000c500a6cd4209), phy(2), device_name(0x5000c500a6cd4208)
[   10.635761]  end_device-1:0:2: add: handle(0x0036), sas_addr(0x5000c500a6cd4209)
[   10.645968] mpt3sas_cm0: handle(0x37) sas_address(0x5000c500a6ef2df9) port_type(0x1)
[   10.674561] scsi 1:0:3:0: SSP: handle(0x0037), sas_addr(0x5000c500a6ef2df9), phy(3), device_name(0x5000c500a6ef2df8)
[   10.698480]  end_device-1:0:3: add: handle(0x0037), sas_addr(0x5000c500a6ef2df9)
[   10.708929] mpt3sas_cm0: handle(0x38) sas_address(0x5000c500a6ef34f5) port_type(0x1)
[   10.738435] scsi 1:0:4:0: SSP: handle(0x0038), sas_addr(0x5000c500a6ef34f5), phy(4), device_name(0x5000c500a6ef34f4)
[   10.762541]  end_device-1:0:4: add: handle(0x0038), sas_addr(0x5000c500a6ef34f5)
[   10.773166] mpt3sas_cm0: handle(0x39) sas_address(0x5000c500a6eed8f5) port_type(0x1)
[   10.805228] scsi 1:0:5:0: SSP: handle(0x0039), sas_addr(0x5000c500a6eed8f5), phy(5), device_name(0x5000c500a6eed8f4)
[   10.829794]  end_device-1:0:5: add: handle(0x0039), sas_addr(0x5000c500a6eed8f5)
[   10.840618] mpt3sas_cm0: handle(0x3a) sas_address(0x5000c500a6cd0c39) port_type(0x1)
[   10.874171] scsi 1:0:6:0: SSP: handle(0x003a), sas_addr(0x5000c500a6cd0c39), phy(6), device_name(0x5000c500a6cd0c38)
[   10.898895]  end_device-1:0:6: add: handle(0x003a), sas_addr(0x5000c500a6cd0c39)
[   10.909786] mpt3sas_cm0: handle(0x3b) sas_address(0x5000c500a6ef28b1) port_type(0x1)
[   10.942802] scsi 1:0:7:0: SSP: handle(0x003b), sas_addr(0x5000c500a6ef28b1), phy(7), device_name(0x5000c500a6ef28b0)
[   10.978521]  end_device-1:0:7: add: handle(0x003b), sas_addr(0x5000c500a6ef28b1)
[   10.989747] mpt3sas_cm0: handle(0x3c) sas_address(0x5000c500a6fb7285) port_type(0x1)
[   11.012840] scsi 1:0:8:0: SSP: handle(0x003c), sas_addr(0x5000c500a6fb7285), phy(8), device_name(0x5000c500a6fb7284)
[   11.037606]  end_device-1:0:8: add: handle(0x003c), sas_addr(0x5000c500a6fb7285)
[   11.048755] mpt3sas_cm0: handle(0x3d) sas_address(0x5000c500a6cd4751) port_type(0x1)
[   11.083032] scsi 1:0:9:0: SSP: handle(0x003d), sas_addr(0x5000c500a6cd4751), phy(9), device_name(0x5000c500a6cd4750)
[   11.108291]  end_device-1:0:9: add: handle(0x003d), sas_addr(0x5000c500a6cd4751)
[   11.119510] mpt3sas_cm0: handle(0x3e) sas_address(0x5000c500a6ccde59) port_type(0x1)
[   11.158114] scsi 1:0:10:0: SSP: handle(0x003e), sas_addr(0x5000c500a6ccde59), phy(10), device_name(0x5000c500a6ccde58)
[   11.227768]  end_device-1:0:10: add: handle(0x003e), sas_addr(0x5000c500a6ccde59)
[   11.238934] mpt3sas_cm0: handle(0x3f) sas_address(0x5000c500a6cc3351) port_type(0x1)
[   11.274511] scsi 1:0:11:0: SSP: handle(0x003f), sas_addr(0x5000c500a6cc3351), phy(11), device_name(0x5000c500a6cc3350)
[   11.299726]  end_device-1:0:11: add: handle(0x003f), sas_addr(0x5000c500a6cc3351)
[   11.311683] mpt3sas_cm0: handle(0x41) sas_address(0x500507681105b4be) port_type(0x1)
[   11.346209] scsi 1:0:12:0: SES: handle(0x0041), sas_addr(0x500507681105b4be), phy(36), device_name(0x0000000000000000)
[   11.391798]  end_device-1:0:12: add: handle(0x0041), sas_addr(0x500507681105b4be)
[   11.402995] mpt3sas_cm0: expander_add: handle(0x0040), parent(0x0033), sas_addr(0x500507681105c1bf), phys(37)
[   11.414200]  expander-1:1: add: handle(0x0040), sas_addr(0x500507681105c1bf)
[   11.429322] mpt3sas_cm0: handle(0x42) sas_address(0x5000c500a6cd4fa9) port_type(0x1)
[   11.460656] scsi 1:0:13:0: SSP: handle(0x0042), sas_addr(0x5000c500a6cd4fa9), phy(0), device_name(0x5000c500a6cd4fa8)
[   11.485676]  end_device-1:1:0: add: handle(0x0042), sas_addr(0x5000c500a6cd4fa9)
[   11.496859] mpt3sas_cm0: handle(0x43) sas_address(0x5000c500a6cc63a9) port_type(0x1)
[   11.528535] scsi 1:0:14:0: SSP: handle(0x0043), sas_addr(0x5000c500a6cc63a9), phy(1), device_name(0x5000c500a6cc63a8)
[   11.564433]  end_device-1:1:1: add: handle(0x0043), sas_addr(0x5000c500a6cc63a9)
[   11.575385] mpt3sas_cm0: handle(0x44) sas_address(0x5000c500a6cc4825) port_type(0x1)
[   11.606689] scsi 1:0:15:0: SSP: handle(0x0044), sas_addr(0x5000c500a6cc4825), phy(2), device_name(0x5000c500a6cc4824)
[   11.631566]  end_device-1:1:2: add: handle(0x0044), sas_addr(0x5000c500a6cc4825)
[   11.642477] mpt3sas_cm0: handle(0x45) sas_address(0x5000c500a6ccd8e5) port_type(0x1)
[   11.670308] scsi 1:0:16:0: SSP: handle(0x0045), sas_addr(0x5000c500a6ccd8e5), phy(3), device_name(0x5000c500a6ccd8e4)
[   11.695113]  end_device-1:1:3: add: handle(0x0045), sas_addr(0x5000c500a6ccd8e5)
[   11.706021] mpt3sas_cm0: handle(0x46) sas_address(0x5000c500a6cc35a1) port_type(0x1)
[   11.739952] scsi 1:0:17:0: SSP: handle(0x0046), sas_addr(0x5000c500a6cc35a1), phy(4), device_name(0x5000c500a6cc35a0)
[   11.809095]  end_device-1:1:4: add: handle(0x0046), sas_addr(0x5000c500a6cc35a1)
[   11.829110] mpt3sas_cm0: handle(0x47) sas_address(0x5000c500a6fbce99) port_type(0x1)
[   11.872317] scsi 1:0:18:0: SSP: handle(0x0047), sas_addr(0x5000c500a6fbce99), phy(5), device_name(0x5000c500a6fbce98)
[   11.897334]  end_device-1:1:5: add: handle(0x0047), sas_addr(0x5000c500a6fbce99)
[   11.908311] mpt3sas_cm0: handle(0x48) sas_address(0x5000c500a6fbaa49) port_type(0x1)
[   11.942198] scsi 1:0:19:0: SSP: handle(0x0048), sas_addr(0x5000c500a6fbaa49), phy(6), device_name(0x5000c500a6fbaa48)
[   11.967264]  end_device-1:1:6: add: handle(0x0048), sas_addr(0x5000c500a6fbaa49)
[   11.978420] mpt3sas_cm0: handle(0x49) sas_address(0x5000c500a6cd436d) port_type(0x1)
[   12.012677] scsi 1:0:20:0: SSP: handle(0x0049), sas_addr(0x5000c500a6cd436d), phy(7), device_name(0x5000c500a6cd436c)
[   12.037931]  end_device-1:1:7: add: handle(0x0049), sas_addr(0x5000c500a6cd436d)
[   12.049038] mpt3sas_cm0: handle(0x4a) sas_address(0x5000c500a6fb9cfd) port_type(0x1)
[   12.082052] scsi 1:0:21:0: SSP: handle(0x004a), sas_addr(0x5000c500a6fb9cfd), phy(8), device_name(0x5000c500a6fb9cfc)
[   12.118324]  end_device-1:1:8: add: handle(0x004a), sas_addr(0x5000c500a6fb9cfd)
[   12.129427] mpt3sas_cm0: handle(0x4b) sas_address(0x5000c500a6cc1d25) port_type(0x1)
[   12.160344] scsi 1:0:22:0: SSP: handle(0x004b), sas_addr(0x5000c500a6cc1d25), phy(9), device_name(0x5000c500a6cc1d24)
[   12.185640]  end_device-1:1:9: add: handle(0x004b), sas_addr(0x5000c500a6cc1d25)
[   12.196826] mpt3sas_cm0: handle(0x4c) sas_address(0x5000c500a6ccf929) port_type(0x1)
[   12.225130] scsi 1:0:23:0: SSP: handle(0x004c), sas_addr(0x5000c500a6ccf929), phy(10), device_name(0x5000c500a6ccf928)
[   12.250430]  end_device-1:1:10: add: handle(0x004c), sas_addr(0x5000c500a6ccf929)
[   12.261573] mpt3sas_cm0: handle(0x4d) sas_address(0x5000c500a6cd46e9) port_type(0x1)
[   12.296311] scsi 1:0:24:0: SSP: handle(0x004d), sas_addr(0x5000c500a6cd46e9), phy(11), device_name(0x5000c500a6cd46e8)
[   12.343572]  end_device-1:1:11: add: handle(0x004d), sas_addr(0x5000c500a6cd46e9)
[   12.363000] mpt3sas_cm0: handle(0x4e) sas_address(0x500507681105c1be) port_type(0x1)
[   12.406980] scsi 1:0:25:0: SES: handle(0x004e), sas_addr(0x500507681105c1be), phy(36), device_name(0x0000000000000000)
[   12.453122]  end_device-1:1:12: add: handle(0x004e), sas_addr(0x500507681105c1be)
[   12.464543] mpt3sas_cm0: expander_add: handle(0x0017), parent(0x0009), sas_addr(0x500507681105b4ff), phys(37)
[   12.475989]  expander-1:2: add: handle(0x0017), sas_addr(0x500507681105b4ff)
[   12.491438] mpt3sas_cm0: handle(0x18) sas_address(0x5000c500a6cc1dea) port_type(0x1)
[   12.530164] scsi 1:0:26:0: SSP: handle(0x0018), sas_addr(0x5000c500a6cc1dea), phy(0), device_name(0x5000c500a6cc1de8)
[   12.555701]  end_device-1:2:0: add: handle(0x0018), sas_addr(0x5000c500a6cc1dea)
[   12.566920] mpt3sas_cm0: handle(0x19) sas_address(0x5000c500a6cc2c42) port_type(0x1)
[   12.599876] scsi 1:0:27:0: SSP: handle(0x0019), sas_addr(0x5000c500a6cc2c42), phy(1), device_name(0x5000c500a6cc2c40)
[   12.625356]  end_device-1:2:1: add: handle(0x0019), sas_addr(0x5000c500a6cc2c42)
[   12.636563] mpt3sas_cm0: handle(0x1a) sas_address(0x5000c500a6cd420a) port_type(0x1)
[   12.672283] scsi 1:0:28:0: SSP: handle(0x001a), sas_addr(0x5000c500a6cd420a), phy(2), device_name(0x5000c500a6cd4208)
[   12.708912]  end_device-1:2:2: add: handle(0x001a), sas_addr(0x5000c500a6cd420a)
[   12.720147] mpt3sas_cm0: handle(0x1b) sas_address(0x5000c500a6ef2dfa) port_type(0x1)
[   12.757044] scsi 1:0:29:0: SSP: handle(0x001b), sas_addr(0x5000c500a6ef2dfa), phy(3), device_name(0x5000c500a6ef2df8)
[   12.782521]  end_device-1:2:3: add: handle(0x001b), sas_addr(0x5000c500a6ef2dfa)
[   12.793795] mpt3sas_cm0: handle(0x1c) sas_address(0x5000c500a6ef34f6) port_type(0x1)
[   12.822449] scsi 1:0:30:0: SSP: handle(0x001c), sas_addr(0x5000c500a6ef34f6), phy(4), device_name(0x5000c500a6ef34f4)
[   12.848004]  end_device-1:2:4: add: handle(0x001c), sas_addr(0x5000c500a6ef34f6)
[   12.859250] mpt3sas_cm0: handle(0x1d) sas_address(0x5000c500a6eed8f6) port_type(0x1)
[   12.947600] scsi 1:0:31:0: SSP: handle(0x001d), sas_addr(0x5000c500a6eed8f6), phy(5), device_name(0x5000c500a6eed8f4)
[   12.972367]  end_device-1:2:5: add: handle(0x001d), sas_addr(0x5000c500a6eed8f6)
[   12.983284] mpt3sas_cm0: handle(0x1e) sas_address(0x5000c500a6cd0c3a) port_type(0x1)
[   13.021185] scsi 1:0:32:0: SSP: handle(0x001e), sas_addr(0x5000c500a6cd0c3a), phy(6), device_name(0x5000c500a6cd0c38)
[   13.046209]  end_device-1:2:6: add: handle(0x001e), sas_addr(0x5000c500a6cd0c3a)
[   13.057187] mpt3sas_cm0: handle(0x1f) sas_address(0x5000c500a6ef28b2) port_type(0x1)
[   13.087062] scsi 1:0:33:0: SSP: handle(0x001f), sas_addr(0x5000c500a6ef28b2), phy(7), device_name(0x5000c500a6ef28b0)
[   13.112304]  end_device-1:2:7: add: handle(0x001f), sas_addr(0x5000c500a6ef28b2)
[   13.123491] mpt3sas_cm0: handle(0x20) sas_address(0x5000c500a6fb7286) port_type(0x1)
[   13.159397] scsi 1:0:34:0: SSP: handle(0x0020), sas_addr(0x5000c500a6fb7286), phy(8), device_name(0x5000c500a6fb7284)
[   13.184482]  end_device-1:2:8: add: handle(0x0020), sas_addr(0x5000c500a6fb7286)
[   13.195703] mpt3sas_cm0: handle(0x21) sas_address(0x5000c500a6cd4752) port_type(0x1)
[   13.231920] scsi 1:0:35:0: SSP: handle(0x0021), sas_addr(0x5000c500a6cd4752), phy(9), device_name(0x5000c500a6cd4750)
[   13.257013]  end_device-1:2:9: add: handle(0x0021), sas_addr(0x5000c500a6cd4752)
[   13.268219] mpt3sas_cm0: handle(0x22) sas_address(0x5000c500a6ccde5a) port_type(0x1)
[   13.304730] scsi 1:0:36:0: SSP: handle(0x0022), sas_addr(0x5000c500a6ccde5a), phy(10), device_name(0x5000c500a6ccde58)
[   13.329879]  end_device-1:2:10: add: handle(0x0022), sas_addr(0x5000c500a6ccde5a)
[   13.341019] mpt3sas_cm0: handle(0x23) sas_address(0x5000c500a6cc3352) port_type(0x1)
[   13.377545] scsi 1:0:37:0: SSP: handle(0x0023), sas_addr(0x5000c500a6cc3352), phy(11), device_name(0x5000c500a6cc3350)
[   13.402648]  end_device-1:2:11: add: handle(0x0023), sas_addr(0x5000c500a6cc3352)
[   13.414546] mpt3sas_cm0: handle(0x25) sas_address(0x500507681105b4fe) port_type(0x1)
[   13.448855] scsi 1:0:38:0: SES: handle(0x0025), sas_addr(0x500507681105b4fe), phy(36), device_name(0x0000000000000000)
[   13.497919]  end_device-1:2:12: add: handle(0x0025), sas_addr(0x500507681105b4fe)
[   13.509290] mpt3sas_cm0: expander_add: handle(0x0024), parent(0x0017), sas_addr(0x500507681105c1ff), phys(37)
[   13.520376]  expander-1:3: add: handle(0x0024), sas_addr(0x500507681105c1ff)
[   13.535632] mpt3sas_cm0: handle(0x26) sas_address(0x5000c500a6cd4faa) port_type(0x1)
[   13.565803] scsi 1:0:39:0: SSP: handle(0x0026), sas_addr(0x5000c500a6cd4faa), phy(0), device_name(0x5000c500a6cd4fa8)
[   13.591256]  end_device-1:3:0: add: handle(0x0026), sas_addr(0x5000c500a6cd4faa)
[   13.602599] mpt3sas_cm0: handle(0x27) sas_address(0x5000c500a6cc63aa) port_type(0x1)
[   13.631839] scsi 1:0:40:0: SSP: handle(0x0027), sas_addr(0x5000c500a6cc63aa), phy(1), device_name(0x5000c500a6cc63a8)
[   13.657394]  end_device-1:3:1: add: handle(0x0027), sas_addr(0x5000c500a6cc63aa)
[   13.668783] mpt3sas_cm0: handle(0x28) sas_address(0x5000c500a6cc4826) port_type(0x1)
[   13.702987] scsi 1:0:41:0: SSP: handle(0x0028), sas_addr(0x5000c500a6cc4826), phy(2), device_name(0x5000c500a6cc4824)
[   13.728597]  end_device-1:3:2: add: handle(0x0028), sas_addr(0x5000c500a6cc4826)
[   13.740010] mpt3sas_cm0: handle(0x29) sas_address(0x5000c500a6ccd8e6) port_type(0x1)
[   13.770118] scsi 1:0:42:0: SSP: handle(0x0029), sas_addr(0x5000c500a6ccd8e6), phy(3), device_name(0x5000c500a6ccd8e4)
[   13.795684]  end_device-1:3:3: add: handle(0x0029), sas_addr(0x5000c500a6ccd8e6)
[   13.807019] mpt3sas_cm0: handle(0x2a) sas_address(0x5000c500a6cc35a2) port_type(0x1)
[   13.845297] scsi 1:0:43:0: SSP: handle(0x002a), sas_addr(0x5000c500a6cc35a2), phy(4), device_name(0x5000c500a6cc35a0)
[   13.870865]  end_device-1:3:4: add: handle(0x002a), sas_addr(0x5000c500a6cc35a2)
[   13.882199] mpt3sas_cm0: handle(0x2b) sas_address(0x5000c500a6fbce9a) port_type(0x1)
[   13.915941] scsi 1:0:44:0: SSP: handle(0x002b), sas_addr(0x5000c500a6fbce9a), phy(5), device_name(0x5000c500a6fbce98)
[   13.941567]  end_device-1:3:5: add: handle(0x002b), sas_addr(0x5000c500a6fbce9a)
[   13.952987] mpt3sas_cm0: handle(0x2c) sas_address(0x5000c500a6fbaa4a) port_type(0x1)
[   13.990714] scsi 1:0:45:0: SSP: handle(0x002c), sas_addr(0x5000c500a6fbaa4a), phy(6), device_name(0x5000c500a6fbaa48)
[   14.016296]  end_device-1:3:6: add: handle(0x002c), sas_addr(0x5000c500a6fbaa4a)
[   14.027744] mpt3sas_cm0: handle(0x2d) sas_address(0x5000c500a6cd436e) port_type(0x1)
[   14.061943] scsi 1:0:46:0: SSP: handle(0x002d), sas_addr(0x5000c500a6cd436e), phy(7), device_name(0x5000c500a6cd436c)
[   14.087476]  end_device-1:3:7: add: handle(0x002d), sas_addr(0x5000c500a6cd436e)
[   14.098965] mpt3sas_cm0: handle(0x2e) sas_address(0x5000c500a6fb9cfe) port_type(0x1)
[   14.136987] scsi 1:0:47:0: SSP: handle(0x002e), sas_addr(0x5000c500a6fb9cfe), phy(8), device_name(0x5000c500a6fb9cfc)
[   14.162814]  end_device-1:3:8: add: handle(0x002e), sas_addr(0x5000c500a6fb9cfe)
[   14.174444] mpt3sas_cm0: handle(0x2f) sas_address(0x5000c500a6cc1d26) port_type(0x1)
[   14.211800] scsi 1:0:48:0: SSP: handle(0x002f), sas_addr(0x5000c500a6cc1d26), phy(9), device_name(0x5000c500a6cc1d24)
[   14.237617]  end_device-1:3:9: add: handle(0x002f), sas_addr(0x5000c500a6cc1d26)
[   14.249119] mpt3sas_cm0: handle(0x30) sas_address(0x5000c500a6ccf92a) port_type(0x1)
[   14.286028] scsi 1:0:49:0: SSP: handle(0x0030), sas_addr(0x5000c500a6ccf92a), phy(10), device_name(0x5000c500a6ccf928)
[   14.311886]  end_device-1:3:10: add: handle(0x0030), sas_addr(0x5000c500a6ccf92a)
[   14.323376] mpt3sas_cm0: handle(0x31) sas_address(0x5000c500a6cd46ea) port_type(0x1)
[   14.360726] scsi 1:0:50:0: SSP: handle(0x0031), sas_addr(0x5000c500a6cd46ea), phy(11), device_name(0x5000c500a6cd46e8)
[   14.386566]  end_device-1:3:11: add: handle(0x0031), sas_addr(0x5000c500a6cd46ea)
[   14.398701] mpt3sas_cm0: handle(0x32) sas_address(0x500507681105c1fe) port_type(0x1)
[   14.433774] scsi 1:0:51:0: SES: handle(0x0032), sas_addr(0x500507681105c1fe), phy(36), device_name(0x0000000000000000)
[   14.480180]  end_device-1:3:12: add: handle(0x0032), sas_addr(0x500507681105c1fe)
[   14.493309] mpt3sas_cm0: port enable: SUCCESS

 

[jgreco]

24.00.01.01 Intel-provided firmware.

I don't know if that's the right firmware. For the 9300 series, it's supposed to be P16.00.12.00. I have a feeling you're on the wrong firmware but I don't know for certain what the correct firmware for the 9400's is.

 

[sunshine931]

I don't know if that's the right firmware. For the 9300 series, it's supposed to be P16.00.12.00. I have a feeling you're on the wrong firmware but I don't know for certain what the correct firmware for the 9400's is.

Are you suggesting it's the wrong firmware for the HBA, or that it's the wrong firmware for that HBA for TrueNAS?

The HBA is definitely a rebranded 9400-16e (it's stamped on the board itself), and it reports as an Intel RSP3GD016J. This is the firmware download page for the card, and the version is 24.x.

I should have included this output too - as you can see, storcli is happy -

Code:

root@truenas[~/util]# storcli show
CLI Version = 007.1504.0000.0000 June 22, 2020
Operating system = Linux 5.15.107+truenas
Status Code = 0
Status = Success
Description = None

Number of Controllers = 2
Host Name = truenas
Operating System  = Linux 5.15.107+truenas
StoreLib IT Version = 07.1503.0200.0000
StoreLib IR3 Version = 16.12-0

System Overview :
===============

--------------------------------------------------------------------------
Ctl Model          Ports PDs DGs DNOpt VDs VNOpt BBU sPR DS EHS ASOs Hlth
--------------------------------------------------------------------------
  0 ServeRAIDM5110     8   1   1     0   1     0 N/A On  -  Y      3 Opt
--------------------------------------------------------------------------


IT System Overview :
==================

-------------------------------------------------------------------------
Ctl Model      AdapterType   VendId DevId SubVendId SubDevId PCI Address
-------------------------------------------------------------------------
  1 RSP3GD016J   SAS3416(B0) 0x1000  0xAC    0x8086   0x3020 00:0c:00:00
-------------------------------------------------------------------------

Ctl=Controller Index|DGs=Drive groups|VDs=Virtual drives|Fld=Failed
PDs=Physical drives|DNOpt=Array NotOptimal|VNOpt=LD NotOptimal|Opt=Optimal
Msng=Missing|Dgd=Degraded|NdAtn=Need Attention|Unkwn=Unknown
sPR=Scheduled Patrol Read|DS=DimmerSwitch|EHS=Emergency Spare Drive
Y=Yes|N=No|ASOs=Advanced Software Options|BBU=Battery backup unit/CV
Hlth=Health|Safe=Safe-mode boot|CertProv-Certificate Provision mode



root@truenas[~/util]# storcli /c1 show all
CLI Version = 007.1504.0000.0000 June 22, 2020
Operating system = Linux 5.15.107+truenas
Controller = 1
Status = Success
Description = None


Basics :
======
Controller = 1
Adapter Type =   SAS3416(B0)
Model = RSP3GD016J
Serial Number = SP81900103
Current System Date/time = 06/15/2023 08:46:05
Concurrent commands supported = 6656
SAS Address =  500605b00dd93380
PCI Address = 00:0c:00:00
Mfg Date = 00/00/00
Rework Date = 00/00/00
Revision No = N/A


Version :
=======
Firmware Package Build = 24.00.01.01
Firmware Version = 24.00.01.00
Bios Version = _24.00.00.00
NVDATA Version = 24.00.16.31
PSOC Version = 05689001
Driver Name = mpt3sas
Driver Version = 39.100.00.00


PCI Version :
===========
Vendor Id = 0x1000
Device Id = 0xAC
SubVendor Id = 0x8086
SubDevice Id = 0x3020
Host Interface = PCIE
Device Interface = SAS-12G
Bus Number = 12
Device Number = 0
Function Number = 0
Domain ID = 0


Pending Images in Flash :
=======================
Image name = No pending images


Status :
======
Controller Status = OK
Memory Correctable Errors = 0
Memory Uncorrectable Errors = 0
Bios was not detected during boot = No
Controller has booted into safe mode = No
Controller has booted into certificate provision mode = No


Supported Adapter Operations :
============================
Alarm Control = No
Cluster Support = No
Self Diagnostic = No
Deny SCSI Passthrough = No
Deny SMP Passthrough = No
Deny STP Passthrough = No
Support more than 8 Phys = Yes
FW and Event Time in GMT = No
Support Enclosure Enumeration = Yes
Support Allowed Operations = Yes
Support Multipath = Yes
Support Security = No
Support Config Page Model = No
Support the OCE without adding drives = No
support EKM = No
Snapshot Enabled = No
Support PFK = No
Support PI = No
Support Shield State = No
Support Set Link Speed = No
Support JBOD = No
Disable Online PFK Change = No
Real Time Scheduler = No
Support Reset Now = No
Support Emulated Drives = No
Support Secure Boot = No
Support Platform Security = No


HwCfg :
=====
ChipRevision =  B0
BatteryFRU = N/A
Front End Port Count = 1
Backend Port Count = 21
Serial Debugger = Absent
NVRAM Size = 0KB
Flash Size = 16MB
On Board Memory Size = 0MB
On Board Expander = Absent
Temperature Sensor for ROC = Present
Temperature Sensor for Controller = Absent
Current Size of CacheCade (GB) = 0
Current Size of FW Cache (MB) = 0
ROC temperature(Degree Celcius) = 48


Policies :
========

Policies Table :
==============

------------------------------------------------
Policy                          Current Default
------------------------------------------------
Predictive Fail Poll Interval   0 sec
Interrupt Throttle Active Count 0
Interrupt Throttle Completion   0 us
Rebuild Rate                    0 %     30%
PR Rate                         0 %     30%
BGI Rate                        0 %     30%
Check Consistency Rate          0 %     30%
Reconstruction Rate             0 %     30%
Cache Flush Interval            0s
------------------------------------------------

Flush Time(Default) = 4s
Drive Coercion Mode = none
Auto Rebuild = Off
Battery Warning = Off
ECC Bucket Size = 0
ECC Bucket Leak Rate (hrs) = 0
Restore HotSpare on Insertion = Off
Expose Enclosure Devices = Off
Maintain PD Fail History = Off
Reorder Host Requests = On
Auto detect BackPlane = SGPIO/i2c SEP
Load Balance Mode = None
Security Key Assigned = Off
Disable Online Controller Reset = Off
Use drive activity for locate = Off


Boot :
====
Max Drives to Spinup at One Time = 0
Maximum number of direct attached drives to spin up in 1 min = 0
Delay Among Spinup Groups (sec) = 0
Allow Boot with Preserved Cache = On


Defaults :
========
Phy Polarity = 0
Phy PolaritySplit = 0
Cached IO = Off
Default spin down time (mins) = 0
Coercion Mode = None
ZCR Config = Unknown
Max Chained Enclosures = 0
Direct PD Mapping = No
Restore Hot Spare on Insertion = No
Expose Enclosure Devices = No
Maintain PD Fail History = No
Zero Based Enclosure Enumeration = No
Disable Puncturing = No
Un-Certified Hard Disk Drives = Block
SMART Mode = Mode 6
Enable LED Header = No
LED Show Drive Activity = No
Dirty LED Shows Drive Activity = No
EnableCrashDump = No
Disable Online Controller Reset = No
Treat Single span R1E as R10 = No
Power Saving option = Enable
TTY Log In Flash = No
Auto Enhanced Import = No
Enable Shield State = No
Time taken to detect CME = 60 sec


Capabilities :
============
Supported Drives = SAS, SATA
Enable JBOD = Yes
Max Parallel Commands = 6656
Max SGE Count = 128
Max Data Transfer Size = 32 sectors
Max Strips PerIO = 0
Max Configurable CacheCade Size = 0
Min Strip Size = 512Bytes
Max Strip Size = 512Bytes

Scheduled Tasks = NA

Security Protocol properties :
============================
Security Protocol = None

root@truenas[~/util]#

 

[jgreco]

Are you suggesting it's the wrong firmware for the HBA, or that it's the wrong firmware for that HBA for TrueNAS?

Don't know. Almost everyone here are cheapskates and we almost exclusively buy the 9200 or 9300's. I don't know if the 9400's have been sold by iXsystems and I have no idea what firmware they may have certified.

The HBA is definitely a rebranded 9400-16e (it's stamped on the board itself), and it reports as an Intel RSP3GD016J. This is the firmware download page for the card, and the version is 24.x.

Has nothing to do with nothing, other than it is one firmware version that will work on the board. I am guessing that this is showing up under the mpr driver, which we expect to use 16.00.12.00 with.

I should have included this output too - as you can see, storcli is happy -

Of course storcli is happy. If it can find a card, it's by definition happy. Big deal. Your problem is probably that the driver is probably expecting a phase 16 firmware. The firmware and the driver on these things are paired together and if there is a significant enough difference, may not even be able to talk at all -- which could be the problem you're seeing. You're on phase 24 and the driver is on phase 16. I would not even expect phase 17 firmware to be stable with a phase 16 driver; we actually know anything other than P16.00.12.00 (or possibly P16.00.10.00 if you don't need the fix introduced in subrev 12) will not work correctly. So you walk in with phase 24 and I basically go !!!???? Dunno!

 

[sunshine931]

Don't know. Almost everyone here are cheapskates and we almost exclusively buy the 9200 or 9300's. I don't know if the 9400's have been sold by iXsystems and I have no idea what firmware they may have certified.



Has nothing to do with nothing, other than it is one firmware version that will work on the board. I am guessing that this is showing up under the mpr driver, which we expect to use 16.00.12.00 with.



Of course storcli is happy. If it can find a card, it's by definition happy. Big deal. Your problem is probably that the driver is probably expecting a phase 16 firmware. The firmware and the driver on these things are paired together and if there is a significant enough difference, may not even be able to talk at all -- which could be the problem you're seeing. You're on phase 24 and the driver is on phase 16. I would not even expect phase 17 firmware to be stable with a phase 16 driver; we actually know anything other than P16.00.12.00 (or possibly P16.00.10.00 if you don't need the fix introduced in subrev 12) will not work correctly. So you walk in with phase 24 and I basically go !!!???? Dunno!

Ok gotcha - so there could be a firmware vs. driver mismatch. I had assumed that latest-greatest TrueNAS Scale would include the latest-greatest drivers.

lspci says:

Code:

root@truenas[~/util]# lspci| grep -i lsi
0c:00.0 Serial Attached SCSI controller: Broadcom / LSI SAS3416 Fusion-MPT Tri-Mode I/O Controller Chip (IOC) (rev 01)
1b:00.0 RAID bus controller: Broadcom / LSI MegaRAID SAS 2208 [Thunderbolt] (rev 05)

driver in use is:

Code:

root@truenas[~/util]# lspci -vvv -s 0c:00.0 | grep -i driver
        Kernel driver in use: mpt3sas

driver version is:

Code:

root@truenas[~/util]# modinfo mpt3sas | grep -i ^version:
version:        39.100.00.00

And that 24.x firmware is validated against driver version 43.00.00.00 - not 39.

Looking at my other options (Intel offers the older firmware too), here are the Firmware vs. Validated Driver options.

Firmware	Validated Driver
24.00.01.01 (Phase 24)	Linux 43.00.00.00
23.00.00.01 (Phase 23)	Linux 42.00.00.00
22.00.00.02 (Phase 22)	Linux 41.00.00.00
21.00.00.01 (Phase 21)	Linux 40.00.00.00
18.00.01.03 (Phase 18)	Linux 37.00.00.00
14.00.02.02 (Phase 14)	Linux 33.00.01.00

Unfortunately, there is no Intel-branded firmware that lines up with driver 39.x

I assume I shouldn't try to update TrueNAS Scale to use driver version 43...

I'm happy to try Phase 21 and Phase 18 firmware to see if they behave differently, since they're the closest option I have to driver version 39.. but do you have a recommendation for a firmware version? Should I seek-out non-Intel firmware that is validated specifically for driver version 39?

Thanks, appreciate your insight into this.

 

[jgreco]

Ok gotcha - so there could be a firmware vs. driver mismatch. I had assumed that latest-greatest TrueNAS Scale would include the latest-greatest drivers.

This may be a brain-o on your part. The goal of TrueNAS isn't to have the latest-greatest of anything, but rather something that works as close to 100% reliably as is possible. There is no reason to think that the "latest-greatest" firmware or driver is the most appropriate or error-free thing, and it looks like what happened is that the 9300 firmware development came to a grinding halt at some point (16.00.10.00 to be exact) and that the 9400 series stuff kept releasing new firmwares. Without spending a bunch of time looking into it, I have no idea what these new versions brought to the game, but the point is that introducing the P17 {, P18, P19,... P24} compatible drivers would have broken compatibility with the widely used and completely sufficient 9300 series controllers. So what you propose ("latest-greatest") is a suggestion for unnecessary brokenness for many sites, whereas your controller can probably be fixed by installing the correct firmware.

And that 24.x firmware is validated against driver version 43.00.00.00 - not 39.

Yeah, dunno. I come from the BSD world, where LSI wrote the driver too and the driver and firmware are generally phased together. There were a few exceptions, but typically you ran the phase 16 (for ex.) driver with the phase 16 firmware. Linux as usual does its own thing for reasons I am not aware of. Possibly because it became too difficult to reconcile with the firmware phase...?

I'm happy to try Phase 21 and Phase 18 firmware to see if they behave differently, since they're the closest option I have to driver version 39.. but do you have a recommendation for a firmware version? Should I seek-out non-Intel firmware that is validated specifically for driver version 39?

It's unclear. If we assume that the phase numbering for 9300 and 9400 are different, then what you suggest makes sense, and it may even be worth it to look up the LSI firmware for your equivalent card and see if LSI can hint you into a more specific phase validated for driver 39.

Either way, I'm guessing that as you get closer to a compatible phase firmware, your card will suddenly "light up". I just don't have any of the newest cards because I am plenty happy with my cheapie $80 9300-based cards. My own view is that SAS is dying and I'm just not willing to invest much in it.

 

[sunshine931]

It's unclear. If we assume that the phase numbering for 9300 and 9400 are different, then what you suggest makes sense, and it may even be worth it to look up the LSI firmware for your equivalent card and see if LSI can hint you into a more specific phase validated for driver 39.

Makes sense, thanks. I've found the 9400-16e firmware (direct from LSI), and Phase 20 matches-up with driver version 39. I'll flash my HBA with that firmware and report back.

 

[jgreco]

Makes sense, thanks. I've found the 9400-16e firmware (direct from LSI), and Phase 20 matches-up with driver version 39. I'll flash my HBA with that firmware and report back.

Fantastic. Inquiring minds want to know! If I can get clarity on the correct firmware, I'll be happy to add it to the HBA article.

 

[sunshine931]

OK I have some results to share.

I've tested both Phase 20 firmware (which should work with driver version 39) and Phase 16 firmware, and continue to have issues with sas3ircu (same error as with Phase 24 firmware). Storcli continues to works just fine, with all versions of firmware tested.

I've only been flashing the card with UEFI firmware, no legacy BIOS firmware. I don't think that'd be my problem, but for the sake of being thorough I'm going to flash my card with both UEFI and BIOS firmware to see.

 

[jgreco]

I've only been flashing the card with UEFI firmware, no legacy BIOS firmware. I don't think that'd be my problem, but for the sake of being thorough I'm going to flash my card with both UEFI and BIOS firmware to see.

Would be shocked if that was the issue here.

So let's rewind a bit. I've re-read your original message again and want to comment a bit.

You have a 9400-16e which should be based on the SAS3416 RoC. These tokens, along with "16" for phase 16, are the things I was skimming for when I looked at your setup, and because you were on Linux, I looked to make sure that this was supposed to show up as a FreeBSD-equivalent to the "mpr" driver which supports it.

Code:

     These controllers are supported by the mpr driver:

     ⢠  Broadcom Ltd./Avago Tech (LSI) SAS 3004 (4 Port SAS)
     ⢠  Broadcom Ltd./Avago Tech (LSI) SAS 3008 (8 Port SAS)
     ⢠  Broadcom Ltd./Avago Tech (LSI) SAS 3108 (8 Port SAS)
     ⢠  Broadcom Ltd./Avago Tech (LSI) SAS 3216 (16 Port SAS)
     ⢠  Broadcom Ltd./Avago Tech (LSI) SAS 3224 (24 Port SAS)
     ⢠  Broadcom Ltd./Avago Tech (LSI) SAS 3316 (16 Port SAS)
     ⢠  Broadcom Ltd./Avago Tech (LSI) SAS 3324 (24 Port SAS)
     ⢠  Broadcom Ltd./Avago Tech (LSI) SAS 3408 (8 Port SAS/PCIe)
     ⢠  Broadcom Ltd./Avago Tech (LSI) SAS 3416 (16 Port SAS/PCIe)
     ⢠  Broadcom Ltd./Avago Tech (LSI) SAS 3508 (8 Port SAS/PCIe)
     ⢠  Broadcom Ltd./Avago Tech (LSI) SAS 3516 (16 Port SAS/PCIe)
     ⢠  Broadcom Ltd./Avago Tech (LSI) SAS 3616 (16 Port SAS/PCIe)
     ⢠  Broadcom Ltd./Avago Tech (LSI) SAS 3708 (8 Port SAS/PCIe)
     ⢠  Broadcom Ltd./Avago Tech (LSI) SAS 3716 (16 Port SAS/PCIe)

So I walked into this expecting a Linux equivalent to the FreeBSD mpr HBA driver. Indeed, I saw some stuff in your output such as

Code:

-------------------------------------------------------------------------
Ctl Model      AdapterType   VendId DevId SubVendId SubDevId PCI Address
-------------------------------------------------------------------------
  1 RSP3GD016J   SAS3416(B0) 0x1000  0xAC    0x8086   0x3020 00:0c:00:00
-------------------------------------------------------------------------

which again made me think "okay, 3416 HBA".

But I wonder if this is misleading. In theory storcli does not work with HBA's, but we knew LSI was working on rebuilding the mrsas (FreeBSD name) driver to be a unified driver for both RAID and HBA. So I kinda ignored where it said

Code:

IT System Overview :
==================

-------------------------------------------------------------------------
Ctl Model      AdapterType   VendId DevId SubVendId SubDevId PCI Address
-------------------------------------------------------------------------
  1 RSP3GD016J   SAS3416(B0) 0x1000  0xAC    0x8086   0x3020 00:0c:00:00
-------------------------------------------------------------------------

Because I saw the "IT System" and figured this was probably cool, it's probably integrated the support for the newer card into your Linux-equivalent mrsas driver. But this also represents a big question mark. I should probably have recognized this and referred you to the Linux mpt3sas driver (which is I think the Linux equivalent to the mrsas).

I still think I was right in not expecting mpt3ircu not to work, if the card has "mrsas" compatible firmware running on it. That would also not work on FreeBSD on an mrsas firmware card.

But I'm a bit blocked as to where to go from here, and I apologize if I went off the tracks and wasted any of your time here.

My real concern is this: where did you get this card from? Is it a known legitimate source? There are fake cards running around on the used market, and I fear very much I might have spent a lot of time wasting your time debugging a "firmware" issue when instead maybe we should have been wondering if the card is even legit.

Fake server cards

This resource was originally created by user: jgreco on the TrueNAS Community Forums Archive. Please DM this account or comment in this thread to claim it. Some time ago, forum frequenter @artlessknave kindly sent me a pair of LSI HBA cards that had failed to work out in a FreeNAS build. Having...

www.truenas.com

 

[Ericloewe]

I'm building a lab with an IBM x3650 M4.

Don't skip the system firmware updates. They fscked up one or more voltage regulators in early firmware, leading to dead motherboards after a few years of use (a certain number of reboots?).

I can access the drives just fine (and am in the middle of removing Type 2 protection from them), but I can't seem to use sas3ircu.

You might have a sas3ircu problem rather than a firmware/driver problem. Is there a newer version of sas3ircu on Broadcom's website?

 

[Ericloewe]

But I wonder if this is misleading. In theory storcli does not work with HBA's, but we knew LSI was working on rebuilding the mrsas (FreeBSD name) driver to be a unified driver for both RAID and HBA.

I don't think that's what happened. My understanding is that the RAID stack team decided that mfi was terrible and started from scratch with a more sane, less early-2000s RAID design (mrsas), which also included support for direct-attach disks. This was apparently pioneered on at least some SAS 2.5 products. Now they're getting rid of the two separate teams (and starting almost from scratch again) with a new unified driver for 9600-series cards.

 

[sunshine931]

Don't skip the system firmware updates. They fscked up one or more voltage regulators in early firmware, leading to dead motherboards after a few years of use (a certain number of reboots?).

Yes, I began by applying the latest firmware for everything in the system from IBM/ Lenovo :)

You might have a sas3ircu problem rather than a firmware/driver problem. Is there a newer version of sas3ircu on Broadcom's website?

I'm testing now with a fresh Ubuntu install.

 

[sunshine931]

My real concern is this: where did you get this card from? Is it a known legitimate source? There are fake cards running around on the used market, and I fear very much I might have spent a lot of time wasting your time debugging a "firmware" issue when instead maybe we should have been wondering if the card is even legit.

Like many here, I'm on a budget and shop online. I try to source my parts only from reputable ebay sellers, located in North America, who have a history of selling refurbished server infra (ie - they're getting gear from datacenters as they refresh their hardware).

I believe this is a genuine Intel card - everything about it just "feels right", having handled many a card in my day. Nothing sketchy about it, and I've examined it after watching the linked "spot a fake" video.

I'm contacting the seller to confirm the history on this card just to be safe..

 

[sunshine931]

I've completed testing with a fresh Ubuntu 22.04, and same issue.. storcli works, sas3ircu does not.

I'm starting to think sas3ircu is not intended for this HBA - when I search LSI's site for utilities for it, I get only storcli.

 

[jgreco]

I'm starting to think sas3ircu is not intended for this HBA - when I search LSI's site for utilities for it, I get only storcli.

Well, I think that's probably the case. It is quite possible that the driver was unified on Linux but still runs the mpr (dedicated HBA) on FreeBSD, or maybe the mpr manpage is just wrong on FreeBSD. Or maybe there used to be mpr firmware for the card. Unfortunately, none of that is particularly helpful to you since we're speculating. I still get the feeling something may be wrong with the card. Are you keeping it nice and cool? LSI cards are notorious for needing the airflow.

 

[nKk]

I get one 9400-8i Tri-Mode SAS HBA for new project running on FreeNAS Core.
I check and mpr driver is used but "sas3ircu" can't found HBA.
"storcli" show the HBA, enclosure and connected disks.

On my other TrueNAS Core systems I have old HBA's and "sas3ircu" fond them but storcli can't.

I think that for Tri-Mode HBA's only "storcli" can working with them and "sas3ircu" support old generations that "storcli" can't.

 

[Ericloewe]

I'm starting to think sas3ircu is not intended for this HBA - when I search LSI's site for utilities for it, I get only storcli.

I think that's what's going on - they possibly dropped support for it in later firmware/driver versions than we typically see in 93xx cards. I'd be curious to know if it's a controller thing, a driver thing or a firmware thing.

Linux mpt3sas driver (which is I think the Linux equivalent to the mrsas).

It's actually equivalent to mpr. Linux has a megaraid_sas driver which I think is broadly equivalent to mrsas, but implements the mfi interface as well for older controllers, instead of relegating them to the legacy mfi driver.

 

[sunshine931]

Well, I think that's probably the case. It is quite possible that the driver was unified on Linux but still runs the mpr (dedicated HBA) on FreeBSD, or maybe the mpr manpage is just wrong on FreeBSD. Or maybe there used to be mpr firmware for the card. Unfortunately, none of that is particularly helpful to you since we're speculating. I still get the feeling something may be wrong with the card. Are you keeping it nice and cool? LSI cards are notorious for needing the airflow.

I've confirmed with the seller that these were working pulls, from Intel-branded servers (and this is an Intel-braded HBA), and that he believes based on his source for the hardware that there's no question that it's genuine Intel gear.

It's in an IBM x3550 M4 with lots of cooling, so I don't think it's overheating.

Given the other person's reply re: also only being able to use storcli, I'm going to conclude that this is a case of "use storcli, dummy", and will do that.

I've decided to flash to the latest firmware provided by Intel, phase 24 firmware. So far everything seems happy, ZFS build/ stress-testing planned for next week.

Thanks everyone for your input, and hopefully this thread helps someone in the future.