LSI 9300-8e not seeing disks in MD1400

[rniedzial]

Long time lurker, first time poster. I've got a problem with my LSI-9300-8e seeings disks attached to a Dell PowerVault MD1400. The LSI is flashed w/ IT firmware and running inside of a Dell R730. I also have a Dell PERC H810 in the R730 and I've confirmed that it does see the drives when in iDRAC (however because this is a RAID card they don't show up in the OS, which I believe is the expected behavior).

I've tried 3 separate pairs of cables now, w/ the latest ones being a pair of 10Gtek 12G External Mini SAS HD SFF-8644 to SFF-8644. So I think I got it narrowed down to either user error or this LSI card having issues. I got it off ebay so ...

I guess my question is, should my setup work? :) Here is output from a few of the troubleshooting commands:

I'm booted into Ubuntu Server 23.04 right now. I have a bunch of SSD disks in there and some NVME drives attached directly to the R730, no other drives show up.

Code:

sudo lsblk
NAME                      MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS
loop0                       7:0    0    73M  1 loop /snap/core22/607
loop1                       7:1    0   163M  1 loop /snap/lxd/24643
loop2                       7:2    0 164.3M  1 loop /snap/lxd/24758
loop3                       7:3    0  49.8M  1 loop /snap/snapd/18596
loop4                       7:4    0  53.2M  1 loop /snap/snapd/18933
sda                         8:0    0 894.3G  0 disk
├─sda1                      8:1    0     1G  0 part /boot/efi
├─sda2                      8:2    0     2G  0 part /boot
└─sda3                      8:3    0 891.2G  0 part
  └─ubuntu--vg-ubuntu--lv 253:0    0   100G  0 lvm  /
sdb                         8:16   0 931.5G  0 disk
sdc                         8:32   0 931.5G  0 disk
sdd                         8:48   0 931.5G  0 disk
sde                         8:64   0 931.5G  0 disk
sdf                         8:80   0 931.5G  0 disk
sdg                         8:96   0 931.5G  0 disk
sdh                         8:112  0 931.5G  0 disk
sdi                         8:128  0 931.5G  0 disk
sdj                         8:144  0 931.5G  0 disk
sdk                         8:160  0 931.5G  0 disk
sdl                         8:176  0 931.5G  0 disk
sr0                        11:0    1  1024M  0 rom
nvme0n1                   259:0    0 931.5G  0 disk
├─nvme0n1p1               259:1    0  1022M  0 part
├─nvme0n1p2               259:2    0     4G  0 part
├─nvme0n1p3               259:3    0 922.5G  0 part
└─nvme0n1p4               259:4    0     4G  0 part
nvme1n1                   259:5    0 931.5G  0 disk
└─nvme1n1p1               259:6    0 931.5G  0 part
nvme2n1                   259:7    0 931.5G  0 disk
└─nvme2n1p1               259:9    0 931.5G  0 part
nvme3n1                   259:8    0 931.5G  0 disk
└─nvme3n1p1               259:10   0 931.5G  0 part

Flashed to latest 16.00.12 firmware from Truenas forums.

Code:

sudo sas3flash -list
Avago Technologies SAS3 Flash Utility
Version 16.00.00.00 (2017.05.02)
Copyright 2008-2017 Avago Technologies. All rights reserved.

    Adapter Selected is a Avago SAS: SAS3008(C0)

    Controller Number              : 0
    Controller                     : SAS3008(C0)
    PCI Address                    : 00:82:00:00
    SAS Address                    : 500605b-0-0388-bdb0
    NVDATA Version (Default)       : 0e.01.00.08
    NVDATA Version (Persistent)    : 0e.01.00.08
    Firmware Product ID            : 0x2221 (IT)
    Firmware Version               : 16.00.12.00
    NVDATA Vendor                  : LSI
    NVDATA Product ID              : SAS9300-8e
    BIOS Version                   : 08.29.00.00
    UEFI BSD Version               : N/A
    FCODE Version                  : N/A
    Board Name                     : SAS9300-8e
    Board Assembly                 : H3-25460-02B
    Board Tracer Number            : SL301P1372

    Finished Processing Commands Successfully.
    Exiting SAS3Flash.

Code:

sudo sas3ircu 0 display
Avago Technologies SAS3 IR Configuration Utility.
Version 17.00.00.00 (2018.04.02)
Copyright (c) 2009-2018 Avago Technologies. All rights reserved.

Read configuration has been initiated for controller 0
------------------------------------------------------------------------
Controller information
------------------------------------------------------------------------
  Controller type                         : SAS3008
  BIOS version                            : 8.29.00.00
  Firmware version                        : 16.00.12.00
  Channel description                     : 1 Serial Attached SCSI
  Initiator ID                            : 0
  Maximum physical devices                : 1023
  Concurrent commands supported           : 9664
  Slot                                    : 5
  Segment                                 : 0
  Bus                                     : 130
  Device                                  : 0
  Function                                : 0
  RAID Support                            : No
------------------------------------------------------------------------
IR Volume information
------------------------------------------------------------------------
------------------------------------------------------------------------
Physical device information
------------------------------------------------------------------------
------------------------------------------------------------------------
Enclosure information
------------------------------------------------------------------------
  Enclosure#                              : 1
  Logical ID                              : 500605b0:0388bdb0
  Numslots                                : 8
  StartSlot                               : 0
------------------------------------------------------------------------
SAS3IRCU: Command DISPLAY Completed Successfully.
SAS3IRCU: Utility Completed Successfully.

Code:

sudo dmesg | grep mpt3
[    9.012033] mpt3sas version 43.100.00.00 loaded
[    9.019273] mpt3sas_cm0: 63 BIT PCI BUS DMA ADDRESSING SUPPORTED, total mem (396117172 kB)
[    9.083772] mpt3sas_cm0: CurrentHostPageSize is 0: Setting default host page size to 4k
[    9.083808] mpt3sas_cm0: MSI-X vectors supported: 96
[    9.083819] mpt3sas_cm0:  0 56 56
[    9.085185] mpt3sas_cm0: High IOPs queues : disabled
[    9.085190] mpt3sas0-msix0: PCI-MSI-X enabled: IRQ 229
[    9.085193] mpt3sas0-msix1: PCI-MSI-X enabled: IRQ 230
[    9.085195] mpt3sas0-msix2: PCI-MSI-X enabled: IRQ 231
[    9.085198] mpt3sas0-msix3: PCI-MSI-X enabled: IRQ 232
[    9.085200] mpt3sas0-msix4: PCI-MSI-X enabled: IRQ 233
[    9.085202] mpt3sas0-msix5: PCI-MSI-X enabled: IRQ 234
[    9.085205] mpt3sas0-msix6: PCI-MSI-X enabled: IRQ 235
[    9.085207] mpt3sas0-msix7: PCI-MSI-X enabled: IRQ 236
[    9.085210] mpt3sas0-msix8: PCI-MSI-X enabled: IRQ 237
[    9.085212] mpt3sas0-msix9: PCI-MSI-X enabled: IRQ 238
[    9.085214] mpt3sas0-msix10: PCI-MSI-X enabled: IRQ 239
[    9.085217] mpt3sas0-msix11: PCI-MSI-X enabled: IRQ 240
[    9.085219] mpt3sas0-msix12: PCI-MSI-X enabled: IRQ 241
[    9.085222] mpt3sas0-msix13: PCI-MSI-X enabled: IRQ 242
[    9.085224] mpt3sas0-msix14: PCI-MSI-X enabled: IRQ 243
[    9.085227] mpt3sas0-msix15: PCI-MSI-X enabled: IRQ 244
[    9.085229] mpt3sas0-msix16: PCI-MSI-X enabled: IRQ 245
[    9.085231] mpt3sas0-msix17: PCI-MSI-X enabled: IRQ 246
[    9.085233] mpt3sas0-msix18: PCI-MSI-X enabled: IRQ 247
[    9.085236] mpt3sas0-msix19: PCI-MSI-X enabled: IRQ 248
[    9.085238] mpt3sas0-msix20: PCI-MSI-X enabled: IRQ 249
[    9.085240] mpt3sas0-msix21: PCI-MSI-X enabled: IRQ 250
[    9.085242] mpt3sas0-msix22: PCI-MSI-X enabled: IRQ 251
[    9.085244] mpt3sas0-msix23: PCI-MSI-X enabled: IRQ 252
[    9.085247] mpt3sas0-msix24: PCI-MSI-X enabled: IRQ 253
[    9.085249] mpt3sas0-msix25: PCI-MSI-X enabled: IRQ 254
[    9.085251] mpt3sas0-msix26: PCI-MSI-X enabled: IRQ 255
[    9.085253] mpt3sas0-msix27: PCI-MSI-X enabled: IRQ 256
[    9.085256] mpt3sas0-msix28: PCI-MSI-X enabled: IRQ 257
[    9.085258] mpt3sas0-msix29: PCI-MSI-X enabled: IRQ 258
[    9.085260] mpt3sas0-msix30: PCI-MSI-X enabled: IRQ 259
[    9.085263] mpt3sas0-msix31: PCI-MSI-X enabled: IRQ 260
[    9.085265] mpt3sas0-msix32: PCI-MSI-X enabled: IRQ 261
[    9.085267] mpt3sas0-msix33: PCI-MSI-X enabled: IRQ 262
[    9.085269] mpt3sas0-msix34: PCI-MSI-X enabled: IRQ 263
[    9.085272] mpt3sas0-msix35: PCI-MSI-X enabled: IRQ 264
[    9.085274] mpt3sas0-msix36: PCI-MSI-X enabled: IRQ 265
[    9.085276] mpt3sas0-msix37: PCI-MSI-X enabled: IRQ 266
[    9.085278] mpt3sas0-msix38: PCI-MSI-X enabled: IRQ 267
[    9.085281] mpt3sas0-msix39: PCI-MSI-X enabled: IRQ 268
[    9.085283] mpt3sas0-msix40: PCI-MSI-X enabled: IRQ 269
[    9.085285] mpt3sas0-msix41: PCI-MSI-X enabled: IRQ 270
[    9.085287] mpt3sas0-msix42: PCI-MSI-X enabled: IRQ 271
[    9.085289] mpt3sas0-msix43: PCI-MSI-X enabled: IRQ 272
[    9.085292] mpt3sas0-msix44: PCI-MSI-X enabled: IRQ 273
[    9.085294] mpt3sas0-msix45: PCI-MSI-X enabled: IRQ 274
[    9.085296] mpt3sas0-msix46: PCI-MSI-X enabled: IRQ 275
[    9.085298] mpt3sas0-msix47: PCI-MSI-X enabled: IRQ 276
[    9.085300] mpt3sas0-msix48: PCI-MSI-X enabled: IRQ 277
[    9.085302] mpt3sas0-msix49: PCI-MSI-X enabled: IRQ 278
[    9.085305] mpt3sas0-msix50: PCI-MSI-X enabled: IRQ 279
[    9.085307] mpt3sas0-msix51: PCI-MSI-X enabled: IRQ 280
[    9.085309] mpt3sas0-msix52: PCI-MSI-X enabled: IRQ 281
[    9.085311] mpt3sas0-msix53: PCI-MSI-X enabled: IRQ 282
[    9.085314] mpt3sas0-msix54: PCI-MSI-X enabled: IRQ 283
[    9.085316] mpt3sas0-msix55: PCI-MSI-X enabled: IRQ 284
[    9.085318] mpt3sas_cm0: iomem(0x00000000c8140000), mapped(0x00000000f036b9dd), size(65536)
[    9.085324] mpt3sas_cm0: ioport(0x0000000000009000), size(256)
[    9.142647] mpt3sas_cm0: CurrentHostPageSize is 0: Setting default host page size to 4k
[    9.170460] mpt3sas_cm0: scatter gather: sge_in_main_msg(1), sge_per_chain(7), sge_per_io(128), chains_per_io(19)
[    9.176216] mpt3sas_cm0: request pool(0x00000000484c9321) - dma(0x30b2800000): depth(9756), frame_size(128), pool_size(1219 kB)
[    9.371305] mpt3sas_cm0: sense pool(0x00000000852af475) - dma(0x30b4800000): depth(9567), element_size(96), pool_size (896 kB)
[    9.371339] mpt3sas_cm0: sense pool(0x00000000852af475)- dma(0x30b4800000): depth(9567),element_size(96), pool_size(0 kB)
[    9.371943] mpt3sas_cm0: reply pool(0x0000000088b641ca) - dma(0x30b2a00000): depth(9820), frame_size(128), pool_size(1227 kB)
[    9.371981] mpt3sas_cm0: config page(0x0000000000e3b207) - dma(0x30b47c9000): size(512)
[    9.371996] mpt3sas_cm0: Allocated physical memory: size(35896 kB)
[    9.372006] mpt3sas_cm0: Current Controller Queue Depth(9564),Max Controller Queue Depth(9664)
[    9.372019] mpt3sas_cm0: Scatter Gather Elements per IO(128)
[    9.542673] mpt3sas_cm0: _base_display_fwpkg_version: complete
[    9.544018] mpt3sas_cm0: LSISAS3008: FWVersion(16.00.12.00), ChipRevision(0x02), BiosVersion(08.29.00.00)
[    9.544060] mpt3sas_cm0: Protocol=(Initiator,Target), Capabilities=(TLR,EEDP,Snapshot Buffer,Diag Trace Buffer,Task Set Full,NCQ)
[    9.552690] mpt3sas_cm0: sending port enable !!
[   12.097128] mpt3sas_cm0: hba_port entry: 00000000fa42a608, port: 255 is added to hba_port list
[   12.098336] mpt3sas_cm0: host_add: handle(0x0001), sas_addr(0x500605b00388bdb0), phys(8)
[   17.229185] mpt3sas_cm0: port enable: SUCCESS
rafal@730:~$
sudo shutdown -h now

Broadcast message from root@730 on pts/1 (Mon 2023-04-24 00:25:21 UTC):

The system will power off now!

Connection to 192.168.0.70 closed by remote host.
Connection to 192.168.0.70 closed.
 rniedzialkowski ~/Downloads/17.05.06.00_MSM_Linux-x64 took 12m42s  at 2023-04-23 17:25 ❯❯
ssh rafal@192.168.0.70
rafal@192.168.0.70's password:
rafal@730:~$
sudo dmesg | grep mpt3
[sudo] password for rafal:
[    8.864653] mpt3sas version 43.100.00.00 loaded
[    8.871923] mpt3sas_cm0: 63 BIT PCI BUS DMA ADDRESSING SUPPORTED, total mem (396117116 kB)
[    8.941169] mpt3sas_cm0: CurrentHostPageSize is 0: Setting default host page size to 4k
[    8.941195] mpt3sas_cm0: MSI-X vectors supported: 96
[    8.941200] mpt3sas_cm0:  0 56 56
[    8.942502] mpt3sas_cm0: High IOPs queues : disabled
[    8.942507] mpt3sas0-msix0: PCI-MSI-X enabled: IRQ 229
[    8.942509] mpt3sas0-msix1: PCI-MSI-X enabled: IRQ 230
[    8.942512] mpt3sas0-msix2: PCI-MSI-X enabled: IRQ 231
[    8.942514] mpt3sas0-msix3: PCI-MSI-X enabled: IRQ 232
[    8.942517] mpt3sas0-msix4: PCI-MSI-X enabled: IRQ 233
[    8.942519] mpt3sas0-msix5: PCI-MSI-X enabled: IRQ 234
[    8.942521] mpt3sas0-msix6: PCI-MSI-X enabled: IRQ 235
[    8.942524] mpt3sas0-msix7: PCI-MSI-X enabled: IRQ 236
[    8.942526] mpt3sas0-msix8: PCI-MSI-X enabled: IRQ 237
[    8.942529] mpt3sas0-msix9: PCI-MSI-X enabled: IRQ 238
[    8.942531] mpt3sas0-msix10: PCI-MSI-X enabled: IRQ 239
[    8.942533] mpt3sas0-msix11: PCI-MSI-X enabled: IRQ 240
[    8.942536] mpt3sas0-msix12: PCI-MSI-X enabled: IRQ 241
[    8.942538] mpt3sas0-msix13: PCI-MSI-X enabled: IRQ 242
[    8.942541] mpt3sas0-msix14: PCI-MSI-X enabled: IRQ 243
[    8.942543] mpt3sas0-msix15: PCI-MSI-X enabled: IRQ 244
[    8.942545] mpt3sas0-msix16: PCI-MSI-X enabled: IRQ 245
[    8.942547] mpt3sas0-msix17: PCI-MSI-X enabled: IRQ 246
[    8.942549] mpt3sas0-msix18: PCI-MSI-X enabled: IRQ 247
[    8.942552] mpt3sas0-msix19: PCI-MSI-X enabled: IRQ 248
[    8.942554] mpt3sas0-msix20: PCI-MSI-X enabled: IRQ 249
[    8.942556] mpt3sas0-msix21: PCI-MSI-X enabled: IRQ 250
[    8.942558] mpt3sas0-msix22: PCI-MSI-X enabled: IRQ 251
[    8.942560] mpt3sas0-msix23: PCI-MSI-X enabled: IRQ 252
[    8.942562] mpt3sas0-msix24: PCI-MSI-X enabled: IRQ 253
[    8.942565] mpt3sas0-msix25: PCI-MSI-X enabled: IRQ 254
[    8.942567] mpt3sas0-msix26: PCI-MSI-X enabled: IRQ 255
[    8.942569] mpt3sas0-msix27: PCI-MSI-X enabled: IRQ 256
[    8.942572] mpt3sas0-msix28: PCI-MSI-X enabled: IRQ 257
[    8.942574] mpt3sas0-msix29: PCI-MSI-X enabled: IRQ 258
[    8.942576] mpt3sas0-msix30: PCI-MSI-X enabled: IRQ 259
[    8.942578] mpt3sas0-msix31: PCI-MSI-X enabled: IRQ 260
[    8.942581] mpt3sas0-msix32: PCI-MSI-X enabled: IRQ 261
[    8.942583] mpt3sas0-msix33: PCI-MSI-X enabled: IRQ 262
[    8.942585] mpt3sas0-msix34: PCI-MSI-X enabled: IRQ 263
[    8.942587] mpt3sas0-msix35: PCI-MSI-X enabled: IRQ 264
[    8.942589] mpt3sas0-msix36: PCI-MSI-X enabled: IRQ 265
[    8.942592] mpt3sas0-msix37: PCI-MSI-X enabled: IRQ 266
[    8.942594] mpt3sas0-msix38: PCI-MSI-X enabled: IRQ 267
[    8.942596] mpt3sas0-msix39: PCI-MSI-X enabled: IRQ 268
[    8.942598] mpt3sas0-msix40: PCI-MSI-X enabled: IRQ 269
[    8.942600] mpt3sas0-msix41: PCI-MSI-X enabled: IRQ 270
[    8.942603] mpt3sas0-msix42: PCI-MSI-X enabled: IRQ 271
[    8.942605] mpt3sas0-msix43: PCI-MSI-X enabled: IRQ 272
[    8.942607] mpt3sas0-msix44: PCI-MSI-X enabled: IRQ 273
[    8.942609] mpt3sas0-msix45: PCI-MSI-X enabled: IRQ 274
[    8.942611] mpt3sas0-msix46: PCI-MSI-X enabled: IRQ 275
[    8.942613] mpt3sas0-msix47: PCI-MSI-X enabled: IRQ 276
[    8.942616] mpt3sas0-msix48: PCI-MSI-X enabled: IRQ 277
[    8.942618] mpt3sas0-msix49: PCI-MSI-X enabled: IRQ 278
[    8.942620] mpt3sas0-msix50: PCI-MSI-X enabled: IRQ 279
[    8.942622] mpt3sas0-msix51: PCI-MSI-X enabled: IRQ 280
[    8.942624] mpt3sas0-msix52: PCI-MSI-X enabled: IRQ 281
[    8.942627] mpt3sas0-msix53: PCI-MSI-X enabled: IRQ 282
[    8.942629] mpt3sas0-msix54: PCI-MSI-X enabled: IRQ 283
[    8.942631] mpt3sas0-msix55: PCI-MSI-X enabled: IRQ 284
[    8.942633] mpt3sas_cm0: iomem(0x00000000c8140000), mapped(0x0000000074bf9f23), size(65536)
[    8.942639] mpt3sas_cm0: ioport(0x0000000000009000), size(256)
[    9.000397] mpt3sas_cm0: CurrentHostPageSize is 0: Setting default host page size to 4k
[    9.027780] mpt3sas_cm0: scatter gather: sge_in_main_msg(1), sge_per_chain(7), sge_per_io(128), chains_per_io(19)
[    9.033290] mpt3sas_cm0: request pool(0x00000000956d16c8) - dma(0x3096600000): depth(9756), frame_size(128), pool_size(1219 kB)
[    9.198079] mpt3sas_cm0: sense pool(0x0000000003e3e945) - dma(0x309a500000): depth(9567), element_size(96), pool_size (896 kB)
[    9.198109] mpt3sas_cm0: sense pool(0x0000000003e3e945)- dma(0x309a500000): depth(9567),element_size(96), pool_size(0 kB)
[    9.198671] mpt3sas_cm0: reply pool(0x000000001e26e595) - dma(0x3098800000): depth(9820), frame_size(128), pool_size(1227 kB)
[    9.198708] mpt3sas_cm0: config page(0x000000008d557e8b) - dma(0x309a4d7000): size(512)
[    9.198721] mpt3sas_cm0: Allocated physical memory: size(35896 kB)
[    9.198731] mpt3sas_cm0: Current Controller Queue Depth(9564),Max Controller Queue Depth(9664)
[    9.198743] mpt3sas_cm0: Scatter Gather Elements per IO(128)
[    9.368903] mpt3sas_cm0: _base_display_fwpkg_version: complete
[    9.370609] mpt3sas_cm0: LSISAS3008: FWVersion(16.00.12.00), ChipRevision(0x02), BiosVersion(08.29.00.00)
[    9.370646] mpt3sas_cm0: Protocol=(Initiator,Target), Capabilities=(TLR,EEDP,Snapshot Buffer,Diag Trace Buffer,Task Set Full,NCQ)
[    9.379081] mpt3sas_cm0: sending port enable !!
[   11.940275] mpt3sas_cm0: hba_port entry: 0000000083cf4c8a, port: 255 is added to hba_port list
[   11.941518] mpt3sas_cm0: host_add: handle(0x0001), sas_addr(0x500605b00388bdb0), phys(8)
[   17.078205] mpt3sas_cm0: port enable: SUCCESS

​

 

[rniedzial]

I should also mention that I'm running two SFF-8644 cables from the LSI 9300-8E to the first port on the MD1400 controllers.

 

[rniedzial]

I should also mention that I'm running two SFF-8644 cables from the LSI 9300-8E to the first port on the MD1400 controllers.

Thought it might be relevant to mention the drives in the MD1400 are white label HGST SAS drives. (HUS724040ALS640 4 TB SAS Internal 7200RPM 0B26885)

 

[HoneyBadger]

Hello @rniedzial

I can't recall the exact Dell models, but I believe the MD1400 doesn't have any manner of internal RAID logic to it at all, so it shouldn't be presenting any problems that way. Hopefully Dell hasn't done anything like require a specific PERC card to interface with what looks to be a standard SAS shelf.

Have you tried connecting only a single cable to the topmost controller in the MD1400?

 

[jgreco]

I can't recall the exact Dell models, but I believe the MD1400 doesn't have any manner of internal RAID logic to it at all, so it shouldn't be presenting any problems that way. Hopefully Dell hasn't done anything like require a specific PERC card to interface with what looks to be a standard SAS shelf.

I dunno about that. The 2X93X enclosure controller that is one of the options for the MD1400 is listed as having "4GB cache" in a number of places. Consider me skeptical.

 

[Ericloewe]

I've always seen them referred to as "direct-attach", and typically lumped in with the MD1200. Could the cache thing be just nonsense spilling over from the Host RAID controller side?

Is the firmware on the MD1400 up to date?

 

[jgreco]

Could the cache thing be just nonsense spilling over from the Host RAID controller side?

It could, but typically the Dell PERC's have either 1G, 2G, or 8G of cache -- not 4G.

In any case, I didn't find a comprehensive list of what the options for that chassis are, so the point here is that the OP should probably double- and triple-check that the shelf's controller doesn't have any sort of RAID config or caching. If it has any sort of configuration port (ethernet, serial, USB, etc) there is a chance that it is some sort of controller that may not be safe with ZFS. Especially if it has cache. Or support for creating virtual drives.

 

[rniedzial]

Hello @rniedzial

I can't recall the exact Dell models, but I believe the MD1400 doesn't have any manner of internal RAID logic to it at all, so it shouldn't be presenting any problems that way. Hopefully Dell hasn't done anything like require a specific PERC card to interface with what looks to be a standard SAS shelf.

Have you tried connecting only a single cable to the topmost controller in the MD1400?

Success! The enclosure is recognized as well as the disks. Are you not supposed to have both controllers connected to the HBA card????

Code:

sudo sas3ircu 0 display
Avago Technologies SAS3 IR Configuration Utility.
Version 17.00.00.00 (2018.04.02)
Copyright (c) 2009-2018 Avago Technologies. All rights reserved.

Read configuration has been initiated for controller 0
------------------------------------------------------------------------
Controller information
------------------------------------------------------------------------
  Controller type                         : SAS3008
  BIOS version                            : 8.29.00.00
  Firmware version                        : 16.00.12.00
  Channel description                     : 1 Serial Attached SCSI
  Initiator ID                            : 0
  Maximum physical devices                : 1023
  Concurrent commands supported           : 9664
  Slot                                    : 5
  Segment                                 : 0
  Bus                                     : 130
  Device                                  : 0
  Function                                : 0
  RAID Support                            : No
------------------------------------------------------------------------
IR Volume information
------------------------------------------------------------------------
------------------------------------------------------------------------
Physical device information
------------------------------------------------------------------------
Initiator at ID #0

Device is a Hard disk
  Enclosure #                             : 2
  Slot #                                  : 4
  SAS Address                             : 5000c50-0-93ba-5ad5
  State                                   : Ready (RDY)
  Size (in MB)/(in sectors)               : 3815447/7814037167
  Manufacturer                            : HP
  Model Number                            : MB4000JVYZQ
  Firmware Revision                       : HPD1
  Serial No                               : ZC11LJ9S
  Unit Serial No(VPD)                     : ZC11LJ9S0000C732QUKY
  GUID                                    : 5000c50093ba5ad7
  Protocol                                : SAS
  Drive Type                              : SAS_HDD

Device is a Hard disk
  Enclosure #                             : 2
  Slot #                                  : 8
  SAS Address                             : 5000cca-0-5ccf-b0ad
  State                                   : Ready (RDY)
  Size (in MB)/(in sectors)               : 3815447/7814037167
  Manufacturer                            : HGST
  Model Number                            : HUS724040ALS640
  Firmware Revision                       : A1C4
  Serial No                               : PCKP6WLX
  Unit Serial No(VPD)                     : PCKP6WLX
  GUID                                    : 5000cca05ccfb0ac
  Protocol                                : SAS
  Drive Type                              : SAS_HDD

Device is a Enclosure services device
  Enclosure #                             : 2
  Slot #                                  : 12
  SAS Address                             : 5204747-2-9a17-2c7d
  State                                   : Standby (SBY)
  Manufacturer                            : DELL
  Model Number                            : MD1400
  Firmware Revision                       : 1.07
  Serial No                               :
  Unit Serial No(VPD)                     : N/A
  GUID                                    : N/A
  Protocol                                : SAS
  Device Type                             : Enclosure services device
------------------------------------------------------------------------
Enclosure information
------------------------------------------------------------------------
  Enclosure#                              : 1
  Logical ID                              : 500605b0:0388bdb0
  Numslots                                : 8
  StartSlot                               : 0
  Enclosure#                              : 2
  Logical ID                              : 52047472:9a172c00
  Numslots                                : 13
  StartSlot                               : 0
------------------------------------------------------------------------
SAS3IRCU: Command DISPLAY Completed Successfully.
SAS3IRCU: Utility Completed Successfully.

 

[WI_Hedgehog]

The Dell PowerVault MD1400 and MD1420 direct-attached storage has a RAID option as a PERC card. Using a LSI-9300-8e HBA in IT mode (confirmed in @rniedzial's post) should make it direct-attach as @Ericloewe mentioned, though I cannot confirm that.

Here are the purchase options.

 

[Ericloewe]

Are you not supposed to have both controllers connected to the HBA card

The whole dual controller thing makes for a lot of weirdness. There may be a different cable configuration that does work.

 

[jgreco]

Are you not supposed to have both controllers connected to the HBA card????

Generally, no. You should have two separate controllers. SAS devices have relatively simplistic ideas about topology and you need to remember that the intention is to provide two separate paths for redundancy. Because SAS identifies devices by WWN, I would be a bit concerned that a single controller would see paths to a target via two different ports, and I'm not sure exactly how that's expected to work. SAS topologies are typically religiously split into the primary and secondary roles, and I don't know if much is done to allow a SAS controller to function as both a primary and secondary initiator. It's on that list of things I probably knew at one time.

 

[Ericloewe]

To make things more complicated, Dell allows for splitting the chassis into halves run independently by each controller, supposedly.

 

[jgreco]

To make things more complicated, Dell allows for splitting the chassis into halves run independently by each controller, supposedly.

If it's a standard SAS expander, in theory you can do a variety of configuration tricks that assign devices, not just half-and-half. I have only seen weird expander tricks done a few times though.

 

[HoneyBadger]

Success! The enclosure is recognized as well as the disks. Are you not supposed to have both controllers connected to the HBA card????

Glad that got it working. It might have tried to negotiate some manner of SAS wide-porting (x8 lanes) that the MD didn't want to speak. Running a single controller is recommended though as SAS multipath introduces some additional challenges.

To make things more complicated, Dell allows for splitting the chassis into halves run independently by each controller, supposedly.

I don't think the MD1400 series supports that any longer - split-mode was on the MD1200 SAS controllers though, and you had to force it to "unified" or "single host" mode.

 

[rniedzial]

UPDATE

So you guys are going to love this, somewhat embarrassing. Turns out it was user error all along. You can have both cables plugged into the controllers.

1. Since the LSI 9300-8e has a full size bracket its technically installed upside down
2. So when I was plugging in the cables I had them plugged in w/ the pullout tabs facing up, because that made sense :) And they fit that way, and provide a nice click indicating they are in
3. Last night when I was messing around w/ it I had the server up on a table and somehow plugged it in w/ the pull out tabs on the SAS cables facing down (which matches the way the card is inserted in the server).
4. Powered on the server this morning and was looking to test w/ two cables and I looked at the back and it hit me, OMG the single cable I tested last night is upside down!!!!!
5. Plugged in the 2nd cable the same way, and presto, it still works. Granted the devices are reported twice and I have some more testing to do but it matches what the Dell documentation says this enclosure can do in what they call (redundant configuration: https://www.dell.com/support/manual...25884d-9f4b-481f-af91-2bf89a1c9d48&lang=en-us )

Thank you for your help guys!

 

[WI_Hedgehog]

It's not that difficult to plug SFF-8644 in upside-down. Now that someone other than me has admitted to doing it this has been added to the Resources List.

 

[jgreco]

It's not that difficult to plug SFF-8644 in upside-down.

Wow, that's a stupid design flaw I never noticed.

 

[WI_Hedgehog]

The tab on the bottom of the SFF-8644 port mates with a tapered guide slot on the connector; in theory it shouldn't go in wrong though in practice it can and for me has, much to my surprise.


(click image to enlarge)

 

[Ericloewe]

That was unexpected. At least USB Type A and Type B have the decency of clearly not fitting if they're oriented incorrectly.

 

[HoneyBadger]

UPDATE

So you guys are going to love this, somewhat embarrassing. Turns out it was user error all along. You can have both cables plugged into the controllers.

1. Since the LSI 9300-8e has a full size bracket its technically installed upside down
2. So when I was plugging in the cables I had them plugged in w/ the pullout tabs facing up, because that made sense :) And they fit that way, and provide a nice click indicating they are in
3. Last night when I was messing around w/ it I had the server up on a table and somehow plugged it in w/ the pull out tabs on the SAS cables facing down (which matches the way the card is inserted in the server).
4. Powered on the server this morning and was looking to test w/ two cables and I looked at the back and it hit me, OMG the single cable I tested last night is upside down!!!!!
5. Plugged in the 2nd cable the same way, and presto, it still works. Granted the devices are reported twice and I have some more testing to do but it matches what the Dell documentation says this enclosure can do in what they call (redundant configuration: https://www.dell.com/support/manual...25884d-9f4b-481f-af91-2bf89a1c9d48&lang=en-us )

Thank you for your help guys!

We appreciate you coming back to update us!

If you're seeing the devices twice, you've got a SAS multipath configuration. We don't suggest this for a single-controller head unit; it's used for HA setups with TrueNAS Enterprise.

Are your cables connected from the HBA to each of the "top" and "bottom" enclosure controllers in the MD1400? Do you get a correct setup with all drives detected once (single-path) if you connect both cables to the top/primary controller in the MD1400?