SOLVED Promise J630s multipath issues - Multiple HBA = no bueno

Joined
Dec 29, 2014
Messages
1,135
I am having some issues with a new (to me) Promise J630s external JBOD array. It is working, but I am not getting multipath despite having two physical connections to this external array. Here is my system:

FreeNAS- 11.3-U5
Cisco C240 M4SX - 24 x 2.5" SAS/SATA drive bays
Dual E5-2637 v3 @ 3.50GHz
256 GB ECC DRAM
2 x Intel i350 Gigabit NIC for management (using 2 ports in LACP channel)
Chelsio T580-CR dual port 40 Gigabit NIC for storage network (using 1 port)
LSI 3108 based HBA
LSI 9207-8E connected to Promise J630s external enclosure - 16 x 3.5" SAS/SATA bays
Storage pool = 2 RAIDZ2 vdevs of 8 x 3TB HITACHI HUS723030ALS640 SAS drives
Intel Optane 900P SLOG (shared device, separate partition per pool)
LSI 9300-8E connected to HP D2600 external enclosure - 12 x 3.5" SAS/SATA bays
Storage pool 2 (D2600) = 2 RAIDZ2 vdevs of 6 x 2TB 7.2k SAS drives with one spare drive (internal)
Storage pool 3 (internal bays) = 2 RAIDZ2 vdevs of 8 x 1TB 7.2k SATA drives with one spare drive
Intel Optane 900P SLOG (shared device, separate partition per pool)
Boot pool = mirrored 120G SATA SSD drives

I am having two issues, but the most important one is that I am seeing the two ses devices for the J630s, but not multipath device for the disks.
Code:
ses0: <Promise J630s 060=> Fixed Enclosure Services SPC-4 SCSI device
ses1: <Promise J630s 060=> Fixed Enclosure Services SPC-4 SCSI device


I know the 9207-8E can do multipath because it sees those when I originally had D2600 attached to it.

I may have another issue because I don't see the 2nd ses device for the D2600 when it is attached to the 9300-8E. I am trying to retire the HP arrays, so my intent is to leave that attached to the 9207 because both are 6G only. I think those drives are HGST multipath capable, but I guess I am not 100% on that either.
 
Joined
Dec 29, 2014
Messages
1,135
I have some more poking around, and noticed some even more peculiar stuff. The Promise J630s (has a Symantec label on it) doesn't appear to do multipath as best I can tell. The HP D2600 that did multipath correctly when connected to the 9207-8e doesn't do it correctly when connected to the 9300-8e. I did some google and forum searching for ways to get FreeNAS to rebuild the multipath configuration, but I didn't find anything that seemed relevant. I did verify that the HP D2600 would work properly on either single cable from the 9300-8e, but still no multipath. Is this do to the fact that the drives are now on HBA "mpr1" instead of "mps0"? The devices are below:
Code:
mps0: <Avago Technologies (LSI) SAS2308> port 0x6000-0x60ff mem 0xc7040000-0xc704ffff,0xc7000000-0xc703ffff irq 32 at device 0.0 numa-domain 0 on pci4
mps0: Firmware: 20.00.07.00, Driver: 21.02.00.00-fbsd
mps0: IOCCapabilities: 5a85c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,EventReplay,MSIXIndex,HostDisc>
mpr0: <Avago Technologies (LSI) SAS3108_1> port 0x5000-0x50ff mem 0xc6e40000-0xc6e4ffff,0xc6e00000-0xc6e3ffff irq 34 at device 0.0 numa-domain 0 on pci5
mpr0: Firmware: 13.00.08.00, Driver: 18.03.00.00-fbsd
mpr0: IOCCapabilities: 7a85c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,EventReplay,MSIXIndex,HostDisc,FastPath,RDPQArray>
mpr1: <Avago Technologies (LSI) SAS3008> port 0xf000-0xf0ff mem 0xfbe40000-0xfbe4ffff,0xfbe00000-0xfbe3ffff irq 58 at device 0.0 numa-domain 1 on pci15
mpr1: Firmware: 16.00.01.00, Driver: 18.03.00.00-fbsd
mpr1: IOCCapabilities: 7a85c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,EventReplay,MSIXIndex,HostDisc,FastPath,RDPQArray>

mpr0 is in the HBA for the internal bays.
mpr1 is the 9300-8e.
mps0 is the 9207-8e.
 
Joined
Dec 29, 2014
Messages
1,135
This is resolved, but I really don't understand why it didn't work. My fix was to put the D2600 back on the 9207-8e where the multipath devices had been originally created, and move the J630s to the 9300-8e. I moved the data to a different pool, destroyed the old pool, deleted the old multipath devices with gmultipath destroy, and lastly a gmultipath unload. Once all that was completed, I disconnected the D2600 array. The moment I did that, multipath devices were automatically built for at the drives in the J630s. Is there some kind of restriction that multipath can only be on a single HBA? If not, why didn't it build those multipath devices when I connected the 9300-8e to the J630s. My problem is resolved, but I am still really confused. :-(
 
Top