LSI SAS9300-8e stopped working after upgrading from 11.2-U7 to 11.3-RELEASE

[badammans]

I'm running a Dell R710 with X5570 CPUs, LSI SAS9300-8e SAS expanders, QLogic QLE2672 Eth/FCoE cards, and 72GB DDR3 ECC RAM.

Under 11.2 I wasn't able to use the QLogic Eth cards because it didn't have the driver, but the SAS expanders worked great. After the upgrade to 11.3-RELEASE today I can use the Eth cards, but the SAS expanders seem to do nothing. Were the drivers removed for these cards?

 

[amp88]

I've had no problems with a similar card (9207-8e) after upgrading from 11.2-U7 to 11.3. What output do you get when you run the following command in the shell:

Code:

lspci | grep LSI

In my system (with the 9207-8e), I get the following:

Code:

root@freenas[~]# lspci | grep LSI
00:06.0 Serial Attached SCSI controller: Broadcom / LSI SAS2308 PCI-Express Fusion-MPT SAS-2 (rev 05)

If you don't get any output from the above command, it's possible your system has a problem detecting the card (which could be for any number of reasons). You can try running the following command to get more information:

Code:

dmesg | grep mps

This should return information about your card. Here are the first couple of lines from my system:

Code:

root@freenas[~]# dmesg | grep mps
mps0: <Avago Technologies (LSI) SAS2308> port 0xc100-0xc1ff mem 0xf3180000-0xf318ffff,0xf3100000-0xf313ffff irq 40 at device 6.0 on pci0
mps0: Firmware: 20.00.07.00, Driver: 21.02.00.00-fbsd

 

[badammans]

Thank you amp88. Strangely enough the issue was that it didn't like having two of the same SAS card in the same system. Right before I upgraded I had put a second one in so I wouldn't have to daisy chain my two shelves. When I removed the second one, the first was recognized and all drives are connected. This issue is resolved (I guess). I don't know if it is FreeNAS or the server that doesn't like two cards.

 

[amp88]

Thank you amp88. Strangely enough the issue was that it didn't like having two of the same SAS card in the same system. Right before I upgraded I had put a second one in so I wouldn't have to daisy chain my two shelves. When I removed the second one, the first was recognized and all drives are connected. This issue is resolved (I guess). I don't know if it is FreeNAS or the server that doesn't like two cards.

Hmm, I'm glad you managed to find a workaround, but that's a less than ideal solution. My suspicion would be that the limitation is on the server side, because as far as I'm aware there are many FreeNAS systems running happily with multiple cards of the same type (I believe you can even make a storage pool containing drives from multiple different controllers).

It might be worth putting the second card back in and going into the R710's BIOS to see what it detects (i.e. no cards, both cards, or only one of the two cards). If the R710's BIOS detects both cards then it could be a FreeNAS/FreeBSD issue (in which case you might want to try booting a Linux distro from a USB drive and running the lspci | grep LSI command from its terminal to see what output you get).

 

[badammans]

Ubuntu can see both cards just fine at the same time:
04:00.0 Serial Attached SCSI controller: LSI Logic / Symbios Logic SAS3008 PCI-Express Fusion-MPT SAS-3 (rev 02)
05:00.0 Ethernet controller: QLogic Corp. ISP8324 1/10GbE Converged Network Controller (rev 02)
05:00.1 Ethernet controller: QLogic Corp. ISP8324 1/10GbE Converged Network Controller (rev 02)
06:00.0 Ethernet controller: QLogic Corp. ISP8324 1/10GbE Converged Network Controller (rev 02)
06:00.1 Ethernet controller: QLogic Corp. ISP8324 1/10GbE Converged Network Controller (rev 02)
07:00.0 Serial Attached SCSI controller: LSI Logic / Symbios Logic SAS3008 PCI-Express Fusion-MPT SAS-3 (rev 02)

In ubuntu I can use them and see all attached drives.

In FreeNAS it still shows both, though I still cannot see the drives:
04:00.0 Serial Attached SCSI controller: Broadcom / LSI SAS3008 PCI-Express Fusion-MPT SAS-3 (rev 02)
05:00.0 Ethernet controller: QLogic Corp. ISP8324 1/10GbE Converged Network Controller (rev 02)
05:00.1 Ethernet controller: QLogic Corp. ISP8324 1/10GbE Converged Network Controller (rev 02)
06:00.0 Ethernet controller: QLogic Corp. ISP8324 1/10GbE Converged Network Controller (rev 02)
06:00.1 Ethernet controller: QLogic Corp. ISP8324 1/10GbE Converged Network Controller (rev 02)
07:00.0 Serial Attached SCSI controller: Broadcom / LSI SAS3008 PCI-Express Fusion-MPT SAS-3 (rev 02)

However, I moved the freenas boot drive and SAS card into a different dell server (R420) and it worked and saw all drives.

Moved it back to the original server, but with only one of the SAS cards and I still cannot see the drives, though I do see the SAS card:
root@freenas[~]# lspci |grep SAS
03:00.0 RAID bus controller: Broadcom / LSI MegaRAID SAS 2108 [Liberator] (rev 05)
04:00.0 Serial Attached SCSI controller: Broadcom / LSI SAS3008 PCI-Express Fusion-MPT SAS-3 (rev 02)

Not sure why this is happening. Though it isn't the only anomoly I've been seeing with FreeNAS 11.3-STABLE. I also have configured some Eth interfaces, and at least in the GUI it doesn't show it correctly. I'm was hoping there is only an issue with the GUI itself, and on the backend it would be fine.

Here is what I see for the drives in the backend:
root@freenas[~]# geom disk list |grep Geom
Geom name: mfid0
Geom name: mfid1
Geom name: mfid2
Geom name: mfid3
Geom name: mfid4
Geom name: mfid5
Geom name: mfid6
Geom name: mfid7
Geom name: cd0
Geom name: cd1
Geom name: da0
Geom name: da1


The Drives labeled mfidX are from the built-in H700 RAID controller with the drives setup as individual RAID 0's so that I can have the ZFS redundancy on them.

I want to say this is only an issue with the Server hardware, but since it all works under Ubuntu I have to assume this is an issue with FreeNAS drivers. I also have a Mellenox3 IB/40Gb Eth controller that is recognized by both OSes, but neither OS has drivers for it.

That said, FreeNAS 11.3-STABLE did load the driver for this card in a different server and the drives were recognized, so that makes me suspect this to be an incompatibility between the PowerEdge R710 and the FreeBSD OS. My only option is to choose a different server for this config. I've been wanting to buy a new one anyway, just didn't want to have to spend the money yet.

 

[stoffix]

I just updated the FW on my 9300-8i and read about a sas-id or something that has to be different for multiple cards to work together in the same computer. Since I only have one card, and didn’t wipe the firmware first (didn’t crossflash) I didn’t mind. Maybe it’s related to your issue?