PCIe LSI2008 won't show x8 link, stuck at x4

[lowlytech]

Have Truenas 12.0-u8 running on a gigabyte B550M DS3H Rev 1.1 mainboard /w Ryzen 5600x CPU installed. I have no video card installed but in the 1 x16 slot I installed a LSI 2008 hard drive controller and for the life of me I can't get it to move to x8 lanes. I have a Chelsio t422 NIC in the other PCI x4 slot and a bootable nvme. Otherwise I don't have any other hardware installed. Is it typical that this 2008 card doesn't achieve 8 lanes but 4?

Code:

mps0@pci0:10:0:0:       class=0x010400 card=0x11771734 chip=0x00721000 rev=0x03 hdr=0x00
    vendor     = 'Broadcom / LSI'
    device     = 'SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon]'
    class      = mass storage
    subclass   = RAID
    cap 01[50] = powerspec 3  supports D0 D1 D2 D3  current D0
    cap 10[68] = PCI-Express 2 endpoint max data 512(4096) FLR RO NS
                 link x4(x8) speed 5.0(5.0) ASPM disabled(L0s)
    cap 03[d0] = VPD
    cap 05[a8] = MSI supports 1 message, 64 bit
    cap 11[c0] = MSI-X supports 15 messages, enabled
                 Table in map 0x14[0x2000], PBA in map 0x14[0x3800]
    ecap 0001[100] = AER 1 0 fatal 0 non-fatal 0 corrected
    ecap 0004[138] = Power Budgeting 1
    ecap 0010[150] = SR-IOV 1 IOV disabled, Memory Space disabled, ARI disabled
                     0 VFs configured out of 16 supported
                     First VF RID Offset 0x0001, VF RID Stride 0x0001
                     VF Device ID 0x0072
                     Page Sizes: 4096 (enabled), 8192, 65536, 262144, 1048576, 4194304
    ecap 000e[190] = ARI 1

 

[Patrick M. Hausen]

No idea about your last question, but - does your mainboard support bifurcation and if yes, did you check it's not enabled in the BIOS setup?

 

[lowlytech]

Yes bifurcation is an option and it was on auto. I went thru the different modes, but it didn't seem to make a difference.

 

[Patrick M. Hausen]

So "off" is one of the modes and even that did not change anything? Then I'm out of experience and ideas, sorry.

 

[lowlytech]

I don't remember off being an option. I will look again tonight when i get home. Seemed there was auto, then about 4 options for 4x4, 8x2, etc. I will report back and let you know.

 

[Arwen]

This page indicates that your system board has 2 x PCIe x16 physical slots, but one is x16 electrical and the other is x4 electrical;

B550M DS3H (rev. 1.0/1.1/1.2/1.3) Specification | Motherboard - GIGABYTE Global

Lasting Quality from GIGABYTE.GIGABYTE Ultra Durable™ motherboards bring together a unique blend of features and technologies that offer users the absolute ultimate platform for...

www.gigabyte.com

Looks like the x16 electrical is the x16 physical slot closest to the CPU, (it's labeled PCIEX16).

 

[rvassar]

^^^ This...

I have one of these myself. The B550 chipset lies about the x16 PCIe slots. Generally, you get 1 x16 and 1 x4 that looks like a x16... It's kind of a throwback to the days of SLI links between GPU's.

 

[lowlytech]

I finally had a chance to pull the server down. the 2008 controller is definitely in the full 16x slot and not the 4. Going through all the bifurcation options doesn't seem to do anything. I pulled the NVME and Chelsio 422 card just to help ease any PCIe bandwidth or conflicts, and one setting made the 2008 card disappear altogether, otherwise I get that card reading in 2x and 4x but never 8x. I reseated the controller as well. Thinking I may update the BIOS since I am a few versions behind.

 

[rvassar]

I finally had a chance to pull the server down. the 2008 controller is definitely in the full 16x slot and not the 4. Going through all the bifurcation options doesn't seem to do anything. I pulled the NVME and Chelsio 422 card just to help ease any PCIe bandwidth or conflicts, and one setting made the 2008 card disappear altogether, otherwise I get that card reading in 2x and 4x but never 8x. I reseated the controller as well. Thinking I may update the BIOS since I am a few versions behind.

Ok. I can agree with your steps... But... I wouldn't waste too much time here. The mfg. admits in docs they second x16 is actually a x4... So they're playing lane games from the start. Who know's what negotiation steps are playing out in the BIOS as the board comes up? Run gamer HW, get gamer HW problems. And I know that's not the answer you wanted to hear, but... That's the world we live in.

 

[lowlytech]

Yeah when I found out they cheaped out on the 2nd x16 slot to only include pins for x4 I wasn't too happy.

Updated BIOS with newest image, and it didn't change a thing. Took the LSI card out and cleaned the pcie fingers really good and still just 4x link no matter what combination of options I pick. Figure PCIe 2.0 x4 isn't gonna hurt me too bad running 8 4TB WD gold drives is it? RaidZ2 with 2 vdevs.

 

[nKk]

There is a different controllers based on SAS2008. What exactly model do you have? Because some of them can be x4 only.

 

[lowlytech]

I am not exactly sure. I pulled the card and took a picture of both sides if that helps..

 

[jgreco]

There is a different controllers based on SAS2008. What exactly model do you have? Because some of them can be x4 only.

Cards like the 9211-4i that are x4-only typically have the PCIe slot connector to match.

 

[rvassar]

That's a 9220-8i, see the sticker in the second pic, upper left. It should present a PCIe 2.0 x8 interface, which is pretty old. The B550 chipset has a bunch of caveats (I'm actually typing this on a B550-Plus now...) with regards to the x1 slots and the fake x16... If the x1's are occupied the fake x16 becomes a x2, etc... That doesn't seem to fit the described behavior, so... Maybe your board is having trouble negotiating down to v2 speeds? Is it running IT mode firmware? Maybe that's a problem. Maybe the controller is damaged and that's the best it can do because of SI management? Does your BIOS offer any slot negotiation settings? It might just decide to remap the lanes due to the slow v2.0 speeds.

Some stats to ponder... PCIe 2.0 spec is 500MB/sec per lane, so X4 will turn 2000MB/sec. So if you running spinning rust, that's somewhere around 8 HDD's at peak load, maybe a bit less for 7200 RPM drives, etc... So if you're under 8 disks, and not running SSD's, you may not notice. The "why" however is unanswered in that calculus.

 

[lowlytech]

I dug deeper and found out that the option rom section of the motherboard was set to UEFI only. Since this SAS2008 is pre UEFI or the FW that is on it is, it would never show any boot up messages. I went into the card and confirmed the firmware is 20.00.07.00-IT but saw no options on pci link speed or anything similar. Still shows a x4 link. To keep everything the way it was I reenabled the option rom in BIOS back to UEFI.