10 Gig Network card - not recognized on another server with the same OS

[4BlueGentoos]

I am trying to use a BCM57840 Network card – and failing.
It is a 4 port 10G SFP+ NIC, with a PCIe 3.0x8 interface.

I have two servers, with the exact same version of TrueNAS Scale installed.
But the card is only detected on one of them.

Show : The Problem Server (A):

• OS:---------TrueNAS-SCALE-23.10-RC.1
• Model:-----Dell Precision T3500-------Standard Motherboard for this machine
• RAM:-------48 GB DDR3---------------(Twice the quoted max, but it works)
• CPU:--------6c12t @ 3.47GHz----------Intel(R) Xeon(R) CPU X5690
• Ethernet:---Onboard----(1x) 1 GbE----(enp6s0)

----- PCI(e) Devices: -----

• GPU:--------Nvidia Quadro 2000
• NIC1:-------PCIe --------(2x) 2.5GbE---(enp7s0, enp8s0)
• NIC2:-------PCI 32/33---(1x) 1 GbE----(enp9s4)
• NIC3:-------PCI 32/33---(1x) 1 GbE----(enp9s5)
• NIC4:-------PCIe 3.0 X8-(4x) 10 GbE---(?, ?, ?, ?)
• -------------Quad- SFP+ Port, with Broadcom BCM57840 Controller

Show : The working server (B):

• OS:---------TrueNAS-SCALE-23.10-RC.1
• Model:-----Custom Built
• Motherboard: ---------------------------ASRock Z690 Steel Legend/D5
• RAM: ------64 GB DDR5
• CPU: -------6c12t @ 3.30GHz ----------12th Gen Intel(R) Core(TM) i5-12600
• Ethernet:---Onboard----(1x) 1 GbE----(enp12s0)
• ----- PCI(e) Devices: -----
• GPU:--------Nvidia RTX 3060 Ti--------8GB GDDR6
• NIC1:-------PCIe---------(4x) 1 GbE----(enp7s0, enp8s0, enp9s0, enp10s0)
• NIC2:-------PCIe 3.0 X8--(4x) 10 GbE--(enp4s0f0, enp4s0f1, enp4s0f2, enp4s0f3)
• -------------Quad- SFP+ Port, with Broadcom BCM57840 Controller
• NIC3:-------PCIe 3.0 X8--(4x) 10 GbE--(enp13s0f0, enp13s0f1, enp13s0f2, enp13s0f3)
• -------------Quad- SFP+ Port, with Broadcom BCM57840 Controller

----------------------------------------------------------------------------------------------------

So far I have:

• Swapped all 3 cards around - in the working Server (B)
  • All three cards work, no matter what slot.
    • So all three NIC’s are good cards.
  • And we confirm that TrueNAS can, in-fact, load the card drivers perfectly fine.
    • So, there is no problem between the TrueNAS Scale OS and the cards.
• Checked each card - in each available PCIe slot, on the problem Server (A).
  • The BIOS can detect all cards, in all slots.
  • It reports it as an ethernet device with 4 ports.
    • So there is no problem between the BIOS and the cards.
• I have installed many versions of TrueNAS Scale (on both machines):
  • TrueNAS-SCALE-22.12.1
  • TrueNAS-SCALE-22.12.3.2
  • TrueNAS-SCALE-23.10-RC.1

----------------------------------------------------------------------------------------------------

The result is always the same:

• The working server (B):
  • can always read the card in the BIOS
  • Always shows the 4 ports on each card as available interfaces in the Web GUI
    • Immediately after installation, and without any extra effort.
• The problem server (A):
  • Can always read the card in the BIOS
  • But never shows any of the 4 ports in the Web GUI.
  • Always shows up with lspci -v:

02:00.0 Ethernet controller: Broadcom Inc. and subsidiaries BCM57840 NetXtreme II 10 Gigabit Ethernet (rev 11)
Subsystem: Broadcom Inc. and subsidiaries BCM57840 NetXtreme II 10 Gigabit Ethernet
Flags: fast devsel, IRQ 30, IOMMU group 12
Expansion ROM at ec100000 [disabled] [size=512K]
Capabilities: [48] Power Management version 3
Capabilities: [50] Vital Product Data
Capabilities: [a0] MSI-X: Enable- Count=32 Masked-
Capabilities: [ac] Express Endpoint, MSI 00
Capabilities: [100] Advanced Error Reporting
Capabilities: [13c] Device Serial Number 6c-b3-11-ff-fe-5e-9b-10
Capabilities: [150] Power Budgeting <?>
Capabilities: [160] Virtual Channel
Capabilities: [1b8] Alternative Routing-ID Interpretation (ARI)
Capabilities: [1c0] Single Root I/O Virtualization (SR-IOV)
Capabilities: [220] Physical Resizable BAR
Capabilities: [210] Address Translation Service (ATS)
Capabilities: [300] Secondary PCI Express
Kernel modules: bnx2x
02:00.1 Ethernet controller: Broadcom …. Same thing ....
02:00.2 Ethernet controller: Broadcom …. Same thing ....
02:00.3 Ethernet controller: Broadcom …. Same thing ....

• Additionally:
  • Always seems to throw errors in dmesg

root@t3500[~]# dmesg | grep bnx2x
[ 1.888778] bnx2x 0000:02:00.0: msix capability found
[ 1.889444] bnx2x 0000:02:00.0: Cannot find PCI device base address, aborting
[ 1.893441] bnx2x 0000:02:00.1: msix capability found
[ 1.894134] bnx2x 0000:02:00.1: Cannot find PCI device base address, aborting
[ 1.896196] bnx2x 0000:02:00.2: msix capability found
[ 1.898447] bnx2x 0000:02:00.2: Cannot find PCI device base address, aborting
[ 1.900586] bnx2x 0000:02:00.3: msix capability found
[ 1.902821] bnx2x 0000:02:00.3: Cannot find PCI device base address, aborting

----------------------------------------------------------------------------------------------------

I did find on the Dell website that:

“On February 18, 2014, Broadcom and QLogic announced the sale and purchase of certain Ethernet Controller-related assets and entered into an ASIC partnership. As part of the sale, the following Broadcom NetXtreme II(NX2)® controllers*, related adapters and software were sold to QLogic. Broadcom retains all other controller products (e.g NetXtreme).

As of April 7, 2015, Dell has implemented changes to reflect the current ownership. Formerly-Broadcom NX2 adapters are now branded as QLogic. The only change to the products is the labeling. The Dell part numbers, SKUs, and PCI Device IDs are the same as when they were branded Broadcom.

On the Dell support website, related drivers, firmware, and documentation can be recognized by the presence of “QLogic BCM57xx and BCM57xxx” in the download title.

*Applies exclusively to 5706, 5708, 5709, 5716, 57710, 57711, 57712, 57800, 57810 and 57840”

So any help I can get on this Network Card will need to come from Dell, as Broadcom no longer owns it...

Anyway, A little searching took me to this driver: [QLogic_LAN_17.12c.4.2_Linux_Source_19.2.0.11.tar.gz] which I have been trying to install, but TrueNAS does not seem to be very friendly about letting me install a driver from source…
----------------------------------------------------------------------------------------------------

First of all, apparently wget doesn't seem to work for getting this file from Dell.. I get a 403 permission error.

Ultimately, I was able to get it by:

1. Logging into an ubuntu machine on my network
2. Downloading it from a web browser
3. scp that file over to my TrueNAS scale machine.

It wasn't much help, because the installer only has instructions for how to install on SuSE and Redhat.

I tried to push forward anyway, attempting to do a make install , which of course did not work because make is not a recognized command in TrueNAS. **eyeroll**

Then I tried to install make, which I tried to do with ./configure, and of course there is no cc or gcc compiler on TrueNAS either....

So at this point, I feel I may have deviated a bit far from the "TrueNAS" way, because I just caught myself trying to install several new package - just to make one small thing work.

Any advice or direction would be much appreciated.

----------------------------------------------------------------------------------------------------

This post is somewhat related, regarding a BCM57416 - it pointed me towards a driver called bxe(4) on the freeBSD website. But this doesn't seem right to me, since TrueNAS Scale is Debian based.

 

[asap2go]

"Checked each card - in each available PCIe slot, on the problem Server (A).

• The BIOS can detect all cards, in all slots."

With all other cards(e.g. GPU) installed at the same time?
Did you try removing everything except the NIC in question and then booting into TrueNAS?