No success with three different 10 Gb NICs

[mtlynch]

I'm new to 10 Gig Ethernet, and I'm having some trouble getting it working on my TrueNAS Core 13 system.

• Motherboard: ASUS Prime A320I-K
• CPU: AMD Athlon 3000G
• TrueNAS Version: TrueNAS-13.0-U5.1

Attempt #1: MELLANOX CONNECTX-3 EN CX311A

The first NIC I tried was actually the most successful. It was the only one that TrueNAS seemed to recognize:

Code:

# dmesg | grep -i mlx
mlx4_core0: Unable to determine PCI device chain minimum BW
mlx4_en mlx4_core0: Activating port:1
mlxen0: Ethernet address: 24:8a:07:ea:22:10
mlx4_en: mlx4_core0: Port 1: Using 4 TX rings
mlx4_en: mlx4_core0: Port 1: Using 4 RX rings
mlxen0: link state changed to DOWN
mlx4_en: mlxen0: Using 4 TX rings
mlx4_en: mlxen0: Using 4 RX rings
mlx4_en: mlxen0: Initializing port

The NIC showed up as a network interface in TrueNAS under Network > Interfaces, but Link State remained "DOWN." I tried enabling it or turning it on with ifconfig, but nothing worked.

I tried upgrading the BIOS to the latest version on my ASUS Prime motherboard, but the results were the same.

I connected the Mellanox NIC to a Windows machine and re-flashed the firmware, but when I plugged it back in to the TrueNAS system, I got the same result.

Attempt #2: Chelsio T520-LL-CR

From the 10 Gig Networking Primer thread, it seemed like the safest bet was a Chelsio NIC, so I picked one up from eBay.

Sadly, that got even worse results than the Mellanox. It didn't show up in TrueNAS's list of network interfaces at all.

I tried adding these lines to /boot/loader.conf

Code:

t4fw_cfg_load="YES"
t5fw_cfg_load="YES"
t6fw_cfg_load="YES"
if_cxgbe_load="YES"

Same results.

I tried adding them instead as Tunables: Same results.

Attempt #3: Chelsio Dual Port T520-CR

Given that Chelsio is supposed to be widely supported on TrueNAS, I thought maybe I got a dud NIC, so I ordered a different model from a different vendor. Unfortunately, I plugged it in and got the exact same result as the previous NIC.

Any suggestions for things I can try? Am I overlooking something?

 

[nKk]

I think that your PCIe slot are working at x4 mode -> from Asus documentation
"AMD Athlon™ with Radeon™ Vega Graphics Processors
1 x PCIe 3.0/2.0 x16 (x4 mode)"

The NIC's should auto negotiate PCIe mode and working with limited bandwidth but MB can have some other limitations or BIOS settings.

You can try to install Chelsio Dual Port T520-CR and execute in shell:
lspci | grep -i ethernet

paste the output here in <code> tags to check if the card is available.
And you can connect monitor and check if there are messages from Chelsio at POST time.

 

[mtlynch]

Thanks for your suggestions!

You can try to install Chelsio Dual Port T520-CR and execute in shell:
lspci | grep -i ethernet

It unfortunately doesn't seem to pick anything up except the built-in Ethernet on the motherboard:

Code:

truenas# lspci | grep -i ethernet
05:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 15)

I recorded the boot, but I don't see any messages related to the Chelsio NIC:

It does say "Probing 16 block devices..not supported" though I'm not sure if that refers to the PCI slot.

I looked for PCI options in the ASUS BIOS, but I didn't see anything related to x4 mode.