Mellanox Connectx-4 LX failes to see connection

[russmc]

New TrueNAS install, running TrueNAS-13.0-U3.1. I had a Chelsio 10G card installed but wanted to upgrade it to one of the Mellanox 10/25G cards that I had pulled out of another server. The interfaces show up in the console, but show the link state as DOWN, even though I have lights on the card, and the switch (Unifi Agg Pro) shows that it's connected at 10G.

Output of dmesg | grep mlx shows:

Code:

root@truenas[~]# dmesg | grep mlx
mlx5_core0: <mlx5_core> mem 0xf6000000-0xf7ffffff irq 26 at device 0.0 on pci3
mlx5: Mellanox Core driver 3.7.1 (November 2021)uhub7: 2 ports with 2 removable, self powered
mlx5_core0: INFO: mlx5_port_module_event:707:(pid 12): Module 0, status: plugged and enabled
mlx5_core0: INFO: init_one:1660:(pid 0): cannot find SR-IOV PCIe cap
mlx5_core: INFO: (mlx5_core0): E-Switch: Total vports 1, l2 table size(65536), per vport: max uc(1024) max mc(16384)
mlx5_core0: Failed to initialize SR-IOV support, error 2
mlx5_core1: <mlx5_core> mem 0xf4000000-0xf5ffffff irq 25 at device 0.1 on pci3
mlx5_core1: INFO: mlx5_port_module_event:707:(pid 12): Module 1, status: plugged and enabled
mlx5_core1: INFO: init_one:1660:(pid 0): cannot find SR-IOV PCIe cap
mlx5_core: INFO: (mlx5_core1): E-Switch: Total vports 1, l2 table size(65536), per vport: max uc(1024) max mc(16384)
mlx5_core1: Failed to initialize SR-IOV support, error 2

Output of pciconf -lv | grep mlx -A 4:

Code:

root@truenas[~]# pciconf -lv  | grep mlx -A 4
mlx5_core0@pci0:4:0:0:  class=0x020000 rev=0x00 hdr=0x00 vendor=0x15b3 device=0x1015 subvendor=0x15b3 subdevice=0x0016
    vendor     = 'Mellanox Technologies'
    device     = 'MT27710 Family [ConnectX-4 Lx]'
    class      = network
    subclass   = ethernet
mlx5_core1@pci0:4:0:1:  class=0x020000 rev=0x00 hdr=0x00 vendor=0x15b3 device=0x1015 subvendor=0x15b3 subdevice=0x0016
    vendor     = 'Mellanox Technologies'
    device     = 'MT27710 Family [ConnectX-4 Lx]'
    class      = network
    subclass   = ethernet

The server is a Supermicro X8 board.

 

[jgreco]

It may not be compatible with TrueNAS. There is a discussion of well-supported cards over in the 10 Gig Networking Primer resource; the Mellanox are not a fan favorite. They are fiddly and finicky. In addition, they operate in more than one mode (not just Ethernet) so if your card isn't set up correctly, it may be showing symptoms of being "there" but still not willing to work. I don't like them or use them for that reason, and others.