Mellanox ConnectX-2 IB not coming up in FreeNAS 11

[Neek]

Hi, I just bought a pair of used Mellanox ConnectX-2 MHRH2A-XSR 20Gbps cards. I'm unable to get it to come up in FreeNAS. I've seen a few other threads here about different variants of these Mellanox cards but I can't tell if the card is supported or not. I see these boot-up messages from the Mellanox drivers. Any guesses what "mlx4_core0: Required capability exceeded device limits" means? Google and other search engines were surprisingly unhelpful.

Is it possible I need some kind of firmware update for the card? Or that this specific variant of card is unsupported?

Code:

root@vault:~# dmesg | grep mlx
mlx4_core0: <mlx4_core> mem 0xdfa00000-0xdfafffff,0xde000000-0xde7fffff irq 16 at device 0.0 on pci1
mlx4_core: Initializing mlx4_core: Mellanox ConnectX VPI driver v2.1.6 (Dec 13 2017)
mlx4_en mlx4_core0: Activating port:1
mlxen0: Ethernet address: 00:02:c9:4f:f2:08
mlx4_en: mlx4_core0: Port 1: Using 8 TX rings
mlxen0: link state changed to DOWN
mlx4_en: mlx4_core0: Port 1: Using 8 RX rings
mlx4_en: mlxen0: Using 8 TX rings
mlx4_en: mlxen0: Using 8 RX rings
mlx4_en: mlxen0: Initializing port
mlx4_core0: command INIT_PORT (0x9) failed: in_param=0x0, in_mod=0x1, op_mod=0x0, fw status = 0x8
mlx4_core0: Required capability exceeded device limits
mlx4_en: mlxen0: Failed Initializing port
mlx4_en mlx4_core0: Activating port:2
mlxen1: Ethernet address: 00:02:c9:4f:f2:09
mlx4_en: mlx4_core0: Port 2: Using 8 TX rings
mlxen1: link state changed to DOWN
mlx4_en: mlx4_core0: Port 2: Using 8 RX rings
mlx4_en: mlxen1: Using 8 TX rings
mlx4_en: mlxen1: Using 8 RX rings
mlx4_en: mlxen1: Initializing port
mlx4_core0: command INIT_PORT (0x9) failed: in_param=0x0, in_mod=0x2, op_mod=0x0, fw status = 0x8
mlx4_core0: Required capability exceeded device limits
mlx4_en: mlxen1: Failed Initializing port

FreeBSD/FreeNAS does see the device correctly:

Code:

root@vault:~# pciconf -lv | grep Mellanox -C 3
	class	  = serial bus
	subclass   = SMBus
mlx4_core0@pci0:1:0:0:   class=0x0c0600 card=0x002215b3 chip=0x673215b3 rev=0xb0 hdr=0x00
	vendor	 = 'Mellanox Technologies'
	device	 = 'MT26418 [ConnectX VPI PCIe 2.0 5GT/s - IB DDR / 10GigE]'
	class	  = serial bus
em1@pci0:3:0:0:   class=0x020000 card=0x000015d9 chip=0x10d38086 rev=0x00 hdr=0x00

 

[BigDave]

Infiniband is not compatible, you should take a look at our Hardware Recommendations Guide and the 10GB networking primer
in the resources section.

 

[nightshade00013]

I have one of the working Mellanox ConnectX-2 cards. They are not considered ideal but it is cheap.

https://www.ebay.com/itm/351416547732

Code:

pciconf -lv | grep Mellanox -C 3
mlx4_core0@pci0:2:0:0:  class=0x020000 card=0x001515b3 chip=0x675015b3 rev=0xb0
hdr=0x00																		
   vendor	 = 'Mellanox Technologies'										
   device	 = 'MT26448 [ConnectX EN 10GigE, PCIe 2.0 5GT/s]'				 
   class	  = network														
   subclass   = ethernet	 

 

[Neek]

Thanks a lot. I'll get the 10Gb ethernet version of the card. Bummer, these Mellanox IB cards are supposed to be crazy fast and are also dirt cheap.

 

[nightshade00013]

No problem. I will give a word of warning with those cards and Windows 7. The latest driver available from Mellanox does not seem to work correctly even though they are listed. In fact they would not install due to requiring a particular patch and it not showing as installed on the system though it is in fact there through a broader patch. Anyway I ended up grabbing an older driver (4.60 I think) and using it and things have worked just fine. Windows 10 I have not heard of the same issue and Linux Mint 17.7 picked it up with no issues.

 

[JustinClift]

Infiniband is not compatible, you should ...

As a data point for future reference, many Mellanox cards can operate in either Infiniband mode or ethernet mode. The cards listed by the OP seem like they should be capable of ethernet mode.

At a rough guess, they probably needed the firmware to be updated before the ethernet driver would work with them.

I'm a few months late to suggest it in time though. :)