SOLVED mellanox cx3 40gb card not recognized

iliak

Contributor
Joined
Dec 18, 2018
Messages
148
cant configure the mellanox driver

i based on https://www.supermicro.com/products/system/1u/1027/sys-1027r-72brftp.cfm
include 2 1gb and 2 10gb
and i added melnox card 2x40

i see only 4 interfaces

  1. igb0 Intel(R) PRO/1000 Network Connection
  2. igb1 Intel(R) PRO/1000 Network Connection
  3. bxe0 QLogic NetXtreme II BCM57810 10GbE (B1) BXE v:1.78.90
  4. bxe1 QLogic NetXtreme II BCM57810 10GbE (B1) BXE v:1.78.90
the mellanox not found

Code:
# dmesg | grep mlx
mlx4_core0: <mlx4_core> mem 0xdfa00000-0xdfafffff,0xdd800000-0xddffffff irq 32 at device 0.0 numa-domain 0 on pci2
mlx4_core: Mellanox ConnectX core driver v3.4.1 (October 2017)
mlx4_core: Initializing mlx4_core
mlx4_core0: Unable to determine PCI device chain minimum BW


card info model MCX314A-BCCT
 
Last edited:

iliak

Contributor
Joined
Dec 18, 2018
Messages
148
anyone?
 
D

dlavigne

Guest
mlx4_core0: Unable to determine PCI device chain minimum BW

Doesn't look good. Is this card supposed to be supported by FreeBSD? Is there an outstanding upstream bug report or patch coming in 12?
 

iliak

Contributor
Joined
Dec 18, 2018
Messages
148
mlx4_core0: Unable to determine PCI device chain minimum BW

Doesn't look good. Is this card supposed to be supported by FreeBSD? Is there an outstanding upstream bug report or patch coming in 12?

it think the issue is with pci channels but there should be enough
 

Mlovelace

Guru
Joined
Aug 19, 2014
Messages
1,111
it think the issue is with pci channels but there should be enough
Have you tried loading the kernel module manually?
Code:
kldload mlx4en

Post the output of the above command.

If you are able to load the module you'll need to put the interfaces into ethernet mode with the following sysctl commands.
Code:
sysctl sys.device.mlx4_core0.mlx4_port1=eth
sysctl sys.device.mlx4_core0.mlx4_port2=eth
 

iliak

Contributor
Joined
Dec 18, 2018
Messages
148
Have you tried loading the kernel module manually?
Code:
kldload mlx4en

Post the output of the above command.
output:
Code:
kldload mlx4en
kldload: can't load mlx4en: No such file or directory



Code:
#sysctl sys.device.mlx4_core0.mlx4_port1=eth
sys.device.mlx4_core0.mlx4_port1: auto (ib) -> eth
#sysctl sys.device.mlx4_core0.mlx4_port2=eth
sys.device.mlx4_core0.mlx4_port2: auto (ib) -> eth


tried to kldload mlx4en but got same error "can't load mlx4en: No such file or directory"

tried to reboot and it back to infiniband.
btw how i can configure it to stay after boot on eth?
 

iliak

Contributor
Joined
Dec 18, 2018
Messages
148
update:
after running
Code:
#sysctl sys.device.mlx4_core0.mlx4_port1=eth
sys.device.mlx4_core0.mlx4_port1: auto (ib) -> eth
#sysctl sys.device.mlx4_core0.mlx4_port2=eth
sys.device.mlx4_core0.mlx4_port2: auto (ib) -> eth

interfaces are found

but after reboot i must set it again
 
Last edited:

iliak

Contributor
Joined
Dec 18, 2018
Messages
148
You'll need to add the sysctl commands to the tunables in the system tab. The variable will be the "sys.device.mlx4_core0.mlx4_port1" and "sys.device.mlx4_core0.mlx4_port2", the value with be "eth", and the type with be "sysctl" (no quotes when you type them in).
it works after reboot, thanks. now only need to check speed

Can you run
Code:
kldstat -v | grep mlx4
and post the output
Code:
378 mlx4en
377 mlx4
 

alpha754293

Dabbler
Joined
Jul 18, 2019
Messages
47
Sorry for being late to the party, but you might also try using mlxconfig to set your ports so that it's persistent (so that you don't have to set it with each reboot).

The MLNX_OFED drivers should be able to do that, IF your cards aren't set to that already.
 
Top