Mellanox MCX354A-QCBT ConnectX-3 QDR Infiniband not recognized

[kumarmvnk]

Is there any tunable i have to use for this Dual Port - 40/10GBE card ?

Same card words with Windows 10 64bit, however the only difference is the rev 3 is in windows and rev1 of this card is in FreeNAS Server.. Please suggest how to best get it to work and best speeds as Peer to Peer between the two Server and PC.

My two cables are "Mellanox MC2206128-005 - 5M 16ft - Passive Copper Cable 40Gb/S QSFP ".

 

[Samuel Tai]

First, is the card set to come up in InfiniBand or Ethernet mode? InfiniBand is the default, and you'll need to use the Windows drivers to set it to Ethernet mode.

Next, does FreeNAS detect the card on boot? Please provide the output of lspci. If the card is seen, you may need to set a loader tunable mlx4en_load="YES". Note the tunable only takes effect on boot.

 

[kumarmvnk]

Hi Samuel

I am not sure of the First part as to InfiniBand or Ethernet mode ? please suggest how to use Windows drivers to set it to Ethernet mode ?

Yes FreeNAS detects as below
root@MVNK[~]# lspci | grep Mellanox
01:00.0 Network controller: Mellanox Technologies MT27500 Family [ConnectX-3]

i have now set a loader tunable " mlx4en_load="YES" " and rebooted.

But still i don't see any Network/ Interfaces


also i got the beklow ..


[~]# sysctl -a | grep mlx
device mlx4
device mlx4en
device mlx5
device mlx5en
device mlxfw
hw.mlx5.relaxed_ordering_write: 0
hw.mlx5.auto_fw_update: 0
hw.mlx5.fast_unload_enabled: 1
hw.mlx5.prof_sel: 2
hw.mlx5.debug_mask: 0
hw.mlx5.sw_reset_timeout: 1200
hw.mlx5.fw_reset_enable: 1
sys.device.mlx4_core0.hw.board_id: ISL1090110018
sys.device.mlx4_core0.hw.fw_version: 2.30.8020
sys.device.mlx4_core0.mlx4_port2_mtu: 4096
sys.device.mlx4_core0.mlx4_port2: auto (ib)
sys.device.mlx4_core0.mlx4_port1_mtu: 4096
sys.device.mlx4_core0.mlx4_port1: auto (ib)
dev.mlx4_core.0.%domain: 0
dev.mlx4_core.0.%parent: pci2
dev.mlx4_core.0.%pnpinfo: vendor=0x15b3 device=0x1003 subvendor=0x15b3 subdevice=0x0150 class=0x028000
dev.mlx4_core.0.%location: slot=0 function=0 dbsf=pci0:1:0:0 handle=\_SB_.PCI0.BR1A.H000
dev.mlx4_core.0.%driver: mlx4_core
dev.mlx4_core.0.%desc: Mellanox driver (3.5.1)
dev.mlx4_core.%parent:
compat.linuxkpi.mlx4_inline_thold: 104
compat.linuxkpi.mlx4_pfcrx: 0
compat.linuxkpi.mlx4_pfctx: 0
compat.linuxkpi.mlx4_udp_rss: 1
compat.linuxkpi.mlx4_log_mtts_per_seg: 3
compat.linuxkpi.mlx4_use_prio: 0
compat.linuxkpi.mlx4_log_num_vlan: 0
compat.linuxkpi.mlx4_log_num_mac: 7
compat.linuxkpi.mlx4_enable_4k_uar: 0
compat.linuxkpi.mlx4_enable_64b_cqe_eqe: 1
compat.linuxkpi.mlx4_log_num_mgm_entry_size: 10
compat.linuxkpi.mlx4_msi_x: 1
compat.linuxkpi.mlx4_enable_qos: 0
compat.linuxkpi.mlx4_internal_err_reset: 1

 

[Samuel Tai]

sys.device.mlx4_core0.mlx4_port2: auto (ib)
sys.device.mlx4_core0.mlx4_port1: auto (ib)

Create sysctl tunables

Code:

sys.device.mlx4_core0.mlx4_port2: eth
sys.device.mlx4_core0.mlx4_port1: eth

You can also remove the loader tunable mlx4en_load="YES", as the mlx4en driver is compiled into the kernel.

 

[kumarmvnk]

exellant got it as

mlxen0
mlxen1

what should i fill in

MTU ?? i know for 10GBE its 9000 but to try 40GBE - ??

Options

 

[Samuel Tai]

You can now configure these interfaces as usual. The maximum MTU is always 9000.

 

[kumarmvnk]

THanks Samuel

 

[kumarmvnk]

Both
mlxen0 & mlxen1 are always DOWN.

mlxen0
PHYSICAL
DOWN
no
no
10.10.20.2/24


mlxen1
PHYSICAL
DOWN
no
no
10.10.30.2/24

 

[Samuel Tai]

That's because your MC2206128-005 cables are InfiniBand, not Ethernet. You need the MC2210126-005 cables for 40G Ethernet.

 

[kumarmvnk]

ic very finiky i suppose ,,, thanks

 

[kumarmvnk]

That's because your MC2206128-005 cables are InfiniBand, not Ethernet. You need the MC2210126-005 cables for 40G Ethernet.

Sam CAn i know if there is any advantage og having two cables versus one for this dual port ?

 

[Samuel Tai]

What's your 40G Ethernet network topology? What protocols are you using for sharing? There's not enough information in your question to answer.

 

[kumarmvnk]

Well Samuel its a simple Peer to Peer from a windows 10 64 bit PC to Freenas Server... In this case as i had these infiniband cables i got the cards, now that the cards are there I need to get cables that are not Infiniband, wondering i should buy two or one. basically trying to get more than 100mbits to atleast a 8gbits per second transfer from pc to server and have a faster Plex.

 

[Samuel Tai]

Neither Windows file sharing nor Plex can take advantage of multi-path IO. A single cable will suffice. A LAGG interface with 2 cables won't work in practice, as both endpoints' IPs won't change so LAGG will load-balance constantly to a single link.

 

[kumarmvnk]

Thank you, just before i start buying another cable, is there no way I can make use of this "MC2206128-005 cables are InfiniBand ??" is IB not compatible with FreeNAS ?

Is there no magic tunable or setting to get this to work !! : )

 

[Samuel Tai]

No, InfiniBand is wired differently from Ethernet.

 

[kumarmvnk]

That's because your MC2206128-005 cables are InfiniBand, not Ethernet. You need the MC2210126-005 cables for 40G Ethernet.

Samuel Just wanted to let you know that the "MC2206128-005 cable" works only as we created ETH tunables, I had to set the Mellanox adapter settings from default IB to ETH. I got this via Utube,

Thank you for the support.