40GB QSFP adapter for FreeNAS

Status
Not open for further replies.

xlameee

Explorer
Joined
Jun 22, 2018
Messages
87
I just got 8 of Mellanox ConnectX-3 MCX314A-BCCT Infiniband QSFP+ Dual Port 40GbE PCI-e 3.0 I was going to use them with esxi hosts and nexus 3132q-v switch but I've wondering if freeBSD can support this adapter I was planing to install a freeNAS on DELL R320 also any recommendations if I am going to have an issues with R320 and FreeNAS I can always put FreeNAS on some of my esxi servers but I'd like to have it independent on the system for iSCSI shared storage
 

kdragon75

Wizard
Joined
Aug 7, 2016
Messages
2,457
One moment while I google that for you.
 

kdragon75

Wizard
Joined
Aug 7, 2016
Messages
2,457
According to the second (non ad) result http://www.mellanox.com/related-docs/prod_adapter_cards/PB_ConnectX-3_Pro_Card_EN.pdf:
Software Support All Mellanox adapter cards are supported by Windows, Linux distributions, VMware, FreeBSD, Ubuntu, and Citrix XenServer. ConnectX-3 Pro adapters support OpenFabrics-based RDMA protocols and software and are compatible with configuration and management tools from OEMs and operating system vendors
Also https://www.freebsd.org/cgi/man.cgi?query=mlx4en&sektion=4&manpath=freebsd-release-ports:
The mlx4en driver supports the following network adapters:

+o Mellanox ConnectX-2 (ETH)
+o Mellanox ConnectX-3 (ETH)
 

kdragon75

Wizard
Joined
Aug 7, 2016
Messages
2,457
Attempting to run kldload mlx4en seems to give
Code:
[root@FreeNAS ~]# kldload mlx4en																									
kldload: can't load mlx4en: No such file or directory

Looks like it was not included.
 

kdragon75

Wizard
Joined
Aug 7, 2016
Messages
2,457

HoneyBadger

actually does care
Administrator
Moderator
iXsystems
Joined
Feb 6, 2014
Messages
5,112
Should be supported by default in FreeNAS 11 and newer. You may have to set a system tunable to tell each card/port to run in Ethernet mode vs. Infiniband though

Code:
sys.device.mlx4_core0.mlx4_port0 = eth
 

xlameee

Explorer
Joined
Jun 22, 2018
Messages
87
I had unused NODE on Dell C6220 with HP / QLogic QLE7342 InfiniBand 4X QDR PCI-E G2 Dual Port HCA Adapter 583211-B21
I just started to install a freenas so far so good I'll se if this card will do the job first!!!
 

HoneyBadger

actually does care
Administrator
Moderator
iXsystems
Joined
Feb 6, 2014
Messages
5,112
I had unused NODE on Dell C6220 with HP / QLogic QLE7342 InfiniBand 4X QDR PCI-E G2 Dual Port HCA Adapter 583211-B21
I just started to install a freenas so far so good I'll se if this card will do the job first!!!

That specific card (583211-B21) is listed under this page as an HCA only, not a dual-mode IB/Ethernet adapter, so it may not work at all.

https://support.hpe.com/hpsc/doc/public/display?docId=c03225999

But do let us know how this works out. You'll definitely need some tuning to get the most out of it though.
 

xlameee

Explorer
Joined
Jun 22, 2018
Messages
87
now how do I find out witch card is witch ?

This node have an embedded dual nic igb0 igb1 I thing
an mezzanine card intel I believe dual 10G
and QLogic QLE7342 InfiniBand
1 of those 2 addons is ix0 and ix1
 

Attachments

  • 2018-08-13_12-33-44.jpg
    2018-08-13_12-33-44.jpg
    70.6 KB · Views: 627

HoneyBadger

actually does care
Administrator
Moderator
iXsystems
Joined
Feb 6, 2014
Messages
5,112
igbN are your embedded ones, ixN are the 10Gb mezzanine ports, and the QLogic card is not being recognized because I don't believe there's a FreeBSD driver for it yet (or it is supported, but it's in HCA/Infiniband mode and isn't supported in that mode)
 

xlameee

Explorer
Joined
Jun 22, 2018
Messages
87
what about this one intel XL710-QDA2 it is intel and what I know about FreeBSD support most of intel chips ?
or what adapter you will recommend me to get 40G without any modifications just out of the box put it on my server and run it
 

kdragon75

Wizard
Joined
Aug 7, 2016
Messages
2,457
What 40gbe switch gear are you using?
 

kdragon75

Wizard
Joined
Aug 7, 2016
Messages
2,457
@xlameee You may find answers faster by asking the google gods.
 

xlameee

Explorer
Joined
Jun 22, 2018
Messages
87
sorry but google is not a god it is a parasite and my firewalls are blocking most of it. And if I know the way to block that shit entirely I will. I use other search engines anyway I hope here people with experience to help me.
 

HoneyBadger

actually does care
Administrator
Moderator
iXsystems
Joined
Feb 6, 2014
Messages
5,112
what about this one intel XL710-QDA2 it is intel and what I know about FreeBSD support most of intel chips ?
or what adapter you will recommend me to get 40G without any modifications just out of the box put it on my server and run it
XL710 should be supported under the ixl driver which is included in FreeNAS 11

Code:
[root@mushroom] /# kldstat -v | grep ixl
				139 pci/ixlv
				138 pci/ixl
[root@mushroom] /#


However your system and drives will likely need tuning in order to operate at 40GbE effectively - we're talking about adjusting Tx/Rx interrupts, jumbograms beyond the 64K IPv4 limit - and even if you get there, that's 5GB/s - you're looking at multiple high-end NVMe drives if you're talking about sustaining back-end I/O at that level of saturation.

While overkill is its own reward, do you really need 40GbE here over MPIO'd 10GbE?
 
Last edited:

kdragon75

Wizard
Joined
Aug 7, 2016
Messages
2,457
sorry but google is not a god it is a parasite and my firewalls are blocking most of it. And if I know the way to block that crap entirely I will. I use other search engines anyway I hope here people with experience to help me.
You may also try FreeBSD.org or use one of the search engines you like. Also there is no god and there is a high probability that we live in a simulation that resides in a deterministic universe.
 
Status
Not open for further replies.
Top