40GB QSFP adapter for FreeNAS

[xlameee]

I just got 8 of Mellanox ConnectX-3 MCX314A-BCCT Infiniband QSFP+ Dual Port 40GbE PCI-e 3.0 I was going to use them with esxi hosts and nexus 3132q-v switch but I've wondering if freeBSD can support this adapter I was planing to install a freeNAS on DELL R320 also any recommendations if I am going to have an issues with R320 and FreeNAS I can always put FreeNAS on some of my esxi servers but I'd like to have it independent on the system for iSCSI shared storage

 

[kdragon75]

One moment while I google that for you.

 

[kdragon75]

According to the second (non ad) result http://www.mellanox.com/related-docs/prod_adapter_cards/PB_ConnectX-3_Pro_Card_EN.pdf:

Software Support All Mellanox adapter cards are supported by Windows, Linux distributions, VMware, FreeBSD, Ubuntu, and Citrix XenServer. ConnectX-3 Pro adapters support OpenFabrics-based RDMA protocols and software and are compatible with configuration and management tools from OEMs and operating system vendors

Also https://www.freebsd.org/cgi/man.cgi?query=mlx4en&sektion=4&manpath=freebsd-release-ports:

The mlx4en driver supports the following network adapters:

+o Mellanox ConnectX-2 (ETH)
+o Mellanox ConnectX-3 (ETH)

 

[xlameee]

https://www.mellanox.com/related-docs/prod_adapter_cards/PB_ConnectX-3_Pro_Card_EN.pdf

it does say FreeBSD but someone ever tried ?

 

[kdragon75]

Attempting to run kldload mlx4en seems to give

Code:

[root@FreeNAS ~]# kldload mlx4en																									
kldload: can't load mlx4en: No such file or directory

Looks like it was not included.

 

[kdragon75]

I just got 8 of Mellanox ConnectX-3 MCX314A-BCCT

If you have them, just try it.

 

[HoneyBadger]

Should be supported by default in FreeNAS 11 and newer. You may have to set a system tunable to tell each card/port to run in Ethernet mode vs. Infiniband though

Code:

sys.device.mlx4_core0.mlx4_port0 = eth

 

[Johnnie Black]

ConnectX-2 works fine, if the driver is the same X-3 should also work.

 

[xlameee]

I had unused NODE on Dell C6220 with HP / QLogic QLE7342 InfiniBand 4X QDR PCI-E G2 Dual Port HCA Adapter 583211-B21
I just started to install a freenas so far so good I'll se if this card will do the job first!!!

 

[HoneyBadger]

I had unused NODE on Dell C6220 with HP / QLogic QLE7342 InfiniBand 4X QDR PCI-E G2 Dual Port HCA Adapter 583211-B21
I just started to install a freenas so far so good I'll se if this card will do the job first!!!

That specific card (583211-B21) is listed under this page as an HCA only, not a dual-mode IB/Ethernet adapter, so it may not work at all.

https://support.hpe.com/hpsc/doc/public/display?docId=c03225999

But do let us know how this works out. You'll definitely need some tuning to get the most out of it though.

 

[xlameee]

now how do I find out witch card is witch ?

This node have an embedded dual nic igb0 igb1 I thing
an mezzanine card intel I believe dual 10G
and QLogic QLE7342 InfiniBand
1 of those 2 addons is ix0 and ix1

 

[HoneyBadger]

igbN are your embedded ones, ixN are the 10Gb mezzanine ports, and the QLogic card is not being recognized because I don't believe there's a FreeBSD driver for it yet (or it is supported, but it's in HCA/Infiniband mode and isn't supported in that mode)

 

[xlameee]

what about this one intel XL710-QDA2 it is intel and what I know about FreeBSD support most of intel chips ?
or what adapter you will recommend me to get 40G without any modifications just out of the box put it on my server and run it

 

[kdragon75]

What 40gbe switch gear are you using?

 

[xlameee]

NEXUS 3132q-v at my business, NEXUS 3132q at home

 

[kdragon75]

@xlameee You may find answers faster by asking the google gods.

 

[xlameee]

sorry but google is not a god it is a parasite and my firewalls are blocking most of it. And if I know the way to block that shit entirely I will. I use other search engines anyway I hope here people with experience to help me.

 

[HoneyBadger]

what about this one intel XL710-QDA2 it is intel and what I know about FreeBSD support most of intel chips ?
or what adapter you will recommend me to get 40G without any modifications just out of the box put it on my server and run it

XL710 should be supported under the ixl driver which is included in FreeNAS 11

Code:

[root@mushroom] /# kldstat -v | grep ixl
				139 pci/ixlv
				138 pci/ixl
[root@mushroom] /#

However your system and drives will likely need tuning in order to operate at 40GbE effectively - we're talking about adjusting Tx/Rx interrupts, jumbograms beyond the 64K IPv4 limit - and even if you get there, that's 5GB/s - you're looking at multiple high-end NVMe drives if you're talking about sustaining back-end I/O at that level of saturation.

While overkill is its own reward, do you really need 40GbE here over MPIO'd 10GbE?

 

[kdragon75]

sorry but google is not a god it is a parasite and my firewalls are blocking most of it. And if I know the way to block that crap entirely I will. I use other search engines anyway I hope here people with experience to help me.

You may also try FreeBSD.org or use one of the search engines you like. Also there is no god and there is a high probability that we live in a simulation that resides in a deterministic universe.

 

[HoneyBadger]

Also there is no god and there is a high probability that we live in a simulation that resides in a deterministic universe.

Computer, end program.