2022 25gig and 100gig nic options?

[jamiejunk]

Looking for some 25 gig and 100 gig nics for TrueNAS core. Looks like Mellanox is generally bad news from the forum posts I’m reading.
I read over @jgreco 10 gig networking primer. But not a whole lot of info about 25 and 100gig nics other than the mention of Chelsio and Intel.

Looks like these are the current offerings from each company:

Chelsio - 100 GIG:
T62100-LP-CR
T62100-SO-CR

Chelsio - 25 gig:
T6225-SO-CR
T6225-CR

Intel - 100 GIG:
E810-CQDA2

Intel - 25 GIG:
XXV710-DA2

Has anyone had any hands on experience with these? Or maybe others? We’re going to be connecting to a Juniper EX4650. I don’t think we need 100 gig, but if we got the switch ports, why the hell not :)

 

[Elliot Dierksen]

I use Chelsio T580's for 40G connections and they work great!

 

[firesyde424]

We use the Intel 25Gbe cards and have had no issues with them.

 

[Dice]

Looking for some 25 gig and 100 gig nics for TrueNAS core. Looks like Mellanox is generally bad news from the forum posts I’m reading.

Are your readings from scenarios with TN13?

 

[Jessep]

Mellanox starting with connectx-3 cards work well, the connectx-2 cards had driver/firmware issues on FreeNAS.
Mellanox should be firmware updated to current for best results.

Otherwise Intel is the goto brand.

 

[jamiejunk]

@firesyde424

We use the Intel 25Gbe cards and have had no issues with them.

Did you need to install drivers for the Intel cards? Or were they ready to roll?

 

[firesyde424]

@firesyde424

Did you need to install drivers for the Intel cards? Or were they ready to roll?

They worked for us out of the box. No additional drivers required.

 

[Dice]

TN13 is based on FreeBSD 13, there seems to be good support:
Have a closer look at the links, I'll cut out some parts for convenience:

mlx5en(4)

www.freebsd.org

FreeBSD 13.0-RELEASE Hardware Notes

FreeBSD is an operating system used to power modern servers, desktops, and embedded platforms.

www.freebsd.org

The mlx4en(4) driver supports the following network adapters:

• Mellanox ConnectX-2 (ETH)
• Mellanox ConnectX-3 (ETH)

The mlx5en(4) driver supports 100Gb, 50Gb, 40Gb, 25Gb and 10Gb Ethernet adapters. ConnectX-5 supports:10/20/25/40/50/56/100Gb/s speeds. ConnectX-4 supports:10/20/25/40/50/56/100Gb/s speeds. ConnectX-4 LX supports:10/25/40/50Gb/s speeds (and reduced power consumption) :

• Mellanox MCX455A-ECAT
• Mellanox MCX456A-ECAT
• Mellanox MCX415A-CCAT
• Mellanox MCX416A-CCAT
• Mellanox MCX455A-FCAT
• Mellanox MCX456A-FCAT
• Mellanox MCX415A-BCAT
• Mellanox MCX416A-BCAT
• Mellanox MCX4131A-GCAT
• Mellanox MCX4131A-BCAT
• Mellanox MCX4121A-ACAT
• Mellanox MCX4111A-ACAT
• Mellanox MCX4121A-XCAT
• Mellanox MCX4111A-XCAT

The ixgbe(4) driver supports the following cards:

• Intel® 10 Gigabit XF SR/AF Dual Port Server Adapter
• Intel® 10 Gigabit XF SR/LR Server Adapter
• Intel® 82598EB 10 Gigabit AF Network Connection
• Intel® 82598EB 10 Gigabit AT CX4 Network Connection

Most adapters in the Intel Ethernet 700 Series with SFP+/SFP28/QSFP+ cages have firmware that requires that Intel qualified modules are used; these qualified modules are listed below. This qualification check cannot be disabled by the driver.
The ixl(4) driver supports 40Gb Ethernet adapters with these QSFP+ modules:

• Intel 4x10G/40G QSFP+ 40GBASE-SR4 E40GQSFPSR
• Intel 4x10G/40G QSFP+ 40GBASE-LR4 E40GQSFPLR

The ixl(4) driver supports 25Gb Ethernet adapters with these SFP28 modules:

• Intel 10G/25G SFP28 25GBASE-SR E25GSFP28SR
• Intel 10G/25G SFP28 25GBASE-SR E25GSFP28SRX (Extended Temp)

The ixl(4) driver supports 25Gb and 10Gb Ethernet adapters with these SFP+ modules:

• Intel 1G/10G SFP+ SR FTLX8571D3BCV-IT
• Intel 1G/10G SFP+ SR AFBR-703SDZ-IN2
• Intel 1G/10G SFP+ LR FTLX1471D3BCV-IT
• Intel 1G/10G SFP+ LR AFCT-701SDZ-IN2
• Intel 1G/10G SFP+ 10GBASE-SR E10GSFPSR
• Intel 10G SFP+ 10GBASE-SR E10GSFPSRX (Extended Temp)
• Intel 1G/10G SFP+ 10GBASE-LR E10GSFPLR

HARDWARE
The mlx5en driver supports 200Gb, 100Gb, 50Gb, 40Gb, 25Gb and 10Gb ether-
net adapters.

+o ConnectX-6 supports 10/20/25/40/50/56/100Gb/200Gb/s speeds.
+o ConnectX-5 supports 10/20/25/40/50/56/100Gb/s speeds.
+o ConnectX-4 supports 10/20/25/40/50/56/100Gb/s speeds.
+o ConnectX-4 LX supports 10/25/40/50Gb/s speeds and reduced power con-
sumption.

 

[jamiejunk]

TN13 is based on FreeBSD 13, there seems to be good support:
Have a closer look at the links, I'll cut out some parts for convenience:

mlx5en(4)

www.freebsd.org

FreeBSD 13.0-RELEASE Hardware Notes

FreeBSD is an operating system used to power modern servers, desktops, and embedded platforms.

www.freebsd.org

This is super helpful. Thanks! For some reason it didn't dawn on my to look at FreeBSD specs. :facepalm: