10 Gig Networking Primer

[vjunkie]

That may be sort-of true, and to be a bit clearer, 5.0GT/s works out to 4Gbit/sec, so at 4 lanes on the mainboard, a dual 10G card is probably is going to be able to hit the limit of the PCIe bus if you try hard.

But what's the point of worrying about it? You're limited by the board. The only way to bump the speed is to toss it out of an airplane at 30,000 feet. You'll get that extra speed boost, at least for a few minutes, try it if you don't believe me. ;-)

I just bought 2x SolarFlare SFN6122F off ebay, dual 10gb and they are listed as pcie 3.0 x8. Is this correct?

 

[jgreco]

I am reasonably certain the SFN6122F is PCIe 2.0 x8. Information is much harder to find since the AMD/Xilinx borg'ing but I just found a reference in case you need to do battle with a lying eBay seller.

https://www.xilinx.com/content/dam/...ell/8522/doc/Solarflare_UserGuide_Issue20.pdf

Solarflare-now-Xilinx confirms it as a PCIe 2.0 x8 card. But this may not matter too much in most cases. Still a good card.

 

[Exhorder]

Just wanted to drop a note that one of my servers has an onboard BCM57810 chipset. I'm using only one of the two links with an Unifi SFP+ module. The server is running FreeNAS/TrueNAS since 2019 without problems now.

Code:

# grep bxe0 /var/run/dmesg.boot
bxe0: <QLogic NetXtreme II BCM57810 10GbE (B0) BXE v:1.78.91> mem 0xfcff000000-0xfcff7fffff,0xfcfe800000-0xfcfeffffff,0xfcff810000-0xfcff81ffff irq 120 at device 0.0 numa-domain 3 on pci12
bxe0: PCI BAR0 [10] memory allocated: 0xfcff000000-0xfcff7fffff (8388608) -> 0xfffffe0151a60000
bxe0: PCI BAR2 [18] memory allocated: 0xfcfe800000-0xfcfeffffff (8388608) -> 0xfffffe0152260000
bxe0: PCI BAR4 [20] memory allocated: 0xfcff810000-0xfcff81ffff (65536) -> 0xfffffe0152a60000
bxe0: Found 10Gb Fiber media.
bxe0: IFMEDIA flags : 20
bxe0: Using defaults for TSO: 65518/35/2048
bxe0: Ethernet address: e0:d5:5e:95:1e:b2
bxe0: MSI-X vectors Requested 5 and Allocated 5
bxe0: NIC Link is Up, 10000 Mbps full duplex, Flow control: ON - receive & transmit
bxe0: link state changed to UP

EDIT:
iperf3 result from a Linux server with Mellanox ConnectX-3 Pro:

Code:

[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  1.09 GBytes  9.40 Gbits/sec  254   1.28 MBytes
[  5]   1.00-2.00   sec  1.09 GBytes  9.38 Gbits/sec    0   1.38 MBytes
[  5]   2.00-3.00   sec  1.09 GBytes  9.40 Gbits/sec   63   1.29 MBytes
[  5]   3.00-4.00   sec  1.09 GBytes  9.37 Gbits/sec  433   1.17 MBytes
[  5]   4.00-5.00   sec  1.09 GBytes  9.38 Gbits/sec  126   1.30 MBytes
[  5]   5.00-6.00   sec  1.09 GBytes  9.38 Gbits/sec  128    906 KBytes
[  5]   6.00-7.00   sec  1.09 GBytes  9.40 Gbits/sec   63   1.16 MBytes
[  5]   7.00-8.00   sec  1.09 GBytes  9.38 Gbits/sec  252    962 KBytes
[  5]   8.00-9.00   sec  1.09 GBytes  9.37 Gbits/sec  568    667 KBytes
[  5]   9.00-10.00  sec  1.09 GBytes  9.38 Gbits/sec  504    957 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  10.9 GBytes  9.39 Gbits/sec  2391             sender
[  5]   0.00-10.00  sec  10.9 GBytes  9.38 Gbits/sec                  receiver

 

[Ericloewe]

That's very interesting, a bunch of users have complained over the years about Broadcom's driver being outright broken...

 

[Exhorder]

Never had an issue with it. It's an onboard NIC on Gigabyte MZ31-AR0 server mainboard.

 

[jgreco]

That's very interesting, a bunch of users have complained over the years about Broadcom's driver being outright broken...

Broadcom's stuff varies. It's in that class of "if you already have it, try it. If you don't, don't spend any money on it" class of card. Which is actually true of quite a few different cards.

 

[Whattteva]

I see from the resource that the recommended cards like Intel and Chelsio are vendor-locked, but I don't see any mentions of it on the Solarflare card. I think it'd be helpful to have that information on the Solarflare card also.

 

[Elliot Dierksen]

I see from the resource that the recommended cards like Intel and Chelsio are vendor-locked

I definitely experienced the vendor locked optics on a 10G T520. The 40G T580 was not, and neither is the 40G XL710 I am using now. That said, the XL710 was a Cisco OEM'd version.

 

[Constantin]

Used optics are cheap, so even if they are vendor locked, there are usually lots to choose amongst. FS.com and 10Gtek also make nice knockoffs.

 

[jgreco]

Used optics are cheap, so even if they are vendor locked, there are usually lots to choose amongst. FS.com and 10Gtek also make nice knockoffs.

There's also very little reason not to just buy new generic or used vendor-coded optics and jam them in as a cost-of-acquisition sort of thing. There really isn't any good reason to separate the optics from the card in the future, which is just my way of future-proofing against further vendor locking annoyances.