Network Cards - PCIe Slots & Lanes

NugentS

MVP
Joined
Apr 16, 2020
Messages
2,945
Can anyone explain to me why its so hard to get a 10Gb NIC in PCIe3*2 (OK, I have never seen a *2 slot) or a dual in PCIe3*4. They all seem to be PCIe3*8 or PCIe2*8
PCIe3*4 is perfect for a dual port 10Gb Card (actually a triple port card - not that I have ever seen one)

Asus do a PCIe3*4 (single port) based on the Aquantia chipset (I use one in my PC) but I don't think its supported in TN (I haven't tried) and works well under windoze

I know I can fit a *8 in an openbacked *4, or use a dremel but I will admit to be very unkeen on the dremel idea and slightly twitchy about the open backed slot idea

Are manufacturers just being lazy or am I missing something

I suppose that 10Gb is dead or deprecated now, the big boys have moved on to 25Mb, 40Mb, 100Mb or more but still. PCIe lanes are valuable and limited.
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
Because most 10G cards are designed for servers. Servers generally have a bunch of PCIe 2.0 x8 or PCIe 3.0 x8 slots. If you do the math, dual 10G on PCIe 2.0 x8 works out pretty well, rather than making a dual PCIe 2.0 x8 and a dual PCIe 3.0 x4 card, which doubles your SKU count for no obvious value received.

You actually hint at this yourself in your message; "PCIe lanes are valuable and limited." No they're not. On common server platforms, they're reasonably plentiful, especially since most servers are dual CPU (== twice the lanes). This leads to slot design that's sufficient for average uses. x8 can handle dual 10GbE, or an HBA/RAID controller, or NVMe devices, so it's real popular. Therefore cards ALSO get designed for x8. PCIe 2.0 x8 was big enough for dual 10GbE, PCIe 3.0 x8 is big enough for quad 10GbE, PCIe 4.0 x8 is big enough for dual 25GbE. That covers about ten years of PCIe evolution, and the increases in speed tend to work out well with the sorts of hardware needed in a server.

System design on a lane-limited system has always been a PITA. I understand that this is disappointing, but it is also why the desktop/workstation single CPU systems are so much less expensive.

What you're looking for, and what may be showing up in the form of the Aquantia card you reference, are that more consumer/desktop PC's are being outfitted with 10G expansion cards, where x8 slots are less common and manufacturers have filled the expansion area with x1 and x4 PCIe connectors for your typical USB expansion card, sound card, crappy SATA and Realtek ethernet cards, etc. So cards are being manufactured and targeted at the consumer sector, and they'll be crappier Windows-compatible cards, but that's fine. Understand that they're targeting your PC and that server cards are targeting servers, and it all sorta works out.
 

NugentS

MVP
Joined
Apr 16, 2020
Messages
2,945
its why I am putting together a new TN NAS with twin CPU's just to get the extra lanes. Should get the hardware on Monday
As for my next PC, (in a few years I hope) - I am going to have to look at lanes very carefully
 
Top