10GBase-T: Best to avoid it if you can

[jgreco]

jgreco submitted a new resource:

10GBase-T: Best to avoid it if you can - Slower, power-hungry, and more expensive.

I recently wrote a resource on high speed network performance tuning, and in it, I commented "Do not try to use copper 10GBase-T". @Elliot Dierksen asked about this, and I cranked out a pretty comprehensive post on the topic. Here it is in somewhat expanded form.

I am curious about the "Do...

Read more about this resource...

 

[Ericloewe]

I just want to say that I do have scars and that I profoundly dislike 10GBase-t.

 

[Elliot Dierksen]

I just want to say that I do have scars and that I profoundly dislike 10GBase-t.

Inquiring minds want to know...

 

[Ericloewe]

Well, it boils down to two related factors: 10GBase-T is an overpriced dead end.

There are technical drawbacks, as @jgreco has laid out, but those just make it crappier than the SFP+ variants. As servers increasingly move to 10GBase-T for basic connectivity (a move which I understand to be mostly to take advantage of advanced NIC features rather than more bandwidth - think along the lines of SR-IOV, which is limited in capabilities on the Intel I350), you'll note that there hasn't been a meaningful growth of the 10GBase-T switching market.
Why? Because everyone has moved on, 25GbE is the new 10GbE. SFP+ has a clear upgrade path to SFP28 and you can incrementally upgrade your network as needed. 10GBase-T is stuck between a rock and a hard place - there's no upgrade path without replacing everything and maintaining it is monstrously expensive.

Thing is, you could excuse a lot of this if 10GBase-T were cheap. It's not. SFP+ 10GBase-T modules start at around 50 bucks a pop, so interoperability is also rather expensive. But how expensive are switches? Let me grab an example, which is more or less representative of how terrible prices are: The Dell S4128T-ON is a pretty generic 10GBase-T switch, with two major catches: It uses an Intel Atom C2000 CPU and the switch chip is poorly supported by any of the switch OSes out there, so you're stuck with Dell's OS9 and OS10. 28 copper ports is not much, and the two QSFP28 ports limit expandability. Overall, I feel this thing would work well as the "mid-speed" switch for legacy devices, while integrating well with faster 25GbE switches. Well, used ones from reputable sellers are going for 1.5k€ and up, with some crazies asking upwards of 4k€ (sidenote - I expect this to be more than buying new from Dell). Meanwhile, eBay was showing me a listing for an Arista DCS-7050QX-32-R - that's 32 QSFP+ ports - for 600€.

Meanwhile, I'm stuck with a bunch of 10GBaseT gear at work. Expansion, upgrades and replacements have been a massive pain. I can't upgrade wholesale because I'd need to do it in one go and I don't have the budget to blow on such a move. Also, it would take a while to reconfigure everything. That means I need three switches per rack at worst: 1GBase-T, 10GBase-T and 25GbE (supporting SFP+ 10GbE clients as well). Sure, I can consolidate things down somewhat by carefully arranging devices by rack to keep things together, but the overall point stands - I can't upgrade, I can only set up something in parallel. That means more rack space, more types of hardware in stock as spares, more power and more pain.

 

[Elliot Dierksen]

Thanks for the clarification. Both you and @jgreco have provided some valuable information. Even as someone that works primarily in networking, there is a lot of stuff I didn't know about 10GBase-T.

 

[Jamberry]

I would never use it in a business environment, but for a homeuser 10GBase-T can be pretty useful.
Let me play devils advocate for 10GBase-T

it consumes more power than an equivalent SFP+ or DAC setup.

Does it really use that much more power? According to https://www.servethehome.com/qsfp-v-sfp-v-10gbase-t-testing-power-consumption-differences/ cards themselves are comparable. Already installed copper 10Gbit NICs we see more and more, maybe even use less than a Fiber addon card?
This module uses 2,4W https://mikrotik.com/product/s_rj10#fndtn-downloads I think for a home user with 4 of these, the additional 8W power consumption is ok.

Unless you installed Cat7 years ago, your copper plant is unlikely to be suitable for carrying Cat7 at distance

With a old 5e installation I would still get 5Gbit, with CAT6 10Gbit are possible up to 50m I think. I don't know if this is a Swiss thing, but for the last 10y the only cable I saw on construction sites was a Dätwyler CAT7a cable and small Hager wiring closet. So all newer apartments support 10Gbit copper.

Category 3 cable, which any monkey should be able to terminate

While I am not sure about Cat7, Cat6 is still possible to be terminated by a monkey if you use something like this
https://www.digitec.ch/de/s1/produc...ktronikkabel-stecker-17297691#fullscreen=show

used market for 10GBase-T gear is very costly

That is true for the moment.

But at the same time, if you buy new, Copper is for free. Because it is already on the motherboard, while for fiber you have to buy an extra card. Newer Supermicro boards often come with dual 10GBase-T Intel NICs and no Fiber. That probably will also have an impact on the used market 5y down the road.

For inexpensive networking, it seems like we shifted from 10Gbit Fiber to 100Fiber and from 1Gbit copper to 2,5 or 10Gbit copper.
2,5Gbit seems to be the average "for free" new standard, while 10Gbit is for higher end workstations like W790.

Again, this is all only true for home users like me. I totally agree with you for business applications.

 

[jgreco]

Let me play devils advocate for 10GBase-T

Ok.

Does it really use that much more power? According to https://www.servethehome.com/qsfp-v-sfp-v-10gbase-t-testing-power-consumption-differences/ cards themselves are comparable.

Where do you see THAT? If we look at the 10GBase-T based X550-T2, it's about 12 watts, but the SFP+ based X710-DA2 is about 4 watts. Even the very old X520-DA2 is lower than the X550-T2. Bearing in mind that each endpoint consumes more power, if we wanted to estimate that the switchport takes maybe 5 *extra* watts above SFP+ and the client takes maybe 5 *extra*, you only need 5 10GBase-T links to be driving 50 extra watts due to the use of copper. I don't really care to get into a debate about the specific watt values simply because it varies depending on the run length and the silicon on each end; my point is that there is a multiplicative value involving the number of endpoints.

Already installed copper 10Gbit NICs we see more and more, maybe even use less than a Fiber addon card?

No, that's not how it works. The power consumed is distance dependent in the best cases, so it will be true that an unplugged 10GBase-T port uses less power than one that is supporting a 100 meter run. This is generally not true with fiber, where there is a generally fixed amount of power used to drive the laser diode.

While I am not sure about Cat7, Cat6 is still possible to be terminated by a monkey if you use something like this

I don't see what's so special about that. I've been terminating Category cable for ... holy crap, 40 years now. I've trained people and even back in the 5e days, I saw lots of people having problems getting their runs to pass certification by a Fluke cable tester. It's gotten very difficult to get it done properly.

But at the same time, if you buy new, Copper is for free. Because it is already on the motherboard, while for fiber you have to buy an extra card. Newer Supermicro boards often come with dual 10GBase-T Intel NICs and no Fiber. That probably will also have an impact on the used market 5y down the road.

I totally get what you're saying as this was the same thing that happened during the 100Mbps->1Gbps transition. However, it isn't free. You are paying for that port. The Supermicro X12SRW-F board is $395 while the X12SRW-TF is $515, so from my perspective, the addition of 10GBase-T ports is adding $120 to the cost of the board; it ISN'T "free". Plus you are paying for new gear on the switchport side.

Sorry, I think SFP+ still wins even for home users. 10GBase-T is a dead-end technology because it has no obvious place to evolve. It's been around for a decade now and is missing important capabilities such as PoE; the wifi freaks who pushed the insipid 2.5GbE and 5GbE stuff are going to be super-unhappy when they don't have a clear path to 10GbE that can power their access points. The RF issues involved in getting to 20Gbps or 25Gbps are pretty significant.

 

[WI_Hedgehog]

@jgreco : I'd give you a 'Like' but you are way to stingy handing them out, so this mention of appreciation for your efforts will have to do.

 

[Jamberry]

Regarding the power consumption, here is what I think is a realistic scenario for a home user. You have fiber 10Gbit WAN and SFP+ switch. Modem to switch is probably Fiber, because it is in the same room. Then you have also one Server for VMs and Storage. That server is a little bit loud and because of that in a different room. So from the switch you start by using a module that uses 4W more to (30m) to said server which uses 4W on its module and a NIC that uses 6W more. That is a 14W more in total to reuse an existing wiring.

No, that's not how it works.

Not sure if you miss my point or if I am wrong on this one. I could imagine that a integrated NIC uses less power than the same NIC as an external addon card. Could that be true? Because if it is true, it could be that a 10Gbit SFP+ addon card uses around the same as internal 10Gbit copper.

I don't see what's so special about that

Nothing. That is why I don't get why you say

you really need a perfectionist grade punch technique followed by a thorough cable test/certification to get this working reliably.

I get that someone with two left hands can mess up these cheap crimp RJ45 you buy in 100 packs for 50$, but the one I posted?

the addition of 10GBase-T ports is adding $120 to the cost of the board

I think this is more of a Supermicro markup price, but even so, 120$ is pretty cheap. That would get me at best a 5y old used SFP+ card without any warranty where I live. A new Intel X520-DA2 is 300$.
These new Intel 2,5Gbit NICs are basically free. The bigger problem is probably poor drivers. Don't get me wrong, I also like Fiber more but unfortunately SFP+ prices seem to be stagnant, while cooper is getting cheaper and more common.
I don't know why Supermicro would include 10GBase-T in any product (other than maybe the workstation boards) but it is the sad reality.

 

[Ericloewe]

I totally get what you're saying as this was the same thing that happened during the 100Mbps->1Gbps transition. However, it isn't free. You are paying for that port. The Supermicro X12SRW-F board is $395 while the X12SRW-TF is $515, so from my perspective, the addition of 10GBase-T ports is adding $120 to the cost of the board; it ISN'T "free". Plus you are paying for new gear on the switchport side.

To this specific point:

It's often "free" in the sense that full servers from the likes of Supermicro, Tyan, and whatever-the-hell-Gigabyte-is-calling-itself-now are often configured with motherboards that include 10GBase-T (or risers that include said NIC, on some Supermicro systems). At that point, getting rid of it means a custom build (from an integrator or as DIY). Most Supermicro servers these days seem to either provide no networking on-board if they have OCP NIC 3.0 slots, or 10 GBase-T if they don't.

Now, if only Dell and co. included proper Intel I350 NICs instead of the dodgy Broadcoms they all love so much. Broadcom is the Realtek of the server space. At least back in the rNDC days, you could make a point to get only Intel.

 

[jgreco]

That is a 14W more in total to reuse an existing wiring.

Perhaps. But as an infrastructure engineer, I also think about stuff like "what is the power consumption of the unlit ports"; do you have a 12 port 10GBase-T switch? Have you accounted for those? Have you accounted for the second 10GBase-T port on your client gear? This all factors into the power budget, at least for an accurate comparison.

Not sure if you miss my point or if I am wrong on this one. I could imagine that a integrated NIC uses less power than the same NIC as an external addon card. Could that be true? Because if it is true, it could be that a 10Gbit SFP+ addon card uses around the same as internal 10Gbit copper.

I prefer not to do apples-to-oranges comparisons because the variables are generally arbitrary. Modern chipsets with integrated ethernet (which is commonly 20G on something like a Broadwell-DE SoC like the 1537) are incredibly efficient. Earlier chipsets like the 82599 (Intel X520) burned a pile of watts just for existing, but the ethernet chipset watt-burn has nothing to do with the watts attributable to transmission medium type. Primarily the medium power differences come at the PHY and SFP+/transceiver level.

Put differently, if you got a most modern and efficient 10GBase-T chipset and compared it to an X520, yes, the X520 would lose. But that has nothing to do with the transmission medium type, it's just because the X520 silicon runs a barbeque restaurant as a side business.

A modern add-on card's chipset should be relatively low wattage. However, the problem is still that this is in ADDITION to the SFP+ or transceiver in use, so I am a little bothered above by the claim of 2.4 watts for a Mikrotik RJ45 SFP+; it works out to more than that to actually provide a functioning port. The other thing is that the Mikrotik SFP+ will be a length-limited part; you won't be able to do 100M within the available power budget. They specify 30M at 2.7W consumption, which is really quite a bit.

The add-on card is also .... well, an add-on card, so whatever watts are burned are in addition to the approximately 1 watt for the onboard ports (1GbE is really efficient).

So let's try to compose an honest-"ish" answer to your question.

A modern ethernet add-on card like an X710 burns 7 watts to service up to four SFP+. Newer Intel cards are 25G or faster IIRC. A SFP+ for 10G consumes about 0.8 watts. Therefore the minimum additional cost in watts to add 10G SFP+ to an existing 1G server is one watt for idling the mainboard 1GbE, plus 7W, plus 0.8W, or let's call it 9 watts. For this one client endpoint.

The X550-T2 burns 18 watts peak, I can't verify that this is at 100M, but for the sake of argument, I'll concede the possibility. If we then imagine this to be built into a mainboard, replacing 1GbE RJ45 ports, we have an 18 watt peak solution.

And finally there is the X710-AT2 that has a max burn of 9.6W for a dual port card. Again, if on the mainboard, this is a 9.6 watt peak solution.

So this is murky. The latest 10GBase-T ethernet silicon can potentially be within a horseshoe's throw of somewhat older SFP+ ethernet silicon when used in a single-port role.

 

[Etorix]

Sorry, I think SFP+ still wins even for home users. 10GBase-T is a dead-end technology because it has no obvious place to evolve.

These are very valid points, but for consumers "10 GbE" is shiny hot stuff that they get with the Aquantia controller on their new mid-/high-range motherboard or with their Mac mini/Mac Studio. Invariably as 10 Base-T—and almost invariably Aquantia, even Supermicro uses it for its Xeon W-3000 motherboards over the Intel X550/X710 NICs which come standard for server boards.
So users will just use 10 GBase-T as it comes, and because it fits with all their other 1 and 2.5 GBase-T devices and perhaps even cables. Until they eventually find out that 40 GbE, 25 GbE and beyond are all optical and that copper 10 GbE is a dead-end. We are not there, and for the next years the theme here will be to explain how to integrate legacy 1 GbE devices and a computer with 10 GBase-T (can't be changed for SFP+ if it's a Mac) with "future-proof" (if such a thing exists in IT…) optical networking.

 

[danb35]

can't be changed for SFP+ if it's a Mac

Sure it can:

Solo10G SFP+ (10Gb Ethernet Thunderbolt Adapter) - SONNETTECH

10GbE Thunderbolt adapter with short-range SFP+ transceiver. Connects through a RJ45 10GbE port to 10GBASE-T 10GbE networks. For Thunderbolt 4 and 3 computers.

www.sonnettech.com

or

https://www.amazon.com/Gigabit-Fiber-Copper-Ethernet-Converter/dp/B07CNBSG5D

or

100M/1G/2.5G/5G/10GBase-T to 10GBase-X SFP+ Unmanaged Fiber Media Converter - FS.com

FS unmanaged 1x 100M/1G/2.5G/5G/10GBase-T to 1x 10GBase-X SFP+ 10g media converter is compatible with SFP+, BIDI, CWDM transceiver modules. The unmanaged media converter has auto-negotiation and auto-MDI/MDI-X RJ45 port to support cat5/5e/6 cables.

www.fs.com

 

[Etorix]

@danb35 The first option adds a SFP+ port through Thunderbolt; it does not replace the 10 GBase-T port soldered to the motherboard.
The other options converts the copper output to optical… for the cost of several Base-T transceivers for use with a SFP+ switch.

In all cases, I'd rather use a transceiver at the end of a copper cable. It's much cheaper and easier.
My point is that, although I do not challenge that "10 GBase-T is best avoided if one can", many will not, or could not, avoid it.

Incidentally, if someone knows of a European equivalent to fs.com, I'm all ears.

 

[jgreco]

Incidentally, if someone knows of a European equivalent to fs.com, I'm all ears.

The European equivalent to fs.com is, unsurprisingly, fs.com. They operate a warehouse in Germany and offer the same "same-day shipping" options for prebuilts. Custom stuff will still come from their facilities in China.

 

[Jamberry]

do you have a 12 port 10GBase-T switch? Have you accounted for those?

I am currently only evaluating the switch to 10Gbit. But even IF I would switch to 10GBase-T, I would use a SFP+ switch with a 10Gbase-T module.

Put differently, if you got a most modern and efficient 10GBase-T chipset and compared it to an X520, yes, the X520 would lose. But that has nothing to do with the transmission medium type, it's just because the X520 silicon runs a barbeque restaurant as a side business.

Totally get what you are saying, but the thing is, if I would buy a modern SFP+ card, the price would be even more in favor of 10GBase-T.
This is the current pricing for my location:
Mobo markup for 10GBase-T: 120$
X520: 300$
X550: 420$
5y old used X520: 120$

The latest 10GBase-T ethernet silicon can potentially be within a horseshoe's throw of somewhat older SFP+ ethernet silicon when used in a single-port role.

That seems perfectly fine for a lot of home users.

X550 takes 11,2W typical and 13W peak according to data sheet. In my rough calculations, that would be 2W extra for the NIC (11,2W - 9,6W) and 2W extra for the SFP+ GBase-T module (2,7W - 0.8W).

4W power consumption 24/7 is 10$ a year for me. That would take a long time to make up for the 300$ markup (420$ - 120$).

I know these variables a different for everyone, this is just my reality. I will still go for fiber, because that is how my ISP enters my home, but I can totally understand how a video editor would go for 10GBase-T.

 

[Etorix]

You can find Solarflare SF5122/SF6122/SF7122 cards on eBay for (a little under) $50, shipped from China. Reportedly work great with TrueNAS and sip power, contrary to X520.
Even after going through customs this should beat on-board 10GBase-T.

 

[Jamberry]

Thank you for the hint, these Solarflare cards look really promising! Also found out about fs.com, boy do they have cheap cables and modules! This really changes the value proposition for me.

 

[WI_Hedgehog]

OMg, those fiber prices...I'm still in the "used and abused" market. Wait, that's a different account.

 

[jgreco]

OMg, those fiber prices

No, not OMg. You probably want OM3 or OM4. I really like this particular product:

Customized LC-LC OM4 Uniboot BIF Fiber Patch Cable - FS.com

Buy customized LC UPC to LC UPC Duplex 2.0mm OM4 Multimode Uniboot BIF Fiber Patch Cable from reliable fiber patch cable supplier-FS.COM!

www.fs.com

It's bend insensitive (probably don't take that too literally) and MUCH easier to groom than conventional fiber; both fibers are inside the single sheath.