10 Gig Networking Primer

[jgreco]

What you do is you look for information about the seller.

The seller who has a name like shiningrainbow2021 and is selling x520 cards alongside sandals, electric foot massagers, and fashionable purses is a questionable seller who may be getting knockoff crap shipped from Asia by the containerful. They will have usually very positive feedback figures in like the five or six digits.

The seller who has lots of computer parts being sold by "recycleit" and has used Supermicro, Dell, HP chassis available in large quantity that appear to be of a 4-to-8-year vintage, where they've clearly taken a server and broken it up into parts, and are selling the chassis, the memory, the hard drives, the hard drive trays, the add-in cards, cables, maybe network switches and other data center miscellanea. These people are parts'ing up used data center gear, and the stuff is going to be the real deal. They typically have only been on eBay a year or two, don't have a lot of feedback, but the feedback that's there is often less than stellar. This is because those of us who buy professionally in the used market often don't leave feedback, we're just happy with what we get, but there are morons who buy stuff that they don't understand, it's perfectly good stuff but they are not able to use it. So you get this weird effect where they seem to have unreasonable feedback rates until you actually look at the feedback, where I typically roll through and go "moron ... moron ... idjit ... seriously?" I do take lucid, cogent, plausible, properly spelled feedback into account, especially where there is some sort of "DOA warranty" that isn't being honored, or stuff is shipped badly, because that is a red flag.

I'm sorry if that sounds cynical or insulting to eBay buyers, but quite frankly I have had pretty good luck with this sort of strategy. We buy lots of stuff used. Sometimes it doesn't work out, but vetting your sellers to be legitimate recyclers is one of the best ways to avoid fakes.

 

[Revolution]

That helps me a lot, thank you very much for the writeup!

 

[Riddick]

Can someone help me?
I recently ordered the Intel Ethernet X710DA2 SVR with 2 Ubiquiti UF-MM-10G sfp+
Freenas told me the media type is not supported.

I'm not sure, can i use fiber optics with this card? if yes, which sfp+ i should buy? Maybe i just ordered the wrong.

 

[jgreco]

Intel makes the Intel X710DA2, so one would expect Intel optics to work with it.

From Intel's tech support site:

https://www.intel.com/content/www/u...007045/network-and-i-o/ethernet-products.html

It says, in part,

• Intel® Ethernet SFP+ SR Optics and Intel® Ethernet SFP+ LR Optics are the only 10-Gbps optical modules supported. You can purchase these modules separately.
• You can't use other brands of SFP+ (10 Gbps) modules with these adapters.
• Tested modules are listed in the Intel® Product Compatibility Tool.

• Other brands of SFP+ optical modules don't work with the Intel® Ethernet Converged Network Adapter (Intel® Ethernet CNA) X710 Series. You must use the listed Intel® SFP+ SR or LR Optics.

The Product Compatibility Tool can be used to check for compatibility. I would also expect that Intel-coded generic optics from someplace like fs.com would work, as would used eBay optics, both of which should be substantially less expensive than new Intel optics.

 

[Riddick]

Intel makes the Intel X710DA2, so one would expect Intel optics to work with it.

From Intel's tech support site:

https://www.intel.com/content/www/u...007045/network-and-i-o/ethernet-products.html

It says, in part,

• Intel® Ethernet SFP+ SR Optics and Intel® Ethernet SFP+ LR Optics are the only 10-Gbps optical modules supported. You can purchase these modules separately.
• You can't use other brands of SFP+ (10 Gbps) modules with these adapters.
• Tested modules are listed in the Intel® Product Compatibility Tool.

• Other brands of SFP+ optical modules don't work with the Intel® Ethernet Converged Network Adapter (Intel® Ethernet CNA) X710 Series. You must use the listed Intel® SFP+ SR or LR Optics.

The Product Compatibility Tool can be used to check for compatibility. I would also expect that Intel-coded generic optics from someplace like fs.com would work, as would used eBay optics, both of which should be substantially less expensive than new Intel optics.

Thank you! Other question, if i use the Intel SFP+ and on the other side of the fiber the Ubiquiti 10g SFP+ should it work?

 

[Constantin]

Thank you! Other question, if i use the Intel SFP+ and on the other side of the fiber the Ubiquiti 10g SFP+ should it work?

Yes. AFAIK, the vendor lock non-sense only applies to the transceiver. So as long as a "intel" transceiver is shoved into a "Intel" SFP+ slot on one end and a "Mikrotik" transceiver is shoved into a "Mikrotik" SFP+ slot on the other end, the connection should be fine. That's what I have here.

The only complication I have encountered is SFP+ transceivers not playing nice with SFP slots (which I mistakenly thought were SFP+). That was a dumb oversight on my part. In other words, do not assume that a 10GbE transceiver can work in a SFP slot (it won't) nor that a 10GbE transceiver in a SFP+ slot can "throttle down" to a 1GbE connection speed (many don't, and those that do, advertise it). I have some Intel transcievers here that can run at 1 or 10GbE, which is a nice-to-have.

Plenty of folk can sell you either a used genuine Intel transceiver or faked new ones that have a Intel vendor ID burned into them.

 

[danb35]

I headed over to STH to try to come up with a suitable mezzanine dual-10G NIC for my Proxmox hosts on nodes of a Dell PowerEdge C6220 II, and while I was there, ran across this:

Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

NOTE #1: do not PM me with switch questions, they will be ignored - post them in this public thread, where hundreds of other members can also answer, and the answer will be public for future users NOTE #2 06-22-2023: Yes, this post is still up to date and nothing has changed: in fact judging by...

forums.servethehome.com

It's looking like a single ICX6610 could replace my Dell X1052, my Unifi US-8-60W, and my two Mikrotik units. A number of other units are listed there as well, and there are what look like pretty good docs for flashing to the latest firmware and at least basic configuration. Look like some nice options for 10G (and even 40G) connectivity.

 

[jgreco]

Yeah, the Brocade ICX stuff is good, some of it originally came from Foundry (hence the Iron moniker). Would buy it without a second thought, but be careful about licensing. You can probably find cheap licenses as this stuff is all EOL as far as I can tell.

 

[danb35]

You can probably find cheap licenses

Free:

ICX6450 Licensing - Fohdeesha Docs

fohdeesha.com

Apparently for the 7xxx gear it's on the honor system:

ICX7xxx Licensing - Fohdeesha Docs

fohdeesha.com

 

[jgreco]

Possibly not legal, but perhaps the licensing cops aren't going to beat down your door.

Actually I'm really tempted. One of my first gig switches was a Foundry FastIron Workgroup Switch 24+2G which is what we used to aggregate 100Mbps ethernets on Intel ISP1100's into two 1G fiber uplinks. It was a great switch. They started to feel the pressure in the mid-2000's and sold themselves off at some point, alas.

Now that some of the better name brand gear is becoming available, I'm a bit torn. I could really use higher port count 10G switches like the ICX6650 to replace several of the core switches here, most of which are 10G or 10/40. But the availability of cheap 25/100 stuff is probably just a handful of years away. I'm not sure that I mind being "limited" to 10/40 though given the vast choices of available gear, and really network utilization here doesn't tend to be super-heavy.

 

[danb35]

I expect the legality is more of a gray area than might be expected, but as this is well outside my specialty it probably wouldn't be wise for me to opine further. But if the gear isn't being made any more, nor are the licenses being sold, it doesn't seem there'd be a whole lot of reason for any extant company (I guess Ruckus is the current owner of the brand?) to pursue any recourse here.

BTW, I'd appreciate if you could give any input into my thread in OT where I pinged you--I think the question has come down to (1) whether these NICs can be made to do Ethernet, and (2) whether they'd allow splitting the QSFP+ into individual SFP+ lanes (thus allowing the use of a QSFP+-to-SFP+ adapter). The Intel 10G NICs with the proper mezzanine bracket appear to be unobtainium.

 

[jgreco]

BTW, I'd appreciate if you could give any input into my thread in OT where I pinged you

You did?

I think the question has come down to (1) whether these NICs can be made to do Ethernet, and (2) whether they'd allow splitting the QSFP+ into individual SFP+ lanes (thus allowing the use of a QSFP+-to-SFP+ adapter).

I may not be finding the right search thingys for that particular card. Or it may just be crappy support docs.

If we look at a similar generation Mellanox card for Dell, we see

https://web.archive.org/web/2022021...etworkcard_MC[GQ]H29B-XCC_user_manual_1_2.pdf

These cards can usually be configured for either/or Infiniband or Ethernet. The specifics of the card types are a bit opaque to me as I have not been a Mellanox fan; I don't have any insight into this particular card. At that price point, it's the sort of thing that's not worth a lot of time looking into, and I would be tempted to just buy one and see what happens(*)

The ConnectX-2 cards have been slowly losing support in various OS's as they are very late in their lifecycle. Be certain your ${stuff} supports them. I noticed a post over on Level1Techs just so you don't screw yourself. I got hung by the ESXi deprecation of the Linux driver compatibility shim and now I have a bunch of Solarflare cards sitting around doing nothing. At some point I should probably hold a TrueNAS virtual garage sale.

Breakout cables should be a possibility if the driver in question supports it. However, there's examples of where that hasn't been true. It looks like the chipset supports it.

(*)This is how I usually get into trouble. ;-)

 

[danb35]

You did?

Yep, Saturday morning. Maybe notifications didn't work like I'd expected.

I may not be finding the right search thingys for that particular card.

That's validating--at least it isn't just me. And while the PDF you found isn't for exactly the card I'm looking at, it still seems encouraging.

I would be tempted to just buy one and see what happens(*)

That's pretty much where I was at anyway. Ordered one, I'll see what happens with it and hopefully order a few more once I can confirm it works.

very late in their lifecycle.

Doubtless that's why they, like the Chelsio S310 cards, are so cheap.

 

[jgreco]

That's validating--at least it isn't just me.

No. This is a common problem with vendor gear. These bits are designed to be sold along with systems and to come from the factory all configured and deployed, so there may be minimal documentation or no documentation; the entirety of what features are actually supported may have existed only on one web page that described the card as an option for the system with a line that vaguely said "Infiniband/Ethernet support".

People wonder why I create a local cache of files I've found I need...

 

[danb35]

It's looking like a single ICX6610 could replace my Dell X1052, my Unifi US-8-60W, and my two Mikrotik units.

Might as well update--it did, and it works great. As promised in the STH thread, the web UI sucks, but configuring it via the CLI wasn't too painful. I ended up with a 6610 in the server rack (in the workshop 75' from the house), and a 6450 in the network closet in the house. The 6450 is quieter than the Dell 5524P it replaced, but the 6610 runs hot (70+°C) and is a screamer--good thing it's a long way from the house. And Oxidized supports both of them, so I'm able to keep a regular config backup of each switch in a local Gitea repo (which, naturally, tracks any changes).

Only downside compared to the Dells they replace is the lack of a "lifetime" (well, at least until 2040) warranty.

 

[danb35]

Any chance of a separate primer, update to this one, or some other relevant thoughts on 40 GbE? Those 40G ports in the 6610 are calling to me...

 

[Constantin]

...while I was there, ran across this:

Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

NOTE #1: do not PM me with switch questions, they will be ignored - post them in this public thread, where hundreds of other members can also answer, and the answer will be public for future users NOTE #2 06-22-2023: Yes, this post is still up to date and nothing has changed: in fact judging by...

forums.servethehome.com

...

Well, a lot of people seem to have run into this guide. Those switches are now retailing at 2x+ what the guide suggests on eBay. I have no doubt that they're still worth that but they're no longer the absolute bargain the poster suggested.

 

[danb35]

Yeah, prices had gone up quite a bit even when I bought mine in May. I'd really like to find one of the ICX7150-C12Ps, but they've gotten stupid expensive.

 

[Ericloewe]

Well, 40GbE is really just four 10GbE links in a trenchcoat (and sometimes they let kids in and you can use four 10GbE links to separate devices), so I figure much of the conversation stays the same. 25GbE might switch things up a bit with new hardware, though.

 

[Constantin]

Yeah, prices had gone up quite a bit even when I bought mine in May. I'd really like to find one of the ICX7150-C12Ps, but they've gotten stupid expensive.

For that kind of money ($500 and up) you can also get a managed UBNT 8 port switch with 2.5GbE interfaces and two SFP+ slots, brand-new. Granted, likely not as high quality as brocade but for indoor SOHO use likely sufficient.

I’m still considering trying to get the ICX7150 for external POE+ cameras and like assets since there the benefit of better input protection is more apparent. Connecting the underlying switch only via optical to the rest of the IT system also provides a layer of protection.