10 Gig Networking Primer

10 Gig Networking Primer

Scampicfx

Contributor
Joined
Jul 4, 2016
Messages
125
Thanks for all the recommendations so far!

Is there a resource that breaks down the Chelsio part/model numbers a bit? Better stated, something that explains the significance? Chelsio's FAQ at http://www.chelsio.com/general-information/ explains that the S in the model denotes storage and the N a NIC, for example, but what's the significance of that? What are the practical differences between the T3, T4, and T5 chipset? Each is respectively a newer generation, suggesting improvements, but are those improvements significant for a FreeNAS server?

I have one more question regarding this point:
What is the differene between Chelsio T520-CR and T520-LL-CR? I had a look to http://www.chelsio.com/wp-content/uploads/2013/09/Adapter-Selector-Card.pdf but I can't see any difference there, except that the LL has a bit more power consumption than the non-LL.
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,680
The LL is ultra low latency. It's a great card but it isn't likely to matter for FreeNAS.
 

Scampicfx

Contributor
Joined
Jul 4, 2016
Messages
125
jgreco, thank you!

By the way, are there some important things to know when comparing Switches Netgear XS708T (Smart Managed) and XS708E (Web Managed Plus, whatever this means...)? They have similar prices, the XS708T offers a bit more VLANs and stuff, is a bit more quite, consumes a bit less power, has 2x SFP+ Ports... is that all? May I use such a managed switch for testing purposes as a subunit of a subunit in a cascaded unmanaged network or do have managed switches to be on top of network infrastructures?
 

Scampicfx

Contributor
Joined
Jul 4, 2016
Messages
125
I tried to outline it with the attached graphic:

Unbenannt-1.jpg
 

Ericloewe

Server Wrangler
Moderator
Joined
Feb 15, 2014
Messages
20,194
Managed switches can be wherever you want them. However, the closer you get to the top, the more management stuff you generally want.
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,680
There's no "top" or "subunits" in a switched network, except perhaps for specific considerations such as the spanning tree topology, which the switches can figure out for themselves. Any of your switches can be managed or not. Obviously this may limit the capabilities of your network, but there are no hard rules about placement of managed switches.

If you are cascading a large number of 10G switches one after another, this of course creates a lot of forwarding and you might be better off with a different topology.
 

Scampicfx

Contributor
Joined
Jul 4, 2016
Messages
125
Thanks for your replies and your input! I wasn't quite sure about managed switches, since I've never used them so far... Now, everything is clear :)
 

cyberjock

Inactive Account
Joined
Mar 25, 2012
Messages
19,526
We use Intel x520-da2 cards with both 9.3 and 9.10. They are flawlessly reliable with more than a year of usage on 9.3 and a few months on 9.10.

We have done no performance tuning. Performance is good enough - better than the 1GB x 4 LACP we had in place previously - so we elected to stick with the stock configuration. I'm sure we're leaving some bits on the table but we're not well enough versed in BSD network tuning to start twirling knobs.

Cheers,
Matt
+1

I love the sentiment. I also use X520 cards and aside from the bug fixed 2 or so years ago, I've had zero problems.
 

Scampicfx

Contributor
Joined
Jul 4, 2016
Messages
125
Hey guys, I just noticed another point: Signal Encoding... ( Link: http://www.rfwireless-world.com/Terminology/10GBASE-T-vs-10GBASE-R-vs-10GBASE-X-vs-10GBASE-W.html )

10GBASE-T supports 64B/65B signal encoding
- supports transmission over twisted pair cabling

10GBASE-R supports 64B/66B signal encoding
- supports transmission over optical fiber medium
- 10GBASE-SR, 10GBASE-LR, 10GBASE-ER and 10GBASE-LRM

10GBASE-X supports 8B/10B signal encoding
- supports both copper (10GBASE-CX4) as well as fiber optic (10GBASE-LX4) mediums

So, from my point of view 10GBASE-SR fits perfectly for small environments and due to 64B/66B signal encoding there is less overhead compared to 8B/10B. I think jgreco posted a good overview about SR/LR and all that technologies at the first post in this thread.

However, I had a look to the description of the Netgear XS708T which is mentioned at the first page of this thread: It supports...
2x Shared (combo) 10GBASE-T copper/10GBASE-X SFP+ ports supporting 10G/1G speeds
https://netgear.com/images/Products...XS748T_DS_15Apr16.pdf?cid=wmt_netgear_organic

What are the benefits / advantages / disadvantages when comparing 10GBASE-SR to 10GBASE-LX4? It is possible to connect a 10GBASE-LX4 SFP+ module (switch) to a 10GBASE-SR SFP+ module (network card)? Wouldn't it be better to go with 10GBASE-SR in switch as well as in network card in server?

Or am I mixing up all the things completely? xD
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,680
No. Stay away from all the -X crap. It's all legacy crap. CX4 is basically four Infiniband channels at 3Gbps or something like that. LX4 is basically four wavelengths over fiber. LX4 is absolutely not compatible with SR, LR, or anything else for that matter. This is not the stuff you want to be using.

Use SFP+ with 10GBase-SR/LR or 10Gbase-T unless you really like delving into hardware compatibility issues and then like to be disappointed by how your first generation 10G hardware burns watts like a flamethrower while simultaneously is incapable of actually maxxing out 10G.

The reason you may see deals on this gear on eBay is because no one wants it, and they're hoping some sucker (you) who doesn't know any better picks it up and gives them beer money.
 

Scampicfx

Contributor
Joined
Jul 4, 2016
Messages
125
That's a pretty clear answer. Thanks jgreco!

However, there is still one point I'm wondering about:
https://www.thomas-krenn.com/en/pro...t-10gbit-switch-netgear-xs712t-10gbase-t.html

The transceiver modules offered at this shop are SR-/LR-modules... So, does this mean that in this 10GBASE-X switch 10GBASE-SR/-LR SFP+ modules get plugged in, although that isn't a good choice to do? Are there any recommended switches out there, which support 10GBASE-R (SR/LR)?
 
Last edited:

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,680
The transceiver modules offered at this shop are SR-/LR-modules... So, does this mean that in this 10GBASE-X switch 10GBASE-SR/-LR SFP+ modules get plugged in, although it isn't a good choice? Are there any switches out there, which support 10GBASE-R (SR/LR)?

What gives you the idea that this is a "10GBASE-X" (vs "10GBASE-R") switch? And I would strongly suggest you drop that terminology, it's not used by anyone anymore (and never really was to begin with, due to the complexity within each family).
 

Scampicfx

Contributor
Joined
Jul 4, 2016
Messages
125
Have a look:

https://netgear.com/business/products/switches/smart/XS708T.aspx#tab-techspecs

10gbase-x.JPG


The copper parts are 10GBASE-T. The SFP+ ports are 1000/10GBASE-X.

However, one of the SFP+ modules listed in the optional part list is a 10GBASE-SR module:

sfp.JPG

PDF, last page: http://www.downloads.netgear.com/files/GDC/datasheet/en/XS712T-XS728T.pdf

That's the point I don't get... The SFP+ Ports of the switch itself are 10GBASE-X, however the SFP+ Transceiver aren't... So, I'm wondering if it is still recommended to use such a switch, since 10GBASE-X shall be avoided and is just crap ;)
 

Ericloewe

Server Wrangler
Moderator
Joined
Feb 15, 2014
Messages
20,194
The SFP+ Ports of the switch itself are 10GBASE-X
No, the SFP+ ports are SFP+ ports. That's it. All the physical layer details are abstracted away by the SFP+ modules. That's the whole point of SFP modules.
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,680
You've created your own confusion. You asked about "10GBase-X" in the context of a signal encoding type, so I answered your question in the context you provided. "Don't use that, it's total crap."

This in no way suggests that every time you see "10GBase X" that they're talking about CX4 or LX4 (and I pretty much even warned that nobody uses that terminology). CX4 and LX4 are not actually true 10G, they're 4 x 2.5G aggregated together. They function as 10G, after a fashion, yes, but they have to run through a switch or interface chip in order to be "usable".

But there was also "1000Base-X", used to denote a range of 1G ethernet options that were mostly fiber, from the common 1000Base-T UTP copper. In general the 1000Base-X options used modules, usually swappable, sometimes fixed, and so generally people might refer to things that weren't UTP as "1000Base-X" and even be almost completely correct. I would take "10GBase-X" to be a generalization of that for 10G, and it's easy to find examples of such use in industry. The very fact that they say "SFP+" means it is *impossible* for it to be signal encoded as 10GBase-X. There aren't four channels available on SFP/SFP+.

You do NOT want to learn all about this crap. I do not want to REMEMBER all this crap. Just trust me here. Go get yourself some happy SFP+ gear, or happy 10GBase-T gear if you don't mind spending $$$$.

For SFP+ stuff, check to see if you need a manufacturer-specific SFP+ module, and if so, get them. SX fiber is very affordable and hooking fiber between devices is a very highly compatible way to make sure stuff "just works."
 

Scampicfx

Contributor
Joined
Jul 4, 2016
Messages
125
Ahhhh! :D Okay, I got it now! Thanks for claryfing and thanks so much for your help and your detailed answers! 10Gbps, I'm on my way! ;)
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,680
Ahhhh! :D Okay, I got it now! Thanks for claryfing and thanks so much for your help and your detailed answers! 10Gbps, I'm on my way! ;)

You'll enjoy it. One of the reasons I wrote this guide was that I had talked to a number of forum users who were either confused by or anxious to try some of the other things that they had seen suggested for "cheap 10G" on places like ServeTheHome, and often it turns out that these are insanely cheap but still not worth the money you pay for them. You really don't want a janky old CX4 ethernet setup with its Infiniband cabling. It reminds me too much of 10Base-5 with AUI. It might "work" fsvo the word, but good FreeNAS support and actual 10G transfers probably aren't included.

Get a nice Intel X520 or Chelsio T420-CR, an SFP+ capable switch, and life's good. Today, while I haven't tried it, I'd certainly look at the newest Netgear 10Gbase-T switches that have been released as well, as a 10Gbase-T option. Maybe 2016 will be the year of 10Gbase-T. Finally.
 

Mirfster

Doesn't know what he's talking about
Joined
Oct 2, 2015
Messages
3,215

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,680
Side question, what are your thoughts on something like this: Quanta LB6M
I am still kinda holding out for a Dell PowerConnect 8024F but am curious...

No direct experience but I know they're kinda common in the data center. Noisy and hot, probably. There's some backstory you should probably research, I think maybe Quanta got bought out or something odd happened to them, I just don't remember offhand and don't have time to Google-fu it. It's nice to be able to get firmware updates.
 
Top