10 Gig Networking Primer

10 Gig Networking Primer

danb35

Hall of Famer
Joined
Aug 16, 2011
Messages
15,464
Is there a resource that breaks down the Chelsio part/model numbers a bit? Better stated, something that explains the significance? Chelsio's FAQ at http://www.chelsio.com/general-information/ explains that the S in the model denotes storage and the N a NIC, for example, but what's the significance of that? What are the practical differences between the T3, T4, and T5 chipset? Each is respectively a newer generation, suggesting improvements, but are those improvements significant for a FreeNAS server?
 

pclausen

Patron
Joined
Apr 19, 2015
Messages
267
I'm putting together a list of what I need to establish a 10 Gbps fiber connection from a SFP+ port on my switch to my primary workstation located 40 feet away. My FreeNAS server is already connected to the switch @ 10 Gbps via a 2 meter Cisco Twinax copper cable. The FreeNAS has an Intel X520 NIC in it.

My plan is to purchase a 2nd X520 for my workstation. My understanding is that the I should stick with an Intel branded SFP+ optics module, such as this one?

http://www.ebay.com/itm/Intel-FTLX8...9-001-Optic-/401003133918?hash=item5d5da63bde

And I was thinking I would pick up a 15 meter cable from these guys:

http://www.cablesandkits.com/lclc-1...uplex-50125-fiber-patch-cable-15m-p-3796.html

LC - LC Multimode is what I want right?

That just leaves figuring out what SFP+ module to get for the switch side. Switch is a Ubiquiti UniFi switch with a pair of SFP+ uplink ports. Perhaps I can get away with the same SFP+ I plan to put in on the X520 end?

This is the X520 card I already purchased for the FreeNAS server:

http://www.ebay.com/itm/181773127011?_trksid=p2060353.m2749.l2649&ssPageName=STRK:MEBIDX:IT

It did not come with any SFP+ modules. Perhaps there are less expensive options out there that includes SFP+ optics? Perhaps I need to use different search terms to dig one up?
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
Reply got complicated enough you made me move to a Real Computer(tm).

I'm putting together a list of what I need to establish a 10 Gbps fiber connection from a SFP+ port on my switch to my primary workstation located 40 feet away. My FreeNAS server is already connected to the switch @ 10 Gbps via a 2 meter Cisco Twinax copper cable. The FreeNAS has an Intel X520 NIC in it.

My plan is to purchase a 2nd X520 for my workstation. My understanding is that the I should stick with an Intel branded SFP+ optics module, such as this one?

http://www.ebay.com/itm/Intel-FTLX8...9-001-Optic-/401003133918?hash=item5d5da63bde

Correct. While optics with the same part number (except the "-IT" at the end, "-Intel") really ought to work fine, they're vendor locked not to. This is a restriction imposed by the device (ethernet card in this case), not the optic. If you have a device that isn't vendor-locked, like a generic switch, the optics are interchangeable and you can use an Intel, Dell, Cisco, IBM, etc., optic in it.

And I was thinking I would pick up a 15 meter cable from these guys:

http://www.cablesandkits.com/lclc-1...uplex-50125-fiber-patch-cable-15m-p-3796.html

LC - LC Multimode is what I want right?

Yes. I will warn you that they are not the cheapest source, but they have been extremely pleasant to work with over the years, and the products are of good quality. You could probably get away with something less expensive if you're very cost sensitive.

That just leaves figuring out what SFP+ module to get for the switch side. Switch is a Ubiquiti UniFi switch with a pair of SFP+ uplink ports. Perhaps I can get away with the same SFP+ I plan to put in on the X520 end?

I wouldn't, without looking around. I'd suggest browsing the Ubiquiti forums for a little while to see if they require vendor locked optics (my guess: not). The Intels are too pricey. I was picking up Dells at $20-$25/ea shipped but those are BCL's not BCV's.

Actually I just spent a few minutes and it seems people have used the Dells successfully.

https://community.ubnt.com/t5/EdgeSwitch/SFP-modules-compatibility/td-p/1059539/page/2

Not sure exactly what hardware you have, so this is not any guarantee. But if you buy the Dell part and it doesn't work out, I might buy it from you if you're willing to sell it for $25 ("opportunistic bastard!").

This is the X520 card I already purchased for the FreeNAS server:

http://www.ebay.com/itm/181773127011?_trksid=p2060353.m2749.l2649&ssPageName=STRK:MEBIDX:IT

It did not come with any SFP+ modules. Perhaps there are less expensive options out there that includes SFP+ optics? Perhaps I need to use different search terms to dig one up?

Ah well they probably won't be less expensive options but what you're looking for is X520-SR2. The -DA2 variant is "D"irect "A"ttach "2" port, the -SR2 is "S"hort "R"ange. The only real difference is that the SR2 includes the optics.

http://www.ebay.com/itm/Intel-X520-...er-X520-SR2-E10G42BFSR-NIC-Dell-/321890267354
 

pclausen

Patron
Joined
Apr 19, 2015
Messages
267
Appreciate the detailed response!

I found a Dell FTLX8571D3BCL for $20 on eBay with $2.50 shipping, so I snatched it up. I have the 48 port 500W switch, which is the same one that guy you linked to tested it with. If it doesn't work, I'd be more than happy to sell it to you at my cost.

Good tip on the SR2 model of the X520. $209.95 seems like a good deal with optics. I might just snatch that one up and have a spare SFP+ module. Cost of X520-DA + a single module works out to almost the same price, especially when including shipping.

Yes, I saw your other post about cablesandkits and how you liked their attention to detail. I don't mind paying a few extra bucks for that.

Hopefully by the end of next week I'll finally be able to give my FreeNAS server a workout. This might be the excuse I need to invest in a RevoDrive for my workstation, lol.
 

pclausen

Patron
Joined
Apr 19, 2015
Messages
267
My fiber cable came in a few days ago. Here's the package:

fibercable01.JPG


Those dB values, do they represent light loss through the cable or something?
 
Last edited:

Ericloewe

Server Wrangler
Moderator
Joined
Feb 15, 2014
Messages
20,176
Those dB values, do they represent light loss through the cable or something?
Attenuation over distance. From a signal standpoint, they're losses.
 

tvsjr

Guru
Joined
Aug 29, 2015
Messages
959
Attenuation over distance. From a signal standpoint, they're losses.
Slight necropost, but FYI, the "IL" values are insertion loss (the total amount of loss through the fiber and connections). However, "RL" is return loss - in very simple terms, a measure of how good a "match" the fiber is for the wavelength of light intended for it.
 

ondjultomte

Contributor
Joined
Aug 10, 2015
Messages
117
Whats the best card for FreeNAS? Chelsio S310 CR or Mellanox connect X2 ? In the workstation I'm going to use Mellanox since that card works fine under win 10 from what I have heard.
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
Whats the best card for FreeNAS? Chelsio S310 CR or Mellanox connect X2 ? In the workstation I'm going to use Mellanox since that card works fine under win 10 from what I have heard.

The S310 is a crummy old gen card that's PCIe 1.1 and burns maybe 15 watts; there's a reason people are virtually giving them away on eBay.

The T420-CR is at least PCIe 2.0 and uses slightly less power.

If you can dump it in a slot that's actually x8, maybe it doesn't matter too much.
 

ondjultomte

Contributor
Joined
Aug 10, 2015
Messages
117
The S310 is a crummy old gen card that's PCIe 1.1 and burns maybe 15 watts; there's a reason people are virtually giving them away on eBay.

The T420-CR is at least PCIe 2.0 and uses slightly less power.

If you can dump it in a slot that's actually x8, maybe it doesn't matter too much.
Well 420 consume 14w. But it might have better offloading?

What about transivers? Is chelsio vendor specifik? Or can i buy sfps from fiberstore?
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
The T420-CR has two ports and PCIe 2.0, which seems like a better use of 14 watts.

The Chelsios do require their own transceivers, but as with most optics, you can probably find generics that contain the right vendor encoding, but the optics are also available fairly cheaply on eBay if you just keep your eyes open.
 

helloha

Contributor
Joined
Jul 6, 2014
Messages
109
I'm a bit confused by a lot of ebay listings of Chelsio cards, a lot of the time they don't list the Chipset model but the product code?

Like this one 110-1040-20 E0 form this listing: http://www.benl.ebay.be/itm/301798059146?_trksid=p2060353.m1438.l2649&ssPageName=STRK:MEBIDX:IT

The listing says that it's an FC card but I find other listings that says it is a NIC. Is it possible that it is both?

I cannot find any reference on the Chelsio website based on this product code.

After doing some more research I found that this card was also sold as a Netapp X1008A-R6 card.

But even on the Netapp website I find a reference to a NIC and FC is not even mentioned...

I found this in some Cisco documentation "via a NetApp X1008A-R6 2-Port 10GbE NIC TOE FC card that we installed in the NetApp FAS3040 (see Figure 5 for where we installed the X1008A 10 GbE)."

The layout of this card is almost identical to the S320E-CR, it just has a different heatsink.

Also it says that this card has the SFP+ modules attached (2x FTLX8511D3) This is why I'm interested in this deal, transceivers included...

Any info?

Thx!
K.
 

helloha

Contributor
Joined
Jul 6, 2014
Messages
109
Well 420 consume 14w. But it might have better offloading?

What about transivers? Is chelsio vendor specifik? Or can i buy sfps from fiberstore?

Curious what transceivers you ended up using. I just ordered a Chelsio S320E-SR and also ordered Finisar modules FTLX8571D3BCL.

I read they are the same ones as Chelsio sells, but then I read something about vendor encoding...

Cheers!
K.
 
Last edited:

helloha

Contributor
Joined
Jul 6, 2014
Messages
109
Wanted to confirm that the FTLX8571D3BCL modules work with Chelsio cards. But don't play nice with my Cisco SG-300 switch.

However I do have a newbie IT question.

My workstation is now directly connected to the NAS. But I also use the onboard NIC from the workstation to browse the web. How do I manage them to not interfere with each other.

When one is on, the other doesn't work and vica versa. Do I need to put both of them on a different subnet?
 

Pheran

Patron
Joined
Jul 14, 2015
Messages
280
Wanted to confirm that the FTLX8571D3BCL modules work with Chelsio cards. But don't play nice with my Cisco SG-300 switch.

However I do have a newbie IT question.

My workstation is now directly connected to the NAS. But I also use the onboard NIC from the workstation to browse the web. How do I manage them to not interfere with each other.

When one is on, the other doesn't work and vica versa. Do I need to put both of them on a different subnet?

Unless there is an SG300 model I don't know about that includes a 10Gbps port, I'm confused about why you would expect a 10Gbps SFP+ transceiver to work in a 1 Gbps switch?

As for your second question, just choose any subnet that you aren't already using in your network (I used 192.168.250.x, but it's totally arbitrary) and give the 10Gbps NIC on your PC one IP (e.g. 192.168.250.1) and the FreeNAS end another (192.168.250.2). When you map a drive on the PC, make sure you use the 10Gbps FreeNAS IP to do it, or you'll be using your 1G link instead. Do not try to configure any default gateway on your 10G IP. The default gateways stays with the 1G NIC; that's how the system knows how to get to the Internet.
 

Rainwulf

Explorer
Joined
Jul 12, 2015
Messages
67
I am after some chelsio 10gbe base-t cards. Does anyone have a list of the part numbers for them? Its hard to work out from the card descriptions if they are sfp or base-t.
I know that 10gbe copper SFP modules are hilarously expensive. I just want to put together a small 10gbe network without using fibre.
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
I am after some chelsio 10gbe base-t cards. Does anyone have a list of the part numbers for them? Its hard to work out from the card descriptions if they are sfp or base-t.

No it isn't. Except for a very small number of them such as the T520-BT which actually say they're 10Gbase-T, they're all SFP+. Chelsio specializes in high performance, and 10Gbase-T isn't the choice of people wanting high performance due to the poorer design.

I know that 10gbe copper SFP modules are hilarously expensive. I just want to put together a small 10gbe network without using fibre.

Yes, they're hilariously expensive because they don't exist. See

https://forums.freenas.org/index.php?threads/issue-with-x710-10gbe-sfp.29023/page-2#post-210719

where I discuss the sad facts.
 
Top