10 Gig Networking Upgrade

Status
Not open for further replies.
Joined
Nov 11, 2014
Messages
1,174
Here's some of the parts you might be using with a fiber deployment.

On the top, a nicely built OM3 aqua fiber patch. This is a nice quality cable we sourced from "cablesandkits.com", which has several little awesome touches. First, look at how neatly they assemble it for shipping, with the fancy cross-laced twist-ties on both connectors. You can see at a glance that there's some TLC that goes into it. There's a section of heat shrink tubing that prevents further splitting of the fiber a few inches from the connector, and that the connectors are neatly spaced from the end of the fiber. This is something I've found to be often off on cheap fiber - one connector being off by a centimeter or more. And polarity markings, so that you can identify which strand is which at each end.

Grades of fiber below OM3 exist. Most 62.5/125 fiber (OM1) is capable of running 10G for at least two dozen meters, and basic 50/125 fiber (OM2) is sufficient for most SOHO purposes (several dozen meters). OM3 will get you 300 meters. But just like HDMI, it is a digital signal - there's no value to be had in buying a $1000 cable if a $10 cable works without error.

View attachment 7511

Click to enlarge.

On the bottom, an Intel X520-SR2. This card includes SR optics, which are removable from the SFP+ cages. If using optics, Intel does require the use of their optics, and the cards usually come with the Intel variant of the Finisar FTLX8571D3BCV-IT. Do note that other non-Intel optics are not likely to work in the card!

The other end would be an optic plugged into the switch of your choice. Again, these may be vendor-locked. We have the occasional argument about this over on NANOG, but basically most people understand that vendor locking is a transparent ploy to make you buy marked-up optics. So do yourself a favor and buy them on eBay, used, from a reputable reseller, probably for about what a generic optic would be new.

Thank you very much for sharing your experience with this vendor , this is exactly what I was looking for , now I know where I'll by my fiber cables. At least no more wondering for good quality vendor. Now the next step is to find if there is something wrong using Intel E15729 card ? I hear about the card from Cyberjock post and it's fantastic price on ebay.
 
Joined
Nov 11, 2014
Messages
1,174
Here's some of the parts you might be using with a fiber deployment.

On the top, a nicely built OM3 aqua fiber patch. This is a nice quality cable we sourced from "cablesandkits.com", which has several little awesome touches. First, look at how neatly they assemble it for shipping, with the fancy cross-laced twist-ties on both connectors. You can see at a glance that there's some TLC that goes into it. There's a section of heat shrink tubing that prevents further splitting of the fiber a few inches from the connector, and that the connectors are neatly spaced from the end of the fiber. This is something I've found to be often off on cheap fiber - one connector being off by a centimeter or more. And polarity markings, so that you can identify which strand is which at each end.

Grades of fiber below OM3 exist. Most 62.5/125 fiber (OM1) is capable of running 10G for at least two dozen meters, and basic 50/125 fiber (OM2) is sufficient for most SOHO purposes (several dozen meters). OM3 will get you 300 meters. But just like HDMI, it is a digital signal - there's no value to be had in buying a $1000 cable if a $10 cable works without error.

View attachment 7511

Click to enlarge.

On the bottom, an Intel X520-SR2. This card includes SR optics, which are removable from the SFP+ cages. If using optics, Intel does require the use of their optics, and the cards usually come with the Intel variant of the Finisar FTLX8571D3BCV-IT. Do note that other non-Intel optics are not likely to work in the card!

The other end would be an optic plugged into the switch of your choice. Again, these may be vendor-locked. We have the occasional argument about this over on NANOG, but basically most people understand that vendor locking is a transparent ploy to make you buy marked-up optics. So do yourself a favor and buy them on eBay, used, from a reputable reseller, probably for about what a generic optic would be new.

The modules (optics) coming our of the card is this the same module that will go in to the switch (Dell 5524) ?
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,680
Prolly want to use Dell optics for the Dell switch. That's what we mean by vendor lock-in. Pretty sure that the 5524 will take the Dell 0N743D. That's what we're using with the 7xxx and 8xxx Dells. Should be about $25 on eBay.

The big difference between the Dell optic and the Intel optic is a matter of some identification codes embedded in the electronics. It's stupid, but, there ya have it. Life in enterprise-grade-land is a peculiar and annoying thing..
 
Joined
Nov 11, 2014
Messages
1,174
Prolly want to use Dell optics for the Dell switch. That's what we mean by vendor lock-in. Pretty sure that the 5524 will take the Dell 0N743D. That's what we're using with the 7xxx and 8xxx Dells. Should be about $25 on eBay.

The big difference between the Dell optic and the Intel optic is a matter of some identification codes embedded in the electronics. It's stupid, but, there ya have it. Life in enterprise-grade-land is a peculiar and annoying thing..

I got little confused here, order to simply: If I buy Intel X520-SR2 and want to connect to dell 5524 , what modules or optics do i need in order to connect them most compatible way possible ?
 

Dennis.kulmosen

Explorer
Joined
Aug 13, 2013
Messages
96
Just a note about vendor locking. You can buy the supermicro cards based on the Intel chipset, they are not vendor locked and works perfectly in FreeNAS. As switch, Netgear enterprise-grade switches are not vendor-locked. Netgear is a big support of open-standards so they intend to stay away from vendor-locking of SFP+ modules. At least that is what Netgear told me. [emoji4]
But if you already have the switch, then ebay is your friend or buying refurbished modules from the vendor. [emoji4]
 
Joined
Nov 11, 2014
Messages
1,174
Just a note about vendor locking. You can buy the supermicro cards based on the Intel chipset, they are not vendor locked and works perfectly in FreeNAS. As switch, Netgear enterprise-grade switches are not vendor-locked. Netgear is a big support of open-standards so they intend to stay away from vendor-locking of SFP+ modules. At least that is what Netgear told me. [emoji4]
But if you already have the switch, then ebay is your friend or buying refurbished modules from the vendor. [emoji4]

I am trying to understand does module on the nic card has to match the one goes in to the switch ? They are both different brand ? How you connect Intel to Dell then? Is the same module on both ends ?! I am new to fiber and little confused ?!
 

Dennis.kulmosen

Explorer
Joined
Aug 13, 2013
Messages
96
No the SFP modules needs to be the vendor branded for the unit they go into.
So basicly all you need to be sure of is both SFP+ modules are SR (short range) og LR (long range). The fiber cable between doesnt care about vendors. So the card in the FreeNAS can require a SFP module that is vendor branded.
Lets say you have an intel card which then req. An intel branded SFP. Then you put the fibre cable between to a lets say an HP switch which req. an HP branded SFP module. There you have the chain.
All I say in the previous post is that supporting of non-vendor-locking products might be a good thing if you buy new stuff. That is what I do to my clients, and an SFP module is just an SFP module wether is have been vendor branded or not. Some workshops even have a SFP flasher where they can rebrand SFP modules to their liking. [emoji4]
 
Joined
Nov 11, 2014
Messages
1,174
No the SFP modules needs to be the vendor branded for the unit they go into.
So basicly all you need to be sure of is both SFP+ modules are SR (short range) og LR (long range). The fiber cable between doesnt care about vendors. So the card in the FreeNAS can require a SFP module that is vendor branded.
Lets say you have an intel card which then req. An intel branded SFP. Then you put the fibre cable between to a lets say an HP switch which req. an HP branded SFP module. There you have the chain.
All I say in the previous post is that supporting of non-vendor-locking products might be a good thing if you buy new stuff. That is what I do to my clients, and an SFP module is just an SFP module wether is have been vendor branded or not. Some workshops even have a SFP flasher where they can rebrand SFP modules to their liking. [emoji4]

Thank you very much. Now I get it, i think. I was confused that Intel module on a nic card has to be the same module to go in to the switch , because whaever pattern of light the intel module is sending , I was thinking it will need same intel module to listen and understand.
 

Dennis.kulmosen

Explorer
Joined
Aug 13, 2013
Messages
96
Nope. The light in the cable is a standard. SR and LR is what needs to be the same in both ends. And multimode and single mode, and with multimode and single mode comes different types of cables too.
I know its a jungle out there. :smile:
 
Joined
Nov 11, 2014
Messages
1,174
Nope. The light in the cable is a standard. SR and LR is what needs to be the same in both ends. And multimode and single mode, and with multimode and single mode comes different types of cables too.
I know its a jungle out there. :)

Today I already learn about single and multimode, also SR-LR and now I graduate with knowing that I need LC-LC cable OM3 aqua color !:smile:))

Just little trouble still with matching the right modules !:smile:
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,680
I am trying to understand does module on the nic card has to match the one goes in to the switch ? They are both different brand ? How you connect Intel to Dell then? Is the same module on both ends ?! I am new to fiber and little confused ?!

No, the optic has to match the device you're inserting it into. If you've got an Intel 10G card, you probably need an Intel optic in it. If you've got a Cisco switch, you probably need a Cisco optic in it. This is mostly an issue of vendors trying to screw users out of dollars because they charge big bucks for their own special optics.

See, the optics have electronics in them, for diagnostic purposes. In a larger deployment where you're shining light a long distance through several distribution frames, sometimes you'll need to be able to tell if the received light is within usable levels, or if you have to have the fiber guys fix their crap. So you ask the switch:

Code:
#show idprom interface tengigabitethernet 1/0/24

Type.............................. SFP+
Media............................. 10GBASE-SR
Serial Number..................... AJQ0W4M
Dell Qualified.................... Yes
#show fiber-ports optical-transceiver tengigabitethernet 1/0/24

                               Output  Input
Port      Temp Voltage Current Power   Power   TX    LOS
          [C]  [Volt]    [mA]  [dBm]   [dBm]   Fault
--------- ---- ------- ------- ------- ------- ----- ---
Te1/0/24  31.0 3.312       6.8  -2.532  -1.981 No    No

Temp - Internally measured transceiver temperatures.
Voltage - Internally measured supply voltage.
Current - Measured TX bias current.
Output Power - Measured optical output power relative to 1mW.
Input Power - Measured optical power received relative to 1mW.
TX Fault - Transmitter fault.
LOS - Loss of signal.


So this is useful stuff to know, but you'll note in there that there's also "Dell Qualified". In the case of Dell, they don't actually require that the module be a Dell qualified module, but you may not get warranty support if it isn't. The generic Finisars work with the Dell gear. However, you can usually obtain the Dell qualified SR's for about the same price as Finisars if you shop on eBay, and, really, why not do that. But it is possible for vendors to lock you out of being able to use generic optics on their gear, because the optics do have the ability to provide this information to the card or switch as diagnostic information. The smart move is to make sure that if you buy a Foobar device, you put Foobar optics into it.

But that's it. That's all there is to this. Once you're past the optic itself, 10G ethernet is 10G ethernet, and you should be able to run a fiber between your Intel card with its Intel optic and your Cisco switch and its Cisco optic without any further trouble. What goes over the fiber is just pure 10G, and that is very well standardized. Going with SR fiber is the most flexible and compatible way to make sure you can hook up any two devices, as the technology has been out there for a dozen years. The details are admittedly a bit finicky, but you can follow conservative rules and guarantee things will work:

1) Use OM3 fiber everywhere

2) Use the hardware vendor's suggested/required optics

3) Follow all best practices for handling fiber (i.e. no sharp bends)

Got it now?
 
Joined
Nov 11, 2014
Messages
1,174
No, the optic has to match the device you're inserting it into. If you've got an Intel 10G card, you probably need an Intel optic in it. If you've got a Cisco switch, you probably need a Cisco optic in it. This is mostly an issue of vendors trying to screw users out of dollars because they charge big bucks for their own special optics.

See, the optics have electronics in them, for diagnostic purposes. In a larger deployment where you're shining light a long distance through several distribution frames, sometimes you'll need to be able to tell if the received light is within usable levels, or if you have to have the fiber guys fix their crap. So you ask the switch:

Code:
#show idprom interface tengigabitethernet 1/0/24

Type.............................. SFP+
Media............................. 10GBASE-SR
Serial Number..................... AJQ0W4M
Dell Qualified.................... Yes
#show fiber-ports optical-transceiver tengigabitethernet 1/0/24

                               Output  Input
Port      Temp Voltage Current Power   Power   TX    LOS
          [C]  [Volt]    [mA]  [dBm]   [dBm]   Fault
--------- ---- ------- ------- ------- ------- ----- ---
Te1/0/24  31.0 3.312       6.8  -2.532  -1.981 No    No

Temp - Internally measured transceiver temperatures.
Voltage - Internally measured supply voltage.
Current - Measured TX bias current.
Output Power - Measured optical output power relative to 1mW.
Input Power - Measured optical power received relative to 1mW.
TX Fault - Transmitter fault.
LOS - Loss of signal.


So this is useful stuff to know, but you'll note in there that there's also "Dell Qualified". In the case of Dell, they don't actually require that the module be a Dell qualified module, but you may not get warranty support if it isn't. The generic Finisars work with the Dell gear. However, you can usually obtain the Dell qualified SR's for about the same price as Finisars if you shop on eBay, and, really, why not do that. But it is possible for vendors to lock you out of being able to use generic optics on their gear, because the optics do have the ability to provide this information to the card or switch as diagnostic information. The smart move is to make sure that if you buy a Foobar device, you put Foobar optics into it.

But that's it. That's all there is to this. Once you're past the optic itself, 10G ethernet is 10G ethernet, and you should be able to run a fiber between your Intel card with its Intel optic and your Cisco switch and its Cisco optic without any further trouble. What goes over the fiber is just pure 10G, and that is very well standardized. Going with SR fiber is the most flexible and compatible way to make sure you can hook up any two devices, as the technology has been out there for a dozen years. The details are admittedly a bit finicky, but you can follow conservative rules and guarantee things will work:

1) Use OM3 fiber everywhere

2) Use the hardware vendor's suggested/required optics

3) Follow all best practices for handling fiber (i.e. no sharp bends)

Got it now?

Oh yes I got it. Thanks a lot. Now I even know who to buy a cable from . I like to follow the conservative rules for hardware. From what I learn , I am thinking to buy 2 cards Chelsio T420-CR and Intel X520-SR, get the proper modules for each card , connect back to back with LC-LC om3 and it should work right ? They will auto crossover ?, just like connecting 2 rj45 nics back to back ?
 

cyberjock

Inactive Account
Joined
Mar 25, 2012
Messages
19,525
If you buy them and cross-connect them directly, they will auto crossover.
 
Joined
Nov 11, 2014
Messages
1,174
If you buy them and cross-connect them directly, they will auto crossover.

What I mean is: remember back in a day when you connect 2 pc together you need crossover cable not the regular cable you use to connect to switch, but now you connect 2 pc directly with your regular cable and the nic will do auto crossover. Is this same way when connecting to cards with fiber LC-LC ? Same way mean no need to alter or cross fiber wires , just plug then the same way you would plug then in switch and nics will do crossing automatically ?
 

cyberjock

Inactive Account
Joined
Mar 25, 2012
Messages
19,525
What I mean is: remember back in a day when you connect 2 pc together you need crossover cable not the regular cable you use to connect to switch, but now you connect 2 pc directly with your regular cable and the nic will do auto crossover. Is this same way when connecting to cards with fiber LC-LC ? Same way mean no need to alter or cross fiber wires , just plug then the same way you would plug then in switch and nics will do crossing automatically ?

That's what I meant. If you read what I did in my other thread you'd know they auto-crossover because that's what I did. ;)
 
Joined
Nov 11, 2014
Messages
1,174
That's what I meant. If you read what I did in my other thread you'd know they auto-crossover because that's what I did. ;)

Oh I read it. I just wasn't sure how is done. And to prove I'll ask you a question regarding that post:):
You said you first started with 2x Intel E15729 10GB XF connected back to back. This cards are really cheap on ebay and I would like to try them if there no catch, so How did they work for you when you use then ?
 

cyberjock

Inactive Account
Joined
Mar 25, 2012
Messages
19,525
Oh I read it. I just wasn't sure how is done. And to prove I'll ask you a question regarding that post:):
You said you first started with 2x Intel E15729 10GB XF connected back to back. This cards are really cheap on ebay and I would like to try them if there no catch, so How did they work for you when you use then ?

They worked great. I couldn't get full 10Gb. It think I capped out at something like 6 or 7Gb in testing. But it's well known that Intel's 10Gb offerings are good, but not great, on FreeBSD/FreeNAS. They have been rock solid, aside from a driver bug in certain 9.2.1.x builds. But let's face it, I can't get 7Gb/sec through CIFS anyway since it's single threaded, so I don't care. And for the price difference between going Intel and Chelsio is pretty wide, I'll stick with the Intel and the slightly-less-than-10Gb 10Gb.
 
Joined
Nov 11, 2014
Messages
1,174
They worked great. I couldn't get full 10Gb. It think I capped out at something like 6 or 7Gb in testing. But it's well known that Intel's 10Gb offerings are good, but not great, on FreeBSD/FreeNAS. They have been rock solid, aside from a driver bug in certain 9.2.1.x builds. But let's face it, I can't get 7Gb/sec through CIFS anyway since it's single threaded, so I don't care. And for the price difference between going Intel and Chelsio is pretty wide, I'll stick with the Intel and the slightly-less-than-10Gb 10Gb.

That is great news. I should buy 2 immediatly.
Unrelated but important , what is you X9SCM-F bios version ? Can I check my from Freenas shell with command instead of turning off and go to bios ?
 

cyberjock

Inactive Account
Joined
Mar 25, 2012
Messages
19,525
dmidecode will probably tell you your bios version. Mine is kept up-to-date regularly, so I'd be on the latest or one version behind at the worst. Not sure of the version number, but again, I'm not home to check. :P
 
Joined
Nov 11, 2014
Messages
1,174
dmidecode will probably tell you your bios version. Mine is kept up-to-date regularly, so I'd be on the latest or one version behind at the worst. Not sure of the version number, but again, I'm not home to check. :p

I got 2 x Intel E15729 single port cards. I tried to pull the module out while playing with it , but it doesn't seem go come out. Please don't laugh at me , is it removable at all ?:D
 
Status
Not open for further replies.
Top