10 Gig Networking Primer

[Mlovelace]

I bet this would fit. I have both of the intel and chelsio cards so I can check the port/bracket compatibility tomorrow at work.

http://www.ebay.com/itm/like/231500349099?lpid=82&chn=ps&ul_noapp=true

 

[BigDave]

I bet this would fit. I have both of the intel and chelsio cards so I can check the port/bracket compatibility tomorrow at work.

http://www.ebay.com/itm/like/231500349099?lpid=82&chn=ps&ul_noapp=true

The dual ports of the Chelsio T420-CR cards are much closer together than that bracket.
I do hope someone finds a source though, I have two cards here collecting dust because of
the bracket issue.

 

[guc32]

The dual ports of the Chelsio T420-CR cards are much closer together than that bracket.
I do hope someone finds a source though, I have two cards here collecting dust because of
the bracket issue.

As long as the mounting point to the board is the same, I can always cut the divider and mount it.

Please update when you find out Mlovelace.

 

[Mlovelace]

As long as the mounting point to the board is the same, I can always cut the divider and mount it.

Please update when you find out Mlovelace.

The bracket mount points on the PCB are different, the ports on the Chelsio are closer together and start lower on the bracket. Bummer...

 

[jgreco]

The bracket mount points on the PCB are different, the ports on the Chelsio are closer together and start lower on the bracket. Bummer...

Standards, standards, why the hell can't we have standards...

 

[Mlovelace]

Standards, standards, why the hell can't we have standards...

If everything was standardized then no one could have all the pieces of the pie for themselves. I think I finally made some headway with buying non-name brand optical transceivers, speaking of standardized.

 

[guc32]

Chelsio gave me a free full height bracket for my T420-CR. Quick easy email back and forth and they left it for me to pick up.

PM me if you like the email of the person I dealt with.

Chelsio is A+ in my book. :D

 

[jgreco]

Which raises an interesting point, they might feel differently about one bracket than several.

 

[Mlovelace]

If anyone is looking for 10Gbe optical tranceivers for a decent price I got a couple tranceivers from www.fs.com to test and they have been stable/good. For the Chelsio 10Gbe card I tried the finisar compatible tranceiver http://www.fs.com/finisar-ftlx8571d3bcl-p-14945.html with the Extreme Networks 10301 compatible for my switch. I haven't seen any packet drops or latency increases and the network speeds are as good as they were with non-clone name brand transceivers.

Thought I'd mention it since their prices are better then the used market.

 

[Ericloewe]

What's the general experience with El-Cheapo SFPs, anyway? @Mlovelace, your experience seems rather positive.

I mean, I see some El-Cheapo SFPs (non-plus) at 25 bucks and I wonder how the hell they can make a profit off those things. And those Fiberstore guys seem to have their own El-Cheapo 10GBase-LR SFP+ at roughly 30 bucks, which sounds insane.
What's the catch here, besides silly attempts at vendor lock-in? Downrated range (crummy optics introducing extra noise or whatever)? Higher sensitivity to return signal noise? Spontaneous combustion?

 

[Mlovelace]

What's the general experience with El-Cheapo SFPs, anyway? @Mlovelace, your experience seems rather positive.

I mean, I see some El-Cheapo SFPs (non-plus) at 25 bucks and I wonder how the hell they can make a profit off those things. And those Fiberstore guys seem to have their own El-Cheapo 10GBase-LR SFP+ at roughly 30 bucks, which sounds insane.
What's the catch here, besides silly attempts at vendor lock-in? Downrated range (crummy optics introducing extra noise or whatever)? Higher sensitivity to return signal noise? Spontaneous combustion?

I heard about fiberstore from a colleague who has had a pair of 10Gbase-ER SFP+ transceivers in production going on 6 months now. They cost him $180 each when the vendor locked transceivers would have been $2200 each. He has had no issues and they are functionally the same as the non-clone modules. I can report a similar experience on a much shorter time line. So far with these transceivers I've had no issues with signal loss, noise, temperature increase, or packet loss. They have been functionally the same as the non-clone transceivers we have. I can't say if they will last year over year, but at $16-$20 each I don't care if I have to replace them more often.

 

[jgreco]

What's the general experience with El-Cheapo SFPs, anyway? @Mlovelace, your experience seems rather positive.

I mean, I see some El-Cheapo SFPs (non-plus) at 25 bucks and I wonder how the hell they can make a profit off those things. And those Fiberstore guys seem to have their own El-Cheapo 10GBase-LR SFP+ at roughly 30 bucks, which sounds insane.
What's the catch here, besides silly attempts at vendor lock-in? Downrated range (crummy optics introducing extra noise or whatever)? Higher sensitivity to return signal noise? Spontaneous combustion?

Well, there's three things here. One is the original manufacturer's SFP+'s. Two is something like a Finisar generic SFP+. Three is some Shenzhen knockoff.

One and two are 100% The Same Thing in many cases, since most manufacturers are outsourcing optic manufacturing to a specialist. Finisar's the Big Name in optics. So if you get a FTLX8571D3BCL from, let's say, Dell, and one from Finisar, the only difference is that Dell took it from Finisar and stuck a code in its EEPROM that says "Dell" and a serial number, and badged it with a Dell label. The Finisar one is the same in every other way.

Those of us who operate Internet networks will sometimes buy a programmer and buy the Finisars or whatever. https://mailman.nanog.org/pipermail/nanog/2015-August/078209.html For all the reasons @Mlovelace mentions (several thousand $$$$$$$$$$$$$$$$$$$$$$$$$ sometimes), it is substantially cheaper to buy the non-vendor optics. I *guarantee* you that the packets you've sent and received from this web site have gone over reprogrammed optics. The big guys have their own programmers. Us smaller fry, we can't justify a programmer. I actually prefer to buy the manufacturer optics on eBay if I've got vendor locked devices, but we have lots of generic Finisars too. By the way if anyone needs generic Finisar 1Gbps SX SFP (not 10Gbps) optics, got a box of maybe a hundred of the little bastards sitting here.

The third category, we don't actually know that much about the manufacturer of true generic optics, only that there's several of them, maybe even half a dozen. I can't speak to quality, noise, etc. You'd really have to try them out yourself, but really, at a tenth the cost of a vendor optic, you can afford to get three for each side... two for a LACP link (high availability) and one as a spare, and you're still better off than the vendor optic situation as long as they don't actually flame out. Heh.

 

[guc32]

Got the T420-CR today. Yes it did come with 2 Finiser Transceivers. Threw on the Full size bracket, installed in Freenas box. Hooked it up to my switch. Everything is running like butta!!!

Thanks for the help. :D Happy camper here.

 

[CookiesLikeWhoa]

So this might be deserving of it's own thread, but I figured I would start here. I'm looking to get a 10gbe switch and dip my feet into networking.

Thank you to jgreco and BigDave to helping me already and helping to direct me into stuff to look for.

I figure I have a $1500 budget for the switch itself. I will initially have 4 connections that will need the 10gig with more down the road. (probably)

I've been looking at:
NetGear XS712T
Dell X1052
Dell 8024(f)
Gnodal GS4008

The worry I have with some of the larger enterprise grade ones (8024) is how loud they will be and how well everything will play with the nas box I'm building.

The Netgear model seems the simplest, but is only really a smart switch, and not fully managed. It seems it will do what I need right now. It also seems to be the quietest. I'm just worried as things become easier (for me) and my needs grow that I will outgrow it and its limited scope.

The Dell 8024 seems to be a step up from the Netgear in every way but also seems to come with it's own set of issues. I do like that it is fully managed, though I doubt I'll be able to really take advantage of all of that initially, and that it has plenty of growth opportunities built in. I've also heard good things about its interface. My concern is size and noise. From what I've gathered this is a rather loud switch and sizable to boot.

Finally the Gnodal GS4008. I haven't really seen or heard much about it and don't even know if it's a managed switch. All I've really gathered is that it has a lot of growth opportunities and has 8 QSFP+ ports, which I doubt I'll ever need, but is fun to have.

Any direction would be hugely welcomed.

 

[jgreco]

With the exception of the Netgear, anything with all 10G or 10G/40G are *all* enterprise grade switches.

10G switches, including the Netgear, tend to be very noisy, because the maximum dissipation on these boxes is usually 100-300 watts in a 1U form factor.

Of them, the Gnodal unit may use somewhat less power on a per port basis. Gnodal was focusing on building extremely efficient ethernet switches primarily for ToR (top-of-rack) use, and they skipped a lot of the features that might be present on an edge switch that aren't commonly used. They made their own ASICs and reportedly the thing was pretty good. But they're dead now, so there's no new firmware or fixes. They're like "the cheaper Mellanox" or something like that. As long as you keep the management interface on a private net, it's probably a hell of a switch for a home user.

However, none of the big 10G's seem to make a lot of sense if you're a home user with a need for only ~4 ports initially. In the first post, I've suggested the 5524 and N2024 switches because they're relatively inexpensive and provide both 1G and a few 10G ports. The X1052 follows that sort of concept, and could easily be upgraded with an 8024F at a later date, so you should strongly contemplate whether or not this is something that can get you a year or two, at which point there may be better 10G options available.

 

[bartnl]

you could also include the Zyxel XS1920 in your shortlist. I use this at home and imho is relatively quiet.

 

[CookiesLikeWhoa]

With the exception of the Netgear, anything with all 10G or 10G/40G are *all* enterprise grade switches.

10G switches, including the Netgear, tend to be very noisy, because the maximum dissipation on these boxes is usually 100-300 watts in a 1U form factor.

Of them, the Gnodal unit may use somewhat less power on a per port basis. Gnodal was focusing on building extremely efficient ethernet switches primarily for ToR (top-of-rack) use, and they skipped a lot of the features that might be present on an edge switch that aren't commonly used. They made their own ASICs and reportedly the thing was pretty good. But they're dead now, so there's no new firmware or fixes. They're like "the cheaper Mellanox" or something like that. As long as you keep the management interface on a private net, it's probably a hell of a switch for a home user.

However, none of the big 10G's seem to make a lot of sense if you're a home user with a need for only ~4 ports initially. In the first post, I've suggested the 5524 and N2024 switches because they're relatively inexpensive and provide both 1G and a few 10G ports. The X1052 follows that sort of concept, and could easily be upgraded with an 8024F at a later date, so you should strongly contemplate whether or not this is something that can get you a year or two, at which point there may be better 10G options available.

Thanks for the info!

I think I may have been overthinking it a bit trying to avoid making a bad purchase. I think I'll stick with the N2024 since they are quite easy to stack from the looks of it and should provide for what I'm looking for right now.

Though if any one is interested there is a Gnodal GS4008 on ebay for $1,200.00. Seems like a relative deal compared to the pricing on the other ones.

 

[jgreco]

I don't think a single N2024 is a bad place to start, and even two of them probably represents less risk of capital loss than a full 10G switch that you don't really need. The prices on these things are dropping, not as fast as I'd like, but they're coming down.

 

[CookiesLikeWhoa]

That's kind of my hope. I figure I can eat the cost of a N2024 or two and use that for my needs right now and hope by the time I do actually need more than the 4 ports prices will have come down on the switches.

 

[jgreco]

Yeah but you're always going to have things that aren't 10GbE to plug in, so having something like the N2024 is almost a necessity anyways.