10GbE networks cards fiber vs copper?

[DataKeeper]

I see most people using the Chelsio, for example, SFP module 10GbE cards with fiber. Any reason, aside from price, for using this setup over say an intel x540T1 at $299 over copper? I see the cost of 10GBase-T switches dropping over the next few years and even now one can get 10 ports for 8 bills. Still expensive for sure but much cheaper then what's been available.

 

[Darren Myers]

So......i was just about to post about this. I found a 10GB switch that takes modules for fiber...im in the process of testing now, but theres a catch, the switch is old....like 2007 old lol. But so far i have 4 10GB devices connected to it.

I have my FreeNAS, iSCSI SAN, and 2 ESXi hosts all connected via 10GB to the same switch and its all working 100% (thus far) But this switch also only takes modules and has no way to make it connect to a 10/100/1000 switch, so im using as a "data backbone" and if everything goes as planned ill add dual connections on each server + my Mac Pro tower.

I was originally doing daisy chains of 10GB fiber...until i realized ill be doing like 6 daisy chains and having ti play with different subnets + ip schemes

 

[DataKeeper]

I picked up one of these Netgear 48 port 10/10/1000 switches for $110. Comes with 4 SFP ports. Likely use one to link into the Netgear 10 port 10GbE switch I want to add later as I have a few short fiber cables here. Still, I prefer to work with copper myself.

 

[Darren Myers]

I really like working with fiber, i just dont appreciate the cost of 10GB, we have 10GB fiber ALLLLL over the datacenter i work in, the only copper that is used is for management NICs (iLo, ipmi ,etc). The switches we use are 5K a pop....but 10GB been around long enough where it really SHOULD be cheaper.

 

[marbus90]

SFP is not SFP+. Only the latter would support 10Gbe. But there are some cost-effective switches for $300 around with 2x SFP+ ports. Could be used with said Netgear 8 or 12port switches with 1-2x SFP+ uplinks and server interconnection via 10Gbase-T.

 

[j_r0dd]

I picked up one of these Netgear 48 port 10/10/1000 switches for $110. Comes with 4 SFP ports. Likely use one to link into the Netgear 10 port 10GbE switch I want to add later as I have a few short fiber cables here. Still, I prefer to work with copper myself.

SFP maxes out at 4.25gbps. SFP+ is what you need. You can get a Mikrotik http://routerboard.com/CRS226-24G-2SplusRM with two SFP+ ports for $299. The 10G modules are $59 each. So for ~$420 you can get 2 devices with that higher bandwidth.

 

[DataKeeper]

Thanks for the correction between SFP and SFP+ as I didn't realize that. Still, the switch I purchased will do for now and can be used to handle the phone & intercom systems, printers, current set top boxes and home audio controllers.. none of which currently requires that much bandwidth.

 

[j_r0dd]

Thanks for the correction between SFP and SFP+ as I didn't realize that. Still, the switch I purchased will do for now and can be used to handle the phone & intercom systems, printers, current set top boxes and home audio controllers.. none of which currently requires that much bandwidth.

Well you still will get a boost in bandwidth. From what I've read there are 1.25gbps modules for SFP. I've been doing more and more research since I want to add 10GbE for my server. Technically you don't even need to add modules if your server is close enough to the switch. You can get SFP/SFP+ direct-attach cable which run about $30. So for about $650 you can get the switch, direct-attach cable and a dual Intel X710DA2. Not bad considering how much this stuff used to run, but still not exactly cheap.

 

[Ericloewe]

Well you still will get a boost in bandwidth. From what I've read there are 1.25gbps modules for SFP. I've been doing more and more research since I want to add 10GbE for my server. Technically you don't even need to add modules if your server is close enough to the switch. You can get SFP/SFP+ direct-attach cable which run about $30. So for about $650 you can get the switch, direct-attach cable and a dual Intel X710DA2. Not bad considering how much this stuff used to run, but still not exactly cheap.

I'm not sure that SFP supports direct-attach. SFP+ does, but they're not as compatible as running fiber between two modules.

 

[jgreco]

I see most people using the Chelsio, for example, SFP module 10GbE cards with fiber. Any reason, aside from price, for using this setup over say an intel x540T1 at $299 over copper? I see the cost of 10GBase-T switches dropping over the next few years and even now one can get 10 ports for 8 bills. Still expensive for sure but much cheaper then what's been available.

Answered in the 10Gig Primer. SFP+ offers lower latency and lower power consumption, and the ability to use arbitrary optics (many kilometer reach!). 10GBASE-T will eventually win out, but is the VHS/Blu-Ray technology...

 

[wpirobotbuilder]

SFP maxes out at 4.25gbps. SFP+ is what you need. You can get a Mikrotik http://routerboard.com/CRS226-24G-2SplusRM with two SFP+ ports for $299. The 10G modules are $59 each. So for ~$420 you can get 2 devices with that higher bandwidth.

Have you used one of those routers, or know a lot about the quality/performance of them? That looks like a very appealing way to get 10Gb into a network.

BTW there is a less expensive version also with two SFP+ ports, linked here. The difference is in the level of RouterOS license, information about which is here, but it looks like the difference is negligible for most people.

 

[j_r0dd]

Have you used one of those routers, or know a lot about the quality/performance of them? That looks like a very appealing way to get 10Gb into a network.

BTW there is a less expensive version also with two SFP+ ports, linked here. The difference is in the level of RouterOS license, information about which is here, but it looks like the difference is negligible for most people.

I switched to Mikrotik over a year ago for my home set-up. We sell a network package using 2 of their products to our customers. I have had zero issues at home. We did get a bad batch about a year ago where about a half dozen routers just went into a bootloop (not at the same time) and were unrecoverable. That was just a one off issue. You get a lot of bang for the buck with all their products. We haven't used their 10GbE products in the field and probably won't any time in the near future because it just doesn't fit our needs. I will be trying out one of these at home once I have a few extra dollars http://routerboard.com/CCR1009-8G-1S-1Splus which would allow me to consolidate a switch and router.

 

[jgreco]

The question for these sorts of products is what sort of layer 3 capability is it. Switch silicon has gotten cheap enough that a device may be able to support some layer 3 functionality in silicon, but performance is going to suffer for anything complex.

http://routerboard.com/CRS226-24G-2SplusIN

Notice from the bottom performance chart that anything that has to traverse the CPU performs poorly, even large packet traffic.

The Ubiquiti EdgeRouter is also a CPU-based device but seems to have more CPU oomph.

 

[depasseg]

The Ubiquiti ERL runs on MIPS: Dual-Core 500 MHz, MIPS64 with Hardware Acceleration for Packet Processing (from their datasheet). I use a bunch of them. Huge fan.

 

[jgreco]

The Cavium's accelerated packet processing is incredibly effin' impressive. Even the low end box can do forwarding at 1Mpps, whereas the suggested Mikrotik is claiming 25Kpps, 1/40th of the EdgeRouter.

The problem with the EdgeRouter is that it is hard to tell which bits are supported by hardware acceleration and which aren't, so you can slam the wall at supersonic speeds and suddenly the thing is performing like a brick, because it is having to involve non-hardware-accelerated packet processing.

Some features like traffic shaping are poorly developed, and the other pps pain points are not well defined. Also, I'm more of a BSD guy, and I'd have a tendency just to throw a real fast CPU at it, and end up with something that might not be able to do 1Mpps but would perform very well and more evenly across the featureset.

 

[jgreco]

And then this morning Ubiquiti announces the EdgeRouter X ...

https://www.ubnt.com/edgemax/edgerouter-x/

Which appears to have lower performance (130Kpps) but also a great price ($49).

 

[j_r0dd]

And then there is this one that is way faster, but ~$500 http://routerboard.com/CCR1009-8G-1S-1Splus.

 

[jgreco]

Not too bad for $500...

 

[cyberjock]

I just bought a Mikrotik RB951G-2HND and it works "well enough". I can tell it bottlenecks with my 150Mbps internet and just a few simple firewall ruls, its a bit confusing even though I have lots of experience with pfsense, but the price was much lower than building a pfsense box (and much smaller). It throws up all over itself in the performance arena if you think you are going to use it as a VPN client though. I think I've seen 400KB/sec only once for a split second and I'm used to seeing 1MB/sec+ on my pfsense box that is over 1000 miles farther away (and has more latency as a result).

 

[DataKeeper]

OK so the Netgear GS748T switch was delivered and is junk! First thing I noticed was a badly twisted mount ear. Plugging it in resulted in the fans sounding like they were being feed an iron bar. Closer inspection found 2 fans were almost dead and grinding while slowly turning and the 3rd was grinding very loudly as it turned at various speeds. Fans are easy enough to replace so I moved on to test ports. Using new Cat6a cables I plugged its Port1 into my main switch, used Netgear's Smartwizard Discovery software from my desktop to find and assign an IP then pulled up the web interface... all good. Plugged a laptop into port 3 (static 192.168.1.245) and another into port 48 (192.168.1.58 via DHCP). From the laptop in port 48 I used "ping -t 192.168.1.245" got a couple replies, unplugged from port3 and into port5, etc along the top row. At port39 it failed to reply though link/speed leds lit up. OK 1 bad port.. bummer. Continuing on more and more started to fail. Switched out the laptop on port 48 to see if that port was failing however it didn't help. Pulled the switch - switch from port1 and inserted it into port39 and I could no longer pullup the web interface. Switched out cables and tested two more ports on the off chance and, as I knew, still no go. As I placed my hand on the top it was noticeably hot, far more then it should have been. Pulled power, disconnected laptops, cables, fired the thing back into the shipping box and emailed the seller.

Reminded me why I rarely order 2nd hand equipment!