Want to hop up on the 10 GbE train

[IceBoosteR]

Hi all,

I think about replacing my 1GBit network to the next level: 10GBit.
Of course I had a look into the wonderful 10GB Primer, but now I would like to know if I can start with the following parts and thoughts.
What I want to archive: FreeNAS should serve the network with high speeds. I do sometimes see that FreeNAS has enough resources, but the gigabit interfance is the bottleneck. Especially when my servers run (iscsi) and I want to transfer large amount of data. Therefore I would like to use LAGG on my ESXi-servers.

Network card for the FreeNAS server:
https://www.ebay.de/itm/10G-Netzwer...986805?hash=item3b17b7b8b5:g:9nEAAOSwxupbCzpC
(possible use for the second one in an ESXi host also)
or
https://www.ebay.de/itm/Chelsio-N32...380104?hash=item2132186448:g:-jcAAOSwySVaBaYP
with this cable:
https://www.ebay.de/itm/Cisco-5m-10...047967&hash=item236edbda93:g:5C4AAOSwENxXmO4y

As this is PCIe x8 version 1! it can handle up to 2GByte/s which should be enough for one port with full speed hopefully. Maybe there is another option available?

On the switch side I have chosen:
Netgear Insight GC700 Rackmout Gigabit Managed Switch, 24x RJ-45, 2x SFP, 2x SFP+ (GC728X-100)
which is around 300$. It has 2SFP+ 10GBit ports.
Some alternatives are:
TP-Link T1700G JetStream Rackmount Gigabit Smart Stack Switch, 24x RJ-45, 4x SFP+ (T1700G-28TQ)
and
maybe:
D-Link DGS-151 Rackmount Gigabit Smart Stack Switch, 24x RJ-45, 2x SFP, 2x SFP+ (DGS-1510-28)

But the favourite is the Netgear one. Hopefully I can deactivate all the Cloud stuff...
My biggest concern: Can the Netgear talk to the Chelsio card with a Direct-Attached-Cable? Don't want to troubleshoot and find out that I need transreceiver and OM3 cables also, the DAC sounds good to me :)

Any experience with that?

Thank you!

-IceBoosteR

 

[Chris Moore]

My biggest concern: Can the Netgear talk to the Chelsio card with a Direct-Attached-Cable?

I have seen situations where a transceiver was not accepted by the switch or network card because of some vendor lock problem, but I have not seen any issues with a direct attach cable.
This looks like it would work if you are only wanting to connect two systems at 10Gb. If you want to connect more than two at 10Gb, you might want to look for one of these:
https://www.ebay.com/itm/Aruba-Netw...-100-1000-Mobility-Access-Switch/132135155050
You might not need 48 1Gb ports, but it has four 10Gb ports and I have read that they work well. I am thinking of getting one.

 

[IceBoosteR]

Hi Chris,
thanks for your input.
That particular switch is not available where I live and it might be overkill. Even my colleagues think, that 10G would be overkill and 24 ports aswell ;)
So I only plan to connect my FreeNAS box, and maybe later another one, and then the option is there.
I think I will buy the parts and hope that it will work ;=)
-Ice

 

[Chris Moore]

It might be overkill to have 10Gb, but it sure it nice.

 

[IceBoosteR]

It might be overkill to have 10Gb, but it sure it nice.

Absolutly ;)

I am currently checking if those "Smart Cloud" features can be disabled. No need for this is a home network. Therefore I might take the TP Link one, it has 4SFP+ ports, but looks a lot older :O

 

[IceBoosteR]

I have seen also those Mellanox cards:
https://www.ebay.de/itm/10G-Netzwer...213666?hash=item21101c57a2:g:RHAAAOSwB-1YnZIr

If I remember correct, they had some issues with FreeNAS and/or the driver. Is that still true?
thanks

 

[Elliot Dierksen]

I have seen also those Mellanox cards:
https://www.ebay.de/itm/10G-Netzwer...213666?hash=item21101c57a2:g:RHAAAOSwB-1YnZIr

If I remember correct, they had some issues with FreeNAS and/or the driver. Is that still true?

It definitely was in FreeNAS/FreeBSD 9. I don't think it is now, but I moved to Chelsio NIC's when the Mellanox ones were a no go and haven't looked back.

 

[IceBoosteR]

It definitely was in FreeNAS/FreeBSD 9. I don't think it is now, but I moved to Chelsio NIC's when the Mellanox ones were a no go and haven't looked back.

Thanks for the feedback :)
The Mellanox ones are cheaper and full profile, while the Chelsio are only low profile ones.

 

[Elliot Dierksen]

The Mellanox ones are cheaper and full profile, while the Chelsio are only low profile ones.

You can get high or low profile brackets for the Chelsio cards. On the Mellanox ones, it looks like they are supported in FreeBSD 11.1 and later. You might have to force it to load the mlx_en driver/module. I have a vague recollection of getting that to work before moving away from the HP servers.

 

[IceBoosteR]

You can get high or low profile brackets for the Chelsio cards. On the Mellanox ones, it looks like they are supported in FreeBSD 11.1 and later. You might have to force it to load the mlx_en driver/module. I have a vague recollection of getting that to work before moving away from the HP servers.

Have not found a full profile Chelsio card, and I am unable to find a single bracket :/
Mhh I don't want to do much manually and I guess the Chelsion ones just work, so I may go for them.

 

[IceBoosteR]

Wow, I have seen the Chelsio card needs about 15watts! Thats more than the switch I want to buy and is nearly twice as much as my NUC server needs. That might be ne no-go for the Chelsio...

Edit/Update: Just bought the Mellanox ConnectX-2 and a5m Cisco DAC cable.
I have seen in the forum that it is working with FN11 :)

 

[Elliot Dierksen]

Just bought the Mellanox ConnectX-2 and a5m Cisco DAC cable.
I have seen in the forum that it is working with FN11 :)

My experience is that it is switches that are more likely to be picky about Twinax (aka DAC) cable and transceivers than NIC cards, but that is very manufacturer specific. I used the Mellanox cards in 4 or 5 ESXi hosts, and I don't recall them ever refusing to work with any of the Twinax cables I had, and some of those were cheapy chinese ones.

 

[jgreco]

Wow, I have seen the Chelsio card needs about 15watts! Thats more than the switch I want to buy and is nearly twice as much as my NUC server needs. That might be ne no-go for the Chelsio...

Edit/Update: Just bought the Mellanox ConnectX-2 and a5m Cisco DAC cable.
I have seen in the forum that it is working with FN11 :)

The older Intel and Chelsio cards eat a lot of power, and the Mellanox is around 8-10 watts IIRC. Most 10G capable switches will be eating more watts as well than a similar 1G.

My experience is that it is switches that are more likely to be picky about Twinax (aka DAC) cable and transceivers than NIC cards, but that is very manufacturer specific.

To expand upon that, which is a good point, my read on the issue is that switch vendors are very big into vendor-locked optics due to the opportunity to sell them at a premium price. That's not really a big problem on the used market as a pair of used SR optics and a cheap patch can be cheaper than a twinax/DAC these days. However, as an end user you still have to deal with the lock-in mindset of the switch vendor.

It is much harder for them to vendor-lock DAC cables. If you're Cisco or Juniper, you don't really sell network cards. So if you --hypothetically-- sell a vendor-locked switchport that will only take a Cisco DAC and a customer wants to plug that DAC into a (again hypothetical) Intel NIC that only wants to take Intel DAC, you have a real problem. Some vendors such as Mellanox actually do sell both sides of the link, but even there they can't sell every type of endpoint device.

Still, it pays to look at going fiber-and-SFP+. The long-term economics support it if there's any chance you will make changes that would result in different length cables. For the Dell, Intel, and Cisco SFP+'s, I've been picking them up at around $15-$20/ea with some a bit lower. You can get LC/LC BIF OM4 fiber (good to 100Gbps) in a 2 meter length for $7.50/ea. That means about $40 per link, but the SFP+'s are totally reusable and if you end up needing a different cable length, you just order the cheap bit of it, the fiber itself.

 

[Johnnie Black]

I have seen in the forum that it is working with FN11 :)

IIRC they work since at least 9.10, I use them myself and they work mostly fine, there are sometimes issues with jumbo frames enable, but they are almost gone with FreeNAS 11.1.

 

[Elliot Dierksen]

Still, it pays to look at going fiber-and-SFP+.

I agree for a couple reasons. I started out all Twinax, but am mostly fiber-SFP+. Twinax cables are a BEAR to route through the cable management arms for rack mounted servers. Also, if both ends of the links are really fussy about the optics, everybody can get what makes them happy in a fiber-SFP+ world. The length adjustment consideration is non-trivial too. If you should need it, it is fairly easy to get wider variety in length for fiber patch cables.

 

[jgreco]

I agree for a couple reasons. I started out all Twinax, but am mostly fiber-SFP+. Twinax cables are a BEAR to route through the cable management arms for rack mounted servers.

If you look at that picture I linked to, the difference between cat cable, traditional fiber, and modern BIF OM4 fiber is *stunning*.

https://extranet.www.sol.net/files/misc/biffiber.jpg

Not clear if you're referring to the foldy carp-arse per-server management arms that some manufacturers inflict on people, or just cable management in general, but either way, ... amazing.

Also, if both ends of the links are really fussy about the optics, everybody can get what makes them happy in a fiber-SFP+ world. The length adjustment consideration is non-trivial too. If you should need it, it is fairly easy to get wider variety in length for fiber patch cables.

Not to put too fine a point on it, but you can order them to exact lengths fairly inexpensively from several sources, the cheapest of which appears to be fs.com. You do have to dig around on the site to find the ordering page to do it: go under Fiber Optic Cables, then Uniboot LC Cables, then 1m (3ft) LC UPC to LC UPC Duplex 2.0mm OM4 Multimode Uniboot BIF Fiber Patch Cable, PVC (OFNR), then click the "Custom" button.

 

[danb35]

https://www.ebay.com/itm/Aruba-Netw...-100-1000-Mobility-Access-Switch/132135155050

Wow. I was happy with $469 for my Dell X1052 (though that was new, not used). I'm pretty sure I paid more than this for my 5524.

 

[Chris Moore]

I am currently checking if those "Smart Cloud" features can be disabled. No need for this is a home network. Therefore I might take the TP Link one, it has 4SFP+ ports, but looks a lot older :O

I asked the seller about that Aruba switch activation and the amount of noise it makes. They said that, "The switch is functional without activation key. This is just for cloud management and central configuration. These are rated at 42 dB."
I think I am going to order one, if they still have them when I have the cash available.

 

[jgreco]

Wow. I was happy with $469 for my Dell X1052 (though that was new, not used). I'm pretty sure I paid more than this for my 5524.

I updated the 10G Primer with some additional information on fiber and newer pricing on the Dell switches. I don't really care to try to make a purchase guide for switches, but it's nice to see that there are some great options for smaller networks now.

 

[IceBoosteR]

Hi guys,

I have now all the hardware at home and tested it. At first, everythin went fine - but after some minutes my testbench just freezes. I could not get the computer to start again (hangs at Windows bootup). So I dismounted the Mellanox NIC and bootet the system. Evenrything was fine. Mellanox card back in, and the computer freezes after some minutes. I switched to a FreeNAS installation. First all was fine for an hour. But I constantly have issue with "link not up" and when booting the system was freezing again when it was trying to load the driver for the 10G card. I guess I have received a faulty card, right?