10Gb NIC - Anybody using them and what's the performance like? Should I get them?

[mstang1988]

I'm considering putting a 10Gb NIC in my freenas box and running a direct connection to my desktop rig. I'll have to figure out how I can route the traffic as I also need both systems to have their 1Gb ports on the same network.

Anyway, anbody using 10Gb NIC's? Do you have performance numbers? What brand do you recommend? (I know Chelsio and Intel are popular but how about blade engine 3 adapters from Emulex etc?)

 

[louisk]

I'm not clear on why additional network cards are required to be on the same network if they are directly connected. Can you please explain?

Something to keep in mind is that typically, w/o quite a beefy system, you will not get >1g performance.

 

[mstang1988]

I'm not clear on why additional network cards are required to be on the same network if they are directly connected. Can you please explain?

Something to keep in mind is that typically, w/o quite a beefy system, you will not get >1g performance.

Both my desktop and my freenas box need to be on the 1Gb network as they distribute to other clients and/or access the internet. I'm confident that I can beef my NAS up to be > 1Gbps (although unlikely to really utilize the full 10Gb). The read speed from the NAS is already 350MB/s ish when testing zfs with a 24GB file (did a pretty good job blowing out the cache before getting this speed). I move large files between my desktop/nas as they back each other up. I would prefer to do this at 10Gbps. Anyway, I don't want to invest in a 10Gb switch because they are $$$$ but NIC's are becoming more affordable and could be directly connected on their own subnet. Hence both 10Gb and 1Gb and being on the same network.

 

[peterh]

There is no problem bridging 10G and 1G ethernets together. In fact it's o no-issue to have some systems use 10G and some 1G
( and maybe others 100Mb or 10Mb)
And if you yhought you couls connect 2 10G NIC's back-to-back ? Forget it. You will need a box ( call it bridge switch or something else)

 

[mstang1988]

There is no problem bridging 10G and 1G ethernets together. In fact it's o no-issue to have some systems use 10G and some 1G
( and maybe others 100Mb or 10Mb)
And if you yhought you couls connect 2 10G NIC's back-to-back ? Forget it. You will need a box ( call it bridge switch or something else)

We connect 10Gb NIC's back to back at work all the time. There is no reason that you cannot connect 10Gb NIC's (you just flip the fiber cable or use an RJ-45 crossover for 10Gb-T). I do not need or want a box. I understand that I can bridge but that's not what I'm looking to do. I want no additional hardware, thus trying to figure out a method to set priority for the NIC's etc.

 

[paleoN]

Anyway, I don't want to invest in a 10Gb switch because they are $$$$ but NIC's are becoming more affordable and could be directly connected on their own subnet. Hence both 10Gb and 1Gb and being on the same network.

You seem to be misunderstanding something or using terminology incorrectly. You want the 10Gb on a different subnet and directly, exclusively connected. So, they should go on a separate network.

Example:

Code:

Desktop, FreeNAS 1Gb  network: 10.18.0.0/16
Desktop, FreeNAS 10Gb network: 192.168.37.0/24

You would then add an entry in the hosts file in your desktop & FreeNAS respectively for the 192.168.37.0/24 addresses.

 

[mstang1988]

You seem to be misunderstanding something or using terminology incorrectly. You want the 10Gb on a different subnet and directly, exclusively connected. So, they should go on a separate network.

Example:

Code:

Desktop, FreeNAS 1Gb  network: 10.18.0.0/16
Desktop, FreeNAS 10Gb network: 192.168.37.0/24

You would then add an entry in the hosts file in your desktop & FreeNAS respectively for the 192.168.37.0/24 addresses.

You are absolutely correct, I described it incorrectly. Thanks for clarifying! What I meant to say was different subnets but both hosts will be on the 1Gb network and then exclusively connected with a 10Gb NIC on a different subnet. I plan to force (if I can figure out how) the priority of leveraging the 10Gb NIC first. I believe the easiest method with be to modify /etc/hosts to resolve the DNS as the 10Gb port and allow the 1Gb to be resolved via the DNS server. I understand it leverages whichever address is returned first. I also understand you have to force Samba to use both networks. Before I invest in 10Gb I'll likely try it with 1Gb first to make sure I can get it setup properly.