10 Gig Networking Upgrade

[Black Ninja]

I got me very exited about this switch. It has fantastic price , and used 10GB fiber adapters are very cheap on ebay. I only hope is reliable switch. The one the works strong and never have to reboot. I hope the freezing problem you have it's not cause by the switch ?!

 

[Darren Myers]

I got me very exited about this switch. It has fantastic price , and used 10GB fiber adapters are very cheap on ebay. I only hope is reliable switch. The one the works strong and never have to reboot. I hope the freezing problem you have it's not cause by the switch ?!

I dont think its from the switch as it only happens on my SAN, i think the 10Gb card in it is over heating even though the 2U fans are moving a good amount of air i dont think the cards heatsink is getting enough cooling. As i am still able to access my FreeNAS server via 10Gb, but cant ping or access the SAN.

 

[Black Ninja]

I dont think its from the switch as it only happens on my SAN, i think the 10Gb card in it is over heating even though the 2U fans are moving a good amount of air i dont think the cards heatsink is getting enough cooling. As i am still able to access my FreeNAS server via 10Gb, but cant ping or access the SAN.

So the switch is solid. You never had to reboot it once in awhile ? It's weir switch I am trying to learn more about it from researching the web, but input from a owner is has the most value for me.

 

[Darren Myers]

So the switch is solid. You never had to reboot it once in awhile ? It's weir switch I am trying to learn more about it from researching the web, but input from a owner is has the most value for me.

It's only been turned off or rebooted when i have to move other network gear around. But other then that no weird drops or anything, no odd power cycles

 

[Black Ninja]

Thank you very much for the input. I can't decide which way I am going to go: Geting 2 Dell Power Connect 6224 and Stack them , for getting total of 8x10gb ports plus 48 Gb ports , or if I switch to Copper 10Gb , switches netgear offer and not expensive and brand new with several option to chose from. I assume you chose 10Gb fiber because of cheap nics as well ?

 

[Darren Myers]

Thank you very much for the input. I can't decide which way I am going to go: Geting 2 Dell Power Connect 6224 and Stack them , for getting total of 8x10gb ports plus 48 Gb ports , or if I switch to Copper 10Gb , switches netgear offer and not expensive and brand new with several option to chose from. I assume you chose 10Gb fiber because of cheap nics as well ?

I choose the 10Gb because its the fastest thing i can use (kinda), and the parts have all become very cheap-ish (~180$ per dual port card + ~25$ per sfp+), and well its something for me to test and say i have experience with not only in a datacenter environment :D, i am going to add a quad port NIC to my SAN, and then 2 dual port NIC's to my ESXi servers and create a 2Gb fail over incase the 10Gb takes a dump and i cant diagnose it in time, it isnt a perm solution but itll give me time to investigate as it isnt occuring weekly, bi weekly, monthly, etc

 

[Black Ninja]

I choose the 10Gb because its the fastest thing i can use (kinda), and the parts have all become very cheap-ish (~180$ per dual port card + ~25$ per sfp+), and well its something for me to test and say i have experience with not only in a datacenter environment :D, i am going to add a quad port NIC to my SAN, and then 2 dual port NIC's to my ESXi servers and create a 2Gb fail over incase the 10Gb takes a dump and i cant diagnose it in time, it isnt a perm solution but itll give me time to investigate as it isnt occuring weekly, bi weekly, monthly, etc

Do you use Jumbo frames in your 10Gb network ?

 

[Darren Myers]

Do you use Jumbo frames in your 10Gb network ?

Nope, all defaults. i didnt screw with any of that as i read all the jumbo frame documentation

 

[Black Ninja]

I don't know how the people are mange to utilize 10Gb without jumbo, perhaps for iscsi plays different. I have Windows 7 with 4390K with 32Gb ram , and with Cifs share I can't go more that 30% network utilization. My single tread copy process is maxing one of the CPU cores and can't go any faster without jumbo ?! Perhaps you are accessing you Freenas with different protocols or not trying to get faster that 30% utilization ?!

 

[Darren Myers]

I don't know how the people are mange to utilize 10Gb without jumbo, perhaps for iscsi plays different. I have Windows 7 with 4390K with 32Gb ram , and with Cifs share I can't go more that 30% network utilization. My single tread copy process is maxing one of the CPU cores and can't go any faster without jumbo ?! Perhaps you are accessing you Freenas with different protocols or not trying to get faster that 30% utilization ?!

I'm using CIFS on my FreeNAS server, while this isnt a REAL test....i can cut and paste data from my windows servers to the FreeNAS server with 300-400Mbps; depending on the file. But i really didnt expect alot of speed from it as it doesnt need to be fast. The only thing fast is my SAN with 20 10K drives but thats purely for the VM's benefit

 

[Black Ninja]

I'm using CIFS on my FreeNAS server, while this isnt a REAL test....i can cut and paste data from my windows servers to the FreeNAS server with 300-400Mbps; depending on the file. But i really didnt expect alot of speed from it as it doesnt need to be fast. The only thing fast is my SAN with 20 10K drives but thats purely for the VM's benefit

I see. Well when you do your between windows copy-paste scenario with 300-400Mbps do you experience maxing one cpu core ?! Because I think that's the bottleneck from going to 980 Mbps ?!

 

[Darren Myers]

I see. Well when you do your between windows copy-paste scenario with 300-400Mbps do you experience maxing one cpu core ?! Because I think that's the bottleneck from going to 980 Mbps ?!

I have 16 cores at 2.93Ghz, i allocate 8 cores for this particular VM, and my FreeNAS server has another 16 cores at 2.4Ghz. I havent seem them get maxed yet...

 

[Black Ninja]

I have 16 cores at 2.93Ghz, i allocate 8 cores for this particular VM, and my FreeNAS server has another 16 cores at 2.4Ghz. I havent seem them get maxed yet...

Well I have 6 cores at 3.4 Ghz , but the problem is that Cifs only uses one core , if you have 16 core and only use 1 core at 2..93Ghz , it should be even worst results for you , no ? Unless in VM doing things differently but in regular non-vm machine Cifs is single treath and maxes 1 core leaving all the rest not used.

 

[Darren Myers]

Well I have 6 cores at 3.4 Ghz , but the problem is that Cifs only uses one core , if you have 16 core and only use 1 core at 2..93Ghz , it should be even worst results for you , no ? Unless in VM doing things differently but in regular non-vm machine Cifs is single treath and maxes 1 core leaving all the rest not used.

It may indeed be single cored, but i havent found a bottle neck as of yet. The next time i do file copies over to FreeNAS ill watch the CPU %

 

[Black Ninja]

It may indeed be single cored, but i havent found a bottle neck as of yet. The next time i do file copies over to FreeNAS ill watch the CPU %

Check it out. Here is your bottleneck 300Mbps. I think the reason why you can copy with 300Mbps instead of 980 Mbps is the cpu core being at max. I am still testing , but I notice with jumbo enable I can offload my CPU core significantly and I think that is the reason speed can jump to 980Mbps. ( I am only interested in cifs speed at this point) I am still testing , but determined to make 10Gb to be close to 10x times faster , not just little bit faster than 1Gb. We are used to utilize 1Gb above 90% easy most of the time , why not get the same from 10Gb, no ?

 

[Black Ninja]

I add a second nic to freenas 10Gb SR single port and connect back to back with desktop pc.I only gave it IP on different subnet. It is working but I notice in a console I am getting message" ...host name/name mismatch.." and it's refering to my desktop ip on 10Gb link.

What other setting need to be applied when second 10Gb nic is connected back to back with desktop ? I only gave it static ip on different subnet and subnet mask 255.255.255.0 , is there more to it ?

 

[cyberjock]

Nope.. but the fix for host name mismatch is clearly explained in the manual. This isn't limited to having the second 10Gb. It's a matter of whether your network is configured to do host name lookups. Obviously, the second 10Gb network you have doesn't have such a setup, so it is failing.

 

[Black Ninja]

Nope.. but the fix for host name mismatch is clearly explained in the manual. This isn't limited to having the second 10Gb. It's a matter of whether your network is configured to do host name lookups. Obviously, the second 10Gb network you have doesn't have such a setup, so it is failing.

The second nic which is 10Gb is connected directly to the same nic located in the desktop. Like we talk before - back to back with 2 intel E15729. It's not part of any network , it just has an static ip on different subnet. I am confused ?! Are you telling me how to fix this host name lookup situation, or you are trying to confuse me ?:p

 

[cyberjock]

The second nic which is 10Gb is connected directly to the same nic located in the desktop. Like we talk before - back to back with 2 intel E15729. It's not part of any network , it just has an static ip on different subnet. I am confused ?! Are you telling me how to fix this host name lookup situation, or you are trying to confuse me ?:p

You are confused... and I am telling you how to fix it! LOL.

 

[Black Ninja]

You are confused... and I am telling you how to fix it! LOL.

Sounds more like a riddle. Well I didn't get it , tell me what is it ?:(