10 Gig Networking Upgrade

[mjws00]

If there is a place jumbo would work. It should be on a direct connect link between identical nics. But across platforms may negate this somewhat. However, you have more than one issue. You have a client side "lag" issue. If you are maxing out one core that proc is doing a TON of processing to push packets. IIRC that was one reason for jumbo frames at all. i.e push fewer, larger packets and increase overall throughput. So maybe the driver for that gear bottlenecks. 98% utilization with larger frames would tend to suggest this and is a useful result.

So test systematically. What does iperf say? Does it show ~9Gbs? Doe Does it show higher throughput FN to Linux, or FN to 2008 or FN to Win7. How do the protocols compare.?NFS vs CIFS vs ISCSI vs FTP. Are you getting the same results when you hit the ARC? Is it always one big CPU crushing bottleneck on the client (sans jumbo)?

Bottom line is you likely tag some sort of single threaded issues around 3-400MBs from everything I've seen. Even across killer big servers we often see multiple connections or operations being necessary to max out the network.

It just sounds like there is some gear you need to get to know. Search 'Intel x520 poor performance' there is lots of information on how interrupts on this card perform, use of extra buffers, and jumbo packets. Plug and play at full utilization is a pipe dream.

Obviously I defer to cyber and jgreco. But I do think a discussion of issues met on a fresh install is relevant. Even if only pointed to after resolution. This is a good fit for something similar to an M1015 walkthrough, it becomes more useful as the gear continues to drop in price.

 

[jgreco]

Plug and play at full utilization is a pipe dream.

That's not punny at all.

 

[Black Ninja]

If there is a place jumbo would work. It should be on a direct connect link between identical nics. But across platforms may negate this somewhat. However, you have more than one issue. You have a client side "lag" issue. If you are maxing out one core that proc is doing a TON of processing to push packets. IIRC that was one reason for jumbo frames at all. i.e push fewer, larger packets and increase overall throughput. So maybe the driver for that gear bottlenecks. 98% utilization with larger frames would tend to suggest this and is a useful result.

So test systematically. What does iperf say? Does it show ~9Gbs? Doe Does it show higher throughput FN to Linux, or FN to 2008 or FN to Win7. How do the protocols compare.?NFS vs CIFS vs ISCSI vs FTP. Are you getting the same results when you hit the ARC? Is it always one big CPU crushing bottleneck on the client (sans jumbo)?

Bottom line is you likely tag some sort of single threaded issues around 3-400MBs from everything I've seen. Even across killer big servers we often see multiple connections or operations being necessary to max out the network.

It just sounds like there is some gear you need to get to know. Search 'Intel x520 poor performance' there is lots of information on how interrupts on this card perform, use of extra buffers, and jumbo packets. Plug and play at full utilization is a pipe dream.

Thanks a lot for your response. I'll do some more testing but jumbo I think is the big answer for all 10Gb single threat situations. I have identical nics Intel E15729 back to back , Windows 7 to freenas. Here is a test on mapped drive , with only enabled/ disable jumbo frames , nothing else is touched on Windows 7 or Freenas side.

 

[Black Ninja]

Instead of read 150MB/s and max core 1, when jumbo is enabled , read is 500 MB/s (ssd max) and cpu core one is 50%. Can't be happier I almost gave up 10Gb because of this struggle.

 

[mjws00]

So you are maxing out your drive and you have a scenario where jumbo frames is a benefit. Enjoy. How fast will it write to a RAM drive?

 

[Black Ninja]

So you are maxing out your drive and you have a scenario where jumbo frames is a benefit. Enjoy. How fast will it write to a RAM drive?

benefit is understatement. On ram drive I ca utilize 98% of 10Gb without jumbo about 30%. This is Windows 2008 R2 to Windows 7 , because I can't do ram drive in freenas to test with that. I'll get another Samsung 850 and make a Raid0 pool to see what will happens from Freenas to Widows 7 next week perhaps.

 

[Bidule0hm]

Actually you can do it, just search for freebsd ramdisk tutorials ;)

 

[Black Ninja]

Actually you can do it, just search for freebsd ramdisk tutorials ;)

Do you have a particular link if you test it already ?

 

[Bidule0hm]

Never tested, but if you compare 3 or 4 tutos you usually can see quite rapidly which is the best ;)

 

[mjws00]

mdconfig -a -t malloc -s 4096M -u 8

Will give you a 4GB drive at dev/md8

 

[depasseg]

mdconfig -a -t malloc -s 4096M -u 8

But if I do that, I won't have any RAM left for my system to run.

 

[mjws00]

LOL. Says the guy with 128GB. Obviously just for testing... but sometimes you need to get hardware out of the way... also md is not the fastest thing on the planet but should tag 10Gb easily enough.

That 10 drive pool should saturate 10Gb on a big sequential read. Just need RAM to receive to and cyber linked that.

Edit: words, phone blah blah.

 

[Black Ninja]

This switch is said to use between 125-140Watts, so i didnt think that was too terrible. And yea it took me a week or so to gather all the info but in the end i feel it was worth it....i got a 10Gb backbone rocking and rolling lol

Are you still happy with this switch ? Is the switch managed with CLI only or it has web GUI for those not good with CLI ?

 

[Darren Myers]

Are you still happy with this switch ? Is the switch managed with CLI only or it has web GUI for those not good with CLI ?

Managed only by CLI, so far so good. Perfectly happy with it, no issues thus far. My only issue so far is my SAN's 10Gb NIC has froze twice now so i am investigating that now but so far it seems the 10Gb NIC is overheating

 

[Black Ninja]

Managed only by CLI, so far so good. Perfectly happy with it, no issues thus far. My only issue so far is my SAN's 10Gb NIC has froze twice now so i am investigating that now but so far it seems the 10Gb NIC is overheating

To bad is CLI only. Why would your SAN's nic overheats, from you signature your nic is in server chassis ?!

 

[Darren Myers]

To bad is CLI only. Why would your SAN's nic overheats, from you signature your nic is in server chassis ?!

It is in a server chassis, but im thinking the heatsink is a little weak as its processing data from 2 ESXi hosts. And the CLI was VERY easy, i plugged the switch into a old desktop serial connection i had around, reset the default config, changed the default password, enabled telnet, and away i went.

 

[Black Ninja]

It is in a server chassis, but im thinking the heatsink is a little weak as its processing data from 2 ESXi hosts. And the CLI was VERY easy, i plugged the switch into a old desktop serial connection i had around, reset the default config, changed the default password, enabled telnet, and away i went.

How is loud is the switch, any idea of the actual power consumption ( I read the spec about 140Wmax)?

 

[Darren Myers]

How is loud is the switch, any idea of the actual power consumption ( I read the spec about 140Wmax)?

Actual power consumption....mhmm good question, i dont have management NIC's in the APC's the switch is connected too, once i get them in i'd be able to tell ya. As far as noise....just about as noisy as my other 1U stuff lol, it sounds like the power profile of the switch is always on, so the fans are..well always on. but its quiet compared to my 2U and 4U servers

 

[Black Ninja]

Actual power consumption....mhmm good question, i dont have management NIC's in the APC's the switch is connected too, once i get them in i'd be able to tell ya. As far as noise....just about as noisy as my other 1U stuff lol, it sounds like the power profile of the switch is always on, so the fans are..well always on. but its quiet compared to my 2U and 4U servers

I have management nic in my APC but it show combine power of everything. I thought you have something like "kill-a-watt" to check the usage, very nice and cheap device by the way.
I noise is not more than a quiet 2u server then is fine. Just have to learn the CLI I guess ?! If power consumption is low of couarse, I don't want switch that will take more power than my average server.

 

[Darren Myers]

I have management nic in my APC but it show combine power of everything. I thought you have something like "kill-a-watt" to check the usage, very nice and cheap device by the way.
I noise is not more than a quiet 2u server then is fine. Just have to learn the CLI I guess ?! If power consumption is low of couarse, I don't want switch that will take more power than my average server.

Nah no kill-o-watt meter, kinda lazy lol. the CLI is easy, the commands are simple and in the manual; tells you exactly how to get it up and running from scratch.