10Gbe Network Performance Low?

[stefanb]

Hi,

I've poor 10GBE Network-Speed (only talking about networkperformance, not disk performance)
The Freenas and the workstation both using the Intel X540T1, connected thru Netgears XS708E switch - direct with a cat6e patch cables to exclude cable/patchpanel issues.

iperf test:

Code:

[  5] local 192.168.16.201 port 5001 connected with 192.168.16.139 port 61471
[  5]  0.0-10.0 sec  3.08 GBytes  2.65 Gbits/sec

I'm very unhappy with a performance of 2.65 Gbits/sec.
Any ideas how to speed up the network? Are there any additional settings in Freenas to increase the speed?

Regards
S.

 

[cyberjock]

As someone that has had to diagnose quite a few complaints of poor 10Gb performance (some much worse than yours), I'd try these things:

1. Use a quality cable. PERIOD. Can't go cheap and should use the highest quality cable you can buy. Also minimize your length to the maximum extent practical.
2. Do not try to make your own 10Gb cables. This is often the source of problems.
3. Try cross connecting two servers together and take the switch out of the equation. This often proves that the switch has some settting problem.
4. I am not a fan of copper 10Gb. Because of #1 mostly, but some people try to do #2 and then argue "but I've been making them for 1Gb for years... that shouldn't matter". Well, it does. So put away your pride and recognize that pride doesn't make this stuff work. Good proper connections do. Not many people try to do their own fiber cables, so that's one plus for fiber.
5. If the workstation is Windows you need to set your TCP window size to 64KB. The default when doing iperf on Windows is something like 0.5KB or something and it inappropriately makes the test perform extremely poorly. What's funny is that you can do an iperf and get some poor value, but then get better throughput than iperf claims is possible over a CIFS share. That really confuses some users.
6. Try doing 4 parallel connections when doing the test. If 1 connection and 4 connections end up with the same total bandwidth it's often a cabling/connector problem. If 4 connections gives you something like 3.5-4x the throughput it's often the switch not being properly configured.

 

[stefanb]

cyberjock,

thanks for your fast answer - impressive :)
I changed the TCP window-size to 64KB and it looks better - but only a little bit.

Code:

[  5] local 192.168.16.201 port 5001 connected with 192.168.16.139 port 62356
[  5]  0.0-10.0 sec  4.23 GBytes  3.63 Gbits/sec

Currently there are only 2x 10GBE powered PCs - the freenas and the video-workstation, so I can't do the parallel connections.

I think I also have to check if the NIC is in the correct PCIe Slot - could decrease the performance if its not in a PCIe 8x2.0 slot, right?

Regards

S.

 

[hugovsky]

I think cyberjock was trying to tell you to use -p with iperf to make parallel connections.

 

[cyberjock]

I think cyberjock was trying to tell you to use -p with iperf to make parallel connections.

That's exactly what I was recommending... ;)

As for the PCIe slot, even a 4x slot that was PCIe 1.0 should give you 1GB/sec, which is far more than you are getting. So I doubt that's your problem. Besides, personal experience with Intel 10Gb NICs says if you don't have the right 8x slot the card won't work at all.

 

[stefanb]

Hi cyberjock,

I played with the TCP window size:

Code:

[root@filer] ~# iperf -s -w 512KB
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size:  512 KByte
------------------------------------------------------------
[  4] local 192.168.16.201 port 5001 connected with 192.168.16.139 port 62474
[ ID] Interval       Transfer     Bandwidth
[  4]  0.0-10.0 sec  5.69 GBytes  4.88 Gbits/sec

Are tehre any recommendations about the TCP window sizes? In CIFS share, are they set automaticly?
Regards
S.

 

[hugovsky]

You can try iperf -c <<your_server>> -P 4.

 

[stefanb]

ok, I did the 4-threads test, with 64KB - server side:

Code:

[  4]  0.0-10.0 sec  1.65 GBytes  1.42 Gbits/sec
[  5]  0.0-13.0 sec  2.49 GBytes  1.65 Gbits/sec
[  6]  0.0-13.0 sec  2.53 GBytes  1.67 Gbits/sec
[  7]  0.0-13.0 sec  2.53 GBytes  1.67 Gbits/sec
[SUM]  0.0-13.0 sec  9.20 GBytes  6.07 Gbits/sec

client side:

Code:

[ ID] Interval       Transfer     Bandwidth
[  6]  0.0- 7.0 sec  1.65 GBytes  2.03 Gbits/sec
[  3]  0.0-10.0 sec  2.49 GBytes  2.14 Gbits/sec
[  4]  0.0-10.0 sec  2.53 GBytes  2.17 Gbits/sec
[  5]  0.0-10.0 sec  2.53 GBytes  2.18 Gbits/sec
[SUM]  0.0-10.0 sec  9.20 GBytes  7.90 Gbits/sec

single thread server side was:

Code:

    [  5] local 192.168.16.201 port 5001 connected with 192.168.16.139 port 62356
    [  5]  0.0-10.0 sec  4.23 GBytes  3.63 Gbits/sec

 

[cyberjock]

Windows adjusts the window size automatically. Since iperf doesn't follow the same code path the window size has to be set manually.. 64kb is what is recommended for iperf testing (which is why I recommended it above).

Your post with -P 4 seems to suggest you have a problem at the switch level. You should try doing the various tests with the two machines directly connected and then to the switch and see if things change.

The fact that the server and client don't have the same output shows you have packets dropping. So your entire datapath is suspect (even the NICs at this point).

 

[stefanb]

OK, it's getting more complicated than exprected :(
I have to find a time window to connect the client & server directly without the switch.

 

[Mguilicutty]

You might also try some Cat7 cables, Monoprice has them cheap and are generally decent quality

 

[titan_rw]

I also have to force MTU to something >64k:

Code:

------------------------------------------------------------
Client connecting to nas, TCP port 5001
TCP window size: 64.0 KByte (default)
------------------------------------------------------------
[  3] local 192.168.6.2 port 50324 connected with 192.168.6.13 port 5001
[ ID] Interval  Transfer  Bandwidth
[  3]  0.0-10.0 sec  3.28 GBytes  2.81 Gbits/sec

Code:

------------------------------------------------------------
Client connecting to nas, TCP port 5001
TCP window size:  512 KByte
------------------------------------------------------------
[  3] local 192.168.6.2 port 50327 connected with 192.168.6.13 port 5001
[ ID] Interval  Transfer  Bandwidth
[  3]  0.0-10.0 sec  10.9 GBytes  9.38 Gbits/sec

I made my own cable here too. Simply a custom length of 5e with regular rj45 ends on it. No problem at all.

 

[stefanb]

Hi,

first i have to fix the issue with the packets dropping. I have to wait, cause there are only 3 10gbe nics in our office - and one of them has gone to heaven.
Don't ask me how its possible to tear off the RJ45 Port...

S.

 

[stefanb]

Hi,

I changed all cables inside the 10GBE Network with new Cat7 ones - performance is perfect now :)
9.2 and 8.9 Gbits - sounds good for me.
Thanks for all your notes & hints!

S.