[iperf] Direct Intel 10GbE X550-T1 not utilizing full 10 Gbps bandwidth

[bluebird]

Hi all,

First time posting here - very excited to meet all of you and hope you can help solve this issue that I'm currently running into:

Client side:
- OS: Ubuntu 16.04 LTS
- Motherboard: Supermicro X10DRi
- CPU: 2x Intel Xeon E5-2630v4
- RAM: 2x Crucial CT4K16G4RFD424A (128 GB total)
- 10 GbE NIC: Intel X550-T1 (directly attached) with IP address 192.168.100.114/24

Server side:
- FreeNAS 11.1-U4
- Motherboard: Asus Z87-Plus
- CPU: Intel i7-4771
- RAM: 16 GB
- 10 GbE NIC: Intel X550-T1 (directly attached) with IP address 192.168.100.111/24

I've run iperf on the 10 GbE direct connection and unfortunately my 10 GbE bandwidth stays under 3.3 Gbps. According to iperf, the 1GbE NIC bandwidth works as expected at just under 1 Gbps. I've tried tuning the MTU to 9000 and setting the FreeNAS tunables as described at http://45drives.blogspot.com/2016/05/how-to-tune-nas-for-direct-from-server.html with no performance changes.

Below are the iperf results from the client side (running either 1 or 4 parallel client threads):


------------------------------------------------------------
Client connecting to 192.168.100.111, TCP port 5001
TCP window size: 325 KByte (default)
------------------------------------------------------------
[ 3] local 192.168.100.114 port 54162 connected with 192.168.100.111 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.0 sec 3.80 GBytes 3.27 Gbits/sec



------------------------------------------------------------
Client connecting to 192.168.100.111, TCP port 5001
TCP window size: 325 KByte (default)
------------------------------------------------------------
[ 6] local 192.168.100.114 port 54172 connected with 192.168.100.111 port 5001
[ 4] local 192.168.100.114 port 54166 connected with 192.168.100.111 port 5001
[ 5] local 192.168.100.114 port 54168 connected with 192.168.100.111 port 5001
[ 3] local 192.168.100.114 port 54170 connected with 192.168.100.111 port 5001
[ ID] Interval Transfer Bandwidth
[ 4] 0.0-10.0 sec 973 MBytes 816 Mbits/sec
[ 5] 0.0-10.0 sec 973 MBytes 816 Mbits/sec
[ 3] 0.0-10.0 sec 973 MBytes 816 Mbits/sec
[ 6] 0.0-10.0 sec 973 MBytes 816 Mbits/sec
[SUM] 0.0-10.0 sec 3.80 GBytes 3.26 Gbits/sec


Any help would be appreciated! Many thanks!

 

[kdragon75]

Use a larger window size say 512K and UDP to see if it improves.

Edit here some info to dig through https://pleiades.ucsc.edu/hyades/FreeBSD_Network_Tuning

 

[bluebird]

Use a larger window size say 512K and UDP to see if it improves.

Edit here some info to dig through https://pleiades.ucsc.edu/hyades/FreeBSD_Network_Tuning

Thank you for the quick reply!

Just to double check, when you say to increase the window size do you mean to perform the following:

iperf -c 192.168.100.111 -w 512k and/or iperf -c 192.168.100.111 -u -w 512k on the client side, and
iperf -s -w 512k and/or iperf -s -u -w 512k on the server side?

I tested using -w 512k quickly and the buffer size in iperf would not go higher than, 416k/500k on the client/server side, respectively.

Also, the link you provided only shows parameters for up to FreeBSD 10.1. Are those the same for FreeNAS 11.1?

Either way will try a few configurations later today and report back :)

 

[kdragon75]

Hmm Il have to play with it later. As for FreeBSD 10 to 11, that will be some changes. Exactly what, I don't know. Its still work reading as all of the same ideas and concepts are still at play. One thing to research would be any kind of tcp offload and making sure its well supported and enabled. You should also keep an eye on the CPU usage during the testing. I doubt that's the issue as you have decent CPUs but it worth checking to see if one or two cores are getting pegged.

 

[bluebird]

No luck. Results for UDP iperf is terrible. Tried rebooting and swapping out the CAT-6 cable and results stayed the same. CPU load on both client and server sides were no where near 100% (even when looking at individual CPU threads with htop). I thought it may be an issue from the client side but at this point I suspect something else is going on:

Latest tuning for both the client and server sides:

MTU = 9000 for both client and server (verified with ifconfig)

Ubuntu 16.04 Client (/etc/sysctl.conf):

net.ipv4.tcp_window_scaling = 1
net.core.wmem_max = 4194304
net.core.rmem_max = 12582912
net.ipv4.tcp_rmem = 4096 87380 4194304
net.ipv4.tcp_wmem = 4096 87380 4194304


FreeNAS 11.1-U4 (sysctl)

kern.ipc.maxsockbuf=16777216
net.inet.ip.intr_queue_maxlen=2048
net.inet.tcp.recvbuf_inc=524288
net.inet.tcp.recvbuf_max=16777216
net.inet.tcp.recvspace=4194304
net.inet.tcp.sendbuf_inc=32768
net.inet.tcp.sendbuf_max=16777216
net.inet.tcp.sendspace=2097152
net.route.netisr_maxqlen=2048


Results using iperf -c 192.168.100.111 and iperf -s:
Client:

Client connecting to 192.168.100.111, TCP port 5001
TCP window size: 325 KByte (default)
------------------------------------------------------------
[ 3] local 192.168.100.114 port 59282 connected with 192.168.100.111 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.0 sec 3.80 GBytes 3.27 Gbits/sec

Server:

------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 4.00 MByte (default)
------------------------------------------------------------
[ 4] local 192.168.100.111 port 5001 connected with 192.168.100.114 port 59282
[ ID] Interval Transfer Bandwidth
[ 4] 0.0-10.0 sec 3.80 GBytes 3.26 Gbits/sec


Results using iperf -c 192.168.100.111 -w 512k and iperf -s -w 512k:
Client:

------------------------------------------------------------
Client connecting to 192.168.100.111, TCP port 5001
TCP window size: 1.00 MByte (WARNING: requested 512 KByte)
------------------------------------------------------------
[ 3] local 192.168.100.114 port 59286 connected with 192.168.100.111 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.0 sec 3.80 GBytes 3.27 Gbits/sec

Server:

------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 500 KByte
------------------------------------------------------------
[ 4] local 192.168.100.111 port 5001 connected with 192.168.100.114 port 59286
[ ID] Interval Transfer Bandwidth
[ 4] 0.0-10.0 sec 3.80 GBytes 3.26 Gbits/sec


Results using iperf -c 192.168.100.111 -u and iperf -s -u:
Client:

------------------------------------------------------------
Client connecting to 192.168.100.111, UDP port 5001
Sending 1470 byte datagrams
UDP buffer size: 208 KByte (default)
------------------------------------------------------------
[ 3] local 192.168.100.114 port 46142 connected with 192.168.100.111 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.0 sec 1.25 MBytes 1.05 Mbits/sec
[ 3] Sent 893 datagrams

Server:

------------------------------------------------------------
Server listening on UDP port 5001
Receiving 1470 byte datagrams
UDP buffer size: 41.1 KByte (default)
------------------------------------------------------------
[ 3] local 192.168.100.111 port 5001 connected with 192.168.100.114 port 46142
[ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams
[ 3] 0.0-10.0 sec 1.25 MBytes 1.05 Mbits/sec 0.004 ms 2147481862/214748275
5 (1e+02%)


Results using iperf -c 192.168.100.111 -u -w 512k and iperf -s -u -w 512k:
Client:

------------------------------------------------------------
Client connecting to 192.168.100.111, UDP port 5001
Sending 1470 byte datagrams
UDP buffer size: 1.00 MByte (WARNING: requested 512 KByte)
------------------------------------------------------------
[ 3] local 192.168.100.114 port 50977 connected with 192.168.100.111 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.0 sec 1.25 MBytes 1.05 Mbits/sec
[ 3] Sent 893 datagrams


Server:

------------------------------------------------------------
Server listening on UDP port 5001
Receiving 1470 byte datagrams
UDP buffer size: 500 KByte
------------------------------------------------------------
[ 3] local 192.168.100.111 port 5001 connected with 192.168.100.114 port 50977
[ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams
[ 3] 0.0-10.0 sec 1.25 MBytes 1.05 Mbits/sec 0.006 ms 2147481862/214748275
5 (1e+02%)