My question is how to convert iPerf setting into FreeBSD and win 7 settable parameters. I am a noob to FreeBSD and FreeNAS and so far the things I have tried haven't worked.
I have been doing performance tests on my recently constructed FreeNAS server running 9.2.0-RELEASE-x64 (ab098f4) on a Supermicro X10SLH-F motherboard with an E3-1230V3 processor, 24GB RAM, and M1015 controller with 24TB RAIDZ2 (5 3TB RED + 3 3TB Seagate NAS). The NAS will be used mostly for media storage and backup.
The client is Windows 7 SP1 platform with an I5-2500K, 16GB RAM, 256MB SSD and an Intel Gigiabit CT Desktop adapter connected to an 8-port 1Gb switch shared with the server.
FreeNAS autotune installed the following sysctls:
net.inet.tcp.sendbuf_max=2097152
net.inet.tcp.recvbuf_max=2097152
kern.ipc.maxsockbuf=2097152
Transfers of large files from the FreeNAS server to the client SSD exceed 95MB/s, great.
However, transfers from the SSD to the sever are closer to 45MB/s which is well below what the system should be capable of achieving.
Testing with iPerf (jPerf 2.0.2 on the client) over UDP showed transfer rates in both directions are very close the 1Gb/s theoretical limit, confirming there are no network hardware issues.
Testing with iPerf over TCP show a transfers from server to client at over well 900Mb/s,, again great. But the iPerf client to server test below shows there is a problem in upload direction (the .01 MByte window size ifs actually Windows 8kB default size):
I have spent some time tuning parameters, primarily on the PC, most of which had little effect. The only parameter that had a significant impact was the TCP window size which when set on both the client and the server resolved the issue with 512k appearing to be the optimum value.
Here is a run with the just the client window set to 512k, a noticeable improvement:
Here is a test with just the server window set to 512k which is actually a little slower :
But put them together and here is a test with both the client and server windows set to 512k:
It is unclear to me why it is necessary to set both windows on a one way transfer to achieve the desired performance but this is what the tests seem to indicate. Some how these 512k iPerf settings seem to resolve the problem but I am suspicious that both windows needed to be 512k for one direction while no new setting were needed for the other direction.
On the Windows 7 side MS implemented a "next generation TCP/IP stack" for Windows 7 SP1 which it suppose to magically adjust windows sizes as needed, there is not even a register entry for TCP windows size. I tried setting various options using TCPOptmizer.exe but none of them yielded results close the iPerf's. I assume iPerf is programmatically setting window size on the PC but this would probably have to be done on a per connection basis. I don't see how to implement it on the PC.
I would appreciate some pointers on translating the iPerf results into usable action. If there is something else going on that I'm missing please let me know.
Thanks
I have been doing performance tests on my recently constructed FreeNAS server running 9.2.0-RELEASE-x64 (ab098f4) on a Supermicro X10SLH-F motherboard with an E3-1230V3 processor, 24GB RAM, and M1015 controller with 24TB RAIDZ2 (5 3TB RED + 3 3TB Seagate NAS). The NAS will be used mostly for media storage and backup.
The client is Windows 7 SP1 platform with an I5-2500K, 16GB RAM, 256MB SSD and an Intel Gigiabit CT Desktop adapter connected to an 8-port 1Gb switch shared with the server.
FreeNAS autotune installed the following sysctls:
net.inet.tcp.sendbuf_max=2097152
net.inet.tcp.recvbuf_max=2097152
kern.ipc.maxsockbuf=2097152
Transfers of large files from the FreeNAS server to the client SSD exceed 95MB/s, great.
However, transfers from the SSD to the sever are closer to 45MB/s which is well below what the system should be capable of achieving.
Testing with iPerf (jPerf 2.0.2 on the client) over UDP showed transfer rates in both directions are very close the 1Gb/s theoretical limit, confirming there are no network hardware issues.
Testing with iPerf over TCP show a transfers from server to client at over well 900Mb/s,, again great. But the iPerf client to server test below shows there is a problem in upload direction (the .01 MByte window size ifs actually Windows 8kB default size):
Code:
bin/iperf.exe -c 192.168.44.44 -P 1 -i 2 -p 5001 -f m -t 20 ------------------------------------------------------------ Client connecting to 192.168.44.44, TCP port 5001 TCP window size: 0.01 MByte (default) ------------------------------------------------------------ [156] local 192.168.44.32 port 51395 connected with 192.168.44.44 port 5001 [ ID] Interval Transfer Bandwidth [156] 0.0- 2.0 sec 81.3 MBytes 341 Mbits/sec [156] 2.0- 4.0 sec 65.8 MBytes 276 Mbits/sec [156] 4.0- 6.0 sec 82.3 MBytes 345 Mbits/sec [156] 6.0- 8.0 sec 82.2 MBytes 345 Mbits/sec [156] 8.0-10.0 sec 81.4 MBytes 341 Mbits/sec [156] 10.0-12.0 sec 81.9 MBytes 344 Mbits/sec [156] 12.0-14.0 sec 82.3 MBytes 345 Mbits/sec [156] 14.0-16.0 sec 80.0 MBytes 336 Mbits/sec [156] 16.0-18.0 sec 80.9 MBytes 339 Mbits/sec [156] 18.0-20.0 sec 81.0 MBytes 340 Mbits/sec [156] 0.0-20.0 sec 799 MBytes 335 Mbits/sec Done.
I have spent some time tuning parameters, primarily on the PC, most of which had little effect. The only parameter that had a significant impact was the TCP window size which when set on both the client and the server resolved the issue with 512k appearing to be the optimum value.
Here is a run with the just the client window set to 512k, a noticeable improvement:
Code:
bin/iperf.exe -c 192.168.44.44 -P 1 -i 2 -p 5001 -w 512.0K -f m -t 20 ------------------------------------------------------------ Client connecting to 192.168.44.44, TCP port 5001 TCP window size: 0.50 MByte ------------------------------------------------------------ [156] local 192.168.44.32 port 52065 connected with 192.168.44.44 port 5001 [ ID] Interval Transfer Bandwidth [156] 0.0- 2.0 sec 137 MBytes 575 Mbits/sec [156] 2.0- 4.0 sec 143 MBytes 599 Mbits/sec [156] 4.0- 6.0 sec 138 MBytes 578 Mbits/sec [156] 6.0- 8.0 sec 138 MBytes 578 Mbits/sec [156] 8.0-10.0 sec 142 MBytes 594 Mbits/sec [156] 10.0-12.0 sec 146 MBytes 614 Mbits/sec [156] 12.0-14.0 sec 124 MBytes 518 Mbits/sec [156] 14.0-16.0 sec 122 MBytes 512 Mbits/sec [156] 16.0-18.0 sec 145 MBytes 610 Mbits/sec [156] 18.0-20.0 sec 142 MBytes 594 Mbits/sec [156] 0.0-20.0 sec 1376 MBytes 577 Mbits/sec Done.
Here is a test with just the server window set to 512k which is actually a little slower :
Code:
bin/iperf.exe -c 192.168.44.44 -P 1 -i 2 -p 5001 -f m -t 20 ------------------------------------------------------------ Client connecting to 192.168.44.44, TCP port 5001 TCP window size: 0.01 MByte (default) ------------------------------------------------------------ [156] local 192.168.44.32 port 52383 connected with 192.168.44.44 port 5001 [ ID] Interval Transfer Bandwidth [156] 0.0- 2.0 sec 62.1 MBytes 261 Mbits/sec [156] 2.0- 4.0 sec 79.2 MBytes 332 Mbits/sec [156] 4.0- 6.0 sec 75.5 MBytes 317 Mbits/sec [156] 6.0- 8.0 sec 80.6 MBytes 338 Mbits/sec [156] 8.0-10.0 sec 78.5 MBytes 329 Mbits/sec [156] 10.0-12.0 sec 60.7 MBytes 255 Mbits/sec [156] 12.0-14.0 sec 78.3 MBytes 329 Mbits/sec [156] 14.0-16.0 sec 72.8 MBytes 305 Mbits/sec [156] 16.0-18.0 sec 45.9 MBytes 193 Mbits/sec [156] 18.0-20.0 sec 80.6 MBytes 338 Mbits/sec [156] 0.0-20.0 sec 714 MBytes 299 Mbits/sec Done.
But put them together and here is a test with both the client and server windows set to 512k:
Code:
bin/iperf.exe -c 192.168.44.44 -P 1 -i 2 -p 5001 -w 512.0K -f m -t 20 ------------------------------------------------------------ Client connecting to 192.168.44.44, TCP port 5001 TCP window size: 0.50 MByte ------------------------------------------------------------ [156] local 192.168.44.32 port 53119 connected with 192.168.44.44 port 5001 [ ID] Interval Transfer Bandwidth [156] 0.0- 2.0 sec 207 MBytes 868 Mbits/sec [156] 2.0- 4.0 sec 224 MBytes 937 Mbits/sec [156] 4.0- 6.0 sec 216 MBytes 907 Mbits/sec [156] 6.0- 8.0 sec 226 MBytes 947 Mbits/sec [156] 8.0-10.0 sec 222 MBytes 932 Mbits/sec [156] 10.0-12.0 sec 226 MBytes 948 Mbits/sec [156] 12.0-14.0 sec 207 MBytes 869 Mbits/sec [156] 14.0-16.0 sec 224 MBytes 941 Mbits/sec [156] 16.0-18.0 sec 226 MBytes 948 Mbits/sec [156] 18.0-20.0 sec 220 MBytes 924 Mbits/sec [156] 0.0-20.0 sec 2198 MBytes 922 Mbits/sec Done.
It is unclear to me why it is necessary to set both windows on a one way transfer to achieve the desired performance but this is what the tests seem to indicate. Some how these 512k iPerf settings seem to resolve the problem but I am suspicious that both windows needed to be 512k for one direction while no new setting were needed for the other direction.
On the Windows 7 side MS implemented a "next generation TCP/IP stack" for Windows 7 SP1 which it suppose to magically adjust windows sizes as needed, there is not even a register entry for TCP windows size. I tried setting various options using TCPOptmizer.exe but none of them yielded results close the iPerf's. I assume iPerf is programmatically setting window size on the PC but this would probably have to be done on a per connection basis. I don't see how to implement it on the PC.
I would appreciate some pointers on translating the iPerf results into usable action. If there is something else going on that I'm missing please let me know.
Thanks
