<div class="bbWrapper">Hi,<br />
<br />
I've poor 10GBE Network-Speed (only talking about networkperformance, not disk performance)<br />
The Freenas and the workstation both using the Intel X540T1, connected thru Netgears XS708E switch - direct with a cat6e patch cables to exclude cable/patchpanel issues.<br />
<br />
iperf test:<br />
<div class="bbCodeBlock bbCodeCode"><div class="type">Code:</div><pre>[ 5] local 192.168.16.201 port 5001 connected with 192.168.16.139 port 61471
[ 5] 0.0-10.0 sec 3.08 GBytes 2.65 Gbits/sec</pre></div><br />
<br />
I'm very unhappy with a performance of 2.65 Gbits/sec. <br />
Any ideas how to speed up the network? Are there any additional settings in Freenas to increase the speed?<br />
<br />
Regards<br />
S.</div>
<div class="bbWrapper">As someone that has had to diagnose quite a few complaints of poor 10Gb performance (some much worse than yours), I'd try these things:<br />
<br />
1. Use a quality cable. PERIOD. Can't go cheap and should use the highest quality cable you can buy. Also minimize your length to the maximum extent practical.<br />
2. Do not try to make your own 10Gb cables. This is often the source of problems.<br />
3. Try cross connecting two servers together and take the switch out of the equation. This often proves that the switch has some settting problem.<br />
4. I am not a fan of copper 10Gb. Because of #1 mostly, but some people try to do #2 and then argue "but I've been making them for 1Gb for years... that shouldn't matter". Well, it does. So put away your pride and recognize that pride doesn't make this stuff work. Good proper connections do. Not many people try to do their own fiber cables, so that's one plus for fiber.<br />
5. If the workstation is Windows you need to set your TCP window size to 64KB. The default when doing iperf on Windows is something like 0.5KB or something and it inappropriately makes the test perform extremely poorly. What's funny is that you can do an iperf and get some poor value, but then get better throughput than iperf claims is possible over a CIFS share. That really confuses some users.<br />
6. Try doing 4 parallel connections when doing the test. If 1 connection and 4 connections end up with the same total bandwidth it's often a cabling/connector problem. If 4 connections gives you something like 3.5-4x the throughput it's often the switch not being properly configured.</div>
<div class="bbWrapper">cyberjock, <br />
<br />
thanks for your fast answer - impressive :) <br />
I changed the TCP window-size to 64KB and it looks better - but only a little bit.<br />
<br />
<div class="bbCodeBlock bbCodeCode"><div class="type">Code:</div><pre>[ 5] local 192.168.16.201 port 5001 connected with 192.168.16.139 port 62356
[ 5] 0.0-10.0 sec 4.23 GBytes 3.63 Gbits/sec</pre></div><br />
<br />
Currently there are only 2x 10GBE powered PCs - the freenas and the video-workstation, so I can't do the parallel connections. <br />
<br />
I think I also have to check if the NIC is in the correct PCIe Slot - could decrease the performance if its not in a PCIe 8x2.0 slot, right?<br />
<br />
Regards<br />
<br />
S.</div>
<div class="bbWrapper"><blockquote data-attributes="member: 20702" data-quote="hugovsky" data-source="post: 169962"
class="bbCodeBlock bbCodeBlock--expandable bbCodeBlock--quote js-expandWatch">
<div class="bbCodeBlock-title">
<a href="/community/goto/post?id=169962"
class="bbCodeBlock-sourceJump"
rel="nofollow"
data-xf-click="attribution"
data-content-selector="#post-169962">hugovsky said:</a>
</div>
<div class="bbCodeBlock-content">
<div class="bbCodeBlock-expandContent js-expandContent ">
I think cyberjock was trying to tell you to use -p with iperf to make parallel connections.
</div>
<div class="bbCodeBlock-expandLink js-expandLink"><a role="button" tabindex="0">Click to expand...</a></div>
</div>
</blockquote><br />
That's exactly what I was recommending... ;)<br />
<br />
As for the PCIe slot, even a 4x slot that was PCIe 1.0 should give you 1GB/sec, which is far more than you are getting. So I doubt that's your problem. Besides, personal experience with Intel 10Gb NICs says if you don't have the right 8x slot the card won't work at all.</div>
<div class="bbWrapper">Hi cyberjock,<br />
<br />
I played with the TCP window size:<br />
<br />
<div class="bbCodeBlock bbCodeCode"><div class="type">Code:</div><pre>[root@filer] ~# iperf -s -w 512KB
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 512 KByte
------------------------------------------------------------
[ 4] local 192.168.16.201 port 5001 connected with 192.168.16.139 port 62474
[ ID] Interval Transfer Bandwidth
[ 4] 0.0-10.0 sec 5.69 GBytes 4.88 Gbits/sec
</pre></div><br />
<br />
Are tehre any recommendations about the TCP window sizes? In CIFS share, are they set automaticly?<br />
Regards<br />
S.</div>
<div class="bbWrapper">Windows adjusts the window size automatically. Since iperf doesn't follow the same code path the window size has to be set manually.. 64kb is what is recommended for iperf testing (which is why I recommended it above).<br />
<br />
Your post with -P 4 seems to suggest you have a problem at the switch level. You should try doing the various tests with the two machines directly connected and then to the switch and see if things change.<br />
<br />
The fact that the server and client don't have the same output shows you have packets dropping. So your entire datapath is suspect (even the NICs at this point).</div>
<div class="bbWrapper">OK, it's getting more complicated than exprected :(<br />
I have to find a time window to connect the client & server directly without the switch.</div>
<div class="bbWrapper">I also have to force MTU to something >64k:<br />
<br />
<div class="bbCodeBlock bbCodeCode"><div class="type">Code:</div><pre>
------------------------------------------------------------
Client connecting to nas, TCP port 5001
TCP window size: 64.0 KByte (default)
------------------------------------------------------------
[ 3] local 192.168.6.2 port 50324 connected with 192.168.6.13 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.0 sec 3.28 GBytes 2.81 Gbits/sec
</pre></div><br />
<br />
<div class="bbCodeBlock bbCodeCode"><div class="type">Code:</div><pre>
------------------------------------------------------------
Client connecting to nas, TCP port 5001
TCP window size: 512 KByte
------------------------------------------------------------
[ 3] local 192.168.6.2 port 50327 connected with 192.168.6.13 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.0 sec 10.9 GBytes 9.38 Gbits/sec
</pre></div><br />
<br />
I made my own cable here too. Simply a custom length of 5e with regular rj45 ends on it. No problem at all.</div>
<div class="bbWrapper">Hi,<br />
<br />
first i have to fix the issue with the packets dropping. I have to wait, cause there are only 3 10gbe nics in our office - and one of them has gone to heaven.<br />
Don't ask me how its possible to tear off the RJ45 Port...<br />
<br />
S.</div>
<div class="bbWrapper">Hi,<br />
<br />
I changed all cables inside the 10GBE Network with new Cat7 ones - performance is perfect now :)<br />
9.2 and 8.9 Gbits - sounds good for me.<br />
Thanks for all your notes & hints!<br />
<br />
S.</div>
