10Gb NICs and iperf is showing 1Gb speeds

[Visseroth]

OK, let me start with system specs.

System 1 is my old server. I'm trying to migrate off of (Once verified stable and reliable) to my new one.

System 1:
SuperMicro x7dbn
Dual dual core Intel 3.2Ghz w/HT
32GB of RAM
2vdevs x 6 drives RAIDZ2
Chelsio 10Gb Card with Finisar FTLX8571D3BCL Module

Quanta LB4M 48 port switch with 2 10Gb ports. Both ports also have Finisar FTLX8571D3BCL Modules
Blue 850nm LC/LC 1 meter fiber cables connected between them. Slack has been looped in about a 8 inche circle.

System 2:
SuperMicro x8dth-iF
Intel Xeon 3.46Ghz 6 core CPU w/HT
48GB of RAM
2vDevs x 6 drives
Chelsio 10Gb card with Finisar FTLX8571D3BCL Module

Now when I run iPerf from one system to the other I'm seeing about 1 to 1.25Gb/s. Mind you both are connect via fiber at 10Gb full.
I've attach screen shots of my tests, tunables, ect. I'm not really sure what I need to do to get better than 1Gb speed. Any help is greatly appreciated.

 

[Visseroth]

Oh yea, I do not have jumbo frames enabled thus far

 

[Visseroth]

And I'm also trying to transfer everything from System 1 to System 2 over scp and I'm only seeing 30 to 40MB/s. If I transfer something from my desptop to either system samba will saturate the connection at about 60 to 98MB/s with 110 to 120MB/s bursts.

 

[Mlovelace]

How are the switch ports configured?

 

[Visseroth]

Here is a screenshot of the switchports from the show running-config

 

[Mlovelace]

I don't know that switch or it's software, but 10gbe does not typically auto-negotiate, you probably need to configure the port speed/duplex.

 

[Visseroth]

OK, well it did but I will set it 10/full

 

[Visseroth]

I set the ports on the switch to 10/full but I don't see a way to set the ports on the server to 10/full.

At any rate the switch was auto detecting 10/full but even set mannually to 10/full I'm still only seeing 1.2 with system 1 as the server but now I just saw 2.18 with system 2 as the server.

 

[zambanini]

change the window size on ipferf

 

[depasseg]

[side note: you can just paste the image from the clipboard directly into the post. You don't have to save it as a file.]

 

[Visseroth]

From System 1...

Code:

@storage] ~% iperf -s
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size:  256 KByte (default)
------------------------------------------------------------
[  4] local 10.10.10.252 port 5001 connected with 10.10.10.248 port 48283
[ ID] Interval       Transfer     Bandwidth
[  4]  0.0-10.0 sec  1.11 GBytes   954 Mbits/sec
[  5] local 10.10.10.252 port 5001 connected with 10.10.10.248 port 39802
[  5]  0.0-10.0 sec  1.06 GBytes   906 Mbits/sec
^C[user@storage] ~% iperf -c 10.10.10.248
------------------------------------------------------------
Client connecting to 10.10.10.248, TCP port 5001
TCP window size:  257 KByte (default)
------------------------------------------------------------
[  3] local 10.10.10.252 port 20317 connected with 10.10.10.248 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  2.45 GBytes  2.10 Gbits/sec
[user@storage] ~% iperf -c 10.10.10.248
------------------------------------------------------------
Client connecting to 10.10.10.248, TCP port 5001
TCP window size:  257 KByte (default)
------------------------------------------------------------
[  3] local 10.10.10.252 port 10391 connected with 10.10.10.248 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  2.49 GBytes  2.14 Gbits/sec

From System 2...

Code:

[root@storage1] ~# iperf -c 10.10.10.252
------------------------------------------------------------
Client connecting to 10.10.10.252, TCP port 5001
TCP window size:  257 KByte (default)
------------------------------------------------------------
[  3] local 10.10.10.248 port 48283 connected with 10.10.10.252 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  1.11 GBytes   956 Mbits/sec
[root@storage1] ~# iperf -c 10.10.10.252
------------------------------------------------------------
Client connecting to 10.10.10.252, TCP port 5001
TCP window size:  257 KByte (default)
------------------------------------------------------------
[  3] local 10.10.10.248 port 39802 connected with 10.10.10.252 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  1.06 GBytes   908 Mbits/sec
[root@storage1] ~# iperf -s
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size:  256 KByte (default)
------------------------------------------------------------
[  4] local 10.10.10.248 port 5001 connected with 10.10.10.252 port 20317
[ ID] Interval       Transfer     Bandwidth
[  4]  0.0-10.0 sec  2.45 GBytes  2.10 Gbits/sec
[  5] local 10.10.10.248 port 5001 connected with 10.10.10.252 port 10391
[  5]  0.0-10.0 sec  2.49 GBytes  2.14 Gbits/sec

 

[Visseroth]

Just got in the propert Chelsio fiber modules for the Chelsio cards, installed them, tested and had even lower speeds. Went from 1Gb/s to ~800Mb/s and 1Gb/s
Sigh

 

[zambanini]

can you direct connect two freenas boxes without your noisy switch? with twinax,DAC this works. at least to test the situation.

 

[Visseroth]

Results were the same. 800Mb/s to 1.2Gb/s using iperf. Tested both directions.

I thought maybe it could be a cable issue so I tried a orange 1 meter cable I had laying around and got the exact same results. They linked, ran the test both directions. 4 tests total, still poor speed issues.

Could it be that my network cards need some tuning?

Edit: I forgot to mention I tried with the cables I use to connect to the switch and tried with the orange cables. Results were the same for all of the above cables.

Also bear in mind that when transferring from system 1 to system 2 via scp I'm only getting ~40MB/s. One would think I'd be able to get 100MB/s+

 

[zambanini]

regarding scp: take a look at the cpu load use top or whatever. on my dell and hp server, there I also had to change the bios settings regarding the 'energy consuming' option, I set them to full power. with the chelsio freebsd drivers, there was a text file with some tuneables. anyway, we know now: it is not the switch. since i am a lazy guy, I would test the networkspeed with the same hardware and another OS.

 

[Visseroth]

Will do, and I checked the cpu usage on both machines.

System 1:

Code:

96768 root            1  52    0 42700K  5268K select  4  12:39  58.25% ssh
96767 root            1  23    0 40564K  5020K zio->i  1   1:27   5.08% scp

System 2:

Code:

84231 root            1  49    0 76092K  7936K select  8   7:50  37.99% sshd
84247 root            1  20    0 40564K  4944K select 11   0:21   0.78% scp

yea, not a lot, some head room left.

 

[BigDave]

Are you matching the window size from sever to client?

i.e. below:
iperf -s -p 5001 -w 512k
iperf -c server.ip.goes.here -p 5001 -w 512k

 

[Visseroth]

I'm not extremely familiar with iPerf, so no I didn't but I just did and had the same results, 800Mb/s to 1Gb/s.

I also ran -P 8 and -d and received the output...

Code:

[ ID] Interval       Transfer     Bandwidth
[  5]  0.0-10.0 sec   365 MBytes   306 Mbits/sec
[  6]  0.0-10.0 sec   251 MBytes   210 Mbits/sec
[  7]  0.0-10.0 sec   384 MBytes   322 Mbits/sec
[ 10]  0.0-10.0 sec   403 MBytes   338 Mbits/sec
[  8]  0.0-10.0 sec   333 MBytes   279 Mbits/sec
[ 19]  0.0-10.0 sec   161 MBytes   135 Mbits/sec
[ 12]  0.0-10.0 sec   324 MBytes   272 Mbits/sec
[  9]  0.0-10.0 sec   394 MBytes   330 Mbits/sec
[ 11]  0.0-10.0 sec   219 MBytes   183 Mbits/sec
[SUM]  0.0-10.0 sec  2.61 GBytes  2.24 Gbits/sec
[  4]  0.0-10.0 sec   142 MBytes   119 Mbits/sec
[ 13]  0.0-10.0 sec   164 MBytes   137 Mbits/sec
[ 14]  0.0-10.0 sec   139 MBytes   116 Mbits/sec
[ 15]  0.0-10.0 sec   139 MBytes   116 Mbits/sec
[ 16]  0.0-10.0 sec   131 MBytes   110 Mbits/sec
[ 17]  0.0-10.0 sec   150 MBytes   125 Mbits/sec
[ 18]  0.0-10.0 sec   119 MBytes  99.9 Mbits/sec
[SUM]  0.0-10.0 sec  1.12 GBytes   958 Mbits/sec

there was a text file with some tuneables

What was in this text file and where can I get a copy? I've been scouring the web trying to figure this out and have never read any mention of it.

 

[zambanini]

https://forums.freenas.org/index.php?threads/chelsio-t520-cr.22875/

but that does not make ten time the differents.

something else must be wrong. do the test with another OS

 

[zambanini]

what chelsio card are you exactly using, which bus type and so on?