10G network - expected iperf results?

[xyzzy]

I've just set up 10G for the first time in the form of two systems, each with an Intel X710-DA4 connected directly via a single Intel SFP+ 3M passive DAC cable. Both systems are Supermicro X9SRE-F with a E5-1650 V2. One is running FreeNAS 9.10 and the other is running Windows 8.1.

I'm using iperf to test the bandwidth. On the server side, I'm running iperf -s. On the client side, I'm running jperf with the default TCP settings.

With 1 parallel stream, I'm hitting 2.97 Gbits/sec. I need to use 4 parallel streams to max out (around 9.46 Gbits/sec).

Is it expected that:
(1) I need 4 parallel streams to max out?
(2) That the max is around 9.46 Gbits/sec?

Thanks in advance!

 

[c32767a]

I've just set up 10G for the first time in the form of two systems, each with an Intel X710-DA4 connected directly via a single Intel SFP+ 3M passive DAC cable. Both systems are Supermicro X9SRE-F with a E5-1650 V2. One is running FreeNAS 9.10 and the other is running Windows 8.1.

I'm using iperf to test the bandwidth. On the server side, I'm running iperf -s. On the client side, I'm running jperf with the default TCP settings.

With 1 parallel stream, I'm hitting 2.97 Gbits/sec. I need to use 4 parallel streams to max out (around 9.46 Gbits/sec).

Is it expected that:
(1) I need 4 parallels streams to max out?
(2) That the max is around 9.46 Gbits/sec?

Thanks in advance!

That seems low for iperf.

I have FreeNAS on a X9SRL-F with an E5-1620v2 connected back to back with an Gigabyte X99 running Ubuntu 16.04 with an E5-1620v3.. The X9SRL has a Solarflare 10G card, the X99 has an intel X710. I can get 9.8Gb/s with iperf on a back to back link. I didn't do any tuning aside from increasing window sizes.. Otherwise they are vanilla installs.

I don't know 10G tuning on Windows. You could boot a Ubuntu liveCD to see if you can get good perf with your hardware. That would tell you if Windows is the problem.

 

[xyzzy]

That seems low for iperf.

I have FreeNAS on a X9SRL-F with an E5-1620v2 connected back to back with an Gigabyte X99 running Ubuntu 16.04 with an E5-1620v3.. The X9SRL has a Solarflare 10G card, the X99 has an intel X710. I can get 9.8Gb/s with iperf on a back to back link. I didn't do any tuning aside from increasing window sizes.. Otherwise they are vanilla installs.

I don't know 10G tuning on Windows. You could boot a Ubuntu liveCD to see if you can get good perf with your hardware. That would tell you if Windows is the problem.

Thanks for the info. I have a funny feeling it's either Windows or the Java client. I did the same test with the same two boxes but instead of running Windows, I was running VMware ESXi 6.0 U2 (which I was surprised to learn has iperf as part of the base install). In this case, I was able to get 9.4 Gbits/sec with a single stream.

Just out of curiosity, what did you change the window size to?

 

[bigphil]

Are you using 9.10 stable or nightlies train of FreeNAS? The current stable branch uses a very old ixl driver for the X710/XL710 family of cards. The latest 9.10 nightlies or version 10 should have a newer driver. If you stay on 9.10 stable, I'd update the driver to 1.4.27 using the method I posted here. Not that it's going to dramatically improve your performance, but there are some good bug fixes in it.

 

[xyzzy]

Are you using 9.10 stable or nightlies train of FreeNAS? The current stable branch uses a very old ixl driver for the X710/XL710 family of cards. The latest 9.10 nightlies or version 10 should have a newer driver. If you stay on 9.10 stable, I'd update the driver to 1.4.27 using the method I posted here. Not that it's going to dramatically improve your performance, but there are some good bug fixes in it.

I'm using the latest stable version. This is some very good info you provided....I'll definitely try updating the driver using the method you provided. Thank you very much!

 

[c32767a]

Thanks for the info. I have a funny feeling it's either Windows or the Java client. I did the same test with the same two boxes but instead of running Windows, I was running VMware ESXi 6.0 U2 (which I was surprised to learn has iperf as part of the base install). In this case, I was able to get 9.4 Gbits/sec with a single stream.

Just out of curiosity, what did you change the window size to?

I misspoke, I meant socket buffer. When you get to around 10Gb/s the default socket buffer can almost fill a 10G link.
kern.ipc.maxsockbuf=16777216 makes it a bit larger. It's also relevant if you have more than a couple msec of latency on your link.

 

[bigphil]

To go along with what @c32767a said, here is a nice article that explains a few network tuning options. Search for maxsockbuf on that page and read the info.

 

[chris crude]

good information. I'm about to upgrade to some 10G interfaces between my windows 10 and FreeNas box (latest stable version) so I will watch this thread.