10Gb performance issue, 350Mb/s tops

dtom10 · Sep 28, 2017

Hello all,

I'm seeing a weird issue with my 10Gb adapters that I don't really know how to debug.

I have the following setup:

On Freenas I have an Intel X540-T2 adapter linked directly to my desktop using a good quality 10m CAT6e cable. The desktop uses the Intel X550 adapter onboard the Asus X99-E-10G WS

Testing with iperf results in a maximum performance of roughly 350Mbytes/s. On the desktop i have upgraded to the latest driver without seeing any performance improvement, desktop OS is Win10. SMB performance is ~330Mbytes/s

Here's the raw data:
Freenas side

Code:

root@ssdnas# iperf -s -f M
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 0.12 MByte (default)
------------------------------------------------------------
[  4] local 192.168.1.200 port 5001 connected with 192.168.1.10 port 53036
[ ID] Interval	   Transfer	 Bandwidth
[  4]  0.0-10.0 sec  3558 MBytes   356 MBytes/sec

Desktop side

Code:

bin/iperf.exe -c 192.168.1.200 -P 1 -i 1 -p 5001 -f M -t 10
------------------------------------------------------------
Client connecting to 192.168.1.200, TCP port 5001
TCP window size: 0.06 MByte (default)
------------------------------------------------------------
[304] local 192.168.1.10 port 53036 connected with 192.168.1.200 port 5001
[ ID] Interval	   Transfer	 Bandwidth
[304]  0.0- 1.0 sec   357 MBytes   357 MBytes/sec
[304]  1.0- 2.0 sec   353 MBytes   353 MBytes/sec
[304]  2.0- 3.0 sec   357 MBytes   357 MBytes/sec
[304]  3.0- 4.0 sec   356 MBytes   356 MBytes/sec
[304]  4.0- 5.0 sec   354 MBytes   354 MBytes/sec
[304]  5.0- 6.0 sec   355 MBytes   355 MBytes/sec
[304]  6.0- 7.0 sec   357 MBytes   357 MBytes/sec
[304]  7.0- 8.0 sec   357 MBytes   357 MBytes/sec
[304]  8.0- 9.0 sec   354 MBytes   354 MBytes/sec
[304]  9.0-10.0 sec   358 MBytes   358 MBytes/sec
[304]  0.0-10.0 sec  3558 MBytes   355 MBytes/sec
Done.

Driver & ifconfig

Code:

ix0: <Intel(R) PRO/10GbE PCI-Express Network Driver, Version - 3.1.13-k> mem 0xfaa00000-0xfabfffff,0xfa9f0000-0xfa9f3fff irq 17 at device 0.0 on pci1
ix0: Using MSIX interrupts with 9 vectors
ix0: Ethernet address: a0:36:9f:1f:e1:00
ix0: PCI Express Bus: Speed 5.0GT/s Width x8

root@ssdnas# ifconfig ix0
ix0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
		options=e407bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
		ether a0:36:9f:1f:e1:00
		inet 192.168.1.200 netmask 0xffffff00 broadcast 192.168.1.255
		nd6 options=9<PERFORMNUD,IFDISABLED>
		media: Ethernet autoselect (10Gbase-T <full-duplex,rxpause,txpause>)
		status: active

Freenas system:
Intel Xeon CPU E3-1265L V2 @ 2.50GHz
16Gb ECC DDR3
4 X 525Gb Crucial MX300 Raid Stripe
FreeNAS-11.0-U4 (54848d13b)

Any help is greatly appreciated.

dlavigne · Sep 28, 2017

While s slightly different model number, it sounds a lot like this: https://bugs.freenas.org/issues/25937 (see comment #7).

dtom10 · Sep 28, 2017

Doesn't look like a match. For starters it's not the same card, mine's tied to ixgbe. There are no network cuts only the performance. For example, the smb copy of an 8Gb .iso file runs peak at around 320Mb/s, no dips or rises. Looks like there's something capping the speed.

Stux · Sep 28, 2017

Possible a low speed quad core doesn't have enough oomph for more than 3.5gbps.

Silly L chips.

What's you CPU utilization during the iperf, and have you run it in both directions?

dtom10 · Sep 28, 2017

I thought iperf does not bother disks with nuisance. The system is idle. It's not doing anything else. At work I have lower speed xeons with dual 10Gb cards. I would think that the cpu is not the bottleneck here.

Code:

last pid: 79901;  load averages:  0.25,  0.11,  0.03																																   up 0+02:31:59  23:44:04
56 processes:  1 running, 55 sleeping
CPU:  0.2% user,  0.0% nice,  1.9% system,  3.2% interrupt, 94.7% idle
Mem: 146M Active, 457M Inact, 4332M Wired, 11G Free
ARC: 990M Total, 90M MFU, 606M MRU, 160K Anon, 40M Header, 254M Other
Swap: 8192M Total, 8192M Free

  PID USERNAME	THR PRI NICE   SIZE	RES STATE   C   TIME	WCPU COMMAND
79667 root		  4  20	0 26488K  3444K uwait   5   0:09  16.41% iperf

LE: tried both, on the the desktop i'm using jperf

Spearfoot · Sep 28, 2017

Try using jumbo frames.

dtom10 · Sep 28, 2017

Did that when I've set it up, the SMB speed was ~190Mb/s :(

Robert J. McMahon · Sep 29, 2017

dtom10 said:

Hello all,

I'm seeing a weird issue with my 10Gb adapters that I don't really know how to debug.

I have the following setup:

On Freenas I have an Intel X540-T2 adapter linked directly to my desktop using a good quality 10m CAT6e cable. The desktop uses the Intel X550 adapter onboard the Asus X99-E-10G WS

Testing with iperf results in a maximum performance of roughly 350Mbytes/s. On the desktop i have upgraded to the latest driver without seeing any performance improvement, desktop OS is Win10. SMB performance is ~330Mbytes/s

Here's the raw data:
Freenas side

Code:

root@ssdnas# iperf -s -f M
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 0.12 MByte (default)
------------------------------------------------------------
[  4] local 192.168.1.200 port 5001 connected with 192.168.1.10 port 53036
[ ID] Interval	   Transfer	 Bandwidth
[  4]  0.0-10.0 sec  3558 MBytes   356 MBytes/sec

Desktop side

Code:

bin/iperf.exe -c 192.168.1.200 -P 1 -i 1 -p 5001 -f M -t 10
------------------------------------------------------------
Client connecting to 192.168.1.200, TCP port 5001
TCP window size: 0.06 MByte (default)
------------------------------------------------------------
[304] local 192.168.1.10 port 53036 connected with 192.168.1.200 port 5001
[ ID] Interval	   Transfer	 Bandwidth
[304]  0.0- 1.0 sec   357 MBytes   357 MBytes/sec
[304]  1.0- 2.0 sec   353 MBytes   353 MBytes/sec
[304]  2.0- 3.0 sec   357 MBytes   357 MBytes/sec
[304]  3.0- 4.0 sec   356 MBytes   356 MBytes/sec
[304]  4.0- 5.0 sec   354 MBytes   354 MBytes/sec
[304]  5.0- 6.0 sec   355 MBytes   355 MBytes/sec
[304]  6.0- 7.0 sec   357 MBytes   357 MBytes/sec
[304]  7.0- 8.0 sec   357 MBytes   357 MBytes/sec
[304]  8.0- 9.0 sec   354 MBytes   354 MBytes/sec
[304]  9.0-10.0 sec   358 MBytes   358 MBytes/sec
[304]  0.0-10.0 sec  3558 MBytes   355 MBytes/sec
Done.

Driver & ifconfig

Code:

ix0: <Intel(R) PRO/10GbE PCI-Express Network Driver, Version - 3.1.13-k> mem 0xfaa00000-0xfabfffff,0xfa9f0000-0xfa9f3fff irq 17 at device 0.0 on pci1
ix0: Using MSIX interrupts with 9 vectors
ix0: Ethernet address: a0:36:9f:1f:e1:00
ix0: PCI Express Bus: Speed 5.0GT/s Width x8

root@ssdnas# ifconfig ix0
ix0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
		options=e407bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
		ether a0:36:9f:1f:e1:00
		inet 192.168.1.200 netmask 0xffffff00 broadcast 192.168.1.255
		nd6 options=9<PERFORMNUD,IFDISABLED>
		media: Ethernet autoselect (10Gbase-T <full-duplex,rxpause,txpause>)
		status: active

Freenas system:
Intel Xeon CPU E3-1265L V2 @ 2.50GHz
16Gb ECC DDR3
4 X 525Gb Crucial MX300 Raid Stripe
FreeNAS-11.0-U4 (54848d13b)

Any help is greatly appreciated.

Which version of iperf? If iperf 2.0.10 you can use -e on a unix client to get TCP RTT and CWND samples

Spearfoot · Sep 29, 2017

dtom10 said:
Did that when I've set it up, the SMB speed was ~190Mb/s :(

Hmmm... interesting.

I found that using jumbo frames was essential to make 10Gb/e work on my systems, but YMMV. See this post and the one following it:

https://forums.freenas.org/index.ph...ceiving-network-data.53722/page-2#post-373647

Stux · Sep 29, 2017

Me too, and that's a 2.2ghz Xeon. It might've been ESXi too.

dtom10 · Oct 1, 2017

Ok, this being a fresh install and not having to worry about data loss I went on to install a Centos 7 linux on the systemboot hdd. I wanted an unconstrained environment on which to test all sorts of settings and install whatever test software i needed.

What I have found is that indeed. @Spearfoot is right, enabling jumbo frames got me ~560Mb/s of performance instead of the previous ~350Mb/s that I got with defaults. The next thing that got me close to the full speed were the rx/tx ring buffers which came with a default of 512k from the 4096k max. Setting those to max on the desktop and the server produced roughly 8.1Gbits/s testing single transfer one way with iperf3 sending data from my win10 desktop to the Centos based install.

Apparently the same settings in FreeNAS produce the same results so I'm a happy camper :)

MTU is set to 9014 because that's what the desktop card is reporting.

Since FreeNAS embeds iperf2 i used jperf2.0.2 on my desktop

Code:

304] local 192.168.1.10 port 52736 connected with 192.168.1.200 port 5001
[ ID] Interval	   Transfer	 Bandwidth
[304]  0.0- 1.0 sec  1017 MBytes  1017 MBytes/sec
[304]  1.0- 2.0 sec  1002 MBytes  1002 MBytes/sec
[304]  2.0- 3.0 sec  1011 MBytes  1011 MBytes/sec
[304]  3.0- 4.0 sec  1049 MBytes  1049 MBytes/sec
[304]  4.0- 5.0 sec  1048 MBytes  1048 MBytes/sec
[304]  5.0- 6.0 sec  1041 MBytes  1041 MBytes/sec
[304]  6.0- 7.0 sec  1045 MBytes  1045 MBytes/sec
[304]  7.0- 8.0 sec  1078 MBytes  1078 MBytes/sec
[304]  8.0- 9.0 sec   996 MBytes   996 MBytes/sec
[304]  9.0-10.0 sec  1088 MBytes  1088 MBytes/sec
[304]  0.0-10.0 sec  10375 MBytes  1037 MBytes/sec
Done.

One more question, is editing the /boot/loader.conf a supported action or we are not supposed to touch that file. I'm planning on setting some sysctls in there to see if I can take the card to its max throughput.

SweetAndLow · Oct 1, 2017

dtom10 said:
Ok, this being a fresh install and not having to worry about data loss I went on to install a Centos 7 linux on the systemboot hdd. I wanted an unconstrained environment on which to test all sorts of settings and install whatever test software i needed.

What I have found is that indeed. @Spearfoot is right, enabling jumbo frames got me ~560Mb/s of performance instead of the previous ~350Mb/s that I got with defaults. The next thing that got me close to the full speed were the rx/tx ring buffers which came with a default of 512k from the 4096k max. Setting those to max on the desktop and the server produced roughly 8.1Gbits/s testing single transfer one way with iperf3 sending data from my win10 desktop to the Centos based install.

Apparently the same settings in FreeNAS produce the same results so I'm a happy camper :)
View attachment 20862

MTU is set to 9014 because that's what the desktop card is reporting.

Since FreeNAS embeds iperf2 i used jperf2.0.2 on my desktop

Code:
304] local 192.168.1.10 port 52736 connected with 192.168.1.200 port 5001 [ ID] Interval Transfer Bandwidth [304] 0.0- 1.0 sec 1017 MBytes 1017 MBytes/sec [304] 1.0- 2.0 sec 1002 MBytes 1002 MBytes/sec [304] 2.0- 3.0 sec 1011 MBytes 1011 MBytes/sec [304] 3.0- 4.0 sec 1049 MBytes 1049 MBytes/sec [304] 4.0- 5.0 sec 1048 MBytes 1048 MBytes/sec [304] 5.0- 6.0 sec 1041 MBytes 1041 MBytes/sec [304] 6.0- 7.0 sec 1045 MBytes 1045 MBytes/sec [304] 7.0- 8.0 sec 1078 MBytes 1078 MBytes/sec [304] 8.0- 9.0 sec 996 MBytes 996 MBytes/sec [304] 9.0-10.0 sec 1088 MBytes 1088 MBytes/sec [304] 0.0-10.0 sec 10375 MBytes 1037 MBytes/sec Done.

One more question, is editing the /boot/loader.conf a supported action or we are not supposed to touch that file. I'm planning on setting some sysctls in there to see if I can take the card to its max throughput.

You set sysctls in the GUI not the cli.

dtom10 · Oct 1, 2017

Right, the tunables section I presume?

One note to add is that the options for the network interface should not contain that equals sign, replace equals with space. I'm assuming it's an UI bug that accepts the equals sign or a feature gap (e.g.: convert equals to space).

Important Announcement for The TrueNAS Community.

10Gb performance issue, 350Mb/s tops

dtom10

Explorer

dlavigne

Guest

dtom10

Explorer

Stux

MVP

dtom10

Explorer

Spearfoot

He of the long foot

dtom10

Explorer

Robert J. McMahon

Dabbler

Spearfoot

He of the long foot

Stux

MVP

dtom10

Explorer

SweetAndLow

Sweet'NASty

dtom10

Explorer

Similar threads