X10SL7-F and FreeNAS 9.2 Network Performance

[SimonRH]

Evening FreeNAS,

I've recently built a NAS box and installed FreeNAS 9.2 upon it. So far I'm liking the user experience but did experience a networking issue when testing using iperf. The issue resulted in me installing a number of other OSes in an attempt to determine if it's a physical network issue or some problem with the drivers in 9.2.

The build uses a Supermicro X10SL7-F board. This board comes with dual Dual Gigabit Ethernet i210AT Lan ports.

Below I list a sample of the iperf results:

Clean install of FreeNAS 9.2:
simon-mini:Downloads simon$ iperf -c 192.168.130.143 -P 1 -i 1 -p 5001
------------------------------------------------------------
Client connecting to 192.168.130.143, TCP port 5001
TCP window size: 129 KByte (default)
------------------------------------------------------------
[ 4] local 192.168.130.141 port 54652 connected with 192.168.130.143 port 5001
[ ID] Interval Transfer Bandwidth
[ 4] 0.0- 1.0 sec 67.1 MBytes 563 Mbits/sec
[ 4] 1.0- 2.0 sec 20.4 MBytes 171 Mbits/sec
[ 4] 2.0- 3.0 sec 26.5 MBytes 222 Mbits/sec
[ 4] 3.0- 4.0 sec 41.0 MBytes 344 Mbits/sec
[ 4] 4.0- 5.0 sec 76.8 MBytes 644 Mbits/sec
[ 4] 5.0- 6.0 sec 51.9 MBytes 435 Mbits/sec
[ 4] 6.0- 7.0 sec 81.9 MBytes 687 Mbits/sec
[ 4] 7.0- 8.0 sec 88.6 MBytes 743 Mbits/sec
[ 4] 8.0- 9.0 sec 43.4 MBytes 364 Mbits/sec
[ 4] 9.0-10.0 sec 108 MBytes 905 Mbits/sec
[ 4] 0.0-10.0 sec 606 MBytes 508 Mbits/sec


Clean install of FreeNAS 9.1.1:
simon-mini:Downloads simon$ iperf -c 192.168.130.143 -P 1 -i 1 -p 5001
------------------------------------------------------------
Client connecting to 192.168.130.143, TCP port 5001
TCP window size: 129 KByte (default)
------------------------------------------------------------
[ 4] local 192.168.130.141 port 50867 connected with 192.168.130.143 port 5001
[ ID] Interval Transfer Bandwidth
[ 4] 0.0- 1.0 sec 113 MBytes 945 Mbits/sec
[ 4] 1.0- 2.0 sec 112 MBytes 941 Mbits/sec
[ 4] 2.0- 3.0 sec 112 MBytes 942 Mbits/sec
[ 4] 3.0- 4.0 sec 112 MBytes 941 Mbits/sec
[ 4] 4.0- 5.0 sec 112 MBytes 942 Mbits/sec
[ 4] 5.0- 6.0 sec 112 MBytes 941 Mbits/sec
[ 4] 6.0- 7.0 sec 112 MBytes 942 Mbits/sec
[ 4] 7.0- 8.0 sec 112 MBytes 942 Mbits/sec
[ 4] 8.0- 9.0 sec 112 MBytes 941 Mbits/sec
[ 4] 9.0-10.0 sec 112 MBytes 942 Mbits/sec
[ 4] 0.0-10.0 sec 1.10 GBytes 942 Mbits/sec

For the record I booted both Ubuntu 3.10 and NAS4Free and ran the same test and saw similar results (94XMbit/sec) to those of the FreeNAS 9.1.1 install. I later installed FreeNAS 9.2.1 RCs in the hope of a silent fix but saw similar timings to those seen in 9.2.

Has anyone experienced a similar issues? Any suggestions on how to fix the performance regression?

Cheers
Simon

 

[cyberjock]

Interesting. I don't have your NIC, so I can't really validate. :(

 

[crumbz]

I'm running FreeNAS 9.2. The iperf host was running on a Windows 7 computer with a Realtek adapter.

[root@freenas] ~# iperf -c 192.168.1.6 -P 1 -i 1 -p 5001
------------------------------------------------------------
Client connecting to 192.168.1.6, TCP port 5001
TCP window size: 32.5 KByte (default)
------------------------------------------------------------
[ 3] local 192.168.1.11 port 64382 connected with 192.168.1.6 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0- 1.0 sec 85.6 MBytes 718 Mbits/sec
[ 3] 1.0- 2.0 sec 92.5 MBytes 776 Mbits/sec
[ 3] 2.0- 3.0 sec 88.8 MBytes 744 Mbits/sec
[ 3] 3.0- 4.0 sec 88.9 MBytes 746 Mbits/sec
[ 3] 4.0- 5.0 sec 88.8 MBytes 744 Mbits/sec
[ 3] 5.0- 6.0 sec 88.9 MBytes 746 Mbits/sec
[ 3] 6.0- 7.0 sec 88.8 MBytes 744 Mbits/sec
[ 3] 7.0- 8.0 sec 88.8 MBytes 744 Mbits/sec
[ 3] 8.0- 9.0 sec 88.9 MBytes 746 Mbits/sec
[ 3] 9.0-10.0 sec 88.9 MBytes 746 Mbits/sec
[ 3] 0.0-10.0 sec 889 MBytes 746 Mbits/sec

 

[crumbz]

My FreeNAS has been updated a few times though, not a clean install.

 

[cyberjock]

Updating shouldn't normally cause problems. If you use RCs and betas though, that can really FUBAR things.

Ideally an Intel NIC on both sides yields the most useful results. Considering you have a Realtek on your desktop, the 700Mb values you got are pretty much what I'd expect. If you go Intel on both ends you'll pretty much always go over 900Mb every time.

 

[crumbz]

I'll re-test with an Intel NIC in my desktop this weekend and post the results.

 

[SimonRH]

Evening all,

It's Friday evening so had a little more time to investigate and thus provide a small update.

Good Performance (throughput):
msfFreeBSD 9.2 (FreeBSD flash drive image) - dmesg shows igb driver as version 2.3.10
NAS4Free-9.1.0.1.x - dmesg shows driver as 2.3.4
FreeNAS-9.1-STABLE -dmesg shows driver as 2.3.9

Poor Performance:
FreeNAS-9.2 - dmesg shows igb drive9r as 2.4.0

So current speculation is a change in the driver between v 2.3.10 and v 2.4.

Cheers
Simon

 

[cyberjock]

It's possible. What interests me is that nobody else with your NIC has complained. I'd have expected complaints from other people. That makes me thing its something unique to your configuration.

I mean, lets face it. People that go with Intel want maximum performance. And when they don't get it, the show up here bustin' up the place. So I'd have expected more than just you with the problem. I don't have your exact NIC, so I can't really validate this for myself. I can validate that my PCIe card in my test machine that is Intel works just fine. Over 950Mb/sec every test.

Do you have any kind of tunables, sysctls, jumbo frames, LACP/LAGG, anything besides the defaults for the LAN stuff?

And I'd like to see all of your server specs.;)

 

[SimonRH]

Evening Cyberjock,

Yup, don't disagree with your thinking as I too would of expected a few more queries if it was a common problem.

Anyway, to answer some of your questions:

No tunables, no sysctls, no Jumbo frames (MTU is at 1500) etc. Just the default LAN settings from a untampered boot.

Specs of the machine I'm playing with:
Supermicro X10SL7-F
24 GB KVR16LE11/8EF (ECC) RAM
LSI 2308 - Flashed to IT Mode
CPU Xeon e3-1230 v2
Bunch of disks to play around with ZFS.
I boot off a 8GB SanDisk Flash (Have 3 of these hence I’m able to play around with different OS versions rather quickly).

More than willing to accept I'm missing something bleeding obvious.

Cheers
Simon

 

[SimonRH]

Sorry, missed the LACP/LAGG question. Nope, not doing any link aggregation yet.

Cheers
Simon

 

[cyberjock]

I have no recommendations then.

 

[HolyK]

Hi,

i have the same MB (full spec here) and using FreeNAS 9.2 . Second end is my workstation with on-board Realtek NIC

iperf results are:

Code:

bin/iperf.exe -c xxx.xxx.xxx.xxx -P 1 -i 1 -p 5001 -w 512K -f m -t 10
------------------------------------------------------------
Client connecting to xxx.xxx.xxx.xxx, TCP port 5001
TCP window size: 0.50 MByte
------------------------------------------------------------
[156] local xxx.xxx.xxx.xxx port 52341 connected with xxx.xxx.xxx.xxx port 5001
[ ID] Interval      Transfer    Bandwidth
[156]  0.0- 1.0 sec  92.3 MBytes  774 Mbits/sec
[156]  1.0- 2.0 sec  88.9 MBytes  746 Mbits/sec
[156]  2.0- 3.0 sec  84.6 MBytes  710 Mbits/sec
[156]  3.0- 4.0 sec  85.3 MBytes  716 Mbits/sec
[156]  4.0- 5.0 sec  84.5 MBytes  709 Mbits/sec
[156]  5.0- 6.0 sec  84.5 MBytes  709 Mbits/sec
[156]  6.0- 7.0 sec  83.6 MBytes  701 Mbits/sec
[156]  7.0- 8.0 sec  83.7 MBytes  702 Mbits/sec
[156]  8.0- 9.0 sec  82.3 MBytes  690 Mbits/sec
[156]  9.0-10.0 sec  84.2 MBytes  707 Mbits/sec
[156]  0.0-10.0 sec  854 MBytes  716 Mbits/sec
Done.

I see you are testing with small TCP window size. Try it with 512KB, this should utilize the whole network bandwidth

 

[raidflex]

This is what I got with the same board. The second PC has a on-board Realtek NIC and Windows 8.1

Code:

~# iperf -c xxx.xxx.xxx.xxx -P 1 -i 1 -p 5001 -w 512K -f m -t 10
------------------------------------------------------------
Client connecting to xxx.xxx.xxx.xxx, TCP port 5001
TCP window size: 0.50 MByte 
------------------------------------------------------------
[  3] local xxx.xxx.xxx.xxx port 35459 connected with xxx.xxx.xxx.xxx port 5001
[ ID] Interval      Transfer    Bandwidth
[  3]  0.0- 1.0 sec  112 MBytes  937 Mbits/sec
[  3]  1.0- 2.0 sec  112 MBytes  938 Mbits/sec
[  3]  2.0- 3.0 sec  111 MBytes  932 Mbits/sec
[  3]  3.0- 4.0 sec  110 MBytes  922 Mbits/sec
[  3]  4.0- 5.0 sec  110 MBytes  920 Mbits/sec
[  3]  5.0- 6.0 sec  110 MBytes  924 Mbits/sec
[  3]  6.0- 7.0 sec  110 MBytes  924 Mbits/sec
[  3]  7.0- 8.0 sec  110 MBytes  924 Mbits/sec
[  3]  8.0- 9.0 sec  110 MBytes  926 Mbits/sec
[  3]  9.0-10.0 sec  110 MBytes  922 Mbits/sec
[  3]  0.0-10.0 sec  1105 MBytes  927 Mbits/sec

 

[HolyK]

Wow, nice ... its like 90.5% of max theoretical 1Gbit network utilization. Is it direct connect or you have some active switch/router between ?

On my side i guess the guilty one is that on-board Realtek. I have spare PCIe intel card but i didn't had time to test it :/

 

[raidflex]

Wow, nice ... its like 90.5% of max theoretical 1Gbit network utilization. Is it direct connect or you have some active switch/router between ?

On my side i guess the guilty one is that on-board Realtek. I have spare PCIe intel card but i didn't had time to test it :/

This is through a non-managed Netgear gigabit switch. I am also using a Realtek NIC on the Win 8 PC. But driver support is probably much better on Windows then Freenas for Realtek NICs.

 

[Sir.Robin]

Here's mine:

Code:

[NAS01] /mnt/vol01/ftp# iperf -c X.X.X.X -P 1 -i 1 -p 5001 -w 512K -f m -t 10
------------------------------------------------------------
Client connecting to X.X.X.X, TCP port 5001
TCP window size: 0.50 MByte (WARNING: requested 0.50 MByte)
------------------------------------------------------------
[  3] local X.X.X.X port 49441 connected with X.X.X.X port 5001
[ ID] Interval      Transfer    Bandwidth
[  3]  0.0- 1.0 sec  106 MBytes  890 Mbits/sec
[  3]  1.0- 2.0 sec  106 MBytes  887 Mbits/sec
[  3]  2.0- 3.0 sec  104 MBytes  870 Mbits/sec
[  3]  3.0- 4.0 sec  101 MBytes  849 Mbits/sec
[  3]  4.0- 5.0 sec  94.0 MBytes  789 Mbits/sec
[  3]  5.0- 6.0 sec  103 MBytes  861 Mbits/sec
[  3]  6.0- 7.0 sec  101 MBytes  850 Mbits/sec
[  3]  7.0- 8.0 sec  101 MBytes  846 Mbits/sec
[  3]  8.0- 9.0 sec  103 MBytes  861 Mbits/sec
[  3]  9.0-10.0 sec  103 MBytes  861 Mbits/sec
[  3]  0.0-10.0 sec  1022 MBytes  857 Mbits/sec
[NAS01] /mnt/vol01/ftp#

My NAS01 is a VM on vSphere 5.5. PC has intel CT NIC and i have a little unmanaged HP switch between. :)

 

[jagga]

I have too very poor perfomance with this mobo runing ESXI host with Freenas.
Trying to clarify where is a problem.

 

[jagga]

I have problem with network perfomance on X10SL7 (Dual Gigabit Ethernet i210AT Lan ports) with Xeon E3-1230v3.
N4F or FreeNAS installed on ssd. MTU=9000. Win7 Client also have 9k MTU size.
Router support MTU pockets. Cables were replaced.

Do you have any suggestations where is a bottleneck?
Below you can find details.

Code:

Client Win7 SP1:
Realtek RTL8111E (1000mb full duplex)
iperf-2.0.5-2-win32>iperf.exe -c 192.168.1.250 -p 5001 -f
m -w 128k
------------------------------------------------------------
Client connecting to 192.168.1.250, TCP port 5001
TCP window size: 0.12 MByte
------------------------------------------------------------
[ 3] local 192.168.1.144 port 49999 connected with 192.168.1.250 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.0 sec 924 MBytes 775 Mbits/sec
 
Server NAS4FREE 9.2.0.1.972 or latest build of Free NAS:
nas4free: ~ # iperf -s -w 128k
 
igb0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
options=401bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,VLAN_HWTSO>
ether 00:25:90:f2:b7:6c
inet 192.168.1.250 netmask 0xffffff00 broadcast 192.168.1.255
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active

 

[Ericloewe]

I have problem with network perfomance on X10SL7 (Dual Gigabit Ethernet i210AT Lan ports) with Xeon E3-1230v3.
N4F or FreeNAS installed on ssd. MTU=9000. Win7 Client also have 9k MTU size.
Router support MTU pockets. Cables were replaced.

Do you have any suggestations where is a bottleneck?
Below you can find details.

Code:

Client Win7 SP1:
Realtek RTL8111E (1000mb full duplex)

There's your likely problem. Realtek is not known for being good in any way (except cost). If you can, run it with a decent adapter on the client end.

 

[cyberjock]

Fail for going with Realtek.

Also, fail for trying jumbo frames. They don't matter on Gb LAN. It is totally possible to have MTU set to 9000 in Windows and FreeNAS and they still not match which is precisely why jumbo frames cause way more problems than they solve.