X540-T2 10GB Very Slow - 700 to 900 KB/s Transfer

[Brosif_My_Nif]

Hey Guys,

I have been searching through the forums for an answer to my problem, but have yet to find someone experiencing the same issue I am. I am hoping someone here would be kind enough to offer me some pointers.

I am just trying to link my main Windows 10 Desktop with a Intel X540-T1 to my FreeNAS server with a X540-T2 (see specs in signature). The problem is, I am averaging between 700 KB/s and 900 KB/s from FreeNAS to Windows 10.

Windows 10 to FreeNAS is not much better. It doesn't go much higher than 10 MB/s.

HOWEVER, IPERF between the two boxes averages around 5 GB/s.

login as: root
root@192.169.2.10's password:
Last login: Tue Jan 19 13:48:06 2016 from 192.169.2.9
FreeBSD 9.3-RELEASE-p31 (FREENAS.amd64) #0 r288272+81e2595: Mon Jan 18 10:48:44 PST 2016

        FreeNAS (c) 2009-2015, The FreeNAS Development Team
        All rights reserved.
        FreeNAS is released under the modified BSD license.

        For more information, documentation, help or support, go here:
        http://freenas.org
Welcome to FreeNAS
[root@freenas] ~# iperf -s
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 64.0 KByte (default)
------------------------------------------------------------
[  4] local 192.169.2.10 port 5001 connected with 192.169.2.9 port 60005
[ ID] Interval       Transfer     Bandwidth
[  4]  0.0-10.0 sec  4.80 GBytes  4.12 Gbits/sec
[  5] local 192.169.2.10 port 5001 connected with 192.169.2.9 port 60011
[  5]  0.0-10.0 sec  5.04 GBytes  4.33 Gbits/sec
[  4] local 192.169.2.10 port 5001 connected with 192.169.2.9 port 60012
[  4]  0.0-10.0 sec  4.93 GBytes  4.23 Gbits/sec

This is where it gets weird. If I put a gigabit switch between the two X540 NIC's, trasfers jump up to around 110 MB/s. Full gigabit.

What am I missing? Any help is very much appreciated.

Thank you!

 

[diehard]

Whats the output from ifconfig say ?

 

[jgreco]

Damn it I know what that is but I can't remember.

Things to try,

# ifconfig ixgbe0 -tso

Set system tunable "hw.ix.enable_aim" to zero and see if it helps (reboot required, I believe).

 

[Brosif_My_Nif]

Whats the output from ifconfig say ?

I'm sorry, I should have included that originally. See below.

The port I'm trying to get working is: "ix1"

lagg0 is connected to the switch serving clients who only need gigabit.

Code:

[root@freenas] ~# ifconfig
ix0: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=407bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSU       M,TSO4,TSO6,LRO,VLAN_HWTSO>
        ether a0:36:9f:54:ef:fc
        nd6 options=9<PERFORMNUD,IFDISABLED>
        media: Ethernet autoselect
        status: no carrier
ix1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=407bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSU       M,TSO4,TSO6,LRO,VLAN_HWTSO>
        ether a0:36:9f:54:ef:fe
        inet 192.169.2.10 netmask 0xffffff00 broadcast 192.169.2.255
        nd6 options=9<PERFORMNUD,IFDISABLED>
        media: Ethernet autoselect (10Gbase-T <full-duplex>)
        status: active
igb0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSU       M,TSO4,TSO6,VLAN_HWTSO>
        ether 00:25:90:fc:98:d0
        nd6 options=9<PERFORMNUD,IFDISABLED>
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
igb1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSU       M,TSO4,TSO6,VLAN_HWTSO>
        ether 00:25:90:fc:98:d0
        nd6 options=9<PERFORMNUD,IFDISABLED>
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
igb2: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSU       M,TSO4,TSO6,VLAN_HWTSO>
        ether 00:25:90:fc:98:d0
        nd6 options=9<PERFORMNUD,IFDISABLED>
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
igb3: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSU       M,TSO4,TSO6,VLAN_HWTSO>
        ether 00:25:90:fc:98:d0
        nd6 options=9<PERFORMNUD,IFDISABLED>
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
ipfw0: flags=8801<UP,SIMPLEX,MULTICAST> metric 0 mtu 65536
        nd6 options=9<PERFORMNUD,IFDISABLED>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0xa
        inet 127.0.0.1 netmask 0xff000000
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
lagg0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSU       M,TSO4,TSO6,VLAN_HWTSO>
        ether 00:25:90:fc:98:d0
        inet 192.168.2.131 netmask 0xffffff00 broadcast 192.168.2.255
        nd6 options=9<PERFORMNUD,IFDISABLED>
        media: Ethernet autoselect
        status: active
        laggproto lacp lagghash l2,l3,l4
        laggport: igb3 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
        laggport: igb2 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
        laggport: igb1 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
        laggport: igb0 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>

I know you did not ask for it, but the Windows 10 Box seems to see the 10GB link just fine as well...

 

[Brosif_My_Nif]

Damn it I know what that is but I can't remember.

Things to try,

# ifconfig ixgbe0 -tso

Set system tunable "hw.ix.enable_aim" to zero and see if it helps (reboot required, I believe).

Thank you very much for trying to offer your assistance. I do appreciate it. Unfortunately, neither of your suggestions seem to have fixed the problem.

I read a little on the -tso problems with the Intel 10GB cards, and I'm pretty sure that problem was fixed in 9.3.

 

[VictorR]

Hi, I have 3 Intel X540-T2 NICS in my 45 Drives Q30 NAS. I have used them direct-connected to clients, and now through a Netgear XS728T 10Gbe switch
I did find that I would get strange bandwidth if one end of the connection was set to a different frame rate than the other, e.g. mtu 1500 . mtu 9000
Have you checked to make sure both Intel 540's are set to the same frame rate?

 

[jgreco]

Ideally ditch jumbo entirely, there's not much benefit and the potential for a whole lot of trouble.

https://forums.freenas.org/index.php?threads/jumbo-frames-notes.26064/

 

[titan_rw]

I've got a x540 in my windows desktop, and an x540 built in to my X9SRH-7TF. Both copper. No issue with 8-9 gigabit between the two. I also use jumbo frames. I only got 5-6 gigabit without jumbo.

 

[Brosif_My_Nif]

Hi, I have 3 Intel X540-T2 NICS in my 45 Drives Q30 NAS. I have used them direct-connected to clients, and now through a Netgear XS728T 10Gbe switch
I did find that I would get strange bandwidth if one end of the connection was set to a different frame rate than the other, e.g. mtu 1500 . mtu 9000
Have you checked to make sure both Intel 540's are set to the same frame rate?

I tried changing the X540 in my Windows 10 machine to 9000 and I changed the MTU on the FreeNAS to the same. No change in performance. Still 700-900 KB/s via CIFS. I tried them all and ensured they were set to the same on both machines. Still no change.

I'm going to run a new cable on the ground between the two boxes and see if that might do anything. I don't think it will, however, since I can get 5 GB/s via IPERF, but why not.

 

[Brosif_My_Nif]

I've got a x540 in my windows desktop, and an x540 built in to my X9SRH-7TF. Both copper. No issue with 8-9 gigabit between the two. I also use jumbo frames. I only got 5-6 gigabit without jumbo.

Did you have to do any configuring or did it "just work" right out of the box?

 

[titan_rw]

Did you have to do any configuring or did it "just work" right out of the box?

It worked right out of the box at > 1 gigabit. I set some network buffers and stuff higher to achieve consistent high rates, but I got 2-4 gigabit with everything on default.

 

[diehard]

I have X540 but it is with iSCSI between FN and ESXi so i'm not sure i can offer any great advice.

 

[jgreco]

I believe the problems that people have noticed with X540 are protocol agnostic, so I'd expect that all experiences are at least moderately helpful.

 

[Brosif_My_Nif]

Well guys, I think I found a problem. I removed the FreeNAS server from the rack and brought it right next to my Windows 10 box and connected the two X540 NIC's via a 3ft. CAT6a cable. I am now able to get CIFS transfers of about 360 MB/s and IPERF speeds of around 600 MB/s.

So, lesson learned. Always check the cables first. I appreciate all your help! Now I just need to work on tuning these X540's.

 

[VictorR]

I had a similar problem with a bad/intermittent SATA cable to HDDs. I spent a couple of weeks ruling out every other possibility

 

[jgreco]

Well guys, I think I found a problem. I removed the FreeNAS server from the rack and brought it right next to my Windows 10 box and connected the two X540 NIC's via a 3ft. CAT6a cable. I am now able to get CIFS transfers of about 360 MB/s and IPERF speeds of around 600 MB/s.

So, lesson learned. Always check the cables first. I appreciate all your help! Now I just need to work on tuning these X540's.

Don't worry, we've all been there.

Start out by making sure you run things like iperf tests directly on the two machines.

 

[{HD}]

Why would the IPERF and realworld tests be so different? Did you make your longer cable and get everything dialed in?

BTW: Yes, I see this is from 9 months ago.

 

[Brosif_My_Nif]

Why would the IPERF and realworld tests be so different? Did you make your longer cable and get everything dialed in?

BTW: Yes, I see this is from 9 months ago.

I returned the Intel X540's. I spent days trying to tune it. Nothing I did would get above 360 MB/s via CIFS. Eventually I gave up, fell back on gigabit connections for now and said I would try it again down the road.

 

[{HD}]

I returned the Intel X540's. I spent days trying to tune it. Nothing I did would get above 360 MB/s via CIFS. Eventually I gave up, fell back on gigabit connections for now and said I would try it again down the road.

Well this is both reassuring and concerning. I just bought 2 x540-T2's and plan to run direct from Main Client to FreeNAS. I hope I can get IPERF to have a decent result but, I know my actual r/w speeds will be much much slower because I am currently running a SAS backplane which I think limits the speed of my drives to only 3Gbits/s.

I'll post me results after my NICs show up.

 

[{HD}]

Well, I said I would so...

Using the X540-T2s direct with many System-Tunables involved.