X520 speed issue

[TheDubiousDubber]

I have 6x4TB RaidZ2 running 9.10 with Intel x520 10Gbe. I also have Windows 10 workstation setup with single 850 EVO and same x520 NIC. Everything appears to be working after installing the 10Gbe NICs, but transfer speeds from workstation and FreeNAS server are only around 150MB/s. This seems a bit slower than I expected. That's only about 3x what I was getting with 1Gig previously. Not sure what the bottleneck is here, but I imagine it should be higher than what it is. I don't know what if any tweaking is required. I ran a quick search and didn't come up with much. Help would be appreciated.

 

[Ericloewe]

Well, start with the basics. What results do you get with iperf?

 

[TheDubiousDubber]

Using FreeNAS as server and Windows 10 as client I get the following

Code:

C:\iperf>iperf -c 192.168.10.126 -i 2
-------------------------------------------------------
Client connecting to 192.168.10.126, TCP port 5001
TCP window size: 64.0 KByte (default)
-------------------------------------------------------
[   3] local 192.168.10.111 port 56716 connected with 192.168.10.126 port 5001
[ ID] Interval                      Transfer                    Bandwidth
[  3] 0.0- 2.0 sec                240 MBytes              1.01 Gbits/sec
[  3] 2.0- 4.0 sec                240 MBytes              1.01 Gbits/sec

Intervals went up to 10.0 sec but they are all the same.

 

[Ericloewe]

Clearly, you have a network issue, since you're barely exceeding 1Gb/s.

only around 150MB/s. This seems a bit slower than I expected. That's only about 3x what I was getting with 1Gig previously.

That means you had about half the expected bandwidth over GbE...

 

[TheDubiousDubber]

Clearly, you have a network issue, since you're barely exceeding 1Gb/s.

Yes, now to figure out what/why. I'm still learning when it comes to all this stuff, so I'm not really sure where to start, in order to figure out the issue.

 

[Ericloewe]

I'd start with something that didn't change. Any weird firewalls running on Windows? Some more dubious switch?

Cables are also highly suspect. It's within the realm of possibility to have had a bad Cat. 5e cable and now a bad fiber optic/SFP+ copper direct attach cable. I wouldn't expect quite this behavior, though.

 

[TheDubiousDubber]

All of the following have recently been replaced: x520 in Windows client >> fiber optic >> Netgear switch >> fiber optic >> x520 in FreeNAS server. Anything pre-existing is unrelated to networking between the two, so I highly doubt any recently replaced components are at fault.

It's a basic install of Windows 10, so the only firewall is the builtin software firewall in windows. Never messed with it, so I wouldn't think that would be at fault.

I don't doubt my lack of networking knowledge has led to some type of misconfiguration. And it's most likely due to some sort of setup involving FreeNAS, though I don't know where to begin. I've always been terrible with networking. I spend hours doing something that should take minutes, mainly due to a complete lack of know-how.


I ran iperf again. Using FreeNAS as server and Windows as client I'm getting 1Gb/s. When using Windows as server and Freenas as client I'm getting a steady 1.3Gb/s. It appears in doing this the TCP windows size changes from 64Kb to 32.5Kb. Not sure if that is what is making the difference or not.

 

[Mlovelace]

Attach the client and freeNAS server directly without the switch and try iperf again.

 

[jgreco]

Why would you not look to see what rate had been negotiated by the Ethernet adapters?

 

[TheDubiousDubber]

Why would you not look to see what rate had been negotiated by the Ethernet adapters?

Not sure what you mean. In Windows it shows the Ethernet interface as being 10.0 Gbps. It showed as 10Gb full duplex under switch management as well.

Though none of that matters now. I realized the switch had old firmware on it. I figured I'd update to see if that helped any and managed to brick the thing in the process. So, no more 10Gb goodness for me.

*Edit* : Now that I'm back on 1Gig connections I realized I am saturating the link from FreeNAS to Windows transfers. I'm consistently getting transfer speeds of ~115MB/s. I'm currently waiting on a new console cable to arrive so I can get to the CLI of the 10Gig switch since the firmware update botched something.

 

[Mirfster]

Out of curiosity what model is the NetGear switch? Just because it has ports that would accept the connection/cables doesn't mean that it is capable of supporting 10G speeds. Just making sure that you are not overlooking a simple limitation of the switch itself.

 

[Mlovelace]

Not sure what you mean. In Windows it shows the Ethernet interface as being 10.0 Gbps. It showed as 10Gb full duplex under switch management as well.

Though none of that matters now. I realized the switch had old firmware on it. I figured I'd update to see if that helped any and managed to brick the thing in the process. So, no more 10Gb goodness for me.

*Edit* : Now that I'm back on 1Gig connections I realized I am saturating the link from FreeNAS to Windows transfers. I'm consistently getting transfer speeds of ~115MB/s. I'm currently waiting on a new console cable to arrive so I can get to the CLI of the 10Gig switch since the firmware update botched something.

You can still test the 10Gbe links by direct connecting the two adapters without the switch. You will have to configure the IPs manually but at least you can see if the problem was the switch or keep trouble shooting.

 

[jgreco]

You can still test the 10Gbe links by direct connecting the two adapters without the switch. You will have to configure the IPs manually but at least you can see if the problem was the switch or keep trouble shooting.

It is ... unusual ... for the first step in troubleshooting a network to be to start disassembling the network when such basic steps as checking link speeds or (for managed switches) looking at error counts have not been performed...

In particular, when an iperf shows a near perfect gigabit on what's assumed to be a 10G link, that reeks of link speed misdetection, which ripping the network apart isn't likely to solve.

 

[Mlovelace]

It is ... unusual ... for the first step in troubleshooting a network to be to start disassembling the network when such basic steps as checking link speeds or (for managed switches) looking at error counts have not been performed...

In particular, when an iperf shows a near perfect gigabit on what's assumed to be a 10G link, that reeks of link speed misdetection, which ripping the network apart isn't likely to solve.

True, but he can't get into the switch currently so why not test the client/server then when the console cable shows up look at the switch.

 

[Ericloewe]

In particular, when an iperf shows a near perfect gigabit on what's assumed to be a 10G link, that reeks of link speed misdetection, which ripping the network apart isn't likely to solve.

iperf shows more than gigabit, so I'd assume it's linked at "10Gb", just with something really wrong.

 

[jgreco]

True, but he can't get into the switch currently so why not test the client/server then when the console cable shows up look at the switch.

Because you don't need to "get into the switch" to check link speeds.

 

[Mlovelace]

Because you don't need to "get into the switch" to check link speeds.

I don't have any netgear switches, but on my extreme networks and Cisco gear, you have to manually configure 10G and 40G port speeds. Auto negotiate is only for 10/100/1000 and since he can't edit the switch config, what does it hurt testing that 10Gbe is achievable without the switch. At least he test if the problem is the switch.

 

[jgreco]

I don't have any netgear switches, but on my extreme networks and Cisco gear, you have to manually configure 10G and 40G port speeds. Auto negotiate is only for 10/100/1000 and since he can't edit the switch config, what does it hurt testing that 10Gbe is achievable without the switch. At least he test if the problem is the switch.

And to you, disassembling a network is a better starting point than typing "ifconfig ${ifc}" on the UNIX end, and pulling up the interface on the Windows end, and just *looking*?

Dude, let me tell you, anything I can reasonably do from my chair, I do from my chair. I debugged a bad ethernet cable on the other side of the country in about three minutes just two weeks ago.

When my car doesn't start, I don't disassemble the engine first, instead I make sure it's got gas and the battery is good.

 

[Mlovelace]

And to you, disassembling a network is a better starting point than typing "ifconfig ${ifc}" on the UNIX end, and pulling up the interface on the Windows end, and just *looking*?

Dude, let me tell you, anything I can reasonably do from my chair, I do from my chair. I debugged a bad ethernet cable on the other side of the country in about three minutes just two weeks ago.

When my car doesn't start, I don't disassemble the engine first, instead I make sure it's got gas and the battery is good.

Okay, I'll just agree to disagree and walk away from this conversation. Enjoy your weekend :)

 

[TheDubiousDubber]

It's a Netgear GSM7352Sv2 it definitely has 2 10Gb uplinks which are being used. As mentioned the speeds I was getting were between 1.0Gbps and 1.01Gbps one way and 1.3Gbps in the other direction, so they aren't linked at 1Gbps. I'm currently working on setting up a direct connect between the FreeNAS server and Windows workstation on a separate subnet to see if problem persists.

I'm a college student, so I have no experience with networking outside of what I do at home. So obvious procedural tasks for you may be unknown to me. I bought the necessary components, I hooked them up. I didn't see what I expected so I made a post here.

How do I direct connect? I set the 10gig nics up on a separate subnet (192.168.10.1 /30) and windows (192.168.10.2/30) and I can't get a ping to go through. I'm sure there's some networking 101 I'm missing here.

Edit: Never mind. Didn't set the gateway properly.

Edit 2: Direct connect yields same results. Averaging between 1.03Gbps and 1.06Gbps in iperf. Also, when switch was working I did check error counts. Errors, collisions, etc. were all at 0. There were no indications of anything other than a possible hardware issue, or something on Windows side.