X520 speed issue

[jgreco]

It's a Netgear GSM7352Sv2 it definitely has 2 10Gb uplinks which are being used. As mentioned the speeds I was getting were between 1.0Gbps and 1.01Gbps one way and 1.3Gbps in the other direction, so they aren't linked at 1Gbps. I'm currently working on setting up a direct connect between the FreeNAS server and Windows workstation on a separate subnet to see if problem persists.

I'm a college student, so I have no experience with networking outside of what I do at home. So obvious procedural tasks for you may be unknown to me. I bought the necessary components, I hooked them up. I didn't see what I expected so I made a post here.

How do I direct connect? I set the 10gig nics up on a separate subnet (192.168.10.1 /30) and windows (192.168.10.2/30) and I can't get a ping to go through. I'm sure there's some networking 101 I'm missing here.

Edit: Never mind. Didn't set the gateway properly.

You definitely don't need a gateway to connect the two things together - there wouldn't even be a gateway in that case and it isn't clear what you think "properly" would be. You should just be able to "ping 192.168.10.2" from 192.168.10.1 and vice versa and it should work, and just to be sure you're doing it right, disconnect any other connections from the two devices you're connecting together to guarantee that traffic can only flow over that one path.

The "almost exactly 1Gbps" thing is a really big red flag that there's either a 1G link in the path or that maybe there's something asymmetric in the network, such as maybe misconfigured IP networking causing one direction to traverse through 1Gbps networking. Windows is particularly annoying because it has firewalling considerations.

 

[TheDubiousDubber]

Well now I feel like a bit of an idiot, but at least I figured out the culprit. It appears to be solely related to Windows 10 issues. I used the windows workstation to boot a Fedora installation and iperf is returning with 9.46 Gb/s.

 

[Mlovelace]

Well now I feel like a bit of an idiot, but at least I figured out the culprit. It appears to be solely related to Windows 10 issues. I used the windows workstation to boot a Fedora installation and iperf is returning with 9.46 Gb/s.

You shouldn't feel like an idiot, that's how you learn. Now you know it's the client and not the server (and probably not the switch). Now you can look into fixing the windows box.

 

[TheDubiousDubber]

Yes. One of the many reason I hate Windows. Sadly the hardware for 10GbE to my mac is ridiculously expensive, so I settled for running it to my windows workstation.

 

[Ericloewe]

Well now I feel like a bit of an idiot, but at least I figured out the culprit. It appears to be solely related to Windows 10 issues. I used the windows workstation to boot a Fedora installation and iperf is returning with 9.46 Gb/s.

iperf, a few weeks ago, was giving me values around 600Mb/s between my workstation and one of my servers (Intel i210-AT on both sides). So I ran it between the two servers and, of course, bandwidth was ~900Mb/s. Haven't really had the time to look into it yet.

 

[jgreco]

Well now I feel like a bit of an idiot, but at least I figured out the culprit. It appears to be solely related to Windows 10 issues. I used the windows workstation to boot a Fedora installation and iperf is returning with 9.46 Gb/s.

They're computers. They're designed to make you feel like an idiot. That doesn't mean you should actually fall victim to that. We all have days we want to throw these things out the window.

 

[TheDubiousDubber]

At this point I'm waiting on a response from Intel support. Not really sure where else to turn. Though my first response from someone was a question regarding what driver I'm using. Considering I stated in my original post what hardware I was using, the operating system and listed the specific driver used, my hopes are not high.

 

[cyberjock]

You could try using linux to do some basic iperf tests. Even a live CD should be enough to figure out if its hardware or software. ;)

 

[TheDubiousDubber]

You could try using linux to do some basic iperf tests. Even a live CD should be enough to figure out if its hardware or software. ;)

According to Intel support. The driver is a built-in from Windows 10 package. They have no plans for support when it comes to Win10. I ended up purchasing a different card that has been shown to work in Win10 and I'll be moving my x520 over to my ESXi server. So it all worked out in the end. Albeit differently than I'd hoped.

 

[Mlovelace]

According to Intel support. The driver is a built-in from Windows 10 package. They have no plans for support when it comes to Win10. I ended up purchasing a different card that has been shown to work in Win10 and I'll be moving my x520 over to my ESXi server. So it all worked out in the end. Albeit differently than I'd hoped.

In case some has the same issue and finds this thread, what card did you end up going with for the win10 box?

 

[TheDubiousDubber]

I'm sure some in here may cringe at it, but I went with a Mellanox Connectx-2. According to another forum, much like the x520, the default drivers work like crap. However, provided you use the proper driver (WinOF 4.95 driver for 8.1 x64) it works great, and at full speed. The card was as cheap as they get on eBay, so I figured it was worth a shot.

 

[Rand]

Just curious, what hardware do you have in the windows box (cpu?).
I haven't tested Win 10 yet but on Win 8 there were a few settings that needed to be done to get better performance and in the end it was quite CPU dependent.
Meaning the E3-1230v3 was fine, the i3-4330 is significantly slower in the same board.

 

[TheDubiousDubber]

Just curious, what hardware do you have in the windows box (cpu?).
I haven't tested Win 10 yet but on Win 8 there were a few settings that needed to be done to get better performance and in the end it was quite CPU dependent.
Meaning the E3-1230v3 was fine, the i3-4330 is significantly slower in the same board.

It's an Ivy-bridge i5 workstation.

 

[Rand]

Hm that should be sufficient.
I am not sure whether the Win10 driver supports the various config settings the Intel driver support, but if you are still interested I can list mine from my slightly older card (Intel 10 GB AT Server Adapter); I was able to max out the connection with them from Win8 to FreeNas

 

[TheDubiousDubber]

That might be helpful. If anything, I might be able to say for certain that the x520 does not work well with Win10

 

[Mlovelace]

I can spin up a win10 VM on Monday and pass through a X520 to try and reproduce what you were seeing.

 

[Rand]

Free translation from german, so might not match exactly

Interrupt Reduction On
LSO4 on
Performance options
-> Receive buffer 4096
-> Flow Control RX/TX
-> Interrupt reduction rate Adaptive
-> Low Latency Interrupt (LLIs) UIser, for TCP PSH marked packages)
-> Transmit Buffer 16384
RSS On
RSS Queues (Depends on CPU Cores, 4+)
TCP Offload (Yes, all)

 

[TheDubiousDubber]

Free translation from german, so might not match exactly

Interrupt Reduction On
LSO4 on
Performance options
-> Receive buffer 4096
-> Flow Control RX/TX
-> Interrupt reduction rate Adaptive
-> Low Latency Interrupt (LLIs) UIser, for TCP PSH marked packages)
-> Transmit Buffer 16384
RSS On
RSS Queues (Depends on CPU Cores, 4+)
TCP Offload (Yes, all)

So there wasn't much I could change, but I did make some major changes to the Receive/Transmit buffers to the values you use and throughput jumped to 6.4Gb/s. That's a definite improvement over the 1.05Gb/s I was getting before.

Though UDP is still the same abysmal 1Gb/s.

 

[CookiesLikeWhoa]

I'm very curious to see how this turns out as I was going to go with the X520 for my Windows work station.

 

[jgreco]

Guys, the reason the X520 is popular around here is that it's a great server adapter. Intel has historically done a great job writing high performance drivers for their chipsets, and with FreeBSD, getting the right driver has typically meant not getting the latest and greatest Intel chipset because the driver support wouldn't be baked into FreeBSD yet. But once that driver's available, it is typically well-supported moving forward, because Intel understands how servers and server upgrades work. So getting these older cards is a great idea ... if you're running FreeBSD/FreeNAS.

Intel, however, has got to be aware that there isn't a super compelling use model for something like the X520-SR2 on a desktop, so I don't think they're putting lots of effort into producing a Windows 10 compatible driver for their older hardware. You're probably better off looking at some newer generation cards for that.