Extremely poor one-sided network performance

Status
Not open for further replies.

JayanWarden

Dabbler
Joined
Nov 26, 2017
Messages
22
Hello everyone,

I recently rebuilt my FreeNAS server out of real server components.
Everything went fine from pool creation to L2Arc (i know that it may be overkill) and bhyve VM creation.
But I now have the following scenario:
Transfer from my PC to my NAS is running at around 400-500 MBit/s. Not the best, I know.
Transfer from my NAS to my PC is running at merely 15 Mbit/s, sometimes even less. Yes, Mbit.
I am using a SMB share that directly lies in my ZFS pool.

Specs:
Server:
Motherboard: Supermicro X9SRL-F-0 (new)
CPU: Intel Xeon E5 2620 v2 (used from ebay but working perfectly)
RAM: 4 x 16 GB Samsung ECC DDR3 (1333 MHz, also used but free from any errors)
5 x 8 TB Seagate IronWolf Pro in ZFS-Z2

PC: (used for testing with iPerf and data transfer)
Motherboard: MSI Gaming Pro Carbon x299
CPU: Intel Core i7 7740x
32GB RAM (as if that matters)

The server and my PC both use the onboard NICs.
The server has an Intel 82574L Dual GBE Controller, my PC has an Intel I219-v GBE Controller.

The PC and the server are connected through a generic TP-Link GBE 8-port switch and I use cat6 cables for the connection (two 2 meter cables)

Here is a screenshot from windows file transfer from my PC to my server:
7qyERLC.png

Throughput is bouncing between 35 and ~70 MB/s

The server is merely doing anything during this transfer. CPU load is at ~40% (I use gzip-1 compression (I know that this may also be bad)), HDD load is at around 30% for all drives.

But when I try to copy a large single file from my server to my PC, bandwidth drops to merely one MB/s...
lALH9ZV.png


iPerf also shows very interesting results.
In this screenshot my PC was running iPerf as a client connecting to my server with one parallel stream.
Bandwidth is all over the place and averages at around 400 Mbit/s.
QOLDCyh.png


But when I bump the stream count to 10, suddenly the whole gigabit connection is getting used:
lxUdB9x.png


So far so bad. But when I turn things around, everything gets worse.
I now set iPerf on my PC in server mode and tell my server to connect to my PC.
Bandwidth is now very bad, averaging at around 14 Mbit/s for one stream.
N0XQ5u3.png


Even worse, my server cannot connect to my PC when using 10 parallel streams. My server console is telling me it is running into timeouts and my PC does not detect an iPerf connection.
With four streams I can get iPerf to connect, but bandwidth does not really get any better.
D6Cmqr6.png


TL;DR:
My PC does get a bandwidth of around 400 Mbit/s when using one stream to connect to the server. With ten streams I get one whole gigabit of throughput.
My Server however can get only around 14 Mbit/s bandwidth when transferring data to my PC, and around 39 Mbit/s when using four streams.
Ten streams run into a timeout.

How can this be?

Has someone ever seen behaviour like this?
How can I fix this?


Thanks in advance.

I will try to answer any questions that are posted.
 
Last edited by a moderator:

Nick2253

Wizard
Joined
Apr 21, 2014
Messages
1,633
Do you have any kind of link aggregation set up? If you do, I would suggest removing it.

Do you have another device on your network that you can test from? That would allow us to rule out anything with your PC as being the problem.

Let's try to rule out the switch by directly connecting the FreeNAS to your computer (you'll need to set static IPs on both interfaces). Then we can try the connection test again.
 

JayanWarden

Dabbler
Joined
Nov 26, 2017
Messages
22
Do you have any kind of link aggregation set up? If you do, I would suggest removing it.
No, I don't.

Do you have another device on your network that you can test from? That would allow us to rule out anything with your PC as being the problem.
I could test with my laptop connected to the switch / server.
Will do once I get home from work (in about 90 minutes)

Let's try to rule out the switch by directly connecting the FreeNAS to your computer (you'll need to set static IPs on both interfaces). Then we can try the connection test again.
Will do once I get home from work (in about 90 minutes)
 

JayanWarden

Dabbler
Joined
Nov 26, 2017
Messages
22
Okay, here are my findings:

Server (iperf -s) ---> Laptop (iperf -c) DIRECTLY connected to server: 942 MBit/s
Server ---> Laptop OVER SWITCH: 900 MBit/s

Server ---> PC DIRECTLY connected: 3.97 MBit/s (!!!) - yes, I double-checked.
Server ---> PC OVER SWITCH: 11.1 MBit/s

PC (iperf -c, no additional parameters. So it's single stream) ---> Server DIRECTLY connected: 250 MBit/s

So I guess there is something wrong with my PC.
I am using Windows 10, I have never touched the advanced options for my NIC. Just like my laptop. And yet they behave completely different.
Any tips on where to look next?

EDIT: I checked, my PC detects the direct server -> PC link as gigabit-capable.
 

Nick2253

Wizard
Joined
Apr 21, 2014
Messages
1,633
I would suggest updating the driver. Use the driver package directly from Intel. (https://downloadcenter.intel.com/product/82186/Intel-Ethernet-Connection-I219-V)

I had an issue with my Windows 10 PC when it stopped waking up on LAN and about 50% of the time the NIC wouldn't wake up when I woke up the computer from sleep/hibernate. Somehow Windows 10 (I'm assuming because of an update) decided that the Intel driver was not OK and removed it. Reinstalling the Intel driver fixed the problem for me.

Obviously, my symptoms were very different from yours, but it's pretty low-hanging fruit to try fixing.
 

JayanWarden

Dabbler
Joined
Nov 26, 2017
Messages
22
I installed the newest drivers, but speed remained the same. Which is to say, crawling.
 

Nick2253

Wizard
Joined
Apr 21, 2014
Messages
1,633
Based on that data, I think it's safe to assume that the problem is somewhere with the PC.

Before we start down some other troubleshooting paths, let's make sure that it's not the driver. I would recommend locating the NIC in device manager, right click, and "Uninstall" the driver. Reboot your machine. It should come up after reboot, detect the NIC, and install the Intel driver. If it does not, then I would "Uninstall" again, and then run the Intel installer.

I wonder if there is a problem with the ethernet port or cable. Can you try it with the cable you used on your laptop (obviously that's a good cable)? Also (though I know it's not something that a bunch of people have lying around), if you have a PCIe NIC, we could use that to rule out the issue.

To rule out the possibility of some weird Windows/FreeNAS incompatibility, can you connect your PC directly to your laptop, set up a share, and see what kind of performance you get?

Can you try copying a file from a USB drive to your hard drive? This would let us rule out any issues with your controller/SATA cable/drive as being the source of our problems (though I think that this is unlikely your issue).
 

rs225

Guru
Joined
Jun 28, 2014
Messages
878
I remember seeing a suggestion to turn off Large Something Offload under the Advanced tab on the NIC driver. Try that, and see if anything changes. If not, turn it back on.
 

JayanWarden

Dabbler
Joined
Nov 26, 2017
Messages
22
To rule out the possibility of some weird Windows/FreeNAS incompatibility, can you connect your PC directly to your laptop, set up a share, and see what kind of performance you get?
I also though about that before I went to sleep.
When I get home (in about 9 hours) I will see what kind of speeds I get when I connect my PC to my laptop. I'll first start with iPerf tests.

Can you try copying a file from a USB drive to your hard drive? This would let us rule out any issues with your controller/SATA cable/drive as being the source of our problems (though I think that this is unlikely your issue).
Local transfer speeds were never the problem.
I use a Samsung 960 EVO as my system disk an all testing was done on that. I get about 2.6 GB/s sequential bandwidth out of that thing.

I remember seeing a suggestion to turn off Large Something Offload under the Advanced tab on the NIC driver. Try that, and see if anything changes. If not, turn it back on.
I also remember some IPv4 and IPv6 checksum offloading options in there... I will have a play with them once I get home.


But I have something to add to this:
It looks like my PC is having problems ->receiving<- data. BUT: My home internet access clocks in at about 130 MBit/s downstream. This to me is also incoming bandwidth. And I get those 130 MBit/s no problem...
 

Nick2253

Wizard
Joined
Apr 21, 2014
Messages
1,633
I'm leaning towards some weird Samba/Windows issue here. When you get home, can you get us a copy of your smb.conf file?
 

JayanWarden

Dabbler
Joined
Nov 26, 2017
Messages
22
I'm leaning towards some weird Samba/Windows issue here. When you get home, can you get us a copy of your smb.conf file?
I don't think that it's a Samba problem, because that wouldn't reflect in these weird iPerf results.
It must be something network related.
 

Nick2253

Wizard
Joined
Apr 21, 2014
Messages
1,633
That's a good point. I forgot you did iPerf for your testing.

Now I'm re-leaning back to Windows NIC config. Perhaps there's something in the driver that's configured all screwy. I'll be honest, this is a weird one.
 

JayanWarden

Dabbler
Joined
Nov 26, 2017
Messages
22
Okay, I ran iPerf between my Laptop and my PC. Here are the results.

Laptop --> PC: 7.4 MBit/s
PC --> Laptop: 310 MBit/s

I then tinkered with some offload options in my NIC on my PC.
I disabled the following things:
Large-Send-Offload
Flowcontrol
IPv4-CRC-Offload
TCP and UDP Checksum Offload

That literally changed nothing. Speed from Laptop --> PC was 7.86 MBit/s

I then changed out the cables.
I had a spare Cat6 Cable lying around, so I swapped the one connecting my PC to the switch (this was the same cable that I tested my laptop <---> server connection with. So it's a known good cable).
Speed from Laptop --> PC was still just 11.7 MBit/s.


So.
I disabled all offloading options in my NIC and swapped the cable for a known good one. Still no resolve.
 

rs225

Guru
Joined
Jun 28, 2014
Messages
878
On Windows, post output of netsh int tcp show global. It might require Admin.
 

JayanWarden

Dabbler
Joined
Nov 26, 2017
Messages
22
On Windows, post output of netsh int tcp show global. It might require Admin.
Here are the results. But I am sorry that they are in german.

Code:
Globale TCP-Parameter
----------------------------------------------
Zustand der empfangsseitigen Skalierung		  : enabled
Chimney-Auslagerungszustand			   : disabled
Autom. Abstimmungsgrad Empfangsfenster	: normal
Add-On "Überlastungssteuerungsanbieter"  : default
ECN-Funktion					  : disabled
RFC 1323-Zeitstempel				 : disabled
RTO (anfänglich)						 : 3000
Zustand der Empfangssegmentzusammenfügung	: enabled
Nicht-SACK-RTT-Resilienz			 : disabled
Maximale SYN-Neuübertragungen			 : 2
Fast Open						   : enabled
Fast Open-Fallback				  : enabled
Geschwindigkeitsprofil					  : off
 
Last edited by a moderator:

JayanWarden

Dabbler
Joined
Nov 26, 2017
Messages
22
I tried reverting the NIC driver to defaults and I also uninstalled all drivers and let windows handle the NIC, but that also didn't help.
 

JayanWarden

Dabbler
Joined
Nov 26, 2017
Messages
22
Here is a screenshot of all my currently enabled tuneables.
Maybe this could be the culprit...

UycwoTp.png
 

Nick2253

Wizard
Joined
Apr 21, 2014
Messages
1,633
Given the performance issues between the laptop and your desktop, I'm not sure that there's any issues with FreeNAS Or rather, I wouldn't start messing with tuneables until we've got it working outside of FreeNAS.

It's really weird that you can download files just fine to your PC, but iPerf doesn't work. Do you, by chance, have a third-party AV or Firewall set up on your PC? It's possible that something is capturing and analyzing all this traffic, which is why it's slowing down. Could you set up WebDAV on FreeNAS, just to see if maybe there's something about HTTP(S) traffic that is working?
 

JayanWarden

Dabbler
Joined
Nov 26, 2017
Messages
22
Given the performance issues between the laptop and your desktop, I'm not sure that there's any issues with FreeNAS Or rather, I wouldn't start messing with tuneables until we've got it working outside of FreeNAS.

It's really weird that you can download files just fine to your PC, but iPerf doesn't work. Do you, by chance, have a third-party AV or Firewall set up on your PC? It's possible that something is capturing and analyzing all this traffic, which is why it's slowing down. Could you set up WebDAV on FreeNAS, just to see if maybe there's something about HTTP(S) traffic that is working?

There you go:

WebDAV Screenshot from Chrome Download:
ShNBtPR.png

I checked for 5 more minutes. The speed stays at ~500 KB/s

Screenshot from same Chrome instance running speedtest.net (which is maxing out my internet connection):
gpDpyFS.png


Shutting down my Antivirus (Avast Free) did literally nothing.
 

Nick2253

Wizard
Joined
Apr 21, 2014
Messages
1,633
So, just to make sure I didn't miss something: you are able to download files from the internet just fine on your desktop? Not just speedtest, but actual download?
 
Status
Not open for further replies.
Top