Transfers from Freenas server slow down after a few seconds, can't figure out why

[Nephalem]

Hi Everyone,

First post here. I've been playing with my new Freenas server for a while, tweaking it, and so far, performance is terrific (I kinda overkilled it for my needs but that doesn't matter). I have started encountering a VERY annoying problem which I can't solve after a LOT of troubleshooting. I work in IT for a living, and most of my work consists of solving problems, so this is seriously annoying me.

I'll be very transparent from the beginning, I am most certain the issue is not with the Freenas box. I'll explain why later, but first, I'll just post specs about my PC so you get context:

Intel i7 6700K CPU
16 GB of RAM
Asus Z170-A Mobo
Integrated Intel i219-V NIC, but also recently bought a standalone intel i210-T1 NIC because I thought the issue might be there.
Local drives are: 1 x Samsung 950 PRO SSD (NVMe) and 1 x OCZ Vertex 3 SSD

SO, here's the issue... When I copy a large file from the NAS, the transfer rate starts around 115 MB/s, which is very close to the theoretical max of a Gigabit NIC. Then after maybe 5-6 seconds, it all goes to hell... Basically to around 40-50 MB/s.

So first assumption would be, bah, the buffer is filled, then you're throttled by the disk write speed, but if you know anything about the Samsung 950 PRO, you know it's not the issue. I get 952 MB/s sequential write and 2129 MB/s sequential read, with 80k iops write and 265k iops read. There is absolutely no bottleneck there. I get the exact same speeds with the Vertex 3 also, which is capable of writing faster than 100 MB/s for sure. So that's out of the way...

Now, I got plenty of CPU, I got more than enough RAM, and disks are way faster than what the gigabit ethernet can serve me... Now the interesting part is twofold:

FIRST: If I copy a large file to the NAS, I get top speed all the way through, 115 MB/s for as long as it takes for a 50 GB file, no speed drop whatsoever. It's ONLY when I copy FROM the nas that it really sucks. I have tried also from my QNAP NAS, just in case, and get the exact same figures...

SECOND: If I copy a large file FROM the nas to my Home Theater PC (AMD CPU), there I can get 115 MB/s all the way through with the exact same large files. That complicates things, but also confirms that the NAS is way capable of serving 1 Gigabit worth of speed.

I had the issue with the on-board Intel i219 NIC, so I bought an Intel i210 NIC (which I'm using in the NAS). I got the exact same performance. 40-50 MB/s when I copy from the nas, 115 MB/s sustained when I copy TO the nas.

I am really at a loss here. I've disabled large send offload, disabled autotuning, disabled IPv6, I simply have nowhere else to look.

Any help you can provide to fix that issue, you'll be my hero! ;)

Thanks in advance!!

 

[Heracles]

Hey Nephalem,

That really looks like the workstation on which you copy the file is the bottleneck you are looking for. That one is the one with not enough caching memory to speed up much.

You should try to copy from disk to RAM. You can always copy to /dev/null for a test as big as you wish (if you transfer to a Unix system...). In all cases, know that objective, neutral performance test is very difficult to achieve.

Good luck and have fun playing with your system,

 

[Nephalem]

Hey Nephalem,

That really looks like the workstation on which you copy the file is the bottleneck you are looking for. That one is the one with not enough caching memory to speed up much.

You should try to copy from disk to RAM. You can always copy to /dev/null for a test as big as you wish (if you transfer to a Unix system...). In all cases, know that objective, neutral performance test is very difficult to achieve.

Good luck and have fun playing with your system,

Hi Heracles,

I finally found my problem. Worst thing is, I already had tried the fix, so I can't explain why it worked the second time around...

Problem was Windows Defender realtime protection. Turned it off again, and I'm getting steady full gigabit speed.

Oh well! ;)