Windows explorer cause slow transfer

Joined
Jan 18, 2017
Messages
524
TLDR: has anyone ever seen windows explorer slow a transfer because it was trying to update the directory listing (or something similar) after every file transferred?

Today I was moving 21000+ small files for a total of 14gig, half way through the transfer it slowed to below 1MB/s after checking gstat and top (smbd was over 100% again) nothing looked unusual until I got to the ZFS tab. ARC request went up to nearly 7k at the same moment the transfer speed dropped off, assuming it was just another user I left it alone for a while about half an hour later I came back and it was still moving along extremely slowly. Spent another hour going over everything I could think of and came up empty hand, finally gave up and resolved myself to the 9 hours remaining on the transfer which wasn't slowing anything else down and closed the windows explorer window. The moment it closed the transfer jumped to 50MB/s and continued to climb, the zfs graphs also showed an instant drop in ARC_requests, demand and SMBD usage. Windows explorer had the navigation pane up with the directory tree on my FreeNAS showing the destination while the main pane had my local drives displayed, Has anyone seen this happen before?
 
Joined
Apr 9, 2015
Messages
1,258
Never dealt with it myself but a directory with that many files will be a pain anyway since there are so many requests. I do know that windows will attempt to update directory contents every so often automatically and with that much being added it will cause issues as you have already found out.
 
Joined
Jan 18, 2017
Messages
524
I have couple folders which exceed 20k files I'll be upgrading the cpu's a touch from E5645's to X5690's hopefully this will help alleviate the issue but it is exceedingly rare that I move such folders to the server.
 

Mirfster

Doesn't know what he's talking about
Joined
Oct 2, 2015
Messages
3,215
FWIW, I am a fan of using the built-in RoboCopy since that is faster than a normal copy/paste also it can leverage multi-threading.

As far as you copying a lot of small files, that in itself will always be slower (IMHO) compared to copying a ZIP of the files. It mainly has to do with the system keeping track of each transfer/copy.
 
Joined
Jan 18, 2017
Messages
524
I did not even know RoBocopy was included in windows..... I obviously expected to have a slower than normal transfer when doing LOTS of little files compared to an archive of the same size but to have the transfer stall nearly knocked me out of my chair lol
 
Joined
Apr 9, 2015
Messages
1,258

Ericloewe

Server Wrangler
Moderator
Joined
Feb 15, 2014
Messages
20,175
FWIW, I am a fan of using the built-in RoboCopy since that is faster than a normal copy/paste also it can leverage multi-threading.

As far as you copying a lot of small files, that in itself will always be slower (IMHO) compared to copying a ZIP of the files. It mainly has to do with the system keeping track of each transfer/copy.
On Windows 7, it also supports server-side copy, which was only introduced in Windows Explorer in 8.
 
Joined
Jan 18, 2017
Messages
524
FWIW, I am a fan of using the built-in RoboCopy since that is faster than a normal copy/paste also it can leverage multi-threading.

As far as you copying a lot of small files, that in itself will always be slower (IMHO) compared to copying a ZIP of the files. It mainly has to do with the system keeping track of each transfer/copy.

Just had to do another 22k little files that totaled a little over 11 gigs RoBoCopy killed it in a little over 5 minutes and kept SMB under 30 percent usage. I've attached a screen shot of the transfer I did last week of 17K for just over 9 gigs with windows explorer and it took something like 13 hours and had SMB pegged the whole time. Thanks for the heads up on RoBoCopy definitely a better way to go in this situation.
 

Attachments

  • FN17kcopy.png
    FN17kcopy.png
    346.5 KB · Views: 985
Joined
Jan 18, 2017
Messages
524
Finally found a fix to this annoying slow down which did not only affect network folders but local ones as well. This fix improved load times and transfer speeds.
 

Attachments

  • Network folders slow after Windows 10 1803 update.pdf
    164.9 KB · Views: 1,818
Top