Duplicating files is slow

[microbug]

Hi all,

I'm experiencing a strange issue with my main pool. Details:

• Pool is 5 * 4TB WD Blue and 1 * 4TB WD Green in RAIDZ1. 5 are attached directly to the motherboard and 1 to a SATA 3 PCIe card.
• 16GB of DDR3 ECC RAM, Xeon E3-1265Lv3
• Gigabit ethernet connection between client and server (only relevant wrt Samba speeds)
• Running FreeNAS-11.0-U4 (54848d13b)

When using the pool via a remote client (SMB), transfers are around 100MB/s which is about expected (given that there are a couple of switches between the server and client, and the network can be busy with streams etc.). Scrubs run with zpool scrub <pool> are running at 500MB/s according to zpool status -v <pool>. During scrubs the disks are reading at around 100MB/s each, as expected.

However, file duplication runs very slowly. Duplicating a folder of large (5-10GB each) video files through cp -av <source> <dest> or rsync -av <source> <dest> runs at around 80MB/s! I know my pool can do better than that. What could the cause be? I've turned off my other server running all services that could be reading/writing from my pool and network activity in FreeNAS's reporting tab is near zero. There is nothing else, as far as I can see, that would be lowering performance. My snapshot policy is one every 10 mins. I don't have any VMs or jails running so the ARC sucks up around 14.5GB of memory once it's been running a while.

Any ideas?

 

[m0nkey_]

There are a couple of sysctl's you could try to speed up file transfers between datasets. You can try the following tunables.

Variable: vfs.zfs.dirty_data_sync
Value: 536870912
Type: Sysctl
Comment: Dirty data in RAM perform sync to disk (512MB)

Variable: vfs.zfs.txg.timeout
Value: 10
Type: Sysctl
Comment: Flush data less often

These two Sysctl's help speed up transfers between datasets increasing the amount of RAM that can be used by dirty data and changing the interval on how often it get's flushed to disk.