Register for the iXsystems Community to get an ad-free experience and exclusive discounts in our eBay Store.

Slow 10Ge ZFS send/receive

Western Digital Drives - The Preferred Drives of FreeNAS and TrueNAS CORE
Status
Not open for further replies.

Michael Eager

Neophyte
Joined
May 14, 2018
Messages
6
I installed a Chelsio T420-BT 10Gbe in my new FreeNAS Mini and it is connected point-to-point to a similar card in my soon-t-be retired ZFS on Linux server. MTU on both cards is set to 9000. When I initially ran iperf, the link was transferring something over 900 MB/s. A file transfer over SSH ran at about 25 MB/s over either the 1Gbe or 10Gbe connection.

When I initially starting migrating files using zfs send/recv, nc (to avoid SSH overhead), and mbuffer, I was seeing transfers around 200 MB/s, which was pretty reasonable. After migrating several small file systems, I started migration of a 350+ Gb file system. Speed was significantly slower, maybe 20 MB/s, and became slower as the transfer continued, down to less that 5 MB/s. Now even transferring smaller filesystems runs very slow at 3 MB/s.

Anyone have any ideas what might be going on and how to correct it?
 

Ericloewe

Not-very-passive-but-aggressive
Moderator
Joined
Feb 15, 2014
Messages
17,019
If your data is very fragmented, it's always going to be slow. What can you tell us about your pools?
 

Michael Eager

Neophyte
Joined
May 14, 2018
Messages
6
Fragmentation seems to be 10%.

Code:
# zpool list
NAME	  SIZE  ALLOC   FREE  EXPANDSZ   FRAG	CAP  DEDUP  HEALTH  ALTROOT
zfspool  10.9T  3.03T  7.85T		 -	10%	27%  1.00x  ONLINE

-
 

Ericloewe

Not-very-passive-but-aggressive
Moderator
Joined
Feb 15, 2014
Messages
17,019
The fragmentation number doesn't actually provide much information. It's just a measure of the fragmentation of free space, not of data. But that pool looks rather empty, so you might want to check that any parameters you're using in your pipeline make sense for your network. Some of those utilities can have rather broken defaults.
 

Michael Eager

Neophyte
Joined
May 14, 2018
Messages
6
Here is the transfer scheme I use:
On the FreeNAS box:
Code:
 # nc -l 8000 | mbuffer -q -s 128k -m 1G | pv -rtab | zfs receive -vF zfspool/data

On the old server:
Code:
# zfs send zfspool/data | mbuffer -q -s 128K -m 1G | pv -b | nc freenas10G 8000

The sending system has 32Gb RAM, 13Gb free. The receiving system has 16Gb RAM, lots free.

Nothing looks off to me.
 

Ericloewe

Not-very-passive-but-aggressive
Moderator
Joined
Feb 15, 2014
Messages
17,019
The 128k block size sounds dubious to me. Try 256 MB.
 
Status
Not open for further replies.
Top