Slow 10Ge ZFS send/receive

[Michael Eager]

I installed a Chelsio T420-BT 10Gbe in my new FreeNAS Mini and it is connected point-to-point to a similar card in my soon-t-be retired ZFS on Linux server. MTU on both cards is set to 9000. When I initially ran iperf, the link was transferring something over 900 MB/s. A file transfer over SSH ran at about 25 MB/s over either the 1Gbe or 10Gbe connection.

When I initially starting migrating files using zfs send/recv, nc (to avoid SSH overhead), and mbuffer, I was seeing transfers around 200 MB/s, which was pretty reasonable. After migrating several small file systems, I started migration of a 350+ Gb file system. Speed was significantly slower, maybe 20 MB/s, and became slower as the transfer continued, down to less that 5 MB/s. Now even transferring smaller filesystems runs very slow at 3 MB/s.

Anyone have any ideas what might be going on and how to correct it?

 

[Ericloewe]

If your data is very fragmented, it's always going to be slow. What can you tell us about your pools?

 

[Michael Eager]

Fragmentation seems to be 10%.

Code:

# zpool list
NAME	  SIZE  ALLOC   FREE  EXPANDSZ   FRAG	CAP  DEDUP  HEALTH  ALTROOT
zfspool  10.9T  3.03T  7.85T		 -	10%	27%  1.00x  ONLINE

-

 

[Ericloewe]

The fragmentation number doesn't actually provide much information. It's just a measure of the fragmentation of free space, not of data. But that pool looks rather empty, so you might want to check that any parameters you're using in your pipeline make sense for your network. Some of those utilities can have rather broken defaults.

 

[Michael Eager]

Here is the transfer scheme I use:
On the FreeNAS box:

Code:

 # nc -l 8000 | mbuffer -q -s 128k -m 1G | pv -rtab | zfs receive -vF zfspool/data

On the old server:

Code:

# zfs send zfspool/data | mbuffer -q -s 128K -m 1G | pv -b | nc freenas10G 8000

The sending system has 32Gb RAM, 13Gb free. The receiving system has 16Gb RAM, lots free.

Nothing looks off to me.

 

[Ericloewe]

The 128k block size sounds dubious to me. Try 256 MB.