Slow 10Ge ZFS send/receive

Status
Not open for further replies.
Joined
May 14, 2018
Messages
6
I installed a Chelsio T420-BT 10Gbe in my new FreeNAS Mini and it is connected point-to-point to a similar card in my soon-t-be retired ZFS on Linux server. MTU on both cards is set to 9000. When I initially ran iperf, the link was transferring something over 900 MB/s. A file transfer over SSH ran at about 25 MB/s over either the 1Gbe or 10Gbe connection.

When I initially starting migrating files using zfs send/recv, nc (to avoid SSH overhead), and mbuffer, I was seeing transfers around 200 MB/s, which was pretty reasonable. After migrating several small file systems, I started migration of a 350+ Gb file system. Speed was significantly slower, maybe 20 MB/s, and became slower as the transfer continued, down to less that 5 MB/s. Now even transferring smaller filesystems runs very slow at 3 MB/s.

Anyone have any ideas what might be going on and how to correct it?
 

Ericloewe

Server Wrangler
Moderator
Joined
Feb 15, 2014
Messages
20,175
If your data is very fragmented, it's always going to be slow. What can you tell us about your pools?
 
Joined
May 14, 2018
Messages
6
Fragmentation seems to be 10%.

Code:
# zpool list
NAME	  SIZE  ALLOC   FREE  EXPANDSZ   FRAG	CAP  DEDUP  HEALTH  ALTROOT
zfspool  10.9T  3.03T  7.85T		 -	10%	27%  1.00x  ONLINE

-
 

Ericloewe

Server Wrangler
Moderator
Joined
Feb 15, 2014
Messages
20,175
The fragmentation number doesn't actually provide much information. It's just a measure of the fragmentation of free space, not of data. But that pool looks rather empty, so you might want to check that any parameters you're using in your pipeline make sense for your network. Some of those utilities can have rather broken defaults.
 
Joined
May 14, 2018
Messages
6
Here is the transfer scheme I use:
On the FreeNAS box:
Code:
 # nc -l 8000 | mbuffer -q -s 128k -m 1G | pv -rtab | zfs receive -vF zfspool/data

On the old server:
Code:
# zfs send zfspool/data | mbuffer -q -s 128K -m 1G | pv -b | nc freenas10G 8000

The sending system has 32Gb RAM, 13Gb free. The receiving system has 16Gb RAM, lots free.

Nothing looks off to me.
 

Ericloewe

Server Wrangler
Moderator
Joined
Feb 15, 2014
Messages
20,175
The 128k block size sounds dubious to me. Try 256 MB.
 
Status
Not open for further replies.
Top