How to improve Import disk [rsync] throughput?

[ilmarmors]

Build: FreeNAS-9.3-STABLE-201412240734
Platform: Intel(R) Atom(TM) CPU C2750 @ 2.40GHz
Memory: 16331MB
Motherboard: Asrock C2750D4I

I built NAS server for home and started to import data from old NTFS formatted disks that were laying around. I'm using Storage -> "Import Disk" in the FreeNAS GUI. Behind the scenes - FreeNAS mounts partition, runs rsync -aH --info=progress2 --itemize-changes to copy files to specified targed dir in the ZFS pool.

I'm importing from old disks like WD5000AAKS, WD3200JS, WD10EACS having NTFS partitions with mostly big files, shouldn't be fragmented (backup disks, copied once and turned off). Although I don't expect top throughput, but it looks like throughput always hit the same ceiling.

At the beginning I connected external disk via USB 2.0. Import Disk throughput was around 20MB/s max.

Then I connected external disk directly to Marvel 9172 port (as I understand 4 Marvel 9230 ports sometimes give problems). No kernel panics or strange errors.

Import Disk / rsync speed increased to around 49MB/s for SATA connected disk.

To determine raw speed of disk I run dd if=/dev/ada1 of=/dev/null bs=2048k
Then iostat shows ~80MB/s
KB/t tps MB/s
128.00 646 80.77

Then I wanted to find out NTFS via FUSE overhead:
#mount -t ntfs -o ro /dev/ada1s1 /mnt/usbext/
#find /mnt/usbext/ -type f -print -exec sh -c 'dd if="$1" of=/dev/null bs=1024k' {} {} \;

Output of iostat shows ~64MB/s on average
KB/t tps MB/s
128.00 514 64.31

CPU stats (from FreeNAS GUI) during Import Disk, no core was overloaded (checked with command-line top):
Idle: 79.55%
User: 8.68%
System: 11.48%
IRQ: 0.26%

Top shows 3 processes that creates any significant load on CPUs:
PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND
4234 root 1 52 0 66224K 7532K fu_ans 4 10:41 39.36% rsync
4233 root 1 40 0 38424K 19808K fu_msg 0 8:14 30.18% ntfs-3g
4243 root 1 35 0 66224K 3904K select 6 6:11 23.19% rsync

Any ideas if and how "Import Disk" throughput can be increased? Tuning fuse / ntfs-3g mount options, tuning rsync, tuning some params via sysctl? Anything else? I noticed that for reads it was constant 128KB per read issued against the disk(?). Can it be somehow improved by somehow using bigger read size or read ahead, which would be very helpful for sequential reads? Googling shows some hardcode limit of 128KB max read side in FUSE, but those pages was few years old.

 

[ilmarmors]

Out of curiosity tested the same hardware and the same disk with SystemRescueCd 4.1.0.

Raw disk read with dd - ~85MB/s vs ~80MB/s on FreeNAS
Mounted NTFS partition read with find files and dd - ~72-74MB/s vs ~64MB/s on FreeNAS (big non-fragmented files)

 

[cyberjock]

rsync is a single-threaded type of function. That being said, it is very slow on large quantities of data and provides no significant benefit when doing local copies (especially so if doing ZFS to ZFS).

I can't tell from your info what the limitation is, but to be honest I'd just copy the data with the cp command and be done with it. :p