Rsync 10x slower than simple FTP

[samfarkus]

I'm using rsync from windows (Delta Copy) to a Freenas 8.0 server.. it's all sweet and stuff but its slow as frick.. about 10x slower than FTP - 100 Mbit on a gigabit connection.

For the initial xfer over, this takes days since i have terabytes of shiznite..

I tried FTP'ing files over, but rsync does not recognize them and sends over replacement copies, so it doesnt solve anything.

 

[louisk]

What are the specs (CPU and Memory) of your FreeNAS? By default, rsync uses ssh as transport, and the encryption can eat up your CPU and slow your connection down. Have you looked at how much CPU ssh is consuming when you're doing a copy? If this is the case, you need to configure rsync on FreeNAS w/o ssh and you should see a big performance increase.

 

[samfarkus]

BIOSTAR TA890FXE mobo
AMD Sempron 2k CPU
2 gigs ram
2x3TB Hitachi
1x1TB WD
norco 8 hot-swap bay case
zippy 400w PSU

i checked load usage, its only 20% at most
i think rsync is just slow and kinda sucks at big xfers
I'm using UFS not ZFS (cuz of the low RAM)
if more RAM would really help i'd get the RAM, but dont see why it would in this case. When hard drive prices go back to normal i plan on getting 3 more drives and making a zpool5 .. not cuz i think i need it but cuz ZFS is epeen + 1 inch and i could use the length

 

[Milhouse]

Running rsync between two HP N36L servers (one FreeNAS 0.7.2, other FreeNAS 8.0.2, both 8GB, RAID-Z1 and Intel CT NICs) and I've seen consistent 70MB/s transfers.

 

[samfarkus]

[ ID] Interval Transfer Bandwidth
[ 4] 0.0-10.1 sec 540 MBytes 451 Mbits/sec


this is iperf showing that it is sa gigabit link (running while rsync running in the background at 100 megabit speed)

 

[samfarkus]

i turned on --verbose --progress and it's running around 19 MB/s
20% network utilization :(

Deltacopy on windows to freenas. Anyone ever get better rates from a windows box?
i checked resource monitor/top ample resources. I think it's gotta be a problem with deltacopy itself and the cygwin libs

 

[marcusmarcus]

I'm having slow rsync as well. Getting around 13-14MBps transfers compared to if I manually copy and past stuff via CIFS where I get around 60MBps. I thought it might be because I was using rsync to transfer my pictures folder with small files but I tried my video folder as well with large files and it gets the same slow speed. Did you or anyone else figure out a fix for this yet?

 

[GioF_71]

I'm having slow rsync as well. Getting around 13-14MBps transfers compared to if I manually copy and past stuff via CIFS where I get around 60MBps. I thought it might be because I was using rsync to transfer my pictures folder with small files but I tried my video folder as well with large files and it gets the same slow speed. Did you or anyone else figure out a fix for this yet?

I used to use deltacopy. But it has glitches, for example it does not support unicode characters in file names unless you manually replace one of the cygwin dlls with an updated version.
Now I use Microsoft SyncToy and synchronize against a Samba shared folder on the NAS. It also supports two-way synchronization which is very useful for example, if you modify your data on your desktop or you modify data directly on the server through another system (say a laptop).

I schedule synchronization tasks with windows task scheduler.

Cheers

 

[Rogla]

I like to use wget and ftp to sync shares instead of rsync. wget can use ftp as input (import files to FreeNAS), example:

wget --tries=3 --retry-connrefused --timeout=5 --continue --inet4-only --ftp-user=User --ftp-password=Pswd --no-host-directories --recursive --level=0 --no-passive-ftp --no-glob ftp://192.168.0.76/share

 

[William Grzybowski]

Rsync over SSH or Rsyncd?

Note that Rsync over SSH is supposed to be slow because it is CPU bound to encryption, while rsyncd is not...

 

[aruffell]

I am having the same exact issue but I do not think it is related to my hardware. If I use SyncToy or simply copy the files over myself I believe I get up to 80MS/s but with rsync it is so slow it would not saturate a 100Mbit connection.

Some info on the NAS:

FreeNAS-8.2.0-BETA3-x64 (r11188)
AMD Phenom(tm) II X6 1090T Processor
16GB DDR3 RAM
Gygabyte GA-890FXA-UD5 Motherboard
FreeNAS running on dedicated 32GB SSD
10TB total harddrives divided in 4 Zpools
Dual Gigabit ethernet connections to managed switch (LAGG / LACP)

After lots of work to get the above system to stop crashing when moving large amounts of data (and I am still not sure what fixed the issue) I am now trying to speed up rsync otherwise it takes a week to sync up all the data from my main computer. The files vary in size however the most common are 20-35MB each up to Gigabytes each.

My PC is a Windows 7 64bit machine running with an 8 core FX-8150, 16GB Ram and so on. Delta Copy is the rsync client I use because I could not find anything better. Suggestions welcomed!!

Can anyone point me in the right direction?

 

[GioF_71]

I abandoned rsync (also its Deltacopy incarnation) on windows because it is VERY slow, and sometimes it even hangs.

In a mixed environment, if you choose to use synctoy on Windows, please be sure to exclude *SyncToy* files from other rsync tasks (on linux or bsd machines), otherwise automated SyncToyCmd task won't work, because they could not be able to find a hidden file (some *SyncToy* file).

My PC is a Windows 7 64bit machine running with an 8 core FX-8150, 16GB Ram and so on. Delta Copy is the rsync client I use because I could not find anything better. Suggestions welcomed!!

Can anyone point me in the right direction?

 

[aruffell]

Main thing that I do not like / trust in SyncToy is that it has not been updated in years. I wonder if it will go away any moment or whether support will be dropped with the next release of Windows. I do use it in the meantime but I don't want to come to rely on it unless I know it will evolve and be supported in the foreseeable future.

 

[GioF_71]

Main thing that I do not like / trust in SyncToy is that it has not been updated in years. I wonder if it will go away any moment or whether support will be dropped with the next release of Windows. I do use it in the meantime but I don't want to come to rely on it unless I know it will evolve and be supported in the foreseeable future.

Ok, consider any product you like (PureSync may be a good product) to sync your windows pc(s) against Samba shares. Rsync is good to sync bsd/linux machines, just be careful not to delete the hidden files/folders created by the windows application when you sync one bsd/linux machine to another one if the same data is synchronized with a windows machine.

 

[aruffell]

Ok, consider any product you like (PureSync may be a good product) to sync your windows pc(s) against Samba shares. Rsync is good to sync bsd/linux machines, just be careful not to delete the hidden files/folders created by the windows application when you sync one bsd/linux machine to another one if the same data is synchronized with a windows machine.

Great, thank you for the suggestion. I am going to try it right now!

 

[pete_c20]

I had the same fault as above. ie a windows XP rsync client (QTDSync 0.6.18beta) that would not exceed 13-14MB/s on large files. After trying numerous things like bypassing the switch, checking / swapping leads, updating NIC drivers the problem remained. The hardware reported that it was all linked at gigabit speed, but I experimented anyway. Changing the rsync client to DeltaCopy made the transfer speed slightly slower. Verification that the network could actually perform at gigabit speed came when doing a straight filecopy using explorer and a CIFS share, it was at least 5 times the speed.
Using FTP & Filezilla and configuring the FTP service on the freeNAS box was MUCH faster. rsync may still have a use once the main file copy has been done by other means.
Whether or not the speed issue was due to both ends of the rsync link I never had the time to work out.