Very slow rsync dry-run on mounted sambashare.

[tsoj]

I am trying to set up an incremental update from a Linux machine to a samba share on a FreeNAS machine (8GB RAM, ASRock J4105B-ITX, 2x WD WD80EFAX Red 8TB (Mirror pool)), Network connection seems good:

Code:

$ iperf -c freenas
Client connecting to freenas, TCP port 5001
TCP window size:  102 KByte (default)
------------------------------------------------------------
[  3] local 192.168.2.106 port 33680 connected with 192.168.2.109 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec   160 MBytes   134 Mbits/sec

I mounted the share like this:

Code:

sudo mount -t cifs //freenas/backup_share /root/backup_mnt -o username=myuser,password=1234,iocharset=utf8

And just copying files to this mounted share is pretty fast as well, if the files are not too small (when many small files the write speed is maybe 500kB/s, if the files are big enough it's around 20MB/s, probably only limited by the clients wifi card.

However this rsync command takes more than 1h for ~100GB and ~400,000 files (path/to/last_backup, is that big, it is the last backup, at the time of this command there were basically no changes):

Code:

rsync --dry-run -rtDhPvi --modify-window=12000 --size-only --delete --link-dest=path/to/last_backup/ from/ to

I suspect that the issue could be slow directory loading (or however it is called), as this command takes 40+ seconds (even when executed from the FreeNAS machine directly):

Code:

time du -sh path/to/last_backup/

while it only takes 2 seconds on my client machine, on the directory of which I made the backup to path/to/last_backup/

I found this thread: https://www.ixsystems.com/community/threads/cifs-directory-browsing-slow-try-this.27751/
And I tried to disable DOS attributes but it didn't help.

Is this expected behavior of a samba share, and are there any solutions to this?

 

[tsoj]

Okay, I have found the following workaround: I added the paramter "vers=2.1" to the mount command like this:

Code:

 sudo mount -t cifs //freenas/backup_share /root/backup_mnt -o vers=2.1,username=myuser,password=1234,iocharset=utf8 

This helped to bring the time for the rsync command down to less than 20 min.
I also deleted a folder that I don't really need to back up with ~200,000 files which led to rsync finishing in 10 minutes, which is not what I imagined but acceptable.

 

[anodos]

Newer Linux clients may have better metadata performance due to more efficient use of SMB compounding.

 

[tsoj]

I am using a 5.4 Linux kernel for the client, is that considered "new" or could there be a performance increase with 5.8 for example?

 

[tsoj]

My final solution is to use the FreeNAS snaphot feature instead of manually creating incremental backups with rsync and --link-dest, as it seems like operations like "cp -al", which is probably similar to what rsync does with --link-dest, are really slow when done on a mounted samba share (when done fully server side (in a ssh shell for example) it is pretty fast but I haven't found a way to incorporate it into a bash script on the client).

just rsync without --link-dest is fast enough, I also enabled the "autotune"-script (I have 8G of RAM with 2x8TB HDD in mirror) which helped quite a bit in this scenario (didn't help enough for the rsync with --link-dest to become practical).