Rsync errors

[Zaaphod]

I recently added some storage to my FreeNAS system. I wanted to transfer a large directory to the new dataset and I thought I would try rsync. I have never use rsync before and I'm not sure if I'm missing something.

I'm using PuTTY, I loged in to FreeNas with a user that has all permissions and used the command:

Code:

rsync -vah /mnt/Old /mnt/New

It ran for a while listing my directories and folders, but after a while, it just stopped. I waited and waited, and finally decided it had way more than enough time to advance to another file, so I hit CTRL-C
as soon as I did that, I got a whole list of errors like:

Code:

rsync error: received SIGINT, SIGTERM, or SIGHUP (code 20) at rsync.c(632) [sender=3.1.1]
rsync: mkstemp "/mnt/New/Filename.avi.uT731z" failed: Operation not permitted (1)
rsync: mkstemp "/mnt/New/Filename2.avi.BubdoR" failed: Operation not permitted (1)
...
rsync error: received SIGINT, SIGTERM, or SIGHUP (code 20) at io.c(504) [generator=3.1.1]

with one rsync: mkstemp listing for each file with the same failed message. Also when I looked at the new location, all the directories were created, however they were ALL empty. Apparently I was getting the errors and files were not being copied, but I couldn't see the errors until I canceled it.

I checked the permissions with ls -l as follows:

Code:

User@freenas:/mnt/New % ls -l
total 29
drwxrwxr-x+ 2 User  Documents  3 Jun  1 15:05 ./
drwxr-xr-x  4 root  wheel  120 Jun  1 11:57 ../
-rwxrwxr-x+ 1 root  User  0 May 31 17:10 .windows*

So I tried to copy the files with the command

Code:

cp -Rp /mnt/Old /mnt/New

and this worked, but I did get 5 messages like

Code:

cp: chmod: /mnt/New/Folder: Operation not permitted

but only 5 of these, out of hundreds of directiores, and the files within seem to have copied ok, however I did notice that the timestamp on the directories with errors were todays date, not the original directory date as all the other directories were. I'm currently running

Code:

diff -rq /mnt/Old/ /mnt/New/

to make sure all the copies are identical

I would like to figure out what is causing these errors. I had planned on using rsync to synchronize directories to removable storage for use with my laptop when I am out of town.. so rsync to the removable drive before I leave, then rsync back when I return. I would like to use something like Rsync because I do not want any of my client computers to be bothered with this task, I want it to be done on the server because it's running anyway. I'm not sure what I'm doing wrong, if I have some permission issue, or if I'm not formatting the command correctly? any advice is greatly appreciated!

 

[Spearfoot]

There are problems with the rsync '-a' option. See this post for details:

https://forums.freenas.org/index.ph...uring-transfer-9-10-stable.43764/#post-290544

 

[anodos]

You can also view the bug report here: https://bugs.freenas.org/issues/14434

When Windows datasets were changed from aclmode=passthrough to aclmode=restricted you had to add --no-p to rsync.

 

[Zaaphod]

Thank you for the information and the links on this.

So if I am understanding this correctly, it's something to do with the p option that is included when I select a for archive. so to correct this I instruct it to leave off the p for permissions with --no-p like this:

Code:

 rsync -vah --no-p /mnt/Old /mnt/New

with the --no-p, it won't copy permissions, then it would use the permissions of the target directory. Am I understanding this correctly? The permissions being copied aren't really important to me, as long as the files are accessible in the new location as expected. Is there some other utility I should be using for this purpose, or just use rsync with the --no-p?

Also, I was unable to see the errors until I did a ctrl-c to break out of the rsync.. then I got all of them at once. is there a way to see these errors as it is processing? or would I need to redirect output to log files to be able to see these during processing?

 

[Spearfoot]

Ummm... in the link I referenced, I pointed out that '-a' is equivalent to '-rlptgoD' and suggested using the latter options explicitly, leaving off the 'p'. I haven't tried the --no-p option, and didn't know it existed -- there are a bewildering array of rsync options.

The bug pointed out by @anodos has to do with the -A (Windows ACL) option, which is a different option, though probably related as ACL and permissions are inexplicably intertwined in an uncannily unknowable way.

Also, the trailing slash is critical with rsync, else you'll end up with scads of files in unwanted places and other unexpected behavior.

Rsync really is a beast!

You should be able to see errors as they occur, though there will be long pauses as rsync performs calculations, copies data, etc.

I suggest trying something like this, and tweaking as necessary:

Code:

rsync -rltgoDv /mnt/Old/ /mnt/New

 

[anodos]

Ummm... in the link I referenced, I pointed out that '-a' is equivalent to '-rlptgoD' and suggested using the latter options explicitly, leaving off the 'p'. I haven't tried the --no-p option, and didn't know it existed -- there are a bewildering array of rsync options.

The bug pointed out by @anodos has to do with the -A (Windows ACL) option, which is a different option, though probably related as ACL and permissions are inexplicably intertwined in an uncannily unknowable way.

Also, the trailing slash is critical with rsync, else you'll end up with scads of files in unwanted places and other unexpected behavior.

Rsync really is a beast!

You should be able to see errors as they occur, though there will be long pauses as rsync performs calculations, copies data, etc.

I suggest trying something like this, and tweaking as necessary:

Code:

rsync -rltgoDv /mnt/Old/ /mnt/New

Both -a and -A imply -p. Changes to mode bits are prevented on windows datasets. Hence the need to set --no-p to prevent rsync from failing.

 

[Zaaphod]

Thanks again for the info. Should I also use the trailing / after the destination directory as well or does this cause problems?

Is there a way to have rsync write to log files similar to robocopy, or do I just redirect to log files myself if I want them?

 

[anodos]

--log-file=/mnt/Tank/foo.txt

 

[Spearfoot]

Thanks again for the info. Should I also use the trailing / after the destination directory as well or does this cause problems?

I use a trailing slash after the source dataset, and no trailing slash after the target dataset. This makes rsync behave the way I want & expect. YMMV.

Give it a try by writing a script and testing it with some small test datasets to get the hang of how it works. Once you're comfortable with it and have it working the way you want, you'll be ready to move mountains of data...

EDIT: Here's a snippet from one of my local admin scripts, known to work on FreeNAS 9.10 - note the trailing slash on the source dataset and missing slash on the target:

Code:

rsync -rltgoDhv --delete-during --inplace /mnt/tank/sysadmin/ /mnt/tank/systems/felix/boomer/sysadmin

And here is a good rsync reference: https://www.samba.org/ftp/rsync/rsync.html

[ @Zaaphod, whenever I see your name, something makes me think: "Beeblebrox" :) ]

 

[Robert Trevellyan]

testing it with some small test datasets to get the hang of how it works

Right, you have to read and understand the documentation regarding trailing slashes, then confirm your understanding by trying it both ways on source and destination. Use the '-n' option for dry-runs, every time, until you're sure.