Impaired rsync permissions support for Windows datasets

Spearfoot

Spearfoot

He of the long foot
Moderator
Joined
May 13, 2015
Messages
2,478
Rsync fails when copying Windows datasets with the '-p' ('Preserve Permissions') option. And there are no plans to implement this option in the future, because "rsync targets primarily unix-like systems." Frankly, I don't blame the rsync team one bit for avoiding the Windows nightmare. :smile:

By implication, the '-p' option not working means the '-a' (Archive) and '-A' ('Preserve Extended Attributes') options are broken, too. From the rsync guide, here are the three options to be avoided:

Code:
-a, --archive   equals -rlptgoD (no -H, -A -X)
-p, --perms     Preserve permssions
-A, --acls     Preserve ACLs (implies -p)


This doesn't mean you can't use rsync! It just means you have to avoid using these particular options for Windows datasets. And, of course, live with any resulting consequences.

In shell scripts I use the fully enumerated' -a' option set, leaving off the 'p', like this: -rltgoD

For Rsync Tasks in the GUI, uncheck the 'Archive:', 'Preserve permissions:', and 'Preserve extended attributes:' selections:

rsync-task-settings.jpg

I've been using both shell scripts and Rsync Tasks configured this way for nearly a year with good results, so it's not like rsync is unusable.

Details available here: https://redmine.ixsystems.com/issues/7713
 
Last edited:
Spearfoot

Spearfoot

He of the long foot
Moderator
Joined
May 13, 2015
Messages
2,478
Here is an rsync 'wrapper' script I wrote that avoids the problems discussed above. I use it to copy datasets between my three NAS systems (2 x FreeNAS and 1 x Synology Diskstation). You can download it from Pastbin here if you want to use it.

Code:
#!/bin/bash
#######################################################################
# A wrapper for invoking rsync to copy a source dataset to a destination
# dataset with options that are 'known to work' with FreeNAS/FreeBSD
# and Windows datasets. This means we have to avoid the '-p' (preserve
# permissions) option in all of its forms, per bug report 7713:
#
# https://bugs.freenas.org/issues/7713
#
# Command-line parameters:
#  1: Source dataset  R_SRC  (local system)  /mnt/tank/sharename/
#  2: Destination dataset  R_DEST  (local system)  /volume1/sharename
#  3: Log file  R_LOGFILE (local system)  /mnt/tank/sysadmin/log/push-log.log
#
# R_SRC or R_DEST may include user ID and hostname specifier
#
# Example usage, where 'boomer' is a FreeNAS server and 'bertrand' is
# a Synology Diskstation. Note that you should include a trailing slash
# only on the source dataset specifier:
#
# rsync-invoke.sh root@bertrand:/volume1/devtools/ /mnt/tank/devtools /mnt/tank/sysadmin/log/pull-from-bertrand.log
# rsync-invoke.sh /mnt/tank/devtools/ root@bertrand:/volume1/devtools /mnt/tank/sysadmin/log/push-to-bertrand.log
#
# Assumes SSH has been enabled and configured between the source
# and destination hosts.
#
# Invokes rsync with these SSH options to optimize transfer speed:
#
#  "-e ssh -T -c arcfour -o Compression=no -x"
#
# You may need to remove this altogether, or modify the encryption
# specifier (-c arcfour) to use a scheme available on your system.
#
# !!! WARNING !!!
# This script deletes files on the destination that don't exist
# on the source! Edit R_OPTIONS below and remove '--delete-during'
# and '--inplace' if you don't want this behavior!
# 
# Tested with:
#  FreeNAS 9.3-STABLE
#  FreeNAS 9.10-STABLE
#  Synology DSM 5.x (as destination only)
#######################################################################

if [ $# -ne 3 ]
then
  echo "Error: not enough arguments!"
  echo "Usage is: $0 r_src r_dest r_logfile"
  exit 2
fi

R_SRC=$1
R_DEST=$2
R_LOGFILE=$3

# Options:
#  -r  recurse into directories
#  -l  copy symlinks as symlinks
#  -t  preserve modification times
#  -g  preserve group
#  -o  preserve owner
#  -D  preserve device and special files
#  -h  human readable progress
#  -v  increase verbosity

#  --delete-during  receiver deletes during the transfer
#  --inplace  write updated data directly to destination file
#  --log-file  specify log file

R_OPTIONS="-rltgoDhv --delete-during --inplace --log-file="${R_LOGFILE}

# Files to exclude:
#  .windows  FreeNAS ACL settings (?)
#  vmware.log  VMware virtual machine log files
#  vmware-*.log
#  @eaDir/  Synology extended attributes (?)
#  @eaDir
#  Thumbs.db  Windows system files

R_EXCLUDE="--exclude .windows --exclude vmware.log --exclude vmware-*.log --exclude @eaDir/ --exclude @eaDir --exclude Thumbs.db"

echo "$(date) Copy" ${R_SRC} "to" ${R_DEST} >> ${R_LOGFILE}
rsync ${R_OPTIONS} ${R_EXCLUDE} "-e ssh -T -c arcfour -o Compression=no -x" ${R_SRC} ${R_DEST}
echo "$(date) Copy completed" >> ${R_LOGFILE}
exit
 
Last edited:
melloa

melloa

Wizard
Joined
May 22, 2016
Messages
1,749
Spearfoot

Spearfoot

He of the long foot
Moderator
Joined
May 13, 2015
Messages
2,478
P

Penbrock

Dabbler
Joined
May 27, 2017
Messages
26
I am trying this between to Freenas boxes. I downloaded it from pastbin, saved it in /root/ and chmod it with 777. owner is root: wheel
but when I try running it I get "Command not Found". I tried it with just rsync-invoke.sh src dst log, and ./root/rsync-invoke.sh src dst log with no luck
when I run it with sh rsync-invoke.sh src dst log I get ": not founde.sh: rsync-invoke.sh: 85: Syntax error: end of file unexpected (expecting "then")"

What am I doing wrong? I'm sure I missed something simple
Main server version: 9.10.2-u6
Dest server version: 11.2-u2
 
P

Penbrock

Dabbler
Joined
May 27, 2017
Messages
26
Penbrock said:
I am trying this between to Freenas boxes. I downloaded it from pastbin, saved it in /root/ and chmod it with 777. owner is root: wheel
but when I try running it I get "Command not Found". I tried it with just rsync-invoke.sh src dst log, and ./root/rsync-invoke.sh src dst log with no luck
when I run it with sh rsync-invoke.sh src dst log I get ": not founde.sh: rsync-invoke.sh: 85: Syntax error: end of file unexpected (expecting "then")"

What am I doing wrong? I'm sure I missed something simple
Main server version: 9.10.2-u6
Dest server version: 11.2-u2
Click to expand...

Skip that, it must have been something in the downloaded script. I manually pasted the lines into a file and it does run now..

How ever I must have something messed up in my DST field. After scanning all the files it errors out
Code:
./CopyNow.sh: line 89: root@10.0.1.198/mnt/UserHomes/HomeDrives: No such file or directory

According to the information, I did remove "-e ssh -T -c arcfour -o Compression=no -x" with no luck

I would love any help
Here is the command line I use
Code:
./CopyNow.sh /mnt/UserHomes/HomeDrives/ root@10.0.1.198:/mnt/UserHomes/HomeDrives /var/log/rsync-invoke.log
 
Spearfoot

Spearfoot

He of the long foot
Moderator
Joined
May 13, 2015
Messages
2,478
Penbrock said:
Skip that, it must have been something in the downloaded script. I manually pasted the lines into a file and it does run now..

How ever I must have something messed up in my DST field. After scanning all the files it errors out
Code:
./CopyNow.sh: line 89: root@10.0.1.198/mnt/UserHomes/HomeDrives: No such file or directory

According to the information, I did remove "-e ssh -T -c arcfour -o Compression=no -x" with no luck

I would love any help
Here is the command line I use
Code:
./CopyNow.sh /mnt/UserHomes/HomeDrives/ root@10.0.1.198:/mnt/UserHomes/HomeDrives /var/log/rsync-invoke.log
Click to expand...
Does /mnt/UserHomes/HomeDrives exist on the target server at IP 10.0.1.198? If UserHomes is a pool on your source server that you want to copy to your target server, you need to create it on the target server before you try to update it w/ rsync from the source.
 
P

Penbrock

Dabbler
Joined
May 27, 2017
Messages
26
Yes they are on both. Infact I had used the rsync inside of Freenas to copy to the backup server but found out it does not keep the domain permissions. That is why I am trying this way.
I also tried deleting one folder in the UserHomes on the target to see if it just didn't like the fact that the folders and files where created by a different run of rsync
 
Spearfoot

Spearfoot

He of the long foot
Moderator
Joined
May 13, 2015
Messages
2,478
Penbrock said:
Yes they are on both. Infact I had used the rsync inside of Freenas to copy to the backup server but found out it does not keep the domain permissions. That is why I am trying this way.
I also tried deleting one folder in the UserHomes on the target to see if it just didn't like the fact that the folders and files where created by a different run of rsync
Click to expand...
Try changing the shebang (the first line in the script) from #!/bin/bash to #!/bin/sh. This is the only difference I see between the version I posted on pastebin and the version I use every day here on my systems.

The fact that you can't seem to use the ssh options concerns me. I wonder if you somehow have gotten a version of the script with DOS/Windows-style newlines (CR-LF pairs) instead of the plain UNIX-style newlines that FreeBSD/FreeNAS expect. Using any kind of Windows editor to edit the file can cause this.

Have you set up encryption keys so that you can use ssh to open a shell from the source server to the target (10.0.1.198) without entering a password?

Lastly, does the script write anything to the log? If so, post the log here (in 'code' tags). Perhaps that will help with troubleshooting.
 
P

Penbrock

Dabbler
Joined
May 27, 2017
Messages
26
Spearfoot said:
Try changing the shebang (the first line in the script) from #!/bin/bash to #!/bin/sh. This is the only difference I see between the version I posted on pastebin and the version I use every day here on my systems.

The fact that you can't seem to use the ssh options concerns me. I wonder if you somehow have gotten a version of the script with DOS/Windows-style newlines (CR-LF pairs) instead of the plain UNIX-style newlines that FreeBSD/FreeNAS expect. Using any kind of Windows editor to edit the file can cause this.

Have you set up encryption keys so that you can use ssh to open a shell from the source server to the target (10.0.1.198) without entering a password?

Lastly, does the script write anything to the log? If so, post the log here (in 'code' tags). Perhaps that will help with troubleshooting.
Click to expand...
I changed the top line to #!/bin/sh . No change.

I did setup the key to ssh to 10.0.1.198. To test I run ssh root@10.0.1.198 and it does not prompt for a password it just connects. From there if I enter 'ls /mnt/UserHomes/HomeDrives/ ' I do see the sub folders I am trying to copy

Here is the log (top and end due to size)
Code:
Mon Feb 25 06:32:12 CST 2019 Copy /mnt/UserHomes/HomeDrives/ to root@10.0.1.198:/mnt/UserHomes/HomeDrives/
2019/02/25 06:32:12 [51596] building file list
2019/02/25 06:32:12 [51596] dr-xr-xr-x              8 2016/04/18 13:57:55 .
2019/02/25 06:32:12 [51596] drwxrwxrwx            334 2019/02/22 11:59:22 D1
2019/02/25 06:32:12 [51596] d---------              2 2018/08/06 09:30:58 D1/2815
2019/02/25 06:32:12 [51596] d---------              2 2018/08/06 09:33:35 D1/2816
2019/02/25 06:32:12 [51596] d---------              2 2018/08/06 09:34:40 D1/2817
2019/02/25 06:32:12 [51596] d---------              2 2018/12/10 06:36:38 D1/2818
2019/02/25 06:32:12 [51596] d---------              2 2018/08/06 09:38:09 D1/2819
2019/02/25 06:32:12 [51596] d---------              2 2018/08/06 09:39:24 D1/2820
2019/02/25 06:32:12 [51596] d---------              2 2018/08/06 09:40:47 D1/2821
2019/02/25 06:32:12 [51596] d---------              2 2018/08/06 09:43:39 D1/2822
2019/02/25 06:32:12 [51596] d---------              2 2018/08/06 09:44:30 D1/2823
2019/02/25 06:32:12 [51596] d---------              2 2018/08/06 09:45:35 D1/2824
2019/02/25 06:32:12 [51596] d---------              2 2018/08/06 09:49:42 D1/2825
2019/02/25 06:32:12 [51596] d---------              2 2018/08/06 09:50:34 D1/2826
2019/02/25 06:32:12 [51596] d---------              2 2018/08/06 09:51:08 D1/2827
2019/02/25 06:32:12 [51596] d---------              2 2018/08/06 09:51:40 D1/2828
2019/02/25 06:32:12 [51596] d---------              2 2018/08/06 09:52:13 D1/2829
2019/02/25 06:32:12 [51596] d---------              2 2017/03/30 11:23:13 D1/ATTin
2019/02/25 06:32:12 [51596] d---------              2 2017/03/30 11:27:53 D1/aames
2019/02/25 06:32:12 [51596] d---------              2 2017/03/30 11:27:53 D1/abrown
2019/02/25 06:32:12 [51596] d---------              2 2018/04/20 12:38:52 D1/acain
2019/02/25 06:32:12 [51596] d---------              2 2017/03/30 11:32:48 D1/acampanelli
2019/02/25 06:32:12 [51596] d---------              2 2017/03/30 11:23:15 D1/acarter
2019/02/25 06:32:13 [51596] d---------              4 2018/10/15 16:23:34 D1/acounterman
2019/02/25 06:32:13 [51596] -rw-rw-rw-         14.13K 2018/10/05 11:49:41 D1/acounterman/Copy of Amanda  MORE Funnel  2018 (4).xlsx
2019/02/25 06:32:13 [51596] -rw-rw-rw-         19.28K 2018/10/15 16:18:00 D1/acounterman/Mobility MORE Funnel  2018 Master.xlsx
2019/02/25 06:32:13 [51596] d---------              3 2019/01/15 13:36:48 D1/agarrison
2019/02/25 06:32:13 [51596] drwxrwxrwx              4 2019/01/15
……  cut out for size
….  cut end
2019/02/25 06:33:01 [51596] -rw-rw-rw-         64.16K 2017/07/17 09:59:47 IT/kcurtis/rsb01/walnutlawn/backup/duplicati-idcd41821f3e241dabe32496187bca983.dindex.zip.aes
2019/02/25 06:33:01 [51596] -rw-rw-rw-         60.20K 2017/07/17
2019/02/25 06:33:01 [51596] drwxrwxrwx              4 2016/04/05 07:28:54 Programmers
2019/02/25 06:33:01 [51596] d---------              2 2016/04/05 07:28:54 Programmers/jblakey
2019/02/25 06:33:01 [51596] d---------              2 2016/04/04 12:47:50 Programmers/jowen
2019/02/25 06:33:01 [51596] sent 4.28M bytes  received 15.94M bytes  408.50K bytes/sec
2019/02/25 06:33:01 [51596] total size is 1.01T  speedup is 50,093.26
Mon Feb 25 06:33:01 CST 2019 Copy completed



This is the script file I copied.

Code:
#!/bin/sh
#######################################################################
# A wrapper for invoking rsync to copy a source dataset to a destination
# dataset with options that are 'known to work' with FreeNAS/FreeBSD
# and Windows datasets. This means we have to avoid the '-p' (preserve
# permissions) option in all of its forms, per bug report 7713:
#
# https://bugs.freenas.org/issues/7713
#
# Command-line parameters:
#   1: Source dataset       R_SRC     (local system)  /mnt/tank/sharename/
#   2: Destination dataset  R_DEST    (local system)  /volume1/sharename
#   3: Log file             R_LOGFILE (local system) /mnt/tank/sysadmin/log/push-log.log
#
# R_SRC or R_DEST may include user ID and hostname specifier
#
# Example usage, where 'boomer' is a FreeNAS server and 'bertrand' is
# a Synology Diskstation. Note that you should include a trailing slash
# only on the source dataset specifier:
#
# rsync-invoke.sh root@bertrand:/volume1/devtools/ /mnt/tank/devtools
# /mnt/tank/sysadmin/log/pull-from-bertrand.log
# rsync-invoke.sh /mnt/tank/devtools/ root@bertrand:/volume1/devtools
# /mnt/tank/sysadmin/log/push-to-bertrand.log
#
# Assumes SSH has been enabled and configured between the source
# and destination hosts.
#
# Invokes rsync with these SSH options to optimize transfer speed:
#
#   "-e ssh -T -c arcfour -o Compression=no -x"
#
# You may need to remove this altogether, or modify the encryption
# specifier (-c arcfour) to use a scheme available on your system.
#
# !!! WARNING !!!
# This script deletes files on the destination that don't exist
# on the source! Edit R_OPTIONS below and remove '--delete-during'
# and '--inplace' if you don't want this behavior!
#
# Tested with:
#   FreeNAS 9.3-STABLE
#   FreeNAS 9.10-STABLE
#   Synology DSM 5.x (as destination only)
#######################################################################

if [ $# -ne 3 ]
then
  echo "Error: not enough arguments!"
  echo "Usage is: $0 r_src r_dest r_logfile"
  exit 2
fi

R_SRC=$1
R_DEST=$2
R_LOGFILE=$3

# Options:
#   -r  recurse into directories
#   -l  copy symlinks as symlinks
#   -t  preserve modification times
#   -g  preserve group
#   -o  preserve owner
#   -D  preserve device and special files
#   -h  human readable progress
#   -v  increase verbosity

#   --delete-during   receiver deletes during the transfer
#   --inplace         write updated data directly to destination file
#   --log-file        specify log file

R_OPTIONS="-rltgoDhv --delete-during --inplace --log-file="${R_LOGFILE}

# Files to exclude:
#   .windows      FreeNAS ACL settings (?)
#   vmware.log    VMware virtual machine log files
#   vmware-*.log
#   @eaDir/       Synology extended attributes (?)
#   @eaDir
#   Thumbs.db     Windows system files

R_EXCLUDE="--exclude .windows --exclude vmware.log --exclude vmware-*.log --exclude
@eaDir/ --exclude @eaDir --exclude Thumbs.db"

echo "$(date) Copy" ${R_SRC} "to" ${R_DEST} >> ${R_LOGFILE}
rsync ${R_OPTIONS} ${R_EXCLUDE} "-e ssh -T -c arcfour -o Compression=no -x" ${R_SRC}
# rsync ${R_OPTIONS} ${R_EXCLUDE}  ${R_SRC}
${R_DEST}
echo "$(date) Copy completed" >> ${R_LOGFILE}
exit


Any help would be great.
 
Last edited:
Spearfoot

Spearfoot

He of the long foot
Moderator
Joined
May 13, 2015
Messages
2,478
Penbrock said:
I changed the top line to #!/bin/sh . No change.

I did setup the key to ssh to 10.0.1.198. To test I run ssh root@10.0.1.198 and it does not prompt for a password it just connects. From there if I enter 'ls /mnt/UserHomes/HomeDrives/ ' I do see the sub folders I am trying to copy

Here is the log (top and end due to size)
Code:
Mon Feb 25 06:32:12 CST 2019 Copy /mnt/UserHomes/HomeDrives/ to root@10.0.1.198:/mnt/UserHomes/HomeDrives/
2019/02/25 06:32:12 [51596] building file list
2019/02/25 06:32:12 [51596] dr-xr-xr-x              8 2016/04/18 13:57:55 .
2019/02/25 06:32:12 [51596] drwxrwxrwx            334 2019/02/22 11:59:22 D1
2019/02/25 06:32:12 [51596] d---------              2 2018/08/06 09:30:58 D1/2815
2019/02/25 06:32:12 [51596] d---------              2 2018/08/06 09:33:35 D1/2816
2019/02/25 06:32:12 [51596] d---------              2 2018/08/06 09:34:40 D1/2817
2019/02/25 06:32:12 [51596] d---------              2 2018/12/10 06:36:38 D1/2818
2019/02/25 06:32:12 [51596] d---------              2 2018/08/06 09:38:09 D1/2819
2019/02/25 06:32:12 [51596] d---------              2 2018/08/06 09:39:24 D1/2820
2019/02/25 06:32:12 [51596] d---------              2 2018/08/06 09:40:47 D1/2821
2019/02/25 06:32:12 [51596] d---------              2 2018/08/06 09:43:39 D1/2822
2019/02/25 06:32:12 [51596] d---------              2 2018/08/06 09:44:30 D1/2823
2019/02/25 06:32:12 [51596] d---------              2 2018/08/06 09:45:35 D1/2824
2019/02/25 06:32:12 [51596] d---------              2 2018/08/06 09:49:42 D1/2825
2019/02/25 06:32:12 [51596] d---------              2 2018/08/06 09:50:34 D1/2826
2019/02/25 06:32:12 [51596] d---------              2 2018/08/06 09:51:08 D1/2827
2019/02/25 06:32:12 [51596] d---------              2 2018/08/06 09:51:40 D1/2828
2019/02/25 06:32:12 [51596] d---------              2 2018/08/06 09:52:13 D1/2829
2019/02/25 06:32:12 [51596] d---------              2 2017/03/30 11:23:13 D1/ATTin
2019/02/25 06:32:12 [51596] d---------              2 2017/03/30 11:27:53 D1/aames
2019/02/25 06:32:12 [51596] d---------              2 2017/03/30 11:27:53 D1/abrown
2019/02/25 06:32:12 [51596] d---------              2 2018/04/20 12:38:52 D1/acain
2019/02/25 06:32:12 [51596] d---------              2 2017/03/30 11:32:48 D1/acampanelli
2019/02/25 06:32:12 [51596] d---------              2 2017/03/30 11:23:15 D1/acarter
2019/02/25 06:32:13 [51596] d---------              4 2018/10/15 16:23:34 D1/acounterman
2019/02/25 06:32:13 [51596] -rw-rw-rw-         14.13K 2018/10/05 11:49:41 D1/acounterman/Copy of Amanda  MORE Funnel  2018 (4).xlsx
2019/02/25 06:32:13 [51596] -rw-rw-rw-         19.28K 2018/10/15 16:18:00 D1/acounterman/Mobility MORE Funnel  2018 Master.xlsx
2019/02/25 06:32:13 [51596] d---------              3 2019/01/15 13:36:48 D1/agarrison
2019/02/25 06:32:13 [51596] drwxrwxrwx              4 2019/01/15
……  cut out for size
….  cut end
2019/02/25 06:33:01 [51596] -rw-rw-rw-         64.16K 2017/07/17 09:59:47 IT/kcurtis/rsb01/walnutlawn/backup/duplicati-idcd41821f3e241dabe32496187bca983.dindex.zip.aes
2019/02/25 06:33:01 [51596] -rw-rw-rw-         60.20K 2017/07/17
2019/02/25 06:33:01 [51596] drwxrwxrwx              4 2016/04/05 07:28:54 Programmers
2019/02/25 06:33:01 [51596] d---------              2 2016/04/05 07:28:54 Programmers/jblakey
2019/02/25 06:33:01 [51596] d---------              2 2016/04/04 12:47:50 Programmers/jowen
2019/02/25 06:33:01 [51596] sent 4.28M bytes  received 15.94M bytes  408.50K bytes/sec
2019/02/25 06:33:01 [51596] total size is 1.01T  speedup is 50,093.26
Mon Feb 25 06:33:01 CST 2019 Copy completed



This is the script file I copied.

Code:
#!/bin/sh
#######################################################################
# A wrapper for invoking rsync to copy a source dataset to a destination
# dataset with options that are 'known to work' with FreeNAS/FreeBSD
# and Windows datasets. This means we have to avoid the '-p' (preserve
# permissions) option in all of its forms, per bug report 7713:
#
# https://bugs.freenas.org/issues/7713
#
# Command-line parameters:
#   1: Source dataset       R_SRC     (local system)  /mnt/tank/sharename/
#   2: Destination dataset  R_DEST    (local system)  /volume1/sharename
#   3: Log file             R_LOGFILE (local system) /mnt/tank/sysadmin/log/push-log.log
#
# R_SRC or R_DEST may include user ID and hostname specifier
#
# Example usage, where 'boomer' is a FreeNAS server and 'bertrand' is
# a Synology Diskstation. Note that you should include a trailing slash
# only on the source dataset specifier:
#
# rsync-invoke.sh root@bertrand:/volume1/devtools/ /mnt/tank/devtools
# /mnt/tank/sysadmin/log/pull-from-bertrand.log
# rsync-invoke.sh /mnt/tank/devtools/ root@bertrand:/volume1/devtools
# /mnt/tank/sysadmin/log/push-to-bertrand.log
#
# Assumes SSH has been enabled and configured between the source
# and destination hosts.
#
# Invokes rsync with these SSH options to optimize transfer speed:
#
#   "-e ssh -T -c arcfour -o Compression=no -x"
#
# You may need to remove this altogether, or modify the encryption
# specifier (-c arcfour) to use a scheme available on your system.
#
# !!! WARNING !!!
# This script deletes files on the destination that don't exist
# on the source! Edit R_OPTIONS below and remove '--delete-during'
# and '--inplace' if you don't want this behavior!
#
# Tested with:
#   FreeNAS 9.3-STABLE
#   FreeNAS 9.10-STABLE
#   Synology DSM 5.x (as destination only)
#######################################################################

if [ $# -ne 3 ]
then
  echo "Error: not enough arguments!"
  echo "Usage is: $0 r_src r_dest r_logfile"
  exit 2
fi

R_SRC=$1
R_DEST=$2
R_LOGFILE=$3

# Options:
#   -r  recurse into directories
#   -l  copy symlinks as symlinks
#   -t  preserve modification times
#   -g  preserve group
#   -o  preserve owner
#   -D  preserve device and special files
#   -h  human readable progress
#   -v  increase verbosity

#   --delete-during   receiver deletes during the transfer
#   --inplace         write updated data directly to destination file
#   --log-file        specify log file

R_OPTIONS="-rltgoDhv --delete-during --inplace --log-file="${R_LOGFILE}

# Files to exclude:
#   .windows      FreeNAS ACL settings (?)
#   vmware.log    VMware virtual machine log files
#   vmware-*.log
#   @eaDir/       Synology extended attributes (?)
#   @eaDir
#   Thumbs.db     Windows system files

R_EXCLUDE="--exclude .windows --exclude vmware.log --exclude vmware-*.log --exclude
@eaDir/ --exclude @eaDir --exclude Thumbs.db"

echo "$(date) Copy" ${R_SRC} "to" ${R_DEST} >> ${R_LOGFILE}
rsync ${R_OPTIONS} ${R_EXCLUDE} "-e ssh -T -c arcfour -o Compression=no -x" ${R_SRC}
# rsync ${R_OPTIONS} ${R_EXCLUDE}  ${R_SRC}
${R_DEST}
echo "$(date) Copy completed" >> ${R_LOGFILE}
exit


Any help would be great.
Click to expand...
One potential problem I see is that the R_EXCLUDE definition appears to 'wrap':
Code:
R_EXCLUDE="--exclude .windows --exclude vmware.log --exclude vmware-*.log --exclude
@eaDir/ --exclude @eaDir --exclude Thumbs.db"

This could simply be an artifact of the way you copied and pasted the code here on the forum. In any case, it should just be a single line, like this:
Code:
R_EXCLUDE="--exclude .windows --exclude vmware.log --exclude vmware-*.log --exclude @eaDir/ --exclude @eaDir --exclude Thumbs.db"

Looking at the first line of your log file:
Code:
Mon Feb 25 06:32:12 CST 2019 Copy /mnt/UserHomes/HomeDrives/ to root@10.0.1.198:/mnt/UserHomes/HomeDrives/
It looks like you may not be running the script correctly if you intend to copy subdirectories. You should call it without the slash on the destination path, like this:
Code:
./rsync-invoke.sh /mnt/UserHomes/HomeDrives/ root@10.0.1.198:/mnt/UserHomes/HomeDrives
Hope this helps...
 
P

Penbrock

Dabbler
Joined
May 27, 2017
Messages
26
Spearfoot said:
One potential problem I see is that the R_EXCLUDE definition appears to 'wrap':
Code:
R_EXCLUDE="--exclude .windows --exclude vmware.log --exclude vmware-*.log --exclude
@eaDir/ --exclude @eaDir --exclude Thumbs.db"

This could simply be an artifact of the way you copied and pasted the code here on the forum. In any case, it should just be a single line, like this:
Code:
R_EXCLUDE="--exclude .windows --exclude vmware.log --exclude vmware-*.log --exclude @eaDir/ --exclude @eaDir --exclude Thumbs.db"

Looking at the first line of your log file:
Code:
Mon Feb 25 06:32:12 CST 2019 Copy /mnt/UserHomes/HomeDrives/ to root@10.0.1.198:/mnt/UserHomes/HomeDrives/
It looks like you may not be running the script correctly if you intend to copy subdirectories. You should call it without the slash on the destination path, like this:
Code:
./rsync-invoke.sh /mnt/UserHomes/HomeDrives/ root@10.0.1.198:/mnt/UserHomes/HomeDrives
Hope this helps...
Click to expand...

I fixed the wrapped line, and made sure the trailing / is removed. No change.

I think I will delete all the scripts and start from a fresh start. I'm sure I am doing something wrong no one else seems to have an issue with your great script
 
Spearfoot

Spearfoot

He of the long foot
Moderator
Joined
May 13, 2015
Messages
2,478
Penbrock said:
...snip...In fact I had used the rsync inside of Freenas to copy to the backup server but found out it does not keep the domain permissions. That is why I am trying this way.
Click to expand...
I missed this earlier statement of yours.

No matter how you use rsync -- via the GUI or from a shell script -- it's not going to copy Windows permissions; that's the subject of this thread. So I doubt it will have any better success with domain permissions.

Your best bet, once the files are copied to the new server, may simply be to manually reconfigure the permissions.
 
P

Penbrock

Dabbler
Joined
May 27, 2017
Messages
26
OK thanks. Sorry I had thought the script was a work around to get the permissions to work.

Thanks for all the help
 
Spearfoot

Spearfoot

He of the long foot
Moderator
Joined
May 13, 2015
Messages
2,478
lucapsg said:
Old thread, current doubt ...

Does this problem still exist in 11.3-RELEASE?
Click to expand...
I honestly don't know... I installed 11.3-RELEASE on my test system last week and haven't had time to investigate this particular problem.
 
J

JDLH

Dabbler
Joined
Aug 10, 2019
Messages
22
Spearfoot

Spearfoot

He of the long foot
Moderator
Joined
May 13, 2015
Messages
2,478
JDLH said:
As of June 2020, the domain bugs.freenas.org redirects to a newer, Jira-based bug database, which doesn't seem to contain this bug report. But I found what looks like the same bug report at https://redmine.ixsystems.com/issues/7713: "Rsync over SSH 'PUSH' task works, 'PULL' task fails", dated 2015-01-27.
Click to expand...
Yep, that's the bug report I participated in several years ago...
 
P

Phil1295

Explorer
Joined
Sep 20, 2020
Messages
79
I tested and I can confirm that the rsync doesn't still preserve ACL permissions between two datasets on latest FreeNAS version
The worst: target files do not even get the target dataset inherited permissions on create, they get root/wheel permissions

Really makes rsync useless for SMB shares

Wired that such a basic feature is not present / fixed
 
anodos

anodos

Sambassador
iXsystems
Joined
Mar 6, 2014
Messages
9,553
Phil1295 said:
I tested and I can confirm that the rsync doesn't still preserve ACL permissions between two datasets on latest FreeNAS version
The worst: target files do not even get the target dataset inherited permissions on create, they get root/wheel permissions

Really makes rsync useless for SMB shares

Wired that such a basic feature is not present / fixed
Click to expand...
What arguments are you passing to rsync?
 
You must log in or register to reply here.

Important Announcement for the TrueNAS Community.

The TrueNAS Community has now been moved. This forum will now become READ-ONLY for historical purposes. Please feel free to join us on the new TrueNAS Community Forums.

Related topics on forums.truenas.com for thread: "Impaired rsync permissions support for Windows datasets"

Similar threads

G
  • Locked
rsync: mkstemp [...] failed: Operation not permitted (1)
Replies
6
Views
7K
desertrider
D
J
  • Locked
rsync -A from linux box to freeNAS fails
Replies
2
Views
4K
Joao S Veiga
J
Z
rsync copies folders but not files
Replies
5
Views
4K
dlavigne
D
S
Rsync task fails trying to preserve AD ACLs
Replies
1
Views
2K
sgt_jamez
S
STREBLO
  • Locked
Permissions on NFS share and rsync
Replies
1
Views
3K
dlavigne
D
Top