Problem using rsync with rsync daemon

[David Dyer-Bennet]

I'm getting my data off my old Solaris ZFS server onto a FreeNAS box. Zfs send/recv doesn't work because I've got options set that aren't supported by FreeNAS (so the recv errors out). In any case, the transfer takes long enough that being able to interrupt and restart is very valuable. Hence, rsync. And rsync daemon because it's the only way to avoid encryption of the network stream (the stream goes through one switch within my internal network, and it's a "one-shot" deal in that when I get the job completed it stops happening, so I'm not too concerned with security risks) (at 1Gb the fastest encryption slows me down to about 750Mb, not horrible, but I get about 875Mb unencrypted; and when I plug the 10Gb cards together, I expect that to become a much bigger issue, so learning to use rsyncd seems like it's worth my time).

I've had zillions of weird situations where rsync and rsyncd running as root claim they can't set the permissions of a file because...they don't have permission. Some of this is because of anomalies in my existing filesystems, and I can fix those -- the directories that have *no* permissions are artifacts of things that used to be done by ACLs, when devolved to no access when the ACLs were removed, for example. So finding and fixing those is beneficial anyway. (But why can't rsyncd running as root create a directory with no normal permissions? Seems like an unlikely case, but a perfectly legal one.) All sorts of confusing things, and I've been thrashing the configuration trying to find things that work, so some of my feeling of confusion is no doubt due to strange configs when I performed particular tests.

I probably don't understand rsyncd config properly. I'm referring to its man page frequently, and I'm viewing the /etc/local/rsyncd.conf file created by the FreeNAS gui to understand what the gui settings are actually doing, but I'm still not always clear on things.

Anyway, that's the background. Right now, I'm getting this error from this rsync invocation:

Code:

root@FSFS:/export/home/ddb/Documents/fsfs/to-rebma# rsync --recursive --links --times -v --stats -R --partial --append --update --inplace --exclude .zfs/ /zp1/home/./ rsync://fsfs-mod@192.168.1.181/fsfs-backup
sending incremental file list
rsync: failed to read xattr rsync.%stat for "/ddb/Documents/Fandom/Minicon/Mc45/minicon_email_list_20091222.txt" (in fsfs-backup): Permission denied (13)
rsync: failed to read xattr rsync.%stat for "/ddb/Documents/Fandom/Minicon/Mc45/minicon_email_list_20091222.txt" (in fsfs-backup): Permission denied (13)
rsync: failed to read xattr rsync.%stat for "/ddb/Documents/Fandom/Minicon/Mc45/pr1email-end.txt" (in fsfs-backup): Permission denied (13)
# and a few dozen further similar errors

I think that that error message is an rsyncd error (so why is it labeled "rsync:"?). The reason I think that is that fsfs-backup is a user and a subdirectory on the receiving (rsyncd) system, and does not exist at all on the sending system.

On the sending side, that directory and file look like this:

Code:

root@FSFS:/zp1/home/ddb# ls -ld /zp1/home/ddb/Documents/Fandom/Minicon/Mc45
dr-xr-xr-x   2 65535    other         21 Mar 31  2010 /zp1/home/ddb/Documents/Fandom/Minicon/Mc45
root@FSFS:/zp1/home/ddb# ls -l /zp1/home/ddb/Documents/Fandom/Minicon/Mc45/minicon_email_list_20091222.txt
-r--------   1 65535    other      33097 Dec 23  2009 /zp1/home/ddb/Documents/Fandom/Minicon/Mc45/minicon_email_list_20091222.txt

Read-only, and a strange user (but I'm not propagating UID across the rsync), but nothing strange that I can see; in particular, no ACL indicated.

And on the receiving side:

Code:

[ddb@rebma /mnt/Z02/fsfs-backup]$ sudo ls -ld /mnt/Z02/fsfs-backup/ddb/Documents/Fandom/Minicon/Mc45/
drwxr-xr-x  2 fsfs-backup  wheel  21 Mar 31  2010 /mnt/Z02/fsfs-backup/ddb/Documents/Fandom/Minicon/Mc45/
[ddb@rebma /mnt/Z02/fsfs-backup]$

[ddb@rebma /mnt/Z02/fsfs-backup]$ sudo ls -l /mnt/Z02/fsfs-backup/ddb/Documents/Fandom/Minicon/Mc45/minicon_email_list_20091222.txt
-rw-------  1 fsfs-backup  wheel  33097 Dec 23  2009 /mnt/Z02/fsfs-backup/ddb/Documents/Fandom/Minicon/Mc45/minicon_email_list_20091222.txt

Finally, the rsyncd.conf file is:

Code:

[ddb@rebma /mnt/Z02/fsfs-backup]$ cat /etc/local/rsyncd.conf 
use chroot = yes
max connections = 4
pid file = /var/run/rsyncd.pid

[fsfs-backup]
    path = /mnt/Z02/fsfs-backup
    max connections = 4
    uid = fsfs-backup
    gid = fsfs-backup
    comment = Access to fsfs backup area
    read only = false
    write only = false
    hosts allow = 192.168.1.205
fake super = yes

Any ideas, hints, clues, leading questions, or suggestions?

 

[cyberjock]

Rsync is still going to be pretty slow. Even local zpool to a second local zpool only gets you about 150M/sec, and that's on a fairly overpowered system. rsync bottlenecks at a single thread, so it quickly becomes a limiting factor.

Honestly, in your case I'd look at trying to do:

1. local copy using cp between the 2 zpools if you can manage to mount both zpools on the same system.
2. mount an NFS or CIFS share from the other system and copy the data via that route (this should be able to easily saturate Gb LAN, and do more than that, although how much more will depend on other things)

 

[David Dyer-Bennet]

I'm copying between Solaris and FreeNAS, which (long ago) I was told weren't really interoperable at the disk level. I've done local copies when possible -- for the additional backups before I started the really big-deal stuff, for example.

Rsync is single-threaded, but I can run multiple rsyncs on subdirectories of the tree to get around that problem. Either the disks or the network are going to be the ultimate limit.

(Also also, I don't have enough controller ports to mount both pools in any one box.)

My problem is with rsync apparently, even when running as root, being unable to access or update files. What's going on with that? It leaves me deeply suspicious about the whole environment.

 

[cyberjock]

My problem is with rsync apparently, even when running as root, being unable to access or update files. What's going on with that? It leaves me deeply suspicious about the whole environment.

That's a common problem when the root user doesn't have permissions to some files or folders. It's one of the reasons I prefer zfs replication over rsync, but as we've established that's not an option for you at the present time. :/

 

[David Dyer-Bennet]

ZFS replication does appear to depend on the same UID and GIDs being on both ends, but I can manage that well enough on only three servers (mostly only two, even). And for our simple situation, I can even do massive chown -R most of the time -- we're not using any complex ownership or sharing structures.

I've been getting slightly better at NFSV4ACLs as implemented by FreeBSD, too, so I've unscrewed a few things without resorting to the big hammers.

 

[cyberjock]

ZFS replication does appear to depend on the same UID and GIDs being on both ends, but I can manage that well enough on only three servers (mostly only two, even). And for our simple situation, I can even do massive chown -R most of the time -- we're not using any complex ownership or sharing structures.

It most certainly doesn't unless you plan to use the backup server as a production server in case the primary server goes down. But that's not a replication issue. That's a "don't expect users and groups to work if they don't exist" problem (aka PEBCAK). If you don't plan to do that, when you replicate the backup data back to the primary, the production server should still use the same UIDs and GIDs that it did before, so the users will be properly linked. ;)

I've seen this on lots of setups that use AD/LDAP. If the primary uses AD or LDAP, the files and folders will have permissions assigned accordingly. On the backup server if you don't put it on the domain its just a bunch of numbers and such. If you put it on the domain then the UIDs and GIDs are replaced with the appropriate users. :P

Lots of users use the backup server as a canary in a coal mine. They upgrade the backup server to a new version first. If they can still authenticate and use AD/LDAP then all is well and they upgrade the primary. If things don't go well they'll see lots of numbers and know things are borked.