How to debug Rsync failure between 2-FN servers.

[Keith Savoie]

I have two FreeNAS servers, one a backup of the other. On the backup server I have a Rsync Task as mode 'Rsync module' configured to pull from the main server. This has worked without issue for quite some time until recently. Now it no longer syncs and I am in need of a way to debug. On the backup server (FN-9.10.2) I am not seeing any errors in messages. On the server being backed (FN-9.2.1) up I see the following in messages.

Code:

Mar  7 12:35:20 nas01 rsyncd[29125]: rsync: writefd_unbuffered failed to write 4092 bytes to socket [sender]: Broken pipe (32)	
Mar  7 12:35:20 nas01 rsyncd[29125]: rsync error: error in rsync protocol data stream (code 12) at io.c(1532) [sender=3.0.9]

f.y.i. the backup server previously was running FN-9.3.1, I only recently upgraded it to see if it would correct the rsync issue.
Any assistance greatly appreciated!

 

[Robert Trevellyan]

I am in need of a way to debug

My approach to debugging rsync operations is by launching it from the command-line with suitable options for verbosity, logging etc.

rsync: writefd_unbuffered failed to write 4092 bytes to socket [sender]: Broken pipe (32)

Looks like maybe the receiver disconnected.

 

[Keith Savoie]

I rebuilt the rsync/module config between the two servers. If I reboot the rsync(backup) server then enable rsync, it initiates a sync to the main server. However in messages I receive 100s of
rsync: mkstemp "/path/to/file/.filename.ext.RaNdomString" (in nas01) failed: Operation not permitted (1)
This error is only thrown on what looks like temporary files, all files begin with a period and end in a random string (real file names minus the period and random string).
Then at some point the process fails without further error/indication. The only way for the two servers to release their connection is a reboot of one or the other. Even if I stop the rsync service on the backup, the main server does not indicate a disconnect. Only until I reboot the backup will main show a disconnect error.
Is it possible this type of behavior could be caused by wrong/corrupted ACLs on the files/folders of the main server? If so is there any harm by removing all ACLs recursively on the main server via CLI? (e.g. chmod -RN /path/to/dir )

 

[Robert Trevellyan]

That does look like some kind of permissions problem. I don't know whether removing ACLs will fix it.

The general operation of rsync is to build the new file with a temporary name, then discard the original and rename the temporary as a near-atomic operation.

Have you tried some basic, small rsync operations at the command-line?

 

[Keith Savoie]

Just figuring this out, on my main server I have X number of users. As files are being added to storage (via afp) they are owned by their accounts. I have removed a couple of users from the FN box and now notice that files who had those owners now have unknown owners. Instead of a username as owner it now shows a number (e.g.1005).
Question; Do I maybe not have FN configured properly to not associate owners with the file uploaded? What change would I need to make to correct this? and what should the owner of all file/directories be in storage?

 

[Robert Trevellyan]

I have removed a couple of users from the FN box and now notice that files who had those owners now have unknown owners. Instead of a username as owner it now shows a number (e.g.1005).

How would it know the user names if you removed the users? It's displaying the numeric ID, which hasn't changed, it just can't map it to a user name.

what should the owner of all file/directories be in storage?

That depends on what you're trying to achieve in terms of access control. It's also beyond the scope of a thread about rsync.

 

[Keith Savoie]

Sorry, don't mean to creep out of the threads scope, just investigating if my initial config may have anything to do with the rsync failure.

That depends on what you're trying to achieve in terms of access control. It's also beyond the scope of a thread about rsync.

Access control...basically none. Just need all users to have access to all files/directories regardless of which one put it there.

 

[Robert Trevellyan]

In general, rsync should be run as root, so it can complete all necessary ownership and permissions operations.

Access control...basically none.

Maybe you should go with a guest share:
http://doc.freenas.org/9.10/sharing.html#creating-afp-guest-shares

 

[Keith Savoie]

Progress. Under the rsync module on the main server, if I uncheck 'Archive' and 'Preserve permissions' rsync will complete. If either are selected, rsync fails again throwing 100s of ( rsync: mkstemp "path/to/file" failed: Operation not permitted ). I found this bug submission that appears to demonstrate similar behavior. https://bugs.freenas.org/issues/7713.
Would like to know what the underlying issue is.

 

[toolchain]

I'm running into the exact same issue, found another supporting thread that works around that bug

https://forums.freenas.org/index.ph...rmissions-support-for-windows-datasets.43973/