Rsync failing after hours with rsync error: error in rsync protocol data stream (code 12) at io.c(22

[David Dyer-Bennet]

Pulling via rsync from a 9.2.x FreeNAS into a 9.3.1. It runs for hours, transferring a lot of data, and then crashes with some variant of this error. I see it got truncated, so:

rsync error: error in rsync protocol data stream (code 12) at io.c(226)

I see lots of variants of this error here, but if I read them correctly, they're an immediate failure (at least the first 6 I looked at are). This is NOT immediate; it's after transferring a lot of data, hours of transfer on a gigabit LAN.

The command I'm issuing is:

sudo rsync --rsh='ssh -A -c arcfour128' -a --safe-links --stats -h -v --exclude .recycle/ --exclude /mnt/Z01/.system/ root@192.168.1.181:/mnt/Z01 /mnt/zzback/rebma-backup/

(The command is issued on the 9.3.1 box; the source it's pulling from, the system at 192.168.1.181, is the 9.2.x box.)

The '-c arcfour128' is an attempt to reduce the encryption load, which seems to be the limiting factor on transfer speed. TOP shows ssh running 80-100% cpu load (maxing out one core) on the receiving system. With the default encryption this was consuming about 450 Mbit/sec as reported in the graph on the reporting tab, with arcfour I'm getting nearly 600 Mbit/sec. As I do over and over and over again in my life, I desperately wish for a null encryption option in ssh; this stream is running between two boxes plugged into the same unmanaged switch, so the traffic is not going to be publicly exposed. Meanwhile, speed is of the essence, especially since we're doing this copy because the disks in the source server may be heat-compromised (thought they're running fine currently). There's probably a way to make a netcat-based script that acts enough like ssh for rsync to use it, eh? But I haven't done it yet, it only comes up urgently every few years.

The "-A" is vestigial and non-functional; I lost track of the fact that sudo breaks my access to my SSH agent, so trying to forward that connection to the other system won't work. Hence I have to supply the root password for 192.168.1.181 each time I start this command up. I just haven't bothered to remove the -A from the file I use to start it the same each time.

Any ideas? Being rsync, I just start it up again, but I wasted 6 hours of last night since it crashed 2 hours after I went to sleep.

 

[Robert Trevellyan]

Are the standard rsync troubleshooting steps helpful?

 

[David Dyer-Bennet]

I've looked at that of course. #3 seems to describe my symptom, but the steps they suggest look totally irrelevant to my situation -- no slightest risk of idle time, for example (and I don't see how idle time of any significance could even come up in an rsync scenario). And I've held ssh connections into both boxes for many many hours, across many crashes of the connection between them (and they're all single-NIC boxes, my connection is coming in the same port as the one that's failing), which rules out the whole set of external (cable, hub) failures, network stack failures, etc.

 

[David Dyer-Bennet]

It doesn't make a huge difference since I enhanced my script to log any failure and restart the transfer. There's some overhead at the start for sending the file list, but it doesn't seem to be very much.