I thought this information might be useful for people transferring in data for new installs. I didn't see any mention of this particular issue when searching the forum.
I have a newly installed TrueNAS SCALE 21.02 system.
I started to transfer several terabytes of data into the server via rsync/ssh but it was much slower than gigabit due to the added encryption. During that time I transferred several hundred gigabytes of data with no problems.
So I went ahead and setup NFS to try to transfer the data in via rsync on a NFS mount.
However every time I started the rsync it hung after only a few minutes.
When it hung a strace of rsync just shows its waiting on a unix socket.
Looking at the stack of rsync I see:
After digging around I found this is a known bug in rsync itself.
Despite the issue title in my case it wasn't actually hanging in the same spot each time.
"rsync -avv /src/ /dst/" hangs constantly at the same place #159
The last comment in the bug report has a number of options that can be used as a workaround if you aren't able to upgrade to the most recent rsync.
I have a newly installed TrueNAS SCALE 21.02 system.
I started to transfer several terabytes of data into the server via rsync/ssh but it was much slower than gigabit due to the added encryption. During that time I transferred several hundred gigabytes of data with no problems.
So I went ahead and setup NFS to try to transfer the data in via rsync on a NFS mount.
However every time I started the rsync it hung after only a few minutes.
When it hung a strace of rsync just shows its waiting on a unix socket.
# strace -fttTvyys 4096 -p 2935
strace: Process 2935 attached
02:21:39.098242 select(5, [], [4<UNIX:[48402->48401]>], [], {tv_sec=24, tv_usec=453068}) = 0 (Timeout) <24.477928>
02:22:03.576830 select(5, [], [4<UNIX:[48402->48401]>], [], {tv_sec=60, tv_usec=0}) = 0 (Timeout) <60.003174>
02:23:03.580308 select(5, [], [4<UNIX:[48402->48401]>], [], {tv_sec=60, tv_usec=0}) = 0 (Timeout) <60.041621>
Looking at the stack of rsync I see:
# cat /proc/2933/stack
[<0>] poll_schedule_timeout.constprop.11+0x46/0x70
[<0>] do_select+0x60d/0x7a0
[<0>] core_sys_select+0x2b1/0x3f0
[<0>] kern_select+0xb7/0x100
[<0>] __x64_sys_select+0x24/0x30
[<0>] do_syscall_64+0x57/0x190
[<0>] entry_SYSCALL_64_after_hwframe+0x44/0xa9
After digging around I found this is a known bug in rsync itself.
Despite the issue title in my case it wasn't actually hanging in the same spot each time.
"rsync -avv /src/ /dst/" hangs constantly at the same place #159
The last comment in the bug report has a number of options that can be used as a workaround if you aren't able to upgrade to the most recent rsync.