FreeNAS-11.2-U4.1, rsync task is not completed, the process rsync go in D status and is impossible to kill it

[PCorsetti]

I need help to solve this problems
The FreeNAS server from a month giving us this problems:
1) the system doesn't complete the rsync task (we have 77 rsync task scheduled)
2) the system have the folder shared by samba service
3) the system after one or two days of uptime became slowly, the service samba doesn't work properly (is impossible to access the shared folder) and need to be rebooted
After investigation I have see that the rsysnc task is not completed, it started but after few minutes it is locked and the rsync process go in D status and it is impossible to kill it.

for your information I show you the following example:
I start the rsync process defined from web console manually.
After started this is the status of process:

root@freenas-slot08-e9000:~ # ps -auxw | grep rsync
root 8030 43.3 0.0 71300 59080 - R 15:22 0:43.12 /usr/local/bin/rsync -r -t -z -a -p -X --delete-delay --delay-updates --log-file=/mnt/vol1_raidz
root 7777 4.7 0.0 16532 13656 - S 15:21 0:02.87 ssh -p 22 -o BatchMode=yes -o StrictHostKeyChecking=yes -l root CV2BMPAPP1.cc.local rsync --serv
root 8060 1.1 0.0 6260 2216 1 S+ 15:22 0:01.67 tail -200f ./Freenas_rsync_CV2BMPAPP1.log
root 4075 0.0 0.0 7752 2908 - Is 15:07 0:00.00 /usr/local/bin/rsync --daemon --config /usr/local/etc/rsync/rsyncd.conf
root 7774 0.0 0.0 4200 2012 - I 15:21 0:00.00 /usr/bin/lockf -s -t 0 -k /mnt/vol1_raidz2_freenas2/MVNE/BMPAPP1 /usr/local/bin/rsync -r -t -z -
root 7776 0.0 0.0 67204 58020 - S 15:21 0:29.76 /usr/local/bin/rsync -r -t -z -a -p -X --delete-delay --delay-updates --log-file=/mnt/vol1_raidz
root 8238 0.0 0.0 6704 2676 2 S+ 15:24 0:00.00 grep rsync
After few minutes you can see that the process id 8030 pass from status R to status D

root@freenas-slot08-e9000:~ # ps -auxw | grep rsync
root 4075 0.0 0.0 7752 2908 - Is 15:07 0:00.00 /usr/local/bin/rsync --daemon --config /usr/local/etc/rsync/rsyncd.conf
root 7773 0.0 0.0 7108 2976 - I 15:21 0:00.00 /bin/sh -c /usr/bin/lockf -s -t 0 -k "/mnt/vol1_raidz2_freenas2/MVNE/BMPAPP1" /usr/local/bin/rsync -r -t -z -a -p -X --delete-delay --delay-updates --log-file=/mnt/vol1_raidz2_freenas2/Log/Freenas_rsync_CV2BMPAPP1.log -e "ssh -p 22 -o BatchMode=yes -o StrictHostKeyChecking=y
root 7774 0.0 0.0 4200 2012 - I 15:21 0:00.00 /usr/bin/lockf -s -t 0 -k /mnt/vol1_raidz2_freenas2/MVNE/BMPAPP1 /usr/local/bin/rsync -r -t -z -a -p -X --delete-delay --delay-updates --log-file=/mnt/vol1_raidz2_freenas2/Log/Freenas_rsync_CV2BMPAPP1.log -e ssh -p 22 -o BatchMode=yes -o StrictHostKeyChecking=yes root@CV2BMP
root 7776 0.0 0.0 67204 58020 - I 15:21 0:29.76 /usr/local/bin/rsync -r -t -z -a -p -X --delete-delay --delay-updates --log-file=/mnt/vol1_raidz2_freenas2/Log/Freenas_rsync_CV2BMPAPP1.log -e ssh -p 22 -o BatchMode=yes -o StrictHostKeyChecking=yes root@CV2BMPAPP1.cc.local:/ /mnt/vol1_raidz2_freenas2/MVNE/BMPAPP1
root 7777 0.0 0.0 16532 13656 - S 15:21 0:02.87 ssh -p 22 -o BatchMode=yes -o StrictHostKeyChecking=yes -l root CV2BMPAPP1.cc.local rsync --server --sender -logDtpXrze.LsfxC . /
root 8030 0.0 0.0 71300 59084 - D 15:22 0:47.99 /usr/local/bin/rsync -r -t -z -a -p -X --delete-delay --delay-updates --log-file=/mnt/vol1_raidz2_freenas2/Log/Freenas_rsync_CV2BMPAPP1.log -e ssh -p 22 -o BatchMode=yes -o StrictHostKeyChecking=yes root@CV2BMPAPP1.cc.local:/ /mnt/vol1_raidz2_freenas2/MVNE/BMPAPP1
root 8060 0.0 0.0 6260 2216 1 I+ 15:22 0:01.67 tail -200f ./Freenas_rsync_CV2BMPAPP1.log
root 8240 0.0 0.0 6704 2676 2 S+ 15:24 0:00.00 grep rsync
root@freenas-slot08-e9000:~ #

On the attached file:
1) HW information
2) rsync task scheduled example
.

 

[dlavigne]

What's the full output of ifconfig (within code tags)?

Is there anything in /var/log/messages when the rsync fails?

 

[PCorsetti]

Attached you can found ifconfig output and the messages log.

I have verified that:

1) in the messages log file there aren't particolar error

2) this morning I have found that the system working normally, if you perform action on standard filesystem as /var/log or /etc but doesn't work when you perform command on /mnt filesystems the command go in hang

3) I have verified the disk on FreeNAS

4) I have verified the Pools configuration


5) I have found that the Pools is used at 98% with 373 GiB Free

6) we have forced the restart of FreeNAS to enable as to works, after restart the system works normally also in /mnt/vol1_raidz2_freenas2 and reading the logs under /mnt/vol1_raidz2_freenas2/Log I have see that the log files updated yesterday are been lost and the log files are came back to previus version.

 

[dlavigne]

Is this still an issue for you? If so, create a report at bugs.ixsystems.com and post the issue number here.

 

[PCorsetti]

Thanks,

Yes I have, I will it as soon.

I am investigating about it to give more information to the support to help it to solve it.

At moment I suppose that:

when the system have to much rsync process at the same time, rsync lock files and don't release it and stuck the rsync process.

This status produce many problems to the system as the scrub task is locked and the system is much slow