SOLVED Rsync tasks piling up

[michen]

Hello dear forum members,

recently my FreeNAS Server seemed very unresponsive and slow when accessing it through Windows Share (SMB) and then via the Web Interface.
Upon closer investigation I found a bunch of accumulated rsync tasks driving the system into full load. I took a screenshot of top:

I configured rsync as a cronjob, and it is supposed to run every 6 hours to mirror the whole dataset to an offsite backup server.

I should maybe add that it has worked stable for quite a while now and this is the first time the problem occured.
The most recent change to the system was an upgrade to the most current version (FreeNAS11.1-U7).

The system configuration is:

• ASUS P5B Mainboard
• Intel(R) Core(TM)2 Quad CPU Q9450 @ 2.66GHz
• 8192 MB RAM
• Stack of 11 WD Red Series, 3TB each
• onboard RAID, additional PCI Controller
• onboard Intel Network Controller

Does anyone have a clue what could be the cause?

Regards,
Michael

 

[michen]

I dug into this a little more and found out that the tasks did not finish their job, i.e. there was an error message from rsync that I did not catch in the logs. I took a dry run from the command line with the same commands that are in the cronjob and the following error came up:

rsync: write failed on "/mnt/syn112/rsyncback.log": No space left on device (28)
rsync error: error in file IO (code 11) at receiver.c(400) [receiver=3.1.2]
rsync error: error in file IO (code 11) at io.c(1633) [sender=3.1.2]

Thing is, there is more than enough space on the destination drive. Further investigation led me to believe that I was missing the --inplace option with rsync. But then again, logic dictates that if it hadn't been ever necessary before to feed rsync with --inplace the original cause must be something else. However, I am now dry-running rsync with that option to see if it helps.

Anyone else have a clue what is going on?

 

[michen]

Okay, I finally found out the cause of the problem. The rsync destination folder was not mounted as a SMB share.
It was treated as a local directory on the system partition.
That happened because of the reboot (update & restart), and the fact that I did not automatically mount the SMB share upon reboot.
I fixed this through the web interface by adding the mount command as a post-init command (Menu->Tasks->Init/Shutdown Scripts).
That should avoid the problem for future updates.