Cloud Sync (rclone) struggles with millions of files in a directory

feistyviking

Cadet
Joined
Jan 30, 2020
Messages
5
I have been struggling with Cloud Sync and migrating data from an old server to a temporary storage setup. (So I can salvage parts from the old beast.)

Source data is roughly 90TB total with a whopping 13.7 million files.

Connection between the two is over WebDAV and logs in with Cloud Credentials no problem.

I start the transfer and begin PULLing data down from the old server at as fast as the drives will give it to me ~3 Gbps...all looks great! I let that run over-night expecting to see large sums of data on the new server in the morning.

Alas, that is not the case :(

Once rclone get's to about 1.2 million objects in it's log, transfer speeds slowly begins to drop, never fully stops but slows down consistently over time.
1580429348896.png

Above is a graph of network traffic, the spikes are when I restarted the sync to see what would happen. Once I would restart the sync it would start streaming great again, but still once it got a little over 1 million objects "checked" it would begin to slow.

Some googling lead me to this post

https://rclone.org/bugs/#rclone-struggles-with-millions-of-files-in-a-directory

I suspect this is what is happening, so I try and start syncing folder by folder, sure enough, speeds stay rocking as long as the folder has less than 1 million objects in it.

Not sure what can be fixed from the FreeNAS side of things but really more looking to see if anyone else has experienced this/should we put a link on FreeNAS documentation to the known rclone bugs page.
 
Top