feistyviking
Cadet
- Joined
- Jan 30, 2020
- Messages
- 5
I have been struggling with Cloud Sync and migrating data from an old server to a temporary storage setup. (So I can salvage parts from the old beast.)
Source data is roughly 90TB total with a whopping 13.7 million files.
Connection between the two is over WebDAV and logs in with Cloud Credentials no problem.
I start the transfer and begin PULLing data down from the old server at as fast as the drives will give it to me ~3 Gbps...all looks great! I let that run over-night expecting to see large sums of data on the new server in the morning.
Alas, that is not the case :(
Once rclone get's to about 1.2 million objects in it's log, transfer speeds slowly begins to drop, never fully stops but slows down consistently over time.
Above is a graph of network traffic, the spikes are when I restarted the sync to see what would happen. Once I would restart the sync it would start streaming great again, but still once it got a little over 1 million objects "checked" it would begin to slow.
Some googling lead me to this post
https://rclone.org/bugs/#rclone-struggles-with-millions-of-files-in-a-directory
I suspect this is what is happening, so I try and start syncing folder by folder, sure enough, speeds stay rocking as long as the folder has less than 1 million objects in it.
Not sure what can be fixed from the FreeNAS side of things but really more looking to see if anyone else has experienced this/should we put a link on FreeNAS documentation to the known rclone bugs page.
Source data is roughly 90TB total with a whopping 13.7 million files.
Connection between the two is over WebDAV and logs in with Cloud Credentials no problem.
I start the transfer and begin PULLing data down from the old server at as fast as the drives will give it to me ~3 Gbps...all looks great! I let that run over-night expecting to see large sums of data on the new server in the morning.
Alas, that is not the case :(
Once rclone get's to about 1.2 million objects in it's log, transfer speeds slowly begins to drop, never fully stops but slows down consistently over time.
Above is a graph of network traffic, the spikes are when I restarted the sync to see what would happen. Once I would restart the sync it would start streaming great again, but still once it got a little over 1 million objects "checked" it would begin to slow.
Some googling lead me to this post
https://rclone.org/bugs/#rclone-struggles-with-millions-of-files-in-a-directory
I suspect this is what is happening, so I try and start syncing folder by folder, sure enough, speeds stay rocking as long as the folder has less than 1 million objects in it.
Not sure what can be fixed from the FreeNAS side of things but really more looking to see if anyone else has experienced this/should we put a link on FreeNAS documentation to the known rclone bugs page.