Fastest transfer method for FreeNAS Server to FreeNAS Server

[Jerami1981]

I am getting ready to rebuild my backup FreeNAS server and am trying to figure out the fastest method to copy the data from my primary to my secondary server. I will be re-configuring the second server with 2 vdevs, 6 disks each, in a single pool. I have 10 gig connections between all devices involved. I have sampled Rsync, RoboCopy, and File Explorer, as copy methods. On my original tests I was copying from a single vdev 12 disk pool, to a single vdev 12 disk pool. Over File Explorer on my desktop, I was able to sustain around 400 MB/s. Robocopy seemed to hover around 110 MB/s, and Rsync looked to be around 90-100 MB/s. I would have expected Rsync to be the fastest as it was copying from server to server, and not going through an extra device as it does with Robocopy and File Explorer. Do I have something configured poorly in Rsync, or is it likely that is just the throughput via that route. I need to copy around 35-40 TB of data, and if I can get it done in 1-2 days vs 8-10 days, that would be ideal. What should be the fastest method for the transfer of those 3, or is there another alternative not mentioned that would be faster.

 

[wblock]

The standard method is replication: http://doc.freenas.org/11/storage.html#replication-tasks.

rsync is not built for speed but to minimize network traffic. The Windows copy methods would be my absolute last resort.

 

[Jerami1981]

Is there a method that will better utilize my network? Network traffic is not of concern, whereas getting all of my data safely back onto a second storage media as quickly as possible is. Only 5 devices run across the 10G network, everything else flows across a 1G network.

Is there a particular reason why you dislike the Windows options?

 

[wblock]

Is there a method that will better utilize my network?

Yes, replication. This is based on zfs send | zfs recv.

Is there a particular reason why you dislike the Windows options?

Long experience.

 

[Jerami1981]

Awesome. I will take replication for a spin. thanks for the intel.

Fair enough. I will steer clear of that then.

Is replication the best method to use to keep everything in sync after the initial pairing, or is it better to switch over to rsync at that point?

 

[Nick2253]

Is replication the best method to use to keep everything in sync after the initial pairing, or is it better to switch over to rsync at that point?

I would continue to use replication. Depending on what you need, you can do regular snapshots (like nightly, or even hourly), and replicate those over.

 

[Jerami1981]

Will I be able to browse the files on the back up server, the same as the primary server, or will they be some kind of proprietary file, only usable in a recovery scenario?

 

[danb35]

Will I be able to browse the files on the back up server

yes.

 

[Jerami1981]

I got everything set up and let my first replication start. I quickly noticed it was only transferring around 15 MB/s. The weird thing is, I did a few different tests after seeing this. Server A can copy to my desktop, or to Server B via my desktop at around 110 MB/s now. However, Server B, as well as my desktop can transfer to Server A at around 400-500 MB/s. I can also transfer to Server B from my desktop at roughly 400-500 MB/s. Is there any kind of bandwidth testing ability within FreeNAS. I've used iPerf on windows to test, as well as simple file transfers and watched the transfer window. I did direct copies from Server A to Server B via my desktop 2 weeks ago and it averaged around 300 MB/s. At that time both servers were 1 12 Disk Vdev Z2. Now Server 2 has been changed over to 2 6 Disk Vdev Z2, which is why I need to copy everything over again. I can't imagine how changing server B's Vdevs would impact Server A's outbound transfer speeds to a completely independent device (my desktop). Any thoughts on how to diagnose this? At 15 MB/s, its going to take a very long time to transfer nearly 40 TB of data. I would expect the replication task to at least get the 110 MB/s that a simple Windows file transfer from Server A to Server B can accomplish.