High performance migration tool - NetApp to FreeNAS

[Chin-Fah HEOH]

Hi

I am looking for a high performance migration software - hopefully free but not too expensive if commercial, to migrate about 120TB from an old NetApp filer to FreeNAS. I could think of an NFS swing server with 2 x 10GbE networking but given the overhead of NFS, we have a problem. Or I can use rsync which is more efficient that NFS. But to migrate the 120TB will take days.

Hopefully get some good advice here. Thanks in advance.

/CF

 

[HoneyBadger]

Could you not mount the NFS export off the NetApp directly on your FreeNAS machine and copy it that way?

The logistics of a migration are often more complex than just "Ctrl-C, Ctrl-V, wait" especially if access needs to be mostly uninterrupted. Is there potential for a momentary downtime? How fast is the data changing on the source, or is it mostly/completely static? Are there programs accessing the old export, or just end-users?

 

[blanchet]

Maybe you can use several clients to speed up the transfer.

We have copied 100 TB of cold data from our EMC Isilon cluster to a FreeNAS server last month.

We have used 4 NFS clients with 10GbE networking in parallel to maximize the bandwidth from the source:

• each client was connected in NFSv3 to a different Isilon node and to the FreeNAS.
• each client runs rsync between the two NFS shares
• there were only big files ( > 4 GB )
• ZFS compression was disabled

I was very impressed by the performance of the FreeNAS server ( a basic server with 2 x 12-disks raidz3 )
zpool iostat tank1 1 reports 1GB for write bandwidth. Obviously the FreeNAS server could not serve anybody else during the transfer. Finally the transfer has taken less than 2 days.

 

[Chin-Fah HEOH]

Could you not mount the NFS export off the NetApp directly on your FreeNAS machine and copy it that way?

The logistics of a migration are often more complex than just "Ctrl-C, Ctrl-V, wait" especially if access needs to be mostly uninterrupted. Is there potential for a momentary downtime? How fast is the data changing on the source, or is it mostly/completely static? Are there programs accessing the old export, or just end-users?

Yes, I could do it that way via an NFS swing server. Copy between the NetApp mountpoint and the FreeNAS mountpoint. I am also aware of the constant changes to the NetApp side, as well as migration of the mountpoints on the NFS clients but for now, it is about the speed of migration.

We are looking at about 2 days windows. Assuming if I use 2 x 10GbE. I could get about 116-120MB/sec for one 10GbE. Let's say 2 will be 220-240MB/sec. That would be around 19-20TB/day (assuming no mishaps and sustained throughput). We are looking at 5-6 days of transfer.

I am looking for something which can do the transfer faster. I am thinking of commercial folks like DataDobi or Interlock Tech.

Thanks for your input.

 

[Chin-Fah HEOH]

Maybe you can use several clients to speed up the transfer.

We have copied 100 TB of cold data from our EMC Isilon cluster to a FreeNAS server last month.

We have used 4 NFS clients with 10GbE networking in parallel to maximize the bandwidth from the source:

• each client was connected in NFSv3 to a different Isilon node and to the FreeNAS.
• each client runs rsync between the two NFS shares
• there were only big files ( > 4 GB )
• ZFS compression was disabled

I was very impressed by the performance of the FreeNAS server ( a basic server with 2 x 12-disks raidz3 )
zpool iostat tank1 1 reports 1GB for write bandwidth. Obviously the FreeNAS server could not serve anybody else during the transfer. Finally the transfer has taken less than 2 days.

This is good insight. Looks similar to my situation here. Thank you very much

 

[blanchet]

You can also imitate the behavior of SnapMirror when migrating data to a new Netapp filer.

• take a snapshot of the Netapp volume
• copy the snapshot data to the destination
• Run a final rsync to transfer the data that have changed since the Netapp snapshot

 

[styno]

• Run a final rsync to transfer the data that have changed since the Netapp snapshot

Or use NetApp's own XCP tool as they surly intended for ppl to use it for NetApp-to-Any migrations

 

[Chin-Fah HEOH]

You can also imitate the behavior of SnapMirror when migrating data to a new Netapp filer.

• take a snapshot of the Netapp volume
• copy the snapshot data to the destination
• Run a final rsync to transfer the data that have changed since the Netapp snapshot

Thank you. This is good knowledge to consider

 

[Chin-Fah HEOH]

Or use NetApp's own XCP tool as they surly intended for ppl to use it for NetApp-to-Any migrations

Thank you. Good stuff. I go check it out.