Moving 20+tb of data from one FreeNAS box to another

[Demo85]

Want to say upfront i'm not a super literate person with this stuff. I run two FreeNAS boxes that I access from various Win10 machines, I want to move 20tbs of data from one machine to the other and Win10 is being kinda bad about it with timing out connections and whatnot. Don't wanna tie up my main machine for all the time it take to move the files either. I'm looking for another way to move the files. I even set up a Ubuntu VM on my main NAS to see if it was better about moving files around but I can't seem to get Ubuntu to connect to the network to map the drives. I'm just about at wits end. Does anyone know of a way to move these files over in a easy way that won't tie up my main machine for days and avoid the grief from Windows tossing "are you sure" prompts in the middle of the transfer that by the time I click yes my NAS timed out the connection or whatever the heck it does that Windows doesn't like that makes the transfer be nulled out?

 

[Mannekino]

You can create a snapshot of the pool(s) you want to move and then move them to the other FreeNAS server with this command

zfs send <snapshot_name> | ssh <target_server> zfs receive -F <target_dataset>

I'm currently experimenting with this myself but I'm getting about half the speed with the ZFS replication compared to copying files over SMB. Planning to make a thread about that later.

I think this is the easiest way to go about it, I got this information from here.

 

[Demo85]

You can create a snapshot of the pool(s) you want to move and then move them to the other FreeNAS server with this command

zfs send <snapshot_name> | ssh <target_server> zfs receive -F <target_dataset>

I'm currently experimenting with this myself but I'm getting about half the speed with the ZFS replication compared to copying files over SMB. Planning to make a thread about that later.

I think this is the easiest way to go about it, I got this information from here.

Gonna have a real dumb question but when it says target data set what am I filling that with?

 

[Mannekino]

Well it could technically be any dataset on your target FreeNAS server. If it already exist you need to put a -F after the receive part. If the child dataset doesn't exist on the target server it will be automatically created.

So in my example, I have a test FreeNAS server with this setup, this is my target server.

I just used the following command to move a snapshot I created on the source server.

zfs send data/testing@manual-20190210 | pv | ssh freenas-test zfs receive -F data/testing

After this was done I could access the data from my /data/testing dataset on the source server on the new server.

I could have also used this command, and then it would have created a new child dataset on the target server.

zfs send data/testing@manual-20190210 | pv | ssh freenas-test zfs receive data/testing/new_dataset

The pv bit will show the transfer speed you are getting. Moving 20 TB of data will take quite some time.

I've enabled SSH access for root to do this for the time being while I'm learning how for a better way to go about this.

Another option is doing it entirely through the UI with replication tasks, I've used this method myself for moving the data from my old FreeNAS to my new FreeNAS a couple months ago.

https://www.ixsystems.com/documentation/freenas/11.2/tasks.html#replication-tasks

I'm not exactly sure how I did that, it was a while ago, but I believe I setup the replication task as described in the manual and then manually created a single snapshot of the dataset I wanted to replicate. After the snapshot was created the task was triggered and the data began to replicate.

 

[Mannekino]

@Demo85

Did you manage to get anywhere with my information? If you're running into slow transfer speeds you might wanna take a look at my solution in this thread.

https://forums.freenas.org/index.ph...alf-of-smb-transfer-speeds.73664/#post-510948

 

[Mlovelace]

Want to say upfront i'm not a super literate person with this stuff. I run two FreeNAS boxes that I access from various Win10 machines, I want to move 20tbs of data from one machine to the other and Win10 is being kinda bad about it with timing out connections and whatnot. Don't wanna tie up my main machine for all the time it take to move the files either. I'm looking for another way to move the files. I even set up a Ubuntu VM on my main NAS to see if it was better about moving files around but I can't seem to get Ubuntu to connect to the network to map the drives. I'm just about at wits end. Does anyone know of a way to move these files over in a easy way that won't tie up my main machine for days and avoid the grief from Windows tossing "are you sure" prompts in the middle of the transfer that by the time I click yes my NAS timed out the connection or whatever the heck it does that Windows doesn't like that makes the transfer be nulled out?

If both freeNAS servers are on the same network, use netcat to send the data. Here is a thread that discusses what to do.

https://forums.freenas.org/index.php?threads/replication-stream-compression.73563/post-509985

 

[Demo85]

@Demo85

Did you manage to get anywhere with my information? If you're running into slow transfer speeds you might wanna take a look at my solution in this thread.

https://forums.freenas.org/index.ph...alf-of-smb-transfer-speeds.73664/#post-510948

If both freeNAS servers are on the same network, use netcat to send the data. Here is a thread that discusses what to do.

https://forums.freenas.org/index.php?threads/replication-stream-compression.73563/post-509985

I kinda side stepped the issue, once I got my new mobo/cpu for the NAS I slapped in the hard drives and reinstalled FreeNAS then restored the data pool and it just worked.