Copying data from one NAS to another

[iankearns]

I have two FreeNAS servers running FreeNAS-9.2.1.7 and intend of on moving the internal gubbins from one of the servers into a new case with greater drive capacity so I can eventually amalgamate all drives into one chassis.

Once I put the drives from the one server into the new case then I will add an additional drive to then start copying the data from server 2 however when I have done this in the past then the transfer of about 15Gb has been painfully slow as I administer the servers from a Windows PC / browser therefore my question is "what is the fastest method to trasfer data from one FreeNAS server to another?"

TIA

 

[Evertb1]

I am not sure how you have been transferring that 15 GB but that should not have taken a lot of time. These days 15 GB is not much data at all. Is your network free from technical issues? Anyways if I were you I would look into making Rsync working for you to tranfer data from one FreeNAS box to another. It's reliable, very well supported by FreeNAS (I know from experience) and fast enough to do a good job. You can also look into snapshot replication but personally I would not bother for a one time job ( I use that as my backup mechanism).

 

[iankearns]

Apologies I meant 15Tb. I have just found an article on Rsync so will look into that further. Thanks for your reply and advice

I am not sure how you have been transferring that 15 GB but that should not have taken a lot of time. These days 15 GB is not much data at all. Is your network free from technical issues? Anyways if I were you I would look into making Rsync working for you to tranfer data from one FreeNAS box to another. It's reliable, very well supported by FreeNAS (I know from experience) and fast enough to do a good job. You can also look into snapshot replication but personally I would not bother for a one time job ( I use that as my backup mechanism).

 

[Evertb1]

Rsync runs between the both FreeNAS boxes where one acts as the client and the other as the server. You configure and initialize it from the GUI of the boxes (there is a lot of documentation available in and out of the forum). Besides of that, there are plenty YouTube videos available like this one How to setup RSync on FreeNAS (RSync between 2 FreeNAS Servers). It is realy not hard to set up but you need to "study" a thing or two. By the way: no mather how you look at it 15 TB will take some time unless you have a 10g network.

 

[Chris Moore]

If both systems are FreeNAS, rsync is not the fastest option.

You could give more details about the system to help us help you.

Please read:

Forum Guidelines

Forum Rules

Welcome to the TrueNAS Community! Updated 11/28/2023 First and foremost I want to thank you for joining the TrueNAS community! My name is JoshDW19 and I'm the Community Manager here at TrueNAS. The TrueNAS Community is a friendly place to discuss open source software, technology, and products...

www.ixsystems.com

 

[Chris Moore]

FreeNAS servers running FreeNAS-9.2.1.7

You should probably upgrade to the latest version of FreeNAS. Do you have a significant reason for not upgrading?

 

[Chris Moore]

ZFS send and receive with netcat instead of ssh (for trusted networks)
https://www.ixsystems.com/community/threads/replication-stream-compression.73563/post-509985

 

[iankearns]

Hi Chris

It isnt a complex setup as it is only in use at home for a media library and family docs and photos.

Two desktops PC with 4 drives each are being condensed into rack mounted case which has space for 8+ drives.

I will take the motherboard, CPU and memory from the one desktop to go into the rack mounted case and install a SATA expansion card and add a new 6TB drive. I will then look to start transfering data from PC2 to the new empty drive.

I estimate I will be able to take all the data from PC2 onto the existing space of the original 4 drives of PC1 (now in the new case) and the new 6TB drive. Then I can flatten PC2 and use the scrubbed drives in the rack mounted case and expand that NAS as the only one.

I have been copying the data previously using a Windows based client and simply copying folders between locations. I had also tried a command line tool similar to rsync but again from a remote client machine so thought the suggestion of doing it from NAS to NAS would be quicker.

Any help or suggestion is most certainly appreciated

If both systems are FreeNAS, rsync is not the fastest option.

You could give more details about the system to help us help you.

Please read:

Forum Guidelines

Forum Rules

Welcome to the TrueNAS Community! Updated 11/28/2023 First and foremost I want to thank you for joining the TrueNAS community! My name is JoshDW19 and I'm the Community Manager here at TrueNAS. The TrueNAS Community is a friendly place to discuss open source software, technology, and products...

www.ixsystems.com

 

[iankearns]

Hi Chris

I had built a brand new server using the 6TB new drive as a starting point and all seemed to be fine but when trying to access from the media server or any of the clients (Windows and Android) I was get permission denied even though I was setup for guest access. I gave up in the end and continued with the older version which didnt complain.

You should probably upgrade to the latest version of FreeNAS. Do you have a significant reason for not upgrading?

 

[Chris Moore]

I had built a brand new server using the 6TB new drive as a starting point and all seemed to be fine but when trying to access from the media server or any of the clients (Windows and Android) I was get permission denied even though I was setup for guest access

You may have missed a crucial step. There are some fine details involved in getting the sharing configured. Here is a link to a guide:

fester112:intro [danb35's Wiki]

www.familybrown.org

 

[Chris Moore]

I worry about your description of the new system having just a single drive. Is that a temporary space just during the data transfer?

 

[iankearns]

Am happy to give it another go as I bought a new (to me) motherboard, processor and 16Gb memory for the new case / chassis originally so if I can put that to good use then thumbs up.

You may have missed a crucial step. There are some fine details involved in getting the sharing configured. Here is a link to a guide:

fester112:intro [danb35's Wiki]

www.familybrown.org

 

[iankearns]

The use of the single drive iwill be included in a new pool and then data copied to it from existing datasets and the other FreeNAS server. As the drives are emptied then they will be scrubbed, the original data removed and then reused in the new pool.

So start with one 6TB drive in PC2 >
Copy data from pool 1 (comprising of 4TB and 3TB drives but not full) in PC1 to the single drive in PC2 on 'new_pool'.
When copy is complete then scrub delete pool1 and scrub the two drives.
Relocate the two drives (4TB and 3TB) into PC2 and expand 'new_pool' to 13TB (original 6TB plus 4TB and 3TB)
Continue to do this until all drives and data are transferred

I worry about your description of the new system having just a single drive. Is that a temporary space just during the data transfer?

 

[Chris Moore]

So start with one 6TB drive in PC2 >
Copy data from dataset 1 (comprising of 4TB and 3TB drives but not full) in PC1 to the single drive in PC2 on 'new_dataset'.
When copy is complete then scrub delete dataset1 and scrub the two drives.
Relocate the two drives (4TB and 3TB) into PC2 and expand 'new_dataset' to 13TB (original 6TB plus 4TB and 3TB)
Continue to do this until all drives and data are transferred

So, you are not using any redundancy. A single drive failure would take all the data on all the drives. That is not a good idea and it prevents ZFS from doing the thing it is usually desired to do.
Is this your intention? I can point you to some resources that can help you understand ZFS better.

 

[blueether]

I'm with Chris on how it looks like you are going to set up the new NAS.

Defentally read up on ZFS/pools/vdevs and redundency.

How much data do you currently have?
What disks do you have? (Size etc)
What is the current pool layouts?

As a sine note you could just install the old drivs in the new NAS and copy to a new pool (still using ZFS send/recive)

 

[iankearns]

I have never really run with any redundancy (well thats not true, as explained below) cause 95% of my data i sonly media such as film and TV programmes so if they go then it would not be a horrendous hardship. The ISO's that I have are most probably being held onto out of lazyness and I would not be suprised if I dont use them again and they are easily downlaodable (Linux distros etc). The small volume of information I see as important such as family photos and documentation is also held in Google Drive; the local copy is for convenience.

I know redundancy is important to some and it may be something I consider more in the coming year however as the overall drive space I have is approx 30TB made up of 6,4 and 3TB drives then to provide the necessary hardware to provide that redundancy would be an expense I couldnt take on at the moment; in 2020 then I may look at buying newer drives and balancing it out with some RAID mirroring or disk redundancy.

So, you are not using any redundancy. A single drive failure would take all the data on all the drives. That is not a good idea and it prevents ZFS from doing the thing it is usually desired to do.
Is this your intention? I can point you to some resources that can help you understand ZFS better.

 

[iankearns]

You say about installing the drives to a new NAS ; so if I took the drives that are included in a pool and transfer them to a new FreeNAS build then the dataset would automatically bee seen and the data available??

I'm with Chris on how it looks like you are going to set up the new NAS.

Defentally read up on ZFS/pools/vdevs and redundency.

How much data do you currently have?
What disks do you have? (Size etc)
What is the current pool layouts?

As a sine note you could just install the old drivs in the new NAS and copy to a new pool (still using ZFS send/recive)

 

[Chris Moore]

dataset

Your use of terms is incorrect and will likely cause some confusion. I would ask, please take a look at these links to get the correct words.

Slideshow explaining VDev, zpool, ZIL and L2ARC
https://www.ixsystems.com/community...ning-vdev-zpool-zil-and-l2arc-for-noobs.7775/

Terminology and Abbreviations Primer
https://www.ixsystems.com/community/threads/terminology-and-abbreviations-primer.28174/

Overview of ZFS Pools in FreeNAS from the iXsystems blog:
https://www.ixsystems.com/blog/zfs-pools-in-freenas/

 

[Chris Moore]

would automatically bee seen and the data available??

If you move a set of drives from an older FreeNAS to a new build, you would need to import the ZFS pool contained on those drives. It would not be automatic. Once the storage pool contained on the drives was imported, you would have access to any dataset inside that pool. The new FreeNAS would also offer you the option of upgrading the version of ZFS on disk and you would still be able to add an additional drive to the pool. The difference is that the data does not get balanced between the disks. New writes would be spread across all available space (all disks) but the data already in the pool would remain in the existing physical disks.

 

[iankearns]

Understood - was using dataset as a description rather than the term as used in FreeNAS. Have addressed and believe a (z)pool (which is the grouping of VDevs) is more appropriate

Thanks

Ian

Your use of terms is incorrect and will likely cause some confusion. I would ask, please take a look at these links to get the correct words.

Slideshow explaining VDev, zpool, ZIL and L2ARC
https://www.ixsystems.com/community...ning-vdev-zpool-zil-and-l2arc-for-noobs.7775/

Terminology and Abbreviations Primer
https://www.ixsystems.com/community/threads/terminology-and-abbreviations-primer.28174/

Overview of ZFS Pools in FreeNAS from the iXsystems blog:
https://www.ixsystems.com/blog/zfs-pools-in-freenas/