Fastest way to copy (or move) files between shares

[macmuchmore]

Hello - I have 2 shares (each pointing to a different dataset). I would like to move all the files & folders from one share to the other without moving the files over the network (slow). I have seen where it has been suggested to use the cp or mv command. I do not have a lot of experience using the command line (though I am comfortable doing so) and am hoping someone could provide guidance... any help would be appreciated.

 

[JohnDigital]

The most straightforward way to do this is likely mv. Issue this command at the TN host terminal. Adjust command for your actual use case.

mv /mnt/tank/source /mnt/tank/destination

However it wont tell you progress or anything. So a fancier way is to go like this. Again adjust your use case. The command is included with the --dry-run flag.. When your sure youve got it right remove the --dry-run.

rsync -avzhP --remove-source-files /mnt/tank/dataset1 /mnt/tank/dataset2 --dry-run

Then after you are satisfied its doing what you need, run the command without the --dry-run flag, youll need to do this to remove all the empty dierectories (if any).

find /mnt/tank/dataset1 -type d -empty -delete

Good luck.

 

[Pitfrr]

You could also use mc in the terminal. It gives you an interface and works even with remote systems.

 

[Basil Hendroff]

If what you're effectively doing is trying to rename the original dataset, the following approach will not move any files at all:

1. Remove the share attached to the dataset.
2. Rename the dataset e.g. if your pool is named tank then zfs rename tank/old_dataset_name tank/new_dataset_name
3. Set up the share against the renamed dataset.

 

[macmuchmore]

The most straightforward way to do this is likely mv. Issue this command at the TN host terminal. Adjust command for your actual use case.

mv /mnt/tank/source /mnt/tank/destination

However it wont tell you progress or anything. So a fancier way is to go like this. Again adjust your use case. The command is included with the --dry-run flag.. When your sure youve got it right remove the --dry-run.

rsync -avzhP --remove-source-files /mnt/tank/dataset1 /mnt/tank/dataset2 --dry-run

Then after you are satisfied its doing what you need, run the command without the --dry-run flag, youll need to do this to remove all the empty dierectories (if any).

find /mnt/tank/dataset1 -type d -empty -delete

Good luck.

This is very helpful - thank you. Will this preserve the directories in the share?

 

[macmuchmore]

If what you're effectively doing is trying to rename the original dataset, the following approach will not move any files at all:

1. Remove the share attached to the dataset.
2. Rename the dataset e.g. if your pool is named tank then zfs rename tank/old_dataset_name tank/new_dataset_name
3. Set up the share against the renamed dataset.

Thank you for this - unfortunately I my goal is to move the files to a different share (that is assigned a specific dataset).

I do not think my original configuration for datasets and shares is correct and I am fearful the data is in jeopardy due to my misconfiguration (I am not sure if the issue is the way I set them up originally [probably], or if the transition to the 12.0 release [far less likely] had something to do with it). The issue is that my original SMB share is pointing to the pool, and not a dataset.

Because of this, when I open the share, I see the iocage dataset, and one other dataset (that is not in use). On my desktop, I added folders to the share (at the root) and populated them with a large amount of data (media files). My concern is that these files and folders are not part of any dataset, but are just "in the pool" so to speak. I assume it is better to have the data in a dataset, and not in the pool directly - in addition, I would prefer the share to only display the data and not the iocage dataset (or any other dataset for that matter).

I have created a new dataset and share, and would like to move the actual files and folders from the old "pool" share to the new "dataset" share. I can do it directly on my computer, but it will take hours. I am hoping to be able to move them directly on the server, without having to do it over the network.

 

[SweetAndLow]

You should just look at mc( midnight commander) it's a cli tool that kind of had a gui and is built into freenas.

 

[macmuchmore]

The most straightforward way to do this is likely mv. Issue this command at the TN host terminal. Adjust command for your actual use case.

mv /mnt/tank/source /mnt/tank/destination

However it wont tell you progress or anything. So a fancier way is to go like this. Again adjust your use case. The command is included with the --dry-run flag.. When your sure youve got it right remove the --dry-run.

rsync -avzhP --remove-source-files /mnt/tank/dataset1 /mnt/tank/dataset2 --dry-run

Then after you are satisfied its doing what you need, run the command without the --dry-run flag, youll need to do this to remove all the empty dierectories (if any).

find /mnt/tank/dataset1 -type d -empty -delete

Good luck.

Hi - so I tried the mv command and I received the error below - I apologize if I am being a noob about this...

Code:

root@PUG[~]# mv /mnt/Pug/Software /mnt/Pug/PUG2/Software

mv: rename /mnt/Pug/Software to /mnt/Pug/PUG2/Software/Software: Operation not supported

root@PUG[~]#

You should just look at mc( midnight commander) it's a cli tool that kind of had a gui and is built into freenas.

I am looking into it - I hope it does what I a need...

 

[macmuchmore]

You should just look at mc( midnight commander) it's a cli tool that kind of had a gui and is built into freenas.

So I have been trying it and it seems to be very buggy - I select a file to copy and tab to enter and press enter and it kind of freezes.... very frustrating, even though on a couple of files it has finished the copy, even though mc seems to have crashed/frozen.

Basically, this is the functionality I am looking for, but if anyone knows a tool that is easier/less "challenging", please let m know. I thought to try Cyberduck, but I am not sure it will do the copies locally on the server or require them to be done over the network (sftp).

 

[SweetAndLow]

So I have been trying it and it seems to be very buggy - I select a file to copy and tab to enter and press enter and it kind of freezes.... very frustrating, even though on a couple of files it has finished the copy, even though mc seems to have crashed/frozen.

Basically, this is the functionality I am looking for, but if anyone knows a tool that is easier/less "challenging", please let m know. I thought to try Cyberduck, but I am not sure it will do the copies locally on the server or require them to be done over the network (sftp).

How much data is there?

I can't really figure out why a rename operation isn't supported, that's an error I didn't expect. If the data is smallish you can just do a cp -R to copy the files then after it finishes you can verify it worked then remove the original file.

 

[macmuchmore]

How much data is there?

I can't really figure out why a rename operation isn't supported, that's an error I didn't expect. If the data is smallish you can just do a cp -R to copy the files then after it finishes you can verify it worked then remove the original file.

One share is over 7TB (media files). Doing the copy over the network takes 20+hours, which is why I would rather it be done directly on the server, bypassing the network. I tried Double commander, but it uses the network the same if I were to do the copy on my desktop.

 

[JohnDigital]

Hi - so I tried the mv command and I received the error below - I apologize if I am being a noob about this...

Code:

root@PUG[~]# mv /mnt/Pug/Software /mnt/Pug/PUG2/Software

mv: rename /mnt/Pug/Software to /mnt/Pug/PUG2/Software/Software: Operation not supported

root@PUG[~]#

My thought is that this path does not exist?? /mnt/Pug/PUG2/Software Is there a folder or dataset inside /mnt/Pug/ called PUG2?

Also this will have you end up with two "software" folders. Try this.

mv /mnt/Pug/Software /mnt/Pug/PUG2/

 

[SweetAndLow]

My thought is that this path does not exist?? /mnt/Pug/PUG2/Software Is there a folder or dataset inside /mnt/Pug/ called PUG2?

Also this will have you end up with two "software" folders. Try this.

mv /mnt/Pug/Software /mnt/Pug/PUG2/

just did a similar test and yeah something in the destination path didn't exist causing that error. Make sure your destination path is correct and exists.

 

[macmuchmore]

My thought is that this path does not exist?? /mnt/Pug/PUG2/Software Is there a folder or dataset inside /mnt/Pug/ called PUG2?

Also this will have you end up with two "software" folders. Try this.

mv /mnt/Pug/Software /mnt/Pug/PUG2/

I apologize for not previously stating clearly what I am trying to do (which I should have in the first post - but it was late and I was super tired).

Yes, the pool is named Pug (I have a share called Pug, with path /mnt/Pug - which is what I want to change as I do not like having a share that points directly to the pool and not a dataset). The new dataset is inside Pug, named PUG2. The share PUG2 has the path of /mnt/Pug/PUG2. I am trying to move all the files and directories (and sub-directories) in the Software folder located in the Pug share to the Software folder in the PUG2 share. I have already created the Software folder on the PUG2 share.

With this in mind, will the suggestion to use

Code:

mv /mnt/Pug/Software /mnt/Pug/PUG2/

create a new Software directory in the PUG2 share (meaning I would need to delete the Software directory I have already created on that share)?

Also, with my shares and datasets being named the same (I thought might be less confusing), does the mv command refer to the share path or the dataset path (in case in the future I have a dataset named differently from the share)?

 

[JohnDigital]

The above commands are perfect. Do it for each folder you want inside of PUG2. (Notice the trailing slash on the destination directory, its important.)

mv command refers to the absolute path, you can name the share whatever you want.

Also yes will "create" software inside of the PUG2 directory.

This process should be almost instant.

 

[macmuchmore]

The above commands are perfect. Do it for each folder you want inside of PUG2. (Notice the trailing slash on the destination directory, its important.)

mv command refers to the absolute path, you can name the share whatever you want.

Also yes will "create" software inside of the PUG2 directory.

This process should be almost instant.

Excellent - I will try it and report back. But let me create a similar scenario and validate my understanding of what you mean by absolute path.

Pool = Pool1
Dataset = Dataset1
Share1 = Share1
Share2 = Share2
Path for Share1 = /mnt/Pool1
Path for Share2 = /mnt/Pool1/Dataset1
File I am trying to move = Software (which is located in the Pool as above)

In this scenario, which is the correct option:

Code:

mv /mnt/Pool1/Software /mnt/Pool1/Share2/

Code:

mv /mnt/Pool1/Software /mnt/Pool1/Dataset1/

Code:

mv /mnt/Share1/Software /mnt/Pool1/Share2/

Code:

mv /mnt/Share1/Software /mnt/Pool1/Dataset1/

Code:

mv /mnt/Share1/Software /mnt/Share1/Dataset1/

Code:

mv /mnt/Share1/Software /mnt/Share1/Share2/

 

[JohnDigital]

lol, #2

 

[JohnDigital]

In number 2 the dataset is also the share or the "folders" in the dataset can be shared..

 

[JohnDigital]

By absolute path I just mean the directory according to TrueNAS, the share name is just that-- a name so you recognize it.

 

[macmuchmore]

How will I know the command has been completed - will the terminal go back to a regular prompt? Also, will I need to delete the software folder on the pool? Lastly - I am running the command now, but it looks like it is copying the files, not moving them.. is this because it won't delete them until the copy is finished?