Best way to merge multiple datasets with media to a single location

[devsa]

Hello everyone, greetings!!

I have been using freenas/truenas for a while and this forum has always been able to answer whatever questions I had so far. However, I recently setup a new server with bigger hard drives which is up and running.

Here is the issue I am facing:
Since I had smalled hard drives, I had all my movies, songs and tv under folder name "media" and somewhat same directory structure with folders named 'movies' 'tv' 'songs' etc. but I had 3 such folders ('media') residing on 3 different datasets on different pools. Now that I have a bigger pool, I would want to merge all three 'media' folders with their sub-folders to a single dataset but how?

I can create a new dataset and then manually copy all the folders from different locations to it. This will take a long time since I have 2+tb of data and also my biggest concern is that sometimes the copy command spits out an error saying long file name or path or something in those lines.

Or I can use the replicate task but I am not too sure how it would play since the folder names are same so if I would replicate all three datasets to a single dataset, would it merge the sub-folders or files in them since they would be different names or will a new dateset replication overwrite the previous operation?

 

[JohnDigital]

I think you'll want to use rsync on the server itself. You dont want to do it from Windows as a middle man. It should whip along pretty quickly. It will skip duplicate files. It will not overwrite anything, just add to or skip. I would do something along the lines of this from an SSH session to the host. Youll obviously change these so they are correct for your system.

rsync -avh /mnt/sourcepool1/dataset/media /mnt/destpool/dataset/media
rsync -avh /mnt/sourcepool2/dataset/media /mnt/destpool/dataset/media
rsync -avh /mnt/sourcepool3/dataset/media /mnt/destpool/dataset/media

Also with rsync you can essentially test the command by appending --dry-run to the command. It wont actually do any copying but will basically let you know what its going to do, if your command is correct or whatnot..

Then once you are happy remove it and go. 2tb will be about 3 hours at 100MBs and youll likely go alot faster than that. See the rsync manpage, there are tons of options. Its also nice because you'll see the progress and speed.

 

[devsa]

John, thank you for your reply. After posting here and waiting for a day I did use rsync with log file output just to make sure all files go through. Once rsync was done, i used a folder compare tool to double check and then deleted the old datasets.
But I appreciate that you responded. Thank you for that.