Copy or move from one dataset to another

[Zaaphod]

I would like to know the best way of copying or moving either individual files or entire directories from one dataset to another directly on the FreeNAS system.

I don't care for the idea of mapping network drives to this for the following reasons:
1. desktop systems do not have ECC RAM and if I am copying data through the desktop system, all bytes of data must go through the desktop RAM to accomplish this
2. it's slower to send everything over the network and back again
3. it ties up the desktop system until it's done

I realize I can do this via the command line by using SSH and use CP or MV commands, but this is very tedious especially if you are trying to re-organize an entire directory structure or if you are dealing with a lot of long file names. So I'm wondering if there is some easy GUI way to manipulate files directly on the server without them going through the desktop system? I'm basically wanting to do something like "browse datastore" on an ESXi server, where I can open two browsers and copy or move files from one to the other using a GUI interface and not need to type all the long paths and file names.

Any ideas?

 

[Mlovelace]

If you want to server side move files you have to use the cli.

 

[Zaaphod]

Is it possible to somehow use something like midnight commander in an SSH shell ? if so, how would I install it?

 

[Jailer]

You're making this way more complicated than it needs to be. Use the CLI and just cp or mv and move an entire directory, all the files within the directory will move too.

 

[Zaaphod]

I understand that.. the problem is I have 1000 files with long file names and long paths also with long file names that have to all go into different places... there are no good widlcards I could use so it's just way too much typing.

 

[Glorious1]

I don't know how big a risk it really is to do it remotely from your desktop computer. If the CLI approach is too tedious, I would go that way. Just make sure you do it over ethernet rather than wifi.

 

[SweetAndLow]

Can you give an example of what you want to do. This will be simple to just use the cli.

 

[Zaaphod]

I just want to do file manager type stuff... but the long filenames and re-arranging the path structure makes it difficult to do on the command line

I want to sort out a directory that full of a huge mixed up mess of files into nice directory structure... so mostly want to move files but they aren't all going to the same destination.. it will be a ton of stuff like

mv /mnt/Vol_8TB/Username/old folder where Stuff was a huge mess/Presentation to Customer with their Complete Name 12-30-98 about whatever the project name is but sometimes words are abbreviated.AVI /mnt/Vol_8TB/Projects/Customer ID - Complete Customer Name/whatever the project name is/Presentations/12-30-1998.avi

there is no pattern to this.. so the next file could be the source does not

mv /mnt/Vol_8TB/Username/old folder where Stuff was a huge mess/Customer with and Abrev. NM 99 project abbreviated sub assy xyz.dwg /mnt/Vol_8TB/Projects/Customer ID - Complete Customer Name/whatever the project name is/CAD Drawings/Sub-Assembly XYZ Rev 15.3 01-15-1999 .dwg

I can't be typing all that... one typo and files are in the wrong place, or it can't even find the file because I didn't capitalize something or I couldn't tell if a character was a lowercase L or an upper case I since they are both the same in windows. Some stuff is in folders some isn't It's been done wrong for a long time. It needs to be organized, but it's a lot of data and I would really rather not have it all go out the network then back to the server when it should just stay on the server. There are a lot of big files and it will happen way faster if it never goes over the network, and it won't be occupying the host computer that issued the command while the server completes the task.

I can get the source name with a copy > paste to notepad > change all the \'s to /'s > stick /mnt/Vol_8TB/ in front of it > copy it to the clipboard again > paste into shell... but dang it's too tedious. If I had something like midnight commander, which isn't really GUI, but a text based way I could pick out the file from a list, and move it to another location it would greatly simplify the task... I've never used midnight commander but it looks like a Linux version of the old PCSHELL for DOS which would greatly simplify this task and not require very much more resources than the shell. Doing the file management on the server side would be so much faster since the server is just doing a move, not really a copy then delete that is called 'move' which is what seems to happen if I do it over a network share. It would also be safer for the data as well as a bad ram chip on the desktop computer could corrupt everything as it is transferred from the server into the desktop RAM then back over the network back to the server.

It seems like someone should have worked out remote file management so everything doesn't have to travel back to a local computer then back to the server to do this kind of thing.

 

[Glorious1]

OK, why don't you do all your rearranging etc. within the dataset where the stuff is now, on your desktop gui with mounted shares. It shouldn't have to go through your desktop. Then go into the CLI and copy or move the whole cleaned-up bunch of files to the other dataset where you want it, with one or a few simple commands. Then if necessary you can mount that dataset to do any final integration with existing file structure via GUI.

 

[anodos]

OK, why don't you do all your rearranging etc. within the dataset where the stuff is now, on your desktop gui with mounted shares. It shouldn't have to go through your desktop. Then go into the CLI and copy or move the whole cleaned-up bunch of files to the other dataset where you want it, with one or a few simple commands. Then if necessary you can mount that dataset to do any final integration with existing file structure via GUI.

If you are using samba with a windows 8 or newer client, then moving / copying within the same share will happen server-side.

 

[Marcet]

If you rearrange first and then decide to go for CLI, then use rsync, it will be faster that scp ;)

 

[Zaaphod]

OK, why don't you do all your rearranging etc. within the dataset where the stuff is now, on your desktop gui with mounted shares. It shouldn't have to go through your desktop. Then go into the CLI and copy or move the whole cleaned-up bunch of files to the other dataset where you want it, with one or a few simple commands. Then if necessary you can mount that dataset to do any final integration with existing file structure via GUI.

Using the mounted shares seemed like it was making everything go through my client computer. Even moving files within the same share were taking forever.. but I did some tests with the command line and doing moves on FreeNas take way longer than doing moves on my windows computer. Perhaps it's something to do with the way ZFS works that is making it take so long. In windows a move doesn't really move anything at all, it just changes the name of the directory name of the file, it's pretty instant, even on the slowest computers. ZFS must be doing something more complicated because it's taking a lot longer. I do notice even on the command line small files move fast, larger files move slower. I don't think it's doing a move at all, I think it's copying the file then deleting the old one.

If you are using samba with a windows 8 or newer client, then moving / copying within the same share will happen server-side.

I tested this on a few large files with a Windows 10 client and you are correct! They key here is the moving / copying needs to be within the same share as you stated, otherwise it is bringing everything through the client computer. All my datasets are sub-sets of a master dataset, so I just shared the master dataset and I can have a single share to re-arrange with.

If you rearrange first and then decide to go for CLI, then use rsync, it will be faster that scp ;)

I've been meaning to look into rsync I guess that is similar to robocopy. I normally use something like robocopy L:\Data W:\New /MIR /V /NP /NDL /copy: DAT /dcopy:T /r:2 /w:2 /log:log.txt /tee I like having a log and I like keeping my file and directory attributes, after all having a file creation date of the date the file was copied is useless.. it should be the original date the original file was created. I also like the log in case something gives an error.
I'm wondering if using robocopy to move files within the same share would also cause it to happen server-side?

 

[Marcet]

I'm wondering if using robocopy to move files within the same share would also cause it to happen server-side?

Sorry, I can't tell you that. My knowledge of robocopy is very limited.
Try to search on Google or do some testing first.

 

[anodos]

Sorry, I can't tell you that. My knowledge of robocopy is very limited.
Try to search on Google or do some testing first.

Robocopy within same share is server side in Windows 7 and later.

 

[Robert Trevellyan]

I can't be typing all that...

Are you aware of tab-completion of file and folder names, which works in the CLI of almost every modern operating system? In other words, you would type mv /mnt/V and then press tab, and the CLI would automatically expand to mv /Vol_8TB/. Then you might type old folder and press tab again, and it would automatically expand to mv /Vol_8TB/old folder where Stuff was a huge mess/
and so on...

 

[Zaaphod]

Are you aware of tab-completion of file and folder names, which works in the CLI of almost every modern operating system?

No, I was not aware of that... thanks for telling me! That will save me a ton of typing and fixing mistakes. I was navigating to the path I want with windows explorer and cutting the path to the clipboard then pasting it on the command line.. this tab completion will be much faster than all of that.

 

[Robert Trevellyan]

In that case, I'll elaborate just a little. When you press tab, the CLI will complete the file or folder name immediately if the part that you already typed is enough to uniquely identify an item. If not, you can press tab a second time and the CLI will display all possible matches. At that point you can see what additional characters you need to type to get a unique match (assuming you don't just type the whole name).

 

[Marcet]

That's one of the beauty of CLI ;)

 

[Zaaphod]

Excellent! Thanks for the explanation! I tried it out some and it's great! I notice it also works in my windows 10 command prompt.. but that's a little different, it just puts in the first match it finds, but if you push tab again it shows the next match.

 

[clemo]

Thank you all for your advice. Let me add a tip on how it is possible to do it without having to type long paths.

1. In the source dataset, I create a folder in the very top folder (not sure if it is correct to call it root folder) with a simple name, one letter, because I am lazy to type. I do it all within file browser/commander on respective Windows share, e.g. \\192.168.1.10\Clemo\a. I use to call it "transport dir"
2. I move all files and folders, that I intend to move, into transport folder. No matter if I use Ctrl+X or F6 in a file commander, it moves instantly as it would on a local drive (only filesystem is being written, I suppose)
3. now I use the mv command, moving whole transport dir to target dataset, again to the very top folder, e.g. mv /mnt/WD8TB/Privat/a /mnt/WD8TB/Public/ . This takes a while, when dealing with large amount of data or count of files.
4. Afterwards, I open the "a" folder on share of the target dataset and move the files/dirs where I want them to have, no matter how many subfolders deep inside. Again, this happens fast, instantly. Delete "a" once it is empty.

Data is really moved because "free space" doesn't change
Thanks to Glorious1 for inspiration when mentioned the word "rearranging"