SOLVED Move files between shares like you moving files on a phsysically disk

m0nKeY

Cadet
Joined
Jan 17, 2021
Messages
8
Hi,

sorry for the bad "Thread title" but I couldn't explain it any better that short. I just want to what is the best solution to my problem this is my current situation:
  1. I have 2 datasets and 2 separate shares for them.
  2. Both shares are mounted by the same server
  3. The FreeNAS server and the other server run on the same VM server.
If I copy files from the first share to the other, the files are obviously moved over network and it takes some time. On the other hand a "move" on the server directly would be instant.

My question now is: Is there any protocol (NFS, SFTP, SCP...) that moves files between shares like you directly move them on the FreeNAS server? Like it is aware that both source share and destination share are on the same physically disk on the same server?

Greetings
m0nKeY
 

ChrisRJ

Wizard
Joined
Oct 23, 2020
Messages
1,904
You could log on the FreeNAS console and perform a local copy there
 

m0nKeY

Cadet
Joined
Jan 17, 2021
Messages
8
Of course, but that's not what I meant. I meant a more build in solution, as it is permanent and repeating task. The AutoOrganize Plugin of Jellyfin/Emby.
 

sretalla

Powered by Neutrality
Moderator
Joined
Jan 1, 2016
Messages
9,702
a "move" on the server directly would be instant.
Not correct... "moving" a file between two different CoW filesystems will still copy the files, just not over the network. "Instant" will only be the case with files small enough to seem so.

My question now is: Is there any protocol (NFS, SFTP, SCP...) that moves files between shares like you directly move them on the FreeNAS server? Like it is aware that both source share and destination share are on the same physically disk on the same server?
The only way you will get to that is if you have the source and target in the same dataset (a true mv operation) and the SMB protocol does support it working that way.

You will not find (as far as I know) anything that can get to that result between different datasets.

You may be able to write your own web API service that does the mv based on the requested file locations and call that, but refer back to my fist point about it not really being a mv.
 

m0nKeY

Cadet
Joined
Jan 17, 2021
Messages
8
Thank you, that was very helpful. Even though I still have questions. :-D

Not correct... "moving" a file between two different CoW filesystems will still copy the files, just not over the network. "Instant" will only be the case with files small enough to seem so.

What do you exactly mean with CoW filesystem? I assumed a "move" command on a local file system (same drive) would be performed instantly just by changing the "index" of the file in the file system.

The only way you will get to that is if you have the source and target in the same dataset (a true mv operation) and the SMB protocol does support it working that way.

You will not find (as far as I know) anything that can get to that result between different datasets.

You may be able to write your own web API service that does the mv based on the requested file locations and call that, but refer back to my fist point about it not really being a mv.

You mean, if the shares would be part of one ZFS dataset (maybe just different folders in the dataset) the mv would be performed instantly like assumed by me? And this is also supported by SMB?
 

sretalla

Powered by Neutrality
Moderator
Joined
Jan 1, 2016
Messages
9,702
What do you exactly mean with CoW filesystem?
ZFS is a Copy on Write (CoW) filesystem. Read the official words about that in this table: https://www.freebsd.org/doc/handbook/zfs-term.html

I assumed a "move" command on a local file system (same drive) would be performed instantly just by changing the "index" of the file in the file system.
Since each dataset is a different filesystem, the file tables are separate and don't know about each other, so you can't just update the file table if the new location isn't in the same filesystem.

You mean, if the shares would be part of one ZFS dataset (maybe just different folders in the dataset) the mv would be performed instantly like assumed by me? And this is also supported by SMB?
That's more-or-less the theory... test it and find out.
 

m0nKeY

Cadet
Joined
Jan 17, 2021
Messages
8
ZFS is a Copy on Write (CoW) filesystem. Read the official words about that in this table: https://www.freebsd.org/doc/handbook/zfs-term.html

Ok, I read but I'm still unsure if understand this correctly. Does this mean ZFS is always really copying the data first, before deleting it, if you move a file? Because if I move a file from /folder0/subfolder0/ to /folder1/subfolder0/ in a single dataset via SSH, it is moved instantly.

That's more-or-less the theory... test it and find out.

Tried it and it is not moved instantly via SMB. Should it?
 

Samuel Tai

Never underestimate your own stupidity
Moderator
Joined
Apr 24, 2020
Messages
5,398
There’s multiple layers of abstraction here. When you move a file in an SMB share, the file itself doesn’t move, only the directory pointer, unless you have ZFS datasets within the share. In this case, ZFS has to copy the file in the background. So all that’s being changed is the directory pointer. If you force a copy in SMB, then a server-side copy takes place.
 
Joined
Oct 22, 2019
Messages
3,582
Just tried this with a 4GB file, and the move was instant. I even did a round robin as quickly as possible, from one folder to the next to the next to the next, all the way back to the original folder. Every move was instant.

Here is what my test looked like.

  1. Share named "playground" points to the "playground" dataset, root folder
  2. Open up Windows Explorer, navigate to \\server.local\playground\
  3. Create several nested folders, such as:
    • \\server.local\playground\folder1\foldera1
    • \\server.local\playground\folder1\folderb1
    • \\server.local\playground\folder1\folderc1
    • \\server.local\playground\folder2\foldera2
    • \\server.local\playground\folder2\folderb2
    • \\server.local\playground\folder3\foldera3
    • \\server.local\playground\folder3\folderb3
    • \\server.local\playground\folder4\foldera4
  4. Copy a 4GB file from my PC to any of these (network) folders, wait for the transfer to complete
  5. Cut and paste (move) this large file from one folder to the next, to the next, to the next, doesn't matter in which order
  6. Observe that each move is instant, as fast as if it was done via SSH using the "mv" command on my NAS server
As @Samuel Tai noted, do you have datasets within the Share itself? If so, then the file's data would have to be copied in its entirety from one dataset to the other.

However, you point out that via SSH your test was instant, yet the same "move" via SMB took a noticeable amount of time?
 

m0nKeY

Cadet
Joined
Jan 17, 2021
Messages
8
@winnielinnie Thank you, I really appreciate the effort you took.

And yes, I have a sub-dataset within the shared dataset. My configuration is like this:
  1. /shared_dataset/folder1/sub-folder0
  2. /shared_dataset/folder2/
  3. /shared_dataset/unshared_dataset/
and I'm moving sub-folder0 into folder2 via SSH. This is performed instantly and then AutoOrganize moves the file from folder2 into unshared_dataset, which is not instantly. And as you explained, this is because unshared_dataset is not a folder but a dataset, right?

Thank you very much, my problems seems to be explained. :-D Now how I solve this? Is it possible to convert a sub_dataset, within a dataset, into a folder?
 
Joined
Oct 22, 2019
Messages
3,582
this is because unshared_dataset is not a folder but a dataset, right?
Bingo! @Samuel Tai predicted exactly that. You're not moving any files "folder to folder" once you traverse a file system (i.e, a dataset.) They may "appear" as plain folders, but truthfully the root of a file system starts at the dataset, in this case "/unshared_dataset/"

You can apply "mental reminders" by pre-pending your datasets with "ds", such as dsMultimedia and dsArchives.

This way, when you see a "folder" that begins with "ds", you'll recognize it as the root directory of a dataset: it is not "simply another folder".
 

m0nKeY

Cadet
Joined
Jan 17, 2021
Messages
8
Thanks for the hint. I will add prefixes to my dataset.

But as this sub-dataset I created, are not really necessary, I would like to convert them to folders. Is this possible?
 
Joined
Oct 22, 2019
Messages
3,582
Thanks for the hint. I will add prefixes to my dataset.

But as this sub-dataset I created, are not really necessary, I would like to convert them to folders. Is this possible?
You would first have to destroy the dataset (which will delete EVERYTHING that is/was ever saved on it, including snapshots), and then later create a new folder with the same name (or any name) either through your file browser or via SSH.

Please read the bold text again, and proceed with caution.
 

m0nKeY

Cadet
Joined
Jan 17, 2021
Messages
8
Thanks to you all for you quick and kind support. The problem is solved. I removed the sub-dataset and created a folder. Now everything is moved instantly.
 
Top