SOLVED Fastest way to copy or move files to dataset?

[Daisuke]

I want to move my /mnt/default/media dataset files to /mnt/default/media/center dataset, to align with new Scale design. I’m used to Linux ways, rsync, cp, mv. Is there a faster/better way using Scale tools?

 

[danb35]

You could do local replication from the one dataset to the other, but I don't think it would be any faster or better than using rsync/mv/mc.

 

[winnielinnie]

What about the zfs rename command? It works instantaneously. No data will be re-written.

zfs-rename.8 — OpenZFS documentation

openzfs.github.io

It would have to be done in multiple steps.

1. Rename the current dataset to a temporary new name, such as "default/media1"
2. Create the parent dataset (as new) named "default/media"
3. Then rename "default/media1" as "default/media/center"

I can't vouch how well the TrueNAS middleware will handle this change that bypasses the GUI / middleware commands.


UPDATE: I just tried this out on a playground pool and it worked. The files are all in tact under the "re-located" path. TrueNAS Core 13.0-U3.1. Should expect it to work the same under SCALE. But, try it at your own risk.

I did not have to refresh the middleware, nor reboot, nor export/import the pool. However, I did this on a test pool.



Here are the exact steps I took. (Steps 1 through 2 are irrelevant in your case, since I needed to first get started with a dataset and fill it with some sample files.)

1. Using the GUI, create a new dataset: testpool/media
2. Fill this dataset with some sample files under /mnt/testpool/media/
3. Using the command-line, rename the dataset temporarily
   • zfs rename testpool/media testpool/media1
4. Using the GUI, create a new dataset (again): testpool/media
5. Now there exists testpool/media1 and testpool/media
6. Finally, rename testpool/media1 to testpool/media/center
   • zfs rename testpool/media1 testpool/media/center
7. The dataset formerly known as testpool/media1 remains in tact, however, it is now located under testpool/media/center, as well as its contents under /mnt/testpool/media/center/

You may have to double-check the dataset options, especially properties which are "inherited" from the fresh parent.

 

[Daisuke]

I just tried this out on a playground pool and it worked

Amazing info, 30 seconds later:

I'm going to add this into Bluefin Recommended Settings and Optimizations thread. I also opened NAS-119755, to have gcp added into Scale.

Edit: Guide updated, as usual, credit and link provided.

When I renamed the media to media-old dataset, it was reflected immediately into UI. Related to dataset options, they are all preserved. In my case, they all showed as inherited, except the recordsize, which kept the previous 1M custom value I've set (new media dataset was created with 128k recordsize). This reassured me the dataset rename did not affected in any way how data is stored inside the dataset. It was a pain when I moved all files from a 128k recordsize dataset to another 1M recordsize dataset. All this is explained into guide.

 

[mutenroid]

one doubt...... if /mnt/default/media/center has previously files inside, can you execute the last rename?? or empty dataset target is required??

Regards

 

[winnielinnie]

You mean that you already have a "center" dataset under default/media/center?

If that's the case, you're already set.

The zfs command will ("should") prevent you from overriding an existing dataset if you try to rename an old dataset to a new name that already exists as a dataset in your pool.

 

[mutenroid]

You mean that you already have a "center" dataset under default/media/center?

If that's the case, you're already set.

The zfs command will ("should") prevent you from overriding an existing dataset if you try to rename an old dataset to a new name that already exists as a dataset in your pool.

ok, i understand.
So, let me explain my scenario.

My wife organizes her huge collection of photos in a dataset shared by Samba. Therefore, she can decide to move, add, delete, relocate, etc... her photos from Windows.

My intention is to set up Photoprism app to manage this collection (visualizer, tagger, face identifier, album manager, etc...)

So, how can I configure the photo dataset and Photoprism app so that I don't have to disable "Host path safety checks"?

Regards

 

[mutenroid]

ok, solved by myself using indications from @Daisuke (Bluefin Recommended Settings and Optimizations)
Now i have.

New account for my wife and added to group apps, and allow Samba Authentication.
/mnt/tank/photos (share-type=SMB and this path added as SMB share)
/mnt/tank/photos/collection (share-type=Generic and chown to apps:apps)

Photoprism originals path mapped with /mnt/tank/photos/collection and works flawlessly (no error initialializing pods)
Accesing from Windows10 to /mnt/tank/photos using wife account credentials works flawlessly too.
And now she can access to /mnt/tank/photos/collection and manage photos as she wants.

Thanks in advance

 

[fenichelar]

@muternoid using a child dataset doesn't address the underlying permissions issue caused by shared paths. The current version of SCALE has insufficient host path safety check logic so it incorrectly doesn't identify using a child dataset as an issue. The host path safety check logic has been improved so you won't be able to use a child dataset in the next version of SCALE without disabling "Host path safety checks".

 

[mutenroid]

Ohh! And what it's your recommendation about this @fenichelar ?

 

[Shigure]

My question might be related to the discussion here, what's the best way to move files from a Linux system to TrueNAS Scale?
What I'm think about now is setup a NFS share in TrueNAS Scale and mount it to the Linux system then use mc or rsync etc.
The way I'm doing it right now is downloading the files to my Windoes machine then upload to TrueNAS, which shouldn't be the most efficient way to do this, I guess.

 

[winnielinnie]

My question might be related to the discussion here, what's the best way to move files from a Linux system to TrueNAS Scale?

This is a completely separate question. A new thread would be better.

But while we're here:

You can rsync directly from the Linux client to TrueNAS with a user account over SSH.

Something like this, as long as you've got your accounts, permissions, and datasets configured properly.
rsync -avhHxxs --progress /home/shig/mydata/ shig@192.168.1.100:/mnt/mypool/mydata/

No need to make multiple trips through NFS or SMB. Just rsync directly, bypassing everything else.

 

[mutenroid]

This is a completely separate question. A new thread would be better.

But while we're here:

You can rsync directly from the Linux client to TrueNAS with a user account over SSH.

Something like this, as long as you've got your accounts, permissions, and datasets configured properly.
rsync -avhHxxs --progress /home/shig/mydata/ shig@192.168.1.100:/mnt/mypool/mydata/

No need to make multiple trips through NFS or SMB. Just rsync directly, bypassing everything else.

I used rsync in the same way as above

 

[Whattteva]

If you are running ZFS as some Linux distros have started providing that option at install time, you can use the standard zfs send/recv, which quite honestly, I prefer due to it being more efficient; particularly for incremental changes on large files like VM images.

 

[Shigure]

This is a completely separate question. A new thread would be better.

But while we're here:

You can rsync directly from the Linux client to TrueNAS with a user account over SSH.

Something like this, as long as you've got your accounts, permissions, and datasets configured properly.
rsync -avhHxxs --progress /home/shig/mydata/ shig@192.168.1.100:/mnt/mypool/mydata/

No need to make multiple trips through NFS or SMB. Just rsync directly, bypassing everything else.

I used rsync in the same way as above

Just tried and it works perfectly(my directories has Japanese names though, will figure out with exa and ripgrep to get a list). Thanks!

If you are running ZFS as some Linux distros have started providing that option at install time, you can use the standard zfs send/recv, which quite honestly, I prefer due to it being more efficient; particularly for incremental changes on large files like VM images.

I know I can zfs send between different pools but don't know I can even send/recivce with remote machine.
My Linux machine is not running zfs but it will be a good reading for me(found this https://pthree.org/2012/12/20/zfs-administration-part-xiii-sending-and-receiving-filesystems/). Thanks for that tip.

 

[Whattteva]

I know I can zfs send between different pools but don't know I can even send/recivce with remote machine.
My Linux machine is not running zfs but it will be a good reading for me(found this https://pthree.org/2012/12/20/zfs-administration-part-xiii-sending-and-receiving-filesystems/). Thanks for that tip.

Typically, it's done through ssh and instead of the usual:

Code:

zfs send pool1/dataset1@snapshot | zfs recv pool2/dataset2

You do:

Code:

zfs send pool1/dataset1@snapshot | ssh nas2 zfs recv nas2/dataset2

Ideally, you would do an extra step here and pipe it through something like netcat to avoid the overhead of encryption when transferring through a local trusted LAN.

 

[fenichelar]

@mutenroid Either:
1. Don't use a host path for the app, use a share (the same way you would do it if the app was not running on the same host as the files)
2. Disable Host Path Safety Checks (this is essentially what you have done now, using the same host path for shares and apps won't necessarily cause an issue, there is just a potential for an issue)

 

[nutrient2314]

Just a small heads up, user need not create the destination dataset before running zfs rename, it get's created automatically