SOLVED Best Way to Split a Dataset into Two Datasets

[eewiz]

Hello All,

Core 13.0-U5, 32GB Memory, Xeon E3-1200 4-core, 128K SSD boot, 12 8TB spinners pool.

I have a dataset that is getting to large.
Lately, I had to manipulate this dataset, which doubled its Used Space, and pushed the server to 95% Used Space.
Is there a way to move about half of the data from an existing dataset to a newly created dataset?
I would like to move the data in a method that does not increase the Used Space.
Do I have to do a move of the data from the hidden .zfs folder residing under the original dataset?
Should I move the data from the oldest or the newest snapshot?
Or, is there a better way?
I auto-create snapshots daily and keep them for 7 days before automatic deletion.
The snapshot of the original dataset is replicated daily to another server.

Thank You

 

[Arwen]

The snapshots are likely the cause of your excessive space usage. Using a different dataset will not solve the problem because the snapshots are still holding on to any data available at the time the snapshot was take.

You can wait 7 days, and the space should drop back down to previous & normal usage levels. Then you can slowly move files from the large dataset to a new one. Just keep in mind the files "moved" will still exist in the old dataset's snapshots until the snapshot auto-deletion occurs.

If you are confident your data is safe, you can destroy the snapshots sooner. Either to reclaim the space. Or to allow moving files without keeping the copy around for 7 days.

Make sure you use similar or better compression on the new ZFS dataset. Some people over-look ZFS native compression. Then move files and find they take up more space than the original.

 

[eewiz]

Hello Arwen
Thank you for your reply.

I see now that I should have explained better.
When I said "Lately, I had to manipulate this dataset, which doubled its Used Space, and pushed the server to 95% Used Space." I should have explained further that that issue has already been resolved.
My Space Used is back to 58%.
Going through that ordeal, caused me to worry about what may happen in the future.
I don't want an irreparable situation to occur in the future.
One day, as the dataset grows, there will not be enough space to host it's space twice over.
So I thought I would split that dataset into two parts.

So, here I will repeat my questions about splitting a dataset in two when there are no other issues of concern.
I would like to move the data in a method that does not increase the Used Space.
If it's not possible to split a dataset without increasing space for the life of the snapshots, I would still like to know the best way to do the split.
Could I move the data from the hidden .zfs folder residing under the original dataset?
Should I move the data from the oldest or the newest snapshot?
Or, is there a better way?

Thank You

 

[Arwen]

No, you can not "split a dataset".

But, the Unix "mv" command only uses extra space for the file being moved. In essence, a Unix "mv" is a copy and delete function all in one, when moving files across file systems, (which datasets act like file systems).

You don't move data hidden in the ".zfs" folder, as it is part of the dataset's record keeping. It is for share info and snapshot default locations.

The fastest why to move files around, would be to log into the TrueNAS via SSH, use TMUX or SCREEN and then "mv" the files as needed. This is what I do when I need to. External GUIs via Samba or NFS may not "mv" the files, they may copy and delete over the network which takes more time.


Unless their is a reason, I am not sure why you want to move snapshot data. Snapshots exist ONLY in the dataset it was created. So you can't move a file out of snapshot without deleting that snapshot altogether. You CAN copy a file out of a snapshot if you need that older copy. But, you now have 2 x space used for that file. The original snapshot and the new copy.


Now I am not clear on WHY you think a single dataset can't have most of the space. You certainly can not over-subscribe the space by adding new datasets and thinking they have all the pool space too. All datasets and zVols in a pool share the pools disk space, (which you probably know). Unless you are talking about a dataset in a DIFFERENT pool?


Looking at your questions it seems you may want to read up on ZFS datasets and snapshots. I don't have any handy recommendations for reading available.

I once had an long discussion with a well experienced co-worker about ZFS snapshots, and he kept insisting it was similar to Linux LVM or VMWare snapshots. That is not the case. Those need to copy back snapshot data because they act like write transaction logs. Today I would have referenced the appropriate ZFS manual page which is more clear for that specific topic.

 

[eewiz]

Thank you Arwen

Arwen wrote:
The fastest why to move files around, would be to log into the TrueNAS via SSH, use TMUX.

I must assume that you are recommending TMUX because you have experienced an unwanted SSH session disconnection in the past.
I have not noticed this. I have left SSH sessions idle for several days to return to a blinking cursor.
Now I am worried. I don't ever want an SSH session to timeout and abort on its own.
But, I find nothing in the SSH service settings that controls a timeout function.
I see that TMUX permits opening multiple prompts but, I don't understand why it's used.
If I need another prompt, I simply start another SSH session. I have had 4 or 5 SSH sessions open at the same time.
Is that incorrect procedure?

I feel like I should explain what got me here in the first place.
All of the files in /mnt/pool/eds/video had somehow been deleted and an automatic snapshot had already been created after the deletion.
I deleted that current snapshot of the empty "video" dataset.
I expected the "video" dataset to be returned to yesterday's snapshot with all of the files back.
The "video" dataset remained empty.
Then I tried to ROLLBACK that now last, good snapshot, taken that day before the files were deleted and nothing happened.
The "video" dataset remained empty.
Since snapshot removal and then ROLLBACK did not work, I tried CLONE on that good, one-day-old snapshot of the "video" dataset.
I cloned that snapshot of /mnt/pool/eds/video to /mnt/pool/eds/video-clone.
I ssh'd in, then su -, then I mv'd each folder from /mnt/pool/eds/video-clone/folder-name to /mnt/pool/eds/video one folder at a time.
Used Space rose from 58% to 94% during the move. It took about 28 hours to move the full 23.9TB.
It's surprising to me that it took 28 hours to manipulate the snapshots concerned since, the move should have created no new data.
The "video-clone" dataset was emptied and all of the data was correctly moved to the "video" dataset.
I thought, wrongly.
I then went to the GUI and deleted the empty "video-clone" dataset.
I never made a snapshot of the video-clone dataset so I expected the Used Space to return to 58% but, it remained at 94%.
I automatically make and retain snapshots for one week.
Ten days after the data move, all snapshots made before the data move had been automatically removed from the system.
Yet, I was still stuck at 94% Used Space.

winnielinnie wrote:
My suspicion is that at one point you have the dataset pool/eds, and populated it with folders and files. One of these "folders" shares the same name with a new child dataset you create under pool/eds. Upon creation of this new dataset, it is automatically mounted at the path /mnt/pool/eds/name. If "name" is the same as the folder name, then it will "overlap" it, which causes confusion. The original folder under the dataset pool/eds still contains the files; yet you cannot access nor see them because it is currently being "overlapped" by an actual mounted dataset that shares the same name as the folder.

Winnielinnie was correct. I found a folder underneath each dataset with the same name.
I did; "zfs unmount /mnt/pool/eds/video" to find a folder named "video" with the 24TB of moved data in it.
I renamed the video folder to "video_dir" and rebooted TrueNAS.
After verifying that all the datasets were correct, I opened an SSH session and started an "rm -R video_dir".
This returned me to 58% Used Space and did not affect the data in the "video" dataset.
I waited another week for all snapshots and replications to my 2nd TrueNAS box to stabilize.
Everything looks perfect now.

I've done some experimentation on this dataset vs underlying folder of the same name issue.
Here are my results.

Code:

root@shed[/mnt/pool/eds]# ls -l
total 339
drwxrwxrwx+ 16 root  wheel  18 Apr 30 19:37 backup
drwxrwxrwx+ 20 root  wheel  20 Jun  5  2018 common
drwxrwxrwx+ 40 root  wheel  40 Dec  4  2019 drivers
drwxrwxrwx+ 33 root  wheel  33 Mar 26  2021 fixes
drwxrwxrwx+  9 root  wheel   9 Jan  7  2021 games
drwxrwxrwx+  2 root  wheel   3 Jun 14 16:22 keepass
drwxrwxrwx+ 29 root  wheel  32 Nov 21  2019 lib
drwxrwxrwx+ 17 root  wheel  19 Mar 30 00:13 music
drwxrwxrwx+  5 root  wheel   5 Dec  2  2021 rch-ec
drwxrwxrwx+  8 root  wheel   8 May  5  2019 sm
drwxrwxrwx+  7 root  wheel   7 Oct 23  2021 software
drwxrwxrwx+  9 root  wheel   9 Jun 21 07:53 urb
drwxrwxrwx   7 root  wheel   7 May 10 05:35 user
drwxrwxrwx+  7 root  wheel   7 Jun  4 01:02 video

Above is a directory listing of the "/mnt/pool/eds" dataset.
Every item listed above is also a dataset.

Code:

root@shed[/mnt/pool/eds]# cd video
root@shed[/mnt/pool/eds/video]# ls -l
total 205
drwxrwxrwx+  6 root  wheel   6 Apr 18  2022 Classics
drwxrwxrwx+ 28 root  wheel  28 Feb 11  2022 Movies
drwxrwxrwx+ 12 root  wheel  12 Apr 18  2022 Music Videos
drwxrwxrwx+ 18 root  wheel  22 May 14 17:24 Tutorials
drwxrwxrwx+  5 root  wheel   5 Mar 13 16:43 TV Shows

Above is a directory listing of the "/mnt/pool/eds/video" dataset.
Every item listed above is a folder.
Notice that reading these two listings gives no clue as to what is a folder and what is a dataset.

Now I unmount the "video" dataset.

Code:

root@shed[/mnt/pool/eds/video]# cd ..
root@shed[/mnt/pool/eds]# zfs unmount /mnt/pool/eds/video
root@shed[/mnt/pool/eds]# ls -l
total 303
drwxrwxrwx+ 16 root  wheel  18 Apr 30 19:37 backup
drwxrwxrwx+ 20 root  wheel  20 Jun  5  2018 common
drwxrwxrwx+ 40 root  wheel  40 Dec  4  2019 drivers
drwxrwxrwx+ 33 root  wheel  33 Mar 26  2021 fixes
drwxrwxrwx+  9 root  wheel   9 Jan  7  2021 games
drwxrwxrwx+  2 root  wheel   3 Jun 14 16:22 keepass
drwxrwxrwx+ 29 root  wheel  32 Nov 21  2019 lib
drwxrwxrwx+ 17 root  wheel  19 Mar 30 00:13 music
drwxrwxrwx+  5 root  wheel   5 Dec  2  2021 rch-ec
drwxrwxrwx+  8 root  wheel   8 May  5  2019 sm
drwxrwxrwx+  7 root  wheel   7 Oct 23  2021 software
drwxrwxrwx+  9 root  wheel   9 Jun 21 07:53 urb
drwxrwxrwx   7 root  wheel   7 May 10 05:35 user
drwxrwxrwx+  2 root  wheel   2 Jun 19 10:10 video

Again the directory listing.
Notice that there is no way to tell from the above listing that the "video" item is no longer a dataset.
The "video" item is now a folder.

Code:

root@shed[/mnt/pool/eds]# cd video
root@shed[/mnt/pool/eds/video]# ls -l
total 0
root@shed[/mnt/pool/eds/video]#

The directory listing above for the "video" folder is empty.

I reboot "shed" and all is back to normal.
Despite winnielinnie's suspicion, I was sure that I had not populated the "eds" dataset with folders and files before I created datasets in "eds".
By the way "eds" is short for "Encrypted Data Set" which is passphrase encrypted.
Now I can see quite starkly that TrueNAS creates an empty folder underneath every dataset.
Try it for yourself, SSH in and "zfs unmount" a dataset then do an "ls -l" that dataset's name is still there but, now it is not a dataset.
It is an empty folder that TrueNAS put there.
"rm -R" the empty folder and "ls -l" will show that it is gone.
Reboot TrueNAS and all is back to normal.
Then "zfs unmount" and "ls -l" to see that the empty folder is back.
So now, I am absolutely sure that I did not initially create folders under the "eds" dataset before creating the child datasets.
TrueNAS creates them.

So now, back to the giant 24TB dataset that I want to split up.
I want to split it up because I know only heartbreak when it comes to moving data between datasets.
As the dataset continues to grow there will come a day when there will not be enough storage space to CLONE the dataset to a new dataset to effect repairs.
I have no idea why the "mv" from the "video-clone" dataset moved the data to the "video" folder underneath the "video" dataset.
Therefor, I must assume that it will happen again.

That is why I wished to know the best practice for moving some of the folders from the "video" dataset to a newly created "video2" dataset.
Now that I have wrestled with this issue for several weeks it comes to me that it would be less disturbance to users to change the 5 folders that reside in the "video" dataset into 5 datasets.
Users would see no difference.
Then I could perform maintenance on any one of the 5 smaller datasets in the "video" dataset as opposed to having to deal with the entire 24TB in one dataset.

This leads to another question.
Can a windows box via samba delete a dataset, or will TrueNAS refuse?
I know that a windows box can delete the files in a folder or an entire folder through samba.
I created a new "test" dataset below the "video" dataset to test if I can delete it from windows.
It does not show up when I view the "video" dataset from windows.
Now I see why.

Code:

root@plum[/mnt/pool/eds/video]# ls -l
total 224
drwxrwxrwx+  6 root  wheel   6 Apr 18  2022 Classics
drwxrwxrwx+ 28 root  wheel  28 Feb 11  2022 Movies
drwxrwxrwx+ 12 root  wheel  12 Apr 18  2022 Music Videos
d---------+  2 root  wheel   2 Jun 21 19:58 test
drwxrwxrwx+ 18 root  wheel  22 May 14 17:24 Tutorials
drwxrwxrwx+  5 root  wheel   5 Mar 13 16:43 TV Shows
root@plum[/mnt/pool/eds/video]#

The "test" dataset has no BSD permissions.
I also have gained no understanding of what to expect for BSD permissions of newly created files, folders or datasets.
All I can say for sure is that it never seems to acquire the parents permissions.

So I try:

Code:

root@plum[/mnt/pool/eds/video]# chmod -R 777 test
chmod: test: Operation not permitted
root@plum[/mnt/pool/eds/video]#

Nada, I can't change the BSD permissions.
So I have no choice but to delete the "test" dataset and get no closer to the answer to my question.
I'll just chalk it up as another failure in the trial and error world of TrueNAS.
Perhaps someone knows, can a windows box via samba delete a dataset, or will TrueNAS refuse?

And at this point I still know nothing about the best practices to reduce the size of a single dataset.

Thank You
All for now

 

[eewiz]

Arwen,

I did some research into SSH timeout.
From other posts I learned that the SSH config file resides here; "/usr/local/etc/ssh/sshd_config"
Below is the contents of sshd_config.
From reading this file it is my understanding that SSH no longer times out.
Notice the line "# ConnectTimeout 0".

Code:

#       $OpenBSD: ssh_config,v 1.35 2020/07/17 03:43:42 dtucker Exp $

# This is the ssh client system-wide configuration file.  See
# ssh_config(5) for more information.  This file provides defaults for
# users, and the values can be changed in per-user configuration files
# or on the command line.

# Configuration data is parsed as follows:
#  1. command line options
#  2. user-specific file
#  3. system-wide file
# Any configuration value is only changed the first time it is set.
# Thus, host-specific definitions should be at the beginning of the
# configuration file, and defaults at the end.

# Site-wide defaults for some commonly used options.  For a comprehensive
# list of available options, their meanings and defaults, please see the
# ssh_config(5) man page.

# Host *
#   ForwardAgent no
#   ForwardX11 no
#   PasswordAuthentication yes
#   HostbasedAuthentication no
#   GSSAPIAuthentication no
#   GSSAPIDelegateCredentials no
#   BatchMode no
#   CheckHostIP no
#   AddressFamily any
#   ConnectTimeout 0
#   StrictHostKeyChecking ask
#   IdentityFile ~/.ssh/id_rsa
#   IdentityFile ~/.ssh/id_dsa
#   IdentityFile ~/.ssh/id_ecdsa
#   IdentityFile ~/.ssh/id_ed25519
#   Port 22
#   Ciphers aes128-ctr,aes192-ctr,aes256-ctr,aes128-cbc,3des-cbc
#   MACs hmac-md5,hmac-sha1,umac-64@openssh.com
#   EscapeChar ~
#   Tunnel no
#   TunnelDevice any:any
#   PermitLocalCommand no
#   VisualHostKey no
#   VerifyHostKeyDNS yes
#   ProxyCommand ssh -q -W %h:%p gateway.example.com
#   RekeyLimit 1G 1h
#   UserKnownHostsFile ~/.ssh/known_hosts.d/%k

So, unless I am reading it wrong, I understand that SSH no longer times out.

All for now

 

[2twisty]

I occasionally have SSH dropouts. Most of the time they are stable, but I have had enough of them to make me always use TMUX. I wish the old-school "SCREEN" command was available, since I rarely need multiple windows, and I find navigating a windowed interface with keyboard commands to be annoying. How many times have I just run my mouse over and clicked in a TMUX window to try to select it, I cannot count.

 

[Arwen]

@eewiz - Sorry I could not read your posts, they are just too long for my fried brain. (I am participating in the time honored practice of sleep deprivation caused by job.)

But I can say this about TMUX & SCREEN, they are not just useful for network issues, but client issues. Meaning you can reboot your client PC while the server is still moving a huge file. Then re-establish access to exactly where you left off.

Their are some SSH options that help keep network connections alive, depending on the network issues. Here they are;
ServerAliveInterval ServerAliveCountMax

Perhaps someone else can help.

 

[eewiz]

Or, perhaps not.

For those that have made it to here and also do not understand.
The solution can be found here.
https://www.truenas.com/community/t...point-view-differ-from-the-direct-view.110876

All for now