SOLVED How Can a Jail Mount Point View Differ from the Direct View?

[eewiz]

Hello All
Core 13.0-U5, 32GB Memory, Xeon E3-1200 4-core, 128K SSD boot, 1-vdev of 12-8TB spinners RAIDZ2

Here is what I did.
I had a "Video" dataset that contained a "Movies" folder and other folders that I won't mention for brevity.
I renamed the "Movies" folder to "Movies1".
I created a new "Movies" dataset inside the "video" dataset.
I copied all of the files from the "Movies1" folder to the newly created "Movies" dataset.
Used Space went up.
I created a new "Movies@snapshot".
I removed the "Movies1" folder from the "videos" dataset.
I deleted all old "videos@snapshots"
Used Space returned to normal.

The issue is:
No files can be seen in the "Movies" dataset from any mount point.
The files that reside inside the "Movies" dataset, that resides inside the "video" dataset, can not be seen through a mount point.
Why permit datasets within datasets if mount points can't see any of the files in child datasets?

Here is some evidence of my research:
In all of the following directory listings, .... means that everything except "Movies" has been removed for brevity.
DIRECT = View from the Root filesystem DIRECT to the "video" dataset:
MOUNT POINT = View from the Root filesystem through a Jail's MOUNT POINT to the "video" dataset:

DIRECT:

Code:

root@plum[/mnt/pool/eds]# ls -l
total 339
....
drwxrwxrwx+ 10 root  wheel  10 Jun 28 06:51 video

Jail UP - MOUNT POINT:

Code:

root@plum[/mnt/pool/iocage/jails/videoj/root/mnt]# ls -l
total 56
....
drwxrwxrwx+ 10 root  wheel  10 Jun 28 06:51 video

Both viewpoints above have similar results.
"video" is a dataset.

DIRECT:

Code:

root@plum[/mnt/pool/eds]# cd video
root@plum[/mnt/pool/eds/video]# ls -l
total 225
....
drwxrwxrwx+  28 root  wheel   28 Jun 25 03:07 Movies

"Movies" is is also a dataset.

Jail UP - MOUNT POINT:

Code:

root@plum[/mnt/pool/iocage/jails/videoj/root/mnt]# cd video
root@plum[...ol/iocage/jails/videoj/root/mnt/video]# ls -l
total 152
....
drwxrwxrwx+ 2 root  wheel  2 Jun 24 18:58 Movies

The "Movies" item above is that same dataset but now, things have changed.
Please notice that the time stamps differ between the two views.

Code:

drwxrwxrwx+  28 root  wheel   28 Jun 25 03:07 Movies <- DIRECT = Jun 25 03:07
drwxrwxrwx+ 2 root  wheel  2 Jun 24 18:58 Movies <- MOUNT POINT = Jun 24 18:58

Via DIRECT, the "Movies" dataset contains 28 items, . and .. as well as 26 folders.
Via MOUNT POINT, the "Movies" dataset contains only 2 items, . and ..
Via MOUNT POINT, the "Movies" dataset appears to be empty.

Code:

root@plum[...ol/iocage/jails/videoj/root/mnt/video]# cd Movies
root@plum[...ge/jails/videoj/root/mnt/video/Movies]# ls -l
total 0

I don't understand how this can happen.

I removed the "videoj" mount point and rebooted TrueNAS.
I recreated the "videoj" mount point and nothing appears to have changed.
Then I stripped all of the "video" dataset's ACL's from the "Storage/Pools" dialog, Recursive and Traverse.
Now it gets really weird.

DIRECT:

Code:

root@plum[/mnt/pool/eds/video]# ls -l
total 225
....
drwxrwxrwx   28 root  wheel   28 Jun 25 03:07 Movies

Please notice above that there are no longer any ACL's as expected.

Jail UP - MOUNT POINT:

Code:

root@plum[...ol/iocage/jails/videoj/root/mnt/video]# ls -l
total 152
....
drwxrwxrwx+ 2 root  wheel  2 Jun 24 18:58 Movies

Yet, when viewed through the mount point, the ACL's are back.

If I edit ACL's from the "Mount Points of videoj" dialog, I have ACL's
If I edit ACL's from the "Storage/Pools" dialog, I have no ACL's.
How can that happen?

DIRECT:

Code:

root@plum[/mnt/pool/eds/video]# zfs list -t all -r /mnt/pool/eds/video
NAME                                                     USED  AVAIL     REFER  MOUNTPOINT
pool/eds/video                                          28.8T  22.5T     6.41M  /mnt/pool/eds/video
....
pool/eds/video/Movies                                   5.98T  22.5T     5.98T  /mnt/pool/eds/video/Movies
pool/eds/video/Movies@auto-2023-06-25_06-50                0B      -     5.97T  -
pool/eds/video/Movies@auto-2023-06-25_09-10                0B      -     5.97T  -
pool/eds/video/Movies@auto-2023-06-26_09-10                0B      -     5.97T  -
pool/eds/video/Movies@auto-2023-06-27_09-10                0B      -     5.97T  -
pool/eds/video/Movies@auto-2023-06-28_09-10              622K      -     5.98T  -

Jail UP - MOUNT POINT:

Code:

root@plum[/mnt/pool/eds/video]# zfs list -t all -r /mnt/pool/iocage/jails/videoj/root/mnt/video
'/mnt/pool/iocage/jails/videoj/root/mnt/video': not a ZFS filesystem

How can a mount point to the "/mnt/pool/eds/video" dataset show different results from the actual "/mnt/pool/eds/video"?

At this point I realize the reason.
The mount point to .../video/Movies does not see the "Movies" dataset.
It sees the empty "Movies" folder that underlies the "Movies" dataset.
I don't know if this a quirk of TrueNAS or it's the work of ZFS.
But, every ZFS dataset on TrueNAS has an underlying empty folder of the same name.

Try it for yourself.
If you unmount a ZFS dataset from the CLI and do an "ls -l", you will see an empty folder with the same name as the dataset that was just unmounted.
Any dataset, all datasets.
Reboot TrueNAS and your dataset(s) are back.
But know full well that that empty folder that underlies a dataset is still there.

What I need to know is how to fix it so mount points correctly address nested datasets not an underlying empty folder?
If you nest a third dataset like ".../video/Movies/test", the "test" dataset is not at all reachable through a mount point to ".../video".
The trail stops at the empty ".../video/Movies" folder when viewed through a mount point to ".../video".

This would mean that jailed applications cannot work with nested datasets.
I hope that that can't possibly be correct.

Please, does anyone know how to fix this?

Thank You

 

[sretalla]

What I need to know is how to fix it so mount points correctly address nested datasets not an underlying empty folder?
If you nest a third dataset like ".../video/Movies/test", the "test" dataset is not at all reachable through a mount point to ".../video".
The trail stops at the empty ".../video/Movies" folder when viewed through a mount point to ".../video".

This would mean that jailed applications cannot work with nested datasets.
I hope that that can't possibly be correct.

Please, does anyone know how to fix this?

Either:

Don't use nested datasets in your mount points, instead using directory structures in a single dataset.

or

Mount each of the datasets you want to be available in the jail at the point you want them to be available.

 

[eewiz]

Thank you sretalla for your reply.

I would bet that there are millions of users that don't know this one.
If your "video" dataset is full of folders, this works.
Your jailed application will see all data in all folders residing in the "video" dataset.

If your "video" dataset is full of other datasets, only this works.

Although, don't remove the top level "video" mount point or your jailed application will have to deal with each of the child datasets separately.
Through the top level "video" mountpoint, your jailed application can now find all of the data resident in all of the child datasets.
Also, since the jail cannot see anything but the empty folders underlying the child datasets, you will have to enter the final names, like "/Music Videos" when creating the child mount points.
For example:

Please notice that "Music Videos" is in the Source tree but does not appear in the Destination tree.
The jail cannot see the nested "Music Videos" dataset.
I had to type in "/Music Videos" at the end of the Destination value to make it work.

All for now

 

[Patrick M. Hausen]

Mounts are per filesystem and not transitive. Fundamental property of all Unix and Unix-like operating systems.

 

[eewiz]

Hi Patrick
I will apologize in advance.
Simple statements such as "Mounts are per filesystem and not transitive." are not informative to those that have not used unix/linux all their lives.
I spent a day's worth of research as detailed above to determine the solution.
Prior to that effort, I would not have understood your statement.

Interestingly enough, I found another post of yours during my research.
https://www.truenas.com/community/threads/mounting-main-data-set.90197
In which you wrote:

This is by design. Mounts are generally not transitive. Possibly there is some ZFS magic flag that just forces the additional mounts, but I have a different suggestion:
You can create a main dataset and just plain directories instead of sub-datasets in there. Then mount these directories into your jails just like you would with datasets. The jail mounts are created by "nullfs" in FreeBSD and you can mount an arbitrary directory path at some arbitrary other location - no need to have a "filesystem" for that.

At the time I read that, "transitive" flew right over my head, but I did spend some time in a fruitless search for a "ZFS magic flag".

Your "folders" solution is precisely what I was working to avert.
The dataset containing folders was so large that it would almost exhaust all drive space if it had to be maintained.
That's why I had to change the folders into datasets resulting in a dataset containing other datasets.
Originally, it took around 90 hours to replicate the dataset to a backup server over a gigabit link.
I never want to have to repeat another 90 hour "Replicate from scratch", now that the server is online.
Folders cannot be replicated, only datasets.

Now, the dataset contains 8 child datasets instead of 8 folders.
Each of the 8 child datasets are replicated separately.
The parent dataset is no longer replicated nor snapshotted.
A snapshot of the parent dataset is useless because that snapshot would reference only the 8 empty folders that underlie the 8 child datasets.
Snapshots also cannot see child datasets nested within a parent dataset.

I have no clue how, but recently all of the folders in the dataset were deleted.
The storage was pushed to 97% Used Space while restoring the deleted folders.
Of course, deleting snapshots after everything was restored, returned the Used Space to a normal 58%.
I was sweating bullets there for a while.

Now, the worst case "Replicate from scratch" for the largest child dataset will take about 18 hours.
Much more survivable.

All for now

 

[Patrick M. Hausen]

I will apologize in advance.

No need to. I got your point. Sometimes the technical precision and terseness in expression that I am used to come across as condescending but are definitely not intended to be.

 

[eewiz]

Hello All

I have discovered another benefit to a parent dataset full of child datasets.
A child dataset cannot be deleted from windows unlike a child folder, which can be deleted.
Trying to delete a dataset, which of course looks just like any other folder in windows, produces the following error.

I thoroughly appreciate that datasets which look like folders in windows cannot be deleted by accident.
By the way, I am the administrator of my system and, most definitely the one who originally deleted the 8 folders by accident.
Now it is simply impossible to make that mistake again.

I also discovered that even though TrueNAS/ZFS cannot see child datasets that reside within a parent dataset, samba can.
Server Message Block permits windows to see and traverse child datasets no matter what mount points are made or not made on the TrueNAS platform.

All for now