NFS share only able to see some folders

[jblack]

I'm having an issue where the client can only see some folders present in an NFS share. I'm pretty sure it's a permissions issue, but I cannot figure out what I need to change. I'll go into excruciating detail below, but the tl:dr is I created a series of nested datasets (ssdpool > nfs-test > nfs-nested > nfs-final) with default inherited settings. Then I created an NFS share on path /mnt/ssdpool/nfs-test with maproot user root and maproot group sudo. When I mount the NFS share onto a client machine (ubuntu server) I can only see the nfs-nested folder, not the nfs-final folder that's inside it. All folders show owned by root in the TrueNAS shell and the client is using root user directly.

Details:​

These are the steps I took and the results I got.

• Create Datasets in TrueNAS
• Enable NFS service in TrueNAS
• Create NFS share in TrueNAS
• Mount NFS share in client
• Results
• Things I tried
• Hardware details

Create Datasets in TrueNAS​

nfs-test​

Go to Storage > ssdpool > Add Dataset with basic settings:

• Parent path: ssdpool
• Name: nfs-test
• Sync: Inherit (standard)
• Compression level: Inherit (lz4)
• Enable Atime: Inherit (off)
• Encryption Options: Inherit (non-encrypted)
• ZFS Deduplication: Inherit (off)
• Case Sensitivity: Sensitive
• Share Type: Generic

nfs-nested​

Go to Storage > ssdpool > nfs-test > Add Dataset with basic settings:

• Parent path: ssdpool/nfs-test
• Name: nfs-nested
• Sync: Inherit (standard)
• Compression level: Inherit (lz4)
• Enable Atime: Inherit (off)
• Encryption Options: Inherit (non-encrypted)
• ZFS Deduplication: Inherit (off)
• Case Sensitivity: Sensitive
• Share Type: Generic

nfs-final​

Go to Storage > ssdpool > nfs-test > nfs-nested > Add Dataset with basic settings:

• Parent path: ssdpool/nfs-test/nfs-nested
• Name: nfs-final
• Sync: Inherit (standard)
• Compression level: Inherit (lz4)
• Enable Atime: Inherit (off)
• Encryption Options: Inherit (non-encrypted)
• ZFS Deduplication: Inherit (off)
• Case Sensitivity: Sensitive
• Share Type: Generic

I didn't touch any of the advanced settings.
I then went into the TrueNAS shell and ran touch /mnt/ssdpool/nfs-test/nfs-nested/nfs-final/temp.txt to create a file at the end of the chain.

Enable NFS service in TrueNAS​

Go to Shares > UNIX (NFS) Shares > Turn on service
Go to Shares > UNIX (NFS) Shares > Config service with settings:

• Number of servers: 16
• Bind IP Addresses: 192.168.13.180 (the ip of my TrueNAS machine)
• Enable NFSv4
• Check Leave NFSv3 ownership model for NFSv4 and Require Kerberos for NFSv4 unchecked
• Leave Ports options blank
• Check Serve UDP NFS clients
• Check Allow non-root mount
• Leave support >16 groups unchecked

Create NFS share in TrueNAS​

Go to Shares > UNIX (NFS) Shares > Add with advanced settings:

• Path: /mnt/ssdpool/nfs-test
• Enabled: yes
• Read only: no
• Maproot user: root
• Maproot group: sudo
• Mapall user: blank
• Mapall group: blank
• Security: blank
• Networks: none
• Hosts: none

Mount NFS share in client​

Setup a brand new ubuntu 22.04 server virtual machine on a proxmox VE server. Run sudo su to switch to root user. Run apt update, apt upgrade, and apt install nfs-common. Run mkdir /mnt/nfs-mount to make a mount point, followed by mount 192.168.13.180:/mnt/ssdpool/nfs-test /mnt/nfs-mount to mount the share.

Results​

Inspecting the mounted share within the client with ls -l yields:

Code:

root@ubuntu:~# ls -l /mnt/
total 1
drwxr-xr-x 3 root root 3 Feb  2 14:22 nfs-mount
root@ubuntu:~# ls -l /mnt/nfs-mount/
total 1
drwxr-xr-x 2 root root 2 Feb  2 14:33 nfs-nested
root@ubuntu:~# ls -l /mnt/nfs-mount/nfs-nested/
total 0

It sucessfully mounted the NFS share as nfs-test and the nfs-nested folder is visible inside the mounted share, but the nfs-final folder is not, nor is the temp.txt file.
Inspecting the datasets from within the TrueNAS shell yields:

Code:

root@truenas[~]# ls -l /mnt/ssdpool/nfs-test 
total 1
drwxr-xr-x 3 root root 3 Feb  2 06:23 nfs-nested
root@truenas[~]# ls -l /mnt/ssdpool/nfs-test/nfs-nested 
total 1
drwxr-xr-x 2 root root 3 Feb  2 06:31 nfs-final
root@truenas[~]# ls -l /mnt/ssdpool/nfs-test/nfs-nested/nfs-final 
total 1
-rw-r--r-- 1 root root 0 Feb  2 06:31 temp.txt

Which seems to me to indicate that all the folders and files involved are owned by root, same as within the ubuntu client. I see no reason that nfs-test and nfs-nested should be visible to the client, but nfs-final and temp.txt should not.

Things I tried​

• Different client, this time an Alpine Linux VM: no change
• Setting the Maproot group to root instead of sudo: no change
• Mounting nfs-nested directly with umount /mnt/nfs-mount then mount 192.168.13.180:/mnt/ssdpool/nfs-test/nfs-nested /mnt/nfs-mount: sucessfully mounted, but showed as empty, like before.
• Mounting nfs-final directly with umount /mnt/nfs-mount then mount 192.168.13.180:/mnt/ssdpool/nfs-test/nfs-nested/nfs-final /mnt/nfs-mount: failed with
  Code:

  Created symlink /run/systemd/system/remote-fs.target.wants/rpc-statd.service → /lib/systemd/system/rpc-statd.service.
  [*]mount.nfs: mounting 192.168.13.180:/mnt/ssdpool/nfs-test/nfs-nested/nfs-final failed, reason given by server: No such file or directory
  [*]

• Adding the ubuntu client's IP address in add hosts section of the NFS share: no change (as expected)

Hardware​

I don't think this will make a difference, but the rules make it clear that hardware should be listed.
TrueNAS machine:

• Motherboard: generic aliexpress mini-itx embedded NAS board (https://www.aliexpress.us/item/3256804565944286.html)
• CPU: Embedded Intel Celeron N5105
• 16GB DDR4 SODIMM RAM
• 4x4TB HDD (not ssdpool)
• 5x120GB SSD (ssdpool, raid z1)
• quad Intel I225-V 2.5gbe NICs
• I Think that's it

Client machine:

• Ubuntu 22.04 Server Virtual Machine on Proxmox. Also tried Alpine Linux Virtual Machine on Proxmox

I ran into this same issue a few months ago, but after asking for help on r/truenas and getting a bunch of people basically saying 'it's a permissions issue duh' without any further explanation, I gave up on it. I've now run into another situation where it would be very nice and thought I'd take another stab at NFS, but have had no more luck this time than last.
I know this is a very long post, so if you're still here, thank you. Any help at all would be greatly appreciated, tutorials, related forum posts, whatever.

 

[jgreco]

When I mount the NFS share onto a client machine (ubuntu server) I can only see the nfs-nested folder, not the nfs-final folder that's inside it.

Yup. You've correctly described what is happening to you, but not why you think this behaviour is incorrect.

A NFS export is limited to the filesystem (dataset, in ZFS parlance) that is listed in the exports file. It will not serve files on filesystems that are mounted on top of it at a deeper level as though they were part of the filesystem actually exported. You would have to mount those deeper filesystems separately, and make sure that mount attempts for them were ordered correctly in fstab.

 

[jblack]

You would have to mount those deeper filesystems separately

Meaning create a new NFS share for every dataset I want to be accessible? So if I want to be able to access nfs-test, nfs-nested, and nfs-final, I would need to create an individual NFS share for each?

, and make sure that mount attempts for them were ordered correctly in fstab.

Then the individual NFS shares would need to be mounted individually as well? What would be the proper order for mounting these? Would I go top-to-bottom, mounting nfs-test, then nfs-nested, then nfs-final? Or vice versa?

You've answered my question on what's happening, but now I have so many more questions.

 

[jblack]

Meaning create a new NFS share for every dataset I want to be accessible? So if I want to be able to access nfs-test, nfs-nested, and nfs-final, I would need to create an individual NFS share for each?

Then the individual NFS shares would need to be mounted individually as well? What would be the proper order for mounting these? Would I go top-to-bottom, mounting nfs-test, then nfs-nested, then nfs-final? Or vice versa?

You've answered my question on what's happening, but now I have so many more questions.

Nevermind, after some more research and playing around, I think I understand. I simply did not realize how NFS actually worked. It would seem that I do in fact need a share for each individual dataset. I still don't know whether top-to-bottom mounting or bottom-to-top mounting is the way to go, but I can cross that bridge when I get to it, since I don't see any reason I'd need to mount both the nfs-nested and nfs-final folders (in my current project at least).
Seems like NFS is working just fine, but I'm gonna end up with a lot of shares very quickly here.

 

[jgreco]

Meaning create a new NFS share for every dataset I want to be accessible? So if I want to be able to access nfs-test, nfs-nested, and nfs-final, I would need to create an individual NFS share for each?

Yes, that is correct. mountd only works on filesystems (or ZFS "datasets" which appear to the filesystem layer as something very much like one).

Then the individual NFS shares would need to be mounted individually as well? What would be the proper order for mounting these? Would I go top-to-bottom, mounting nfs-test, then nfs-nested, then nfs-final? Or vice versa?

Shortest path first. Now that we're looking at this from the other (client) side, the question becomes, what sort of behaviour are you looking for? Something like Linux crossmnt? FreeBSD doesn't have that, though you can certainly use the automounter (autofs) to help manage mounts, especially ones that would best be transient. I personally feel permanent mounts are best done in fstab. Combining the automounter with a directory service gives you a nice way to be able to rearchitect fleets of clients from a single management point. This works out very nicely if you have lots of mountpoints but only need a few at a time.

 

[jblack]

Shortest path first.

That does make more sense to me than the other way around.

Now that we're looking at this from the other (client) side, the question becomes, what sort of behaviour are you looking for? Something like Linux crossmnt? FreeBSD doesn't have that, though you can certainly use the automounter (autofs) to help manage mounts, especially ones that would best be transient. I personally feel permanent mounts are best done in fstab. Combining the automounter with a directory service gives you a nice way to be able to rearchitect fleets of clients from a single management point. This works out very nicely if you have lots of mountpoints but only need a few at a time.

My main goal with NFS shares is to use it to enable remote Docker volumes through portainer. The idea being that it's easier to move the containers to different machines if the important volumes are already remote. Plus it means that the system running the docker containers doesn't need to actually store the volumes, so the storage for that host system can be very small, even if the docker volumes get big. For this I would need to have lots of mount points all going at once, but portainer has a system built in to manage mounting, so that's not a problem. I'll take a look at automounter though, it seems like something that may become handy in the future.
Thank you very much for all your help!

 

[jgreco]

My main goal with NFS shares is to use it to enable remote Docker volumes through portainer.

Sorry, eyes glazed over by the word "portainer". Not familiar enough with any of that to have any suggestions for you.

I'll take a look at automounter though, it seems like something that may become handy in the future.

Automounter's cool. Every NFS admin should be familiar with it. Doesn't always mean it is the right solution to a problem, but failure to be aware of it only hurts.

Thank you very much for all your help!

Good luck on your NFS mounting adventure. Hope this was enough to get you straightened out.