SOLVED NFS Client - cp/rsync/rclone fails until command is run for a second time

[essinghigh]

Hi All,
Running into a bit of a weird issue where my VM NFS clients are unable to cp a file to the TrueNAS NFS share until the command is run twice.
This also seems to cause issues with rsync, rclone, etc.

Code:

root@isobuild:/media/torrents/complete# cp archlinux-2023.08.01-x86_64.iso /mnt/nfs
cp: cannot create regular file '/mnt/nfs/archlinux-2023.08.01-x86_64.iso': Operation not permitted
root@isobuild:/media/torrents/complete# cp archlinux-2023.08.01-x86_64.iso /mnt/nfs
root@isobuild:/media/torrents/complete#

When the cp command is run for the first time, I get this "Operation not permitted" message, and a 0B file is created at /mnt/nfs.

Code:

root@isobuild:/media/torrents/complete# cp archlinux-2023.08.01-x86_64.iso /mnt/nfs
cp: cannot create regular file '/mnt/nfs/archlinux-2023.08.01-x86_64.iso': Operation not permitted
root@isobuild:/media/torrents/complete# ls -la /mnt/nfs | grep arch
-rwx------  1 root root    0 May 27  1982 archlinux-2023.08.01-x86_64.iso
root@isobuild:/media/torrents/complete#

The second time, the file is copied as expected. If I remove the 0B file and try again, I receive the "Operation not permitted" message again.

Code:

root@isobuild:/media/torrents/complete# cp debian-12.1.0-amd64-netinst.iso /mnt/nfs
cp: cannot create regular file '/mnt/nfs/debian-12.1.0-amd64-netinst.iso': Operation not permitted
root@isobuild:/media/torrents/complete# rm /mnt/nfs/debian-12.1.0-amd64-netinst.iso
root@isobuild:/media/torrents/complete# cp debian-12.1.0-amd64-netinst.iso /mnt/nfs
cp: cannot create regular file '/mnt/nfs/debian-12.1.0-amd64-netinst.iso': Operation not permitted
root@isobuild:/media/torrents/complete#

Not sure what is going on here, seems to be affected on multiple VMs.

Code:

root@debian:~# dd if=/dev/urandom of=test.file bs=1 count=0 seek=1G
0+0 records in
0+0 records out
0 bytes copied, 4.1778e-05 s, 0.0 kB/s
root@debian:~# cp test.file /mnt/nfs
cp: cannot create regular file '/mnt/nfs/test.file': Operation not permitted
root@debian:~# ls -la /mnt/nfs | grep test
.rwx------ 0 root 30 Nov  2004 test.file
root@debian:~# cp test.file /mnt/nfs
root@debian:~# ls -la /mnt/nfs | grep test
.rwx------ 1.1G root  2 Sep 21:51 test.file
root@debian:~#

My config on TrueNAS for the NFS share is as follows (I understand this might not be best practice from a security perspective, was just having a play about with NFS as had not set it up before).

Path: /mnt/data/SMB (Was previously an SMB share)
Both servers are authorized hosts
Read Only: Disabled
Mapall User: root
Mapall Group: root

The dataset is configured with NFSv4 permissions, as follows from when it was configured as an SMB share (this probably isn't the cause but want to add as much info as I can):
Group - Media | Allow, Full Access
User - libvirt-qemu | Allow, Full Access
User - Administrator | Allow, Full Access
User - root | Allow, Full Access
Owner & Group is root.

/etc/exports:

Code:

"/mnt/data/SMB"\
        192.168.1.254(sec=sys,rw,anonuid=0,anongid=0,all_squash,insecure,no_subtree_check)\
        192.168.1.150(sec=sys,rw,anonuid=0,anongid=0,all_squash,insecure,no_subtree_check)\
        192.168.2.2(sec=sys,rw,anonuid=0,anongid=0,all_squash,insecure,no_subtree_check)\
        192.168.2.3(sec=sys,rw,anonuid=0,anongid=0,all_squash,insecure,no_subtree_check)\
        192.168.1.235(sec=sys,rw,anonuid=0,anongid=0,all_squash,insecure,no_subtree_check)

VM(s) /etc/fstab:

Code:

192.168.2.1:/mnt/data/SMB/ /mnt/nfs nfs sync,nolock,mountvers=3,nfsvers=3 0 0

Where the clients can read (and eventually write) to the share I'm guessing it's not a permissions issue, so no idea what could be the cause.

Any pointers would be much appreciated, as can't seem to find anything relevant online with the exception of an IBM support page, which does not seem to be of any use....
Thanks in advance :)

 

[essinghigh]

Okay, looks like the issue is with the dataset, probably permissions structure.
Created a new dataset and am having no issues. Where it was used as an SMB share previously, perhaps I need to migrate the permissions back to POSIX structure instead of SMB/NFSv4? Getting a warning saying it will be "destructive", sounds like great fun! I'll give this a test in a little bit as am not too worried about any potential destruction.

 

[winnielinnie]

Getting a warning saying it will be "destructive", sounds like great fun!

You can create a manual (temporary) snapshot immediately before doing this. If something messes up badly in regards to permissions, you can rollback to the snapshot, and you will restore the ACLs as they previously were.

 

[essinghigh]

Perfect, migrating back to a POSIX style ACL looks to have worked nicely! Using the POSIX-Open preset for the moment - will fine-tune in at some point in the near-future if I don't forget.
Saturating a 10gig link on write too, couldn't manage that with SMB!

For anyone who happens to stumble across this - here's how I solved it:
Create a snapshot of your dataset (thanks @winnielinnie, I always forget about this step!)
Datasets > [Your Dataset] > Edit Dataset Details > Advanced Options > Set ACL Type: POSIX
After this is done, apply permissions recursively
Then, give yourself a pat on the back for having modified the ACL Type without setting your server, your home, and the surrounding area on fire.