FGuillaume
Cadet
- Joined
- Aug 2, 2023
- Messages
- 1
Hello,
I recently ran into an issue trying to use a truenas scale nfs share to share data between pods in my kubernetes cluster.
The share is perfectly accessible from any VM of the server.
However my pods are not able to mount the share and throw the following error :
My truenas nfs service config is the following :
And the shared folder ACL access :
Here are the nfs-server logs when I try to connect to the share :
I initially thought the issue was about my kubernetes persistent volume claim configuration, so I tried a few fixes without any success. When I shell into a container to try to mount the share by hand I get this :
I finally tried to connect to the share using a simple docker container outside of the cluster and got roughly the same output :
Since I got the same error from both outside and inside the cluster, the kubernetes config files should not be the issue here.
Does anyone have an idea about a potential solution to this issue ?
Thanks a lot!
I recently ran into an issue trying to use a truenas scale nfs share to share data between pods in my kubernetes cluster.
The share is perfectly accessible from any VM of the server.
However my pods are not able to mount the share and throw the following error :
Code:
2 secs ago MountVolume.SetUp failed for volume "nfs-volume" : mount failed: exit status 32 Mounting command: mount Mounting arguments: -t nfs 192.168.1.35:/mnt/Fast-Raid-z/andromeda_cluster_storage/certificates/devblog /var/lib/kubelet/pods/e02f2931-0a94-41eb-a989-139c7e1dc010/volumes/kubernetes.io~nfs/nfs-volume Output: mount.nfs: access denied by server while mounting 192.168.1.35:/mnt/Fast-Raid-z/andromeda_cluster_storage/certificates/devblog
My truenas nfs service config is the following :
And the shared folder ACL access :
Here are the nfs-server logs when I try to connect to the share :
Code:
Sep 12 11:45:54 truenas kernel: nfsd4_sequence: slotid 0 Sep 12 11:45:54 truenas kernel: check_slot_seqid enter. seqid 31 slot_seqid 30 Sep 12 11:45:54 truenas kernel: nfsd: fh_verify(20: 00060001 f602fb7a 048fa000 00000000 00000000 00000000) Sep 12 11:45:54 truenas kernel: nfsd: fh_verify(20: 00060001 f602fb7a 048fa000 00000000 00000000 00000000) Sep 12 11:45:54 truenas kernel: --> nfsd4_store_cache_entry slot 00000000bb2be616 Sep 12 11:45:54 truenas kernel: __find_in_sessionid_hashtbl: 1694511163:2078428104:723:0 Sep 12 11:45:54 truenas kernel: nfsd4_sequence: slotid 0 Sep 12 11:45:54 truenas kernel: check_slot_seqid enter. seqid 32 slot_seqid 31 Sep 12 11:45:54 truenas kernel: nfsd: fh_verify(20: 00060001 f602fb7a 048fa000 00000000 00000000 00000000) Sep 12 11:45:54 truenas kernel: nfsd: fh_verify(20: 00060001 f602fb7a 048fa000 00000000 00000000 00000000) Sep 12 11:45:54 truenas kernel: --> nfsd4_store_cache_entry slot 00000000bb2be616 Sep 12 11:45:54 truenas kernel: __find_in_sessionid_hashtbl: 1694511163:2078428104:723:0 Sep 12 11:45:54 truenas kernel: nfsd4_sequence: slotid 0 Sep 12 11:45:54 truenas kernel: check_slot_seqid enter. seqid 33 slot_seqid 32 Sep 12 11:45:54 truenas kernel: nfsd: fh_verify(20: 00060001 f602fb7a 048fa000 00000000 00000000 00000000) Sep 12 11:45:54 truenas kernel: nfsd: fh_verify(20: 00060001 f602fb7a 048fa000 00000000 00000000 00000000) Sep 12 11:45:54 truenas kernel: --> nfsd4_store_cache_entry slot 00000000bb2be616 Sep 12 11:45:54 truenas kernel: __find_in_sessionid_hashtbl: 1694511163:2078428104:723:0 Sep 12 11:45:54 truenas kernel: nfsd4_sequence: slotid 0 Sep 12 11:45:54 truenas kernel: check_slot_seqid enter. seqid 34 slot_seqid 33 Sep 12 11:45:54 truenas kernel: nfsd: fh_verify(20: 00060001 f602fb7a 048fa000 00000000 00000000 00000000) Sep 12 11:45:54 truenas kernel: nfsd: fh_verify(20: 00060001 f602fb7a 048fa000 00000000 00000000 00000000) Sep 12 11:45:54 truenas kernel: --> nfsd4_store_cache_entry slot 00000000bb2be616 Sep 12 11:45:54 truenas kernel: __find_in_sessionid_hashtbl: 1694511163:2078428104:723:0 Sep 12 11:45:54 truenas kernel: nfsd4_sequence: slotid 0 Sep 12 11:45:54 truenas kernel: check_slot_seqid enter. seqid 35 slot_seqid 34 Sep 12 11:45:54 truenas kernel: nfsd: fh_verify(20: 00060001 f602fb7a 048fa000 00000000 00000000 00000000) Sep 12 11:45:54 truenas kernel: nfsd: fh_verify(20: 00060001 f602fb7a 048fa000 00000000 00000000 00000000) Sep 12 11:45:54 truenas kernel: nfsd: fh_verify(20: 00060001 f602fb7a 048fa000 00000000 00000000 00000000) Sep 12 11:45:54 truenas kernel: --> nfsd4_store_cache_entry slot 00000000bb2be616 Sep 12 11:45:54 truenas kernel: nfsd4_destroy_session: 1694511163:2078428104:723:0 Sep 12 11:45:54 truenas kernel: __find_in_sessionid_hashtbl: 1694511163:2078428104:723:0 Sep 12 11:45:54 truenas kernel: nfsd4_umh_cltrack_upcall: cmd: remove Sep 12 11:45:54 truenas kernel: nfsd4_umh_cltrack_upcall: arg: 4c696e7578204e465376342e322067616d6d61 Sep 12 11:45:54 truenas kernel: nfsd4_umh_cltrack_upcall: env0: (null) Sep 12 11:45:54 truenas kernel: nfsd4_umh_cltrack_upcall: env1: (null) Sep 12 11:45:54 truenas kernel: nfsd4_umh_cltrack_upcall: /sbin/nfsdcltrack return value: 0 Sep 12 11:45:56 truenas kernel: nfsd: FSSTAT(3) 20: 00060001 45e6973e b9508200 00000000 00000000 00000000 Sep 12 11:45:56 truenas kernel: nfsd: fh_verify(20: 00060001 45e6973e b9508200 00000000 00000000 00000000) Sep 12 11:45:56 truenas kernel: nfsd: FSSTAT(3) 20: 00060001 e9bed6c2 f6993700 00000000 00000000 00000000 Sep 12 11:45:56 truenas kernel: nfsd: fh_verify(20: 00060001 e9bed6c2 f6993700 00000000 00000000 00000000) Sep 12 11:45:56 truenas kernel: nfsd: WRITE(3) 32: 01060001 45e6973e b9508200 00000000 00000000 0101000a 4096 bytes at 9035235328 stable Sep 12 11:45:56 truenas kernel: nfsd: WRITE(3) 32: 01060001 45e6973e b9508200 00000000 00000000 0101000a 4096 bytes at 9035247616 stable Sep 12 11:45:56 truenas kernel: nfsd: fh_verify(32: 01060001 45e6973e b9508200 00000000 00000000 0101000a) Sep 12 11:45:56 truenas kernel: nfsd: fh_verify(32: 01060001 45e6973e b9508200 00000000 00000000 0101000a) Sep 12 11:45:56 truenas kernel: nfsd: FSSTAT(3) 20: 00060001 14eb2d1c b75fda00 00000000 00000000 00000000 Sep 12 11:45:56 truenas kernel: nfsd: fh_verify(20: 00060001 14eb2d1c b75fda00 00000000 00000000 00000000) Sep 12 11:45:57 truenas kernel: nfsd: WRITE(3) 32: 01060001 45e6973e b9508200 00000000 00000000 0101000a 8192 bytes at 6319927296 stable Sep 12 11:45:57 truenas kernel: nfsd: fh_verify(32: 01060001 45e6973e b9508200 00000000 00000000 0101000a) Sep 12 11:45:57 truenas kernel: nfsd: WRITE(3) 32: 01060001 45e6973e b9508200 00000000 00000000 0101000a 4096 bytes at 6319935488 stable Sep 12 11:45:57 truenas kernel: nfsd: fh_verify(32: 01060001 45e6973e b9508200 00000000 00000000 0101000a) Sep 12 11:46:01 truenas kernel: nfsd: WRITE(3) 32: 01060001 45e6973e b9508200 00000000 00000000 0101000a 4096 bytes at 9035247616 stable Sep 12 11:46:01 truenas kernel: nfsd: fh_verify(32: 01060001 45e6973e b9508200 00000000 00000000 0101000a) Sep 12 11:46:06 truenas kernel: nfsd: FSSTAT(3) 20: 00060001 45e6973e b9508200 00000000 00000000 00000000 Sep 12 11:46:06 truenas kernel: nfsd: fh_verify(20: 00060001 45e6973e b9508200 00000000 00000000 00000000) Sep 12 11:46:06 truenas kernel: nfsd: FSSTAT(3) 20: 00060001 14eb2d1c b75fda00 00000000 00000000 00000000 Sep 12 11:46:06 truenas kernel: nfsd: fh_verify(20: 00060001 14eb2d1c b75fda00 00000000 00000000 00000000) Sep 12 11:46:06 truenas kernel: nfsd: WRITE(3) 32: 01060001 45e6973e b9508200 00000000 00000000 0101000a 8192 bytes at 6319939584 stable Sep 12 11:46:06 truenas kernel: nfsd: WRITE(3) 32: 01060001 45e6973e b9508200 00000000 00000000 0101000a 4096 bytes at 9035247616 stable Sep 12 11:46:06 truenas kernel: nfsd: fh_verify(32: 01060001 45e6973e b9508200 00000000 00000000 0101000a) Sep 12 11:46:06 truenas kernel: nfsd: fh_verify(32: 01060001 45e6973e b9508200 00000000 00000000 0101000a) Sep 12 11:46:06 truenas kernel: nfsd: WRITE(3) 32: 01060001 45e6973e b9508200 00000000 00000000 0101000a 4096 bytes at 6319947776 stable Sep 12 11:46:06 truenas kernel: nfsd: fh_verify(32: 01060001 45e6973e b9508200 00000000 00000000 0101000a) Sep 12 11:46:06 truenas kernel: nfsd: FSSTAT(3) 20: 00060001 e9bed6c2 f6993700 00000000 00000000 00000000 Sep 12 11:46:06 truenas kernel: nfsd: fh_verify(20: 00060001 e9bed6c2 f6993700 00000000 00000000 00000000)
I initially thought the issue was about my kubernetes persistent volume claim configuration, so I tried a few fixes without any success. When I shell into a container to try to mount the share by hand I get this :
Code:
root@devblog-app-deployment-9c7d5c495-rpvt5:/app# mount -vvv 192.168.1.35:/Fast-Raid-z/andromeda_cluster_storage/certificates/devblog /home -o nolock mount.nfs: timeout set for Tue Sep 12 10:04:54 2023 mount.nfs: trying text-based options 'nolock,vers=4.2,addr=192.168.1.35,clientaddr=10.42.0.65' mount.nfs: mount(2): No such file or directory mount.nfs: trying text-based options 'nolock,addr=192.168.1.35' mount.nfs: prog 100003, trying vers=3, prot=6 mount.nfs: trying 192.168.1.35 prog 100003 vers 3 prot TCP port 2049 mount.nfs: prog 100005, trying vers=3, prot=17 mount.nfs: trying 192.168.1.35 prog 100005 vers 3 prot UDP port 38745 mount.nfs: mount(2): Permission denied mount.nfs: access denied by server while mounting 192.168.1.35:/Fast-Raid-z/andromeda_cluster_storage/certificates/devblog
I finally tried to connect to the share using a simple docker container outside of the cluster and got roughly the same output :
Code:
root@0f8c8973d9aa:/home# mount -vvv 192.168.1.35:/mnt/Fast-Raid-z/andromeda_cluster_storage/certificates/devblog /home/test -o nolock mount.nfs: timeout set for Tue Sep 12 09:34:56 2023 mount.nfs: trying text-based options 'nolock,vers=4.2,addr=192.168.1.35,clientaddr=172.17.0.3' mount.nfs: mount(2): Permission denied mount.nfs: trying text-based options 'nolock,vers=4,minorversion=1,addr=192.168.1.35,clientaddr=172.17.0.3' mount.nfs: mount(2): Permission denied mount.nfs: trying text-based options 'nolock,vers=4,addr=192.168.1.35,clientaddr=172.17.0.3' mount.nfs: mount(2): Permission denied mount.nfs: trying text-based options 'nolock,addr=192.168.1.35' mount.nfs: prog 100003, trying vers=3, prot=6 mount.nfs: trying 192.168.1.35 prog 100003 vers 3 prot TCP port 2049 mount.nfs: prog 100005, trying vers=3, prot=17 mount.nfs: trying 192.168.1.35 prog 100005 vers 3 prot UDP port 38745 mount.nfs: access denied by server while mounting 192.168.1.35:/mnt/Fast-Raid-z/andromeda_cluster_storage/certificates/devblog
Since I got the same error from both outside and inside the cluster, the kubernetes config files should not be the issue here.
Does anyone have an idea about a potential solution to this issue ?
Thanks a lot!
