Issues with NFS on TrueNAS Scale

[Kosta]

Hello,

I am confronting an issue when trying to access an NFS share on TrueNAS Scale from an Zidoo Z9x media player - Z9x simply can't access the NFS share on Scale. However, settings should be correct because:
- Z9x can mount an NFS share on another NAS, a Synology
- an alternative Linux server (Ubuntu) can mount an NFS share on the TrueNAS Scale
So, neither seems to be wrongly configured.
I x-checked all relevant settings:
It's enabled, Mapall User root, Security SYS, no restrictions in network or hosts.
NFS is started and no specific settings have been made for NFS.

I also used to have TrueNAS Core, and I know that NFS access on both NAS was working, I had it set up like that. So I believe it is some kind of incompatibility between TrueNAS Scale and Z9x.
I also tried restoring the old VM, without the pool, only a single disk, which I used for a simple NFS share - and Zidoo connected at once.

I also noticed that the NFS-Share (IP-Address) does not appear on the Zidoo when I try with Scale. It does however when I try with Core. It's like Zidoo doesn't even see the share with Scale.

I restored my old TrueNAS Core VM, and tested without any pools, and confirmed that NFS was yet again working as usual. So it is definitely the Scale.

Any ways I can troubleshoot this and get Scale to work? Failing to do that, the only way will be to go back to Core, which I'm not very keen to do, there are reasons I switched. Question is also, if I can import the current ZFS pool under Scale into latest version of Core.

Thank you

 

[morganL]

Puzzling.. can you confirm network connectivity via ping.

Does Z9x support SMB... just as a test?

 

[Kosta]

Yes, network connectivity is given, I can ping Z9x from TrueNAS Scale SSH. Also made sure that I don't use DHCP, so that I can set my domain controller as DNS. Not that it matters really, tried with DHCP too.
I also checked with netstat -tulpn, I can see both 0.0.0.0:111 and <TrueNAS-IP>:2049 ports listening.
I even tried rebooting the firewall (OPNsense, dedicated hardware) and the switch (HP Aruba 1930).

Yes, Z9x does support SMB, however I can't get it to work with an active directory account, even after setting SMB1 and NTLMv1. I know however that this authentication is working, since I have both my PC (which is logged in with the same account) and my Macbook (which has the SMB connected with that same account) and can access shares.

I also didn't get it to work with a local account on TrueNAS Scale, not sure why really. I tried with a completely fresh Dataset and open permissions, also gave the local user complete rights on the share. The only configuration that I got to work was a local user on Synology, even there it failed with a domain user. All these issues with SMB is the exact reason why I always went with Read-Only NFS for Z9x.

 

[samarium]

I think I saw something recently with a mismatch between NFS versions and rpcinfo / rpcbind.

If Z9 is sending rpcinfo request, and not getting a response in a reasonable time, maybe because it is trying to use NFS v 2 or v3? vs just trying to mount and using port 2049 without trying a rpcinfo request like IIRC NFS v4 does? At least that seemed to he happening to the person I was talking to. Slightly different man pages on the linux implimentations that were trying to mount the TNS NFS share.

You might want to use tcpdump to capture a packet trace, and then compare that to what you expect, and preferably to a known working NFS mount.

 

[Kosta]

Can you please just clarify on which device you mean a tcpdump? Never did it with tcpdump on the VM itself (via SSH).

 

[samarium]

TNS seems like the easiest, and would be central as it has the NFS shares you want to mount.

I have not idea about the Z9x, and while it would be preferable to have a complimentary tcpdump on it, that might not be possible if is just an embedded media player.

For the working comparison box, it would also be good to capture the traffic, again as a comparison to see that the same traffic is visible on both ends.

Make sure time is synced too, so timestamps make sense in comparison if they are encountered.

Since you have a working Z9x Syno connection, that may be comparatively informative too.

 

[Kosta]

Well, I did three things now:
1) I port mirrored a port where Z9x is hanging on the switch, and hang my laptop on the mirrored port. I see the traffic. When I attempt to connect to the TrueNAS from Z9x, I see packets going from and to in that moment. If I type nfs into Wireshark filter, nothing comes up.
2) I ran tcpdump -w *file* -i <interface> host <zidoo-ip> and on both Scale and Core, and confirmed that on Scale it fails and on Core was successful, and exported .dmp file on both sessions, and looked at them in Wireshark. Filtering nfs also doesn't show anything.
In fact, I believe I should see something like Network File System blah blah... but no.
Kinda stumped. It might be that I am doing something wrong in Wireshark. I see protocol MOUNT.

And btw. if I do a simple tcpdump on the NAS, I see "sunrpc". Not sure what that means? Maybe Scale can't do sunrpc?

Syno I could do, but have to sniff on the Zidoo port nevertheless, because not sure if I can easily sniff the LACP on the Syno.

Also not sure what you mean by TNC.

 

[Kosta]

I noticed in the dump that jumboframes 9000 on Core was on. Lowered that to 1500, no difference however. Core still connects.

 

[Kosta]

OK sorry, that was me reading Wireshark wrong. I see something peculiar:

Doing tcpdump on the Z9x port...
With Core, it begins with destination port 111, does a usual SYN, ACK and FIN, and then at the next step follow packets with destination port 778, again full SYN, ACK and FIN procedure. And it continues with 778.
Following are MOUNT protocols, with "RPC executed successfully".
However, with Scale, it begins the same, port 111, but then, instead 778, it goes to high ports, 35613.
Also, on MOUNT, it says "Accept State: remote can't support version # (2)"

Also going to Synology, it say "RPC Version: 2".

Does that mean that Zidoo is trying to use NFS2?

 

[Kosta]

Well, again stuck.
I enabled NFSv2 in Scale, I changed:
/etc/default/nfs-kernel-server
RPCNFSDCOUNT="4 -V 2"
(I guess 4 is coming from the number of threads)
Restarted the service...
Made sure NFSv2 is started and listening:
cat /proc/fs/nfsd/versions
+2 instead of -2
rpcinfo -p localhost | fgrep nfs
shown both 2 and 3 listening on 2049

Nothing...

Rebooted TrueNAS, RPCNFSDCOUNT="4 -V 2" goes back to "4", NFSv2 deactivated.

Is there a way to activate NFSv2?
I read that Core is backwards compatible with NFS2, due to FreeBSD implementation. How is it with Scale?

 

[samarium]

Sounds similar, but the thread didn't complete after my suggestions, so I don't know what happened on the other thread.
Talking to rpcbind, probably wanting to talk to mountd rpc, initially to so a V3 mount, then would ran to talk the nfs port. The actual mountd port number is randomish, and not important, that is what portmapper/rpcbind is for to return the correct port.
What does rpcinfo -p core and rpcinfo -p scale show? Identical except for the actual port numbers which can change?
NFS V2 is very old, UDP too IIRC, not surprised it isn't working. NFS 3 and 4 are using TCP IIRC. NFS 3 probably requires client to talk to portapper/rpcbind, while 4 might be able to go direct on port 2049. Those my impressions. Don't have a solution, but you at least have a better idea what is going on, and maybe can work forward.

 

[Kosta]

Not sure what you mean with thread and another thread.
Anyway...

rpcinfo -p core:
program vers proto port service
100000 4 tcp 111 rpcbind
100000 3 tcp 111 rpcbind
100000 2 tcp 111 rpcbind
100000 4 udp 111 rpcbind
100000 3 udp 111 rpcbind
100000 2 udp 111 rpcbind
100000 4 local 111 rpcbind
100000 3 local 111 rpcbind
100000 2 local 111 rpcbind
100005 1 udp 702 mountd
100005 3 udp 702 mountd
100005 1 tcp 702 mountd
100005 3 tcp 702 mountd
100003 2 tcp 2049 nfs
100003 3 tcp 2049 nfs
100024 1 udp 690 status
100024 1 tcp 690 status
100021 0 udp 773 nlockmgr
100021 0 tcp 695 nlockmgr
100021 1 udp 773 nlockmgr
100021 1 tcp 695 nlockmgr
100021 3 udp 773 nlockmgr
100021 3 tcp 695 nlockmgr
100021 4 udp 773 nlockmgr
100021 4 tcp 695 nlockmgr


rpcinfo -p scale:
program vers proto port service
100000 4 tcp 111 portmapper
100000 3 tcp 111 portmapper
100000 2 tcp 111 portmapper
100000 4 udp 111 portmapper
100000 3 udp 111 portmapper
100000 2 udp 111 portmapper
100005 3 udp 40495 mountd
100005 3 tcp 49799 mountd
100024 1 udp 53156 status
100024 1 tcp 43263 status
100003 3 tcp 2049 nfs
100227 3 tcp 2049
100021 1 udp 35194 nlockmgr
100021 3 udp 35194 nlockmgr
100021 4 udp 35194 nlockmgr
100021 1 tcp 44173 nlockmgr
100021 3 tcp 44173 nlockmgr
100021 4 tcp 44173 nlockmgr

You see, in the upper one, 100003 2 tcp 2049 nfs, I guess that makes a difference.

I am aware that NFS2 is old, I will ask in the Z9x forum if there is a chance for NFSv3, but I am afraid I am out of luck here.
I think I have three courses of action:
1) manage to enable NFSv2 on Scale
2) managed to enable NFSv2 on Z9x
3) downgrade to Core
(or change my media player)

 

[samarium]

You see, in the upper one, 100003 2 tcp 2049 nfs, I guess that makes a difference.

I am aware that NFS2 is old, I will ask in the Z9x forum if there is a chance for NFSv3, but I am afraid I am out of luck here.
I think I have three courses of action:
1) manage to enable NFSv2 on Scale
2) managed to enable NFSv2 on Z9x
3) downgrade to Core
(or change my media player)

There was another NFS issue thread, just like this thread, with a similar NFS issue, but it wasn't completed, so I don't know what the resolution was.

Yes, core is showing NFSv2 support, but over TCP only. 778 I'm guessing will probably the network lock manager, or one of the other RPCs.

Scale is currently just showing NFS v3 and 100227 which is nfs_acl RPC. I don't think you gain anything by disabling NFS V4. You only gain if you can enable V2, which I'm guessing isn't complied into the binary. Scale seems to be using high number ports for RPC services by default, probably more recent software, less predictable, more secure, which is probably also why NFSv2 is not supported.

Not sure what would be workable as a workaround. There is a user mode, not kernel mode, uNFSd, from who knows how many years ago, which might have worked well enough to work for you for light duty music serving, but scale is an appliance not a linux server, so difficult to get it to work, even if it compiles these days.

You might run core, or another older linux server in a VM, and but they have to figure out how to import NFS on one hand and also export it as NFSv2 on the other.

You might try running server in a VM, which NFS mounts the share, and then runs SMB non AD on top of the share to export as SMB readonly.

Or you could run core as you say, but eventually core it going to get upgraded too.

 

[Kosta]

I switched back to Core again. I wanted to remake my LUN for my file server anyway, because it grew for some reason (read: user-error) beyond what was needed, so now chosen 3 TiB force size sparse for 2,7TB of data, and also allows me to test how my restore of this VMDK works (Veeam NFS repo on Synology).

In any case many many thanks for helping me through, whatever comes this way, I will most likely have to see I get a player which is NFSv3+ able, but until that, it doesn't cost me anything to go with Core, so that is what I am doing.

But as you said, the option in the future might be running a Linux server "in between", which mounts the Samba-Share from TrueNAS, and exports it read-only with NFSv2. But didn't want to go that route now. Cleaner to run the Core.

Until the next problem - take care!

 

[samarium]

Importing as NFS, and exporting again as NFS might not be possible, I don't know how the code is structured. SMB + NFS either way is a better bet I feel. Either way, as you said, a problem for the future.