NFS Directory Listing Missing Files

[randomadmin]

We are running into an issue where some directory listings on one of our NFS client machines (running Samba) will only show about 20-30 files out of 1500 files or so. Only 1% of the 1500 files change each day and almost all of those are overwrites of existing filenames but the 20-30 files shown have timestamps scattered over the past few months. No obvious reason why those are shown vs the ones that aren’t. All the files have the same permissions plus we know the files still exist because we can see them from other NFS clients or after the affected NFS client machine has its NFS share unmounted/mounted again. We have seen two variations of this.

• It will temporarily happen when our servers are under their peak loads usually just after the top of the hour. We added a debug output to our process looking for files via Samba and we can see a full directory listing at 10:47 PM, then a super short one at 11:03 PM, and then a long directory listing at 11:17 PM. Most of the time we see this scenario is just after the hour (our peak times) whereas when run at more random times they are usually full listings and work fine.
• Sometimes a given directory will get stuck with the short directory listing. So as some point it drops to 30 files or so and stays that way (at least for day straight). We see the short listing using Samba but if we go onto the Samba VM we can directly ls the NFS directory and it shows the same short file listing. So Samba is just showing what NFS shows. Even more odd sometimes one the filenames will be truncated (like VALIDATE.X instead of VALIDATE.XML). As if the directory listing is getting cut short. I can re-upload one of the files in the short list and see the timestamp change but the rest of the files in that NFS are still missing. However other NFS client machines can see all the files in that NFS directory fine. So the files still live on the NFS server as expected. The only way we know how to fix this so far is to unmount and remount the NFS share on the affected Samba machine.

Here is a little more background on our setup. We have multiple SFTP FreeBSD 12.2 VMs mounting our TRUENAS-X10 using NFS (we have tried both NFS 3 and 4). Several hundred SFTP clients will connect on the hour and overwrite about 1% of the files in that SFTP client directory. No two clients should be writing to the same subdirectory under the NFS mount.



We also have a Samba FreeBSD 12.2 VM mounting the same TrueNAS NFS share as the SFTP servers. Within 1 second of the SFTP upload completing we have a process on another server that looks for the new files over SMB to the Samba server share. Ultimately it’s this process that does a directory listing via SMB and sometimes gets the super short directory listing.



The above issues only started after replacing our old setup which was configured the same way as above except with 10 year old physical hardware running FreeBSD 9 and our own FreeBSD based ZFS/NFS server. The new setup with TrueNAS and VMs is much faster than our old configuration and works great except for this missing file issue.



So we are wondering if anyone has seen something like this before and can perhaps guide us as to why these NFS directory listings sometimes get cut short. Thanks.