NT_STATUS_TOO_MANY_OPENED_FILES

[NateroniPizza]

Hello all,

I've got a new TrueNAS Scale installation set up inside of a Proxmox VM. VM details:
-32GB of RAM allocated to VM
-4x vCPU cores
-OS is installed on a 64GB VHD on SSD storage
-Passing through an HBA with 2x 14TB Exos X16 drives on it currently

I've set these two drives up as a ZFS mirror, and created a dataset under them. I've created a SMB network share associated with this dataset.

I am trying to copy about 2.9TB of files (~940,000 files) over to the share. I've tried FreeFileSync, and am now trying Teracopy, but the issue I am running into is that both will get to a certain point of transferring files, and the share will lock up. UnderFreeFileSync it didn't say whether it was an issue with the source or the destination, it was just showing "error_too_many_open_files." Under under Teracopy it specifically says "Target: The system cannot find the path specified." When I try to open the network share, I receive the following error:

I enabled debug level logging, and looking in /var/log/samba4/log.smbd, I am seeing "NT_STATUS_TOO_MANY_OPENED_FILES:"

Note that the system CPU usage is always under 25% and the RAM usage was exactly at 50% before (I increased the ZFS cache last night so that it would use up to 24GB, and as a result, now have only 3GB free - but this was happening with ~16GB of RAM free as well).

Whenever I restart the SMB service, the share becomes accessible again, and I am able to continue file copies.

Looking this error up in the context of TrueNAS doesn't bring anything up.

Any ideas what could be causing this?

Thank you

 

[anodos]

Hello all,

I've got a new TrueNAS Scale installation set up inside of a Proxmox VM. VM details:
-32GB of RAM allocated to VM
-4x vCPU cores
-OS is installed on a 64GB VHD on SSD storage
-Passing through an HBA with 2x 14TB Exos X16 drives on it currently

I've set these two drives up as a ZFS mirror, and created a dataset under them. I've created a SMB network share associated with this dataset.

I am trying to copy about 2.9TB of files (~940,000 files) over to the share. I've tried FreeFileSync, and am now trying Teracopy, but the issue I am running into is that both will get to a certain point of transferring files, and the share will lock up. UnderFreeFileSync it didn't say whether it was an issue with the source or the destination, it was just showing "error_too_many_open_files." Under under Teracopy it specifically says "Target: The system cannot find the path specified." When I try to open the network share, I receive the following error:
View attachment 61256

I enabled debug level logging, and looking in /var/log/samba4/log.smbd, I am seeing "NT_STATUS_TOO_MANY_OPENED_FILES:"
View attachment 61257

Note that the system CPU usage is always under 25% and the RAM usage was exactly at 50% before (I increased the ZFS cache last night so that it would use up to 24GB, and as a result, now have only 3GB free - but this was happening with ~16GB of RAM free as well).

Whenever I restart the SMB service, the share becomes accessible again, and I am able to continue file copies.

Looking this error up in the context of TrueNAS doesn't bring anything up.

Any ideas what could be causing this?

Thank you

It's possible there might be an FD leak somewhere. Please send me a debug via PM.

 

[NateroniPizza]

It's possible there might be an FD leak somewhere. Please send me a debug via PM.

Would that be the SMB log, or is there another you would like?

I will PM them back to you this evening.

 

[anodos]

Would that be the SMB log, or is there another you would like?

I will PM them back to you this evening.

System->Advanced->Save Debug.

 

[NateroniPizza]

System->Advanced->Save Debug.

Thank you. Just had some time on my lunch break, so connected in and grabbed them.

 

[NateroniPizza]

Good morning, all,

Unfortunately, no headway on that front.

I went and blew away the install and the ZFS pool, and started over with a blank slate. Same issue when copying the data over again.

I looked through the logs, and found the following error showing up:
Too many open files, unable to open more! smbd's max open files = 16424

Searching for this error, I see multiple recent threads over on the Unraid forum reporting the same. It has been theorized over there that this may be an issue or a change in a recent version of Samba.

The most promising recommendation I've seen is to increase the "max open files" value in the "smb-extra.conf" file, referenced in this Unraid thread (https://forums.unraid.net/bug-repor...er-in-network-after-6112-update-r2184/?page=2). However, I am unable to find either the smb-extra.conf or the smb.conf files I have seen referenced elsewhere.

Does anyone know where I can go to change this setting?

I suspect that, even if the underlying issue of file being stuck open remains, this would effectively take care of my issue. I've gone through and cleaned out more than half of the small files I have, and it only took two attempts to get everything over. Since I could increase this value by 4x, I'm hoping this will allow me to work around it.