Large transfer stops "Too many files are currently in use" TrueNAS Scale

[dgcdance]

Hi everyone,

I am trying to transfer around 1.10 tb (150,000 files), from a volume mounted as a disk to a Windows VM, to a mounted share in that Windows VM.

Small transfers work fine, but with this transfer, it pauses from time to time, with the error “Too many files are currently in use. Quit one or more programs, and then try again.” Most of the time when trying again, the transfer continues. The VM resources don’t seem to be a problem.

From time to time when the transfer is no longer working with “trying again”, I noticed access to the share was lost from that VM, if I try to access the share through another computer it works.

I have TrueNAS configured with a bridge connecting the main interface and have a static IP. MTU is set to 1500, and I have no idea if this should be changed. I had to use a bridge to access the shares within VMs (I read that was the way of doing it). By the way, my nameserver is set to 1.1.1.1

I’ve gone through a lot of settings, permissions, default share parameters, instead of generic SMB, and much more, and I'm exhausted :(

I am currently testing doing a similar transfer but from another computer, I am copying an almost exact copy of the data to another Share, this is an ongoing test…

Another thing that I noticed is some weird errors on the main console, here you can take a look at the debug info, and some screenshots

https://www.dropbox.com/sh/wytxuyrk9d62kr4/AADQjf9vwun_48o92aFSVNTra?dl=0

I am also having a SMART notification, but when performing LONG SMART test, they appear to have no errors, investigating with scrutiny, they appear to be ok

• Device: /dev/sdd, SMART Failure: WARNING: ascq=0x96.
• Device: /dev/sdb, SMART Failure: WARNING: ascq=0x96.

My setup:

• Dell PowerEdge T430 BIOS 2.16.0
• 2 x 16 Gb ECC DDR4 1866 Mhz
• 2 x Intel Xeon E5-2620 v3 @ 2.4GHz
• 8 x 4tb SAS 6 GiBit 7200 RPM HGST HUS724040ALS640 RZ-2 Vdev
• Dell PERC H730 in HBA mode
• 2 x Integrated Broadcom Gigabit Ethernet BCM5720
• 2 x 120 Gb Sata SSD mirrored boot drive (Consumer Grade)
• 2 x 512 nvme m2 drives (Consumer grade) striped vdev
• TrueNAS Scale 22.12.0
• I have cloudflared to remote access some stuff trough a tunnel

Permissions

Owner:
admin

Group:
Admin



Type:
FILESYSTEM

Sync:
STANDARD

Compression Level:
Inherit (LZ4)

Enable Atime:
OFF

ZFS Deduplication:
OFF

Case Sensitivity:
ON

I appreciate all your support!

 

[dgcdance]

Performing the transfer from the other computer didn't work, I got the same error "Too many files are currently in use". I am now testing copying to another computer, and not to another share.

 

[dgcdance]

Update:

• Pulling those files (+150,000) from TrueNAS share to another computer with a big HDD works
• Copying from TrueNAS share to another share does not work, whether is done from a VM or another computer. Tried with windows copy, robocopy, teracopy, fails
• Copying a large number of files from another computer to TrueNAS share fails.

Eventually, I lose access to that share from the computer or vm im trying to do the copying if I try accessing the share from another computer it works.

 

[jgreco]

Dell PERC H730 in HBA mode

There is no such thing as a PERC H730 in "HBA mode"; the H730 is a RAID controller, no matter what Dell tries to convince you of in their setup utility.

RAID controllers are unsuitable for use with TrueNAS. Please see

What's all the noise about HBAs, and why can't I use a RAID controller?

1) An HBA is a Host Bus Adapter. This is a controller that allows SAS and SATA devices to be attached to, and communicate directly with, a server. RAID controllers typically aggregate several disks into a Virtual Disk abstraction of some sort...

www.truenas.com

Please replace this with a proper controller.

 

[morganL]

Update:

• Pulling those files (+150,000) from TrueNAS share to another computer with a big HDD works
• Copying from TrueNAS share to another share does not work, whether is done from a VM or another computer. Tried with windows copy, robocopy, teracopy, fails
• Copying a large number of files from another computer to TrueNAS share fails.

Eventually, I lose access to that share from the computer or vm im trying to do the copying if I try accessing the share from another computer it works.

There are limits within samba/systemd that you may be touching. https://lists.samba.org/archive/samba/2018-April/215250.html

In particular, the last one is a TrueNAS issues.

• Copying a large number of files from another computer to TrueNAS share fails.

My guess is that 1,000 files never fails... can you check, 20,000 and 100,000 and then report a bug.

I'm a bit surprised that the copy software keeps files open??? So this may not be the issue.

 

[Daisuke]

There is no such thing as a PERC H730 in "HBA mode"

That's the main reason why I purchased a Dell R720xd with a H710 mini PERC, instead of R730xd. I wanted to make sure I get hardware supported by TrueNAS.

 

[dgcdance]

I haven't found a solution, I am re-testing with "protocol = SMB2_02" under the SMB settings. I am also setting up a different TrueNAS instance to see if I can replicate this error. I will keep you posted :)

 

[dgcdance]

My issue might be related to the bridge that I was using to access TrueNAS within the VM. I am currently testing with the VM using a dedicated NIC.

 

[dgcdance]

Update / Workaround

I've tried so much...

Using SSD pool only, n\ew VM's, settings like SMB instead of generic, sync off, a fresh install, and even increasing my ram from 32 Gb to 80 Gb

I am starting to think its something on the Windows side, I tried the following settings,

Disable the Windows Search Service’s Indexing
Turn Off the Remote Differential Compression

I've noticed an increase in performance, faster file discovery, and some transfers peaked 700 mbs, something that I haven't seen before, but eventually, the transfer fails, same error "Too many files are currently in use. Quit one or more programs, and then try again"

I prepared a VM with Ubuntu 22.04.1 LTS, mounted the shares and performed what I've been trying to do, and it worked, perfectly on the first try.

So WTF Windows! (This happened on Windows 10 Pro, 11 Pro, Server 2016, 2019, and 2022).

I was hoping to find the bottom of this problem and share a solution for anybody else having this issue, I will stop messing around in TrueNAS, and when I need to perform huge file copies, I'll just use my Ubuntu VM.

 

[morganL]

Update / Workaround

I've tried so much...

Using SSD pool only, n\ew VM's, settings like SMB instead of generic, sync off, a fresh install, and even increasing my ram from 32 Gb to 80 Gb

I am starting to think its something on the Windows side, I tried the following settings,

Disable the Windows Search Service’s Indexing
Turn Off the Remote Differential Compression

I've noticed an increase in performance, faster file discovery, and some transfers peaked 700 mbs, something that I haven't seen before, but eventually, the transfer fails, same error "Too many files are currently in use. Quit one or more programs, and then try again"

I prepared a VM with Ubuntu 22.04.1 LTS, mounted the shares and performed what I've been trying to do, and it worked, perfectly on the first try.

So WTF Windows! (This happened on Windows 10 Pro, 11 Pro, Server 2016, 2019, and 2022).

I was hoping to find the bottom of this problem and share a solution for anybody else having this issue, I will stop messing around in TrueNAS, and when I need to perform huge file copies, I'll just use my Ubuntu VM.

Just confirming:

With Ubuntu as SMB client to trueNAS.. everything was OK

With Windows as client to trueNAS, you get this error?