Too many open files

[Space]

Hi,

I am running 9.10.2-U3 (the error was also in U2).

I have the following configuration

1. FreeNas in AD
2. Mapped Drives with nsmb.conf and fstab (boot mount with password from file) /media/server1, /media/server2
   
3. Configured a Jail with rsnapshot
4. Published mapped fstab network shares to the local jaib: /mnt/server1 and /mnt/server2
5. Mapped a DataSet to the jail /mnt/rsnapshot

Folder listing /mnt:

drwxrwxr-x 1 root wheel 16384 Jan 1 1970 server1
drwxrwxr-x 1 root wheel 16384 Jan 1 1970 server2
drwxrwxr-x+ 6 root wheel 6 Apr 27 12:30 rsnapshot

Jail storage Map:
/media/server1 to /mnt/server1
/media/server2 to /mnt/server2

Rsnapshot (rsync) was first unable to chown on the target, but after removing extended attributes from rsync, that was resolved..

At the moment the biggest issue I cant past by is the problem of rsync. I am doing the first sync, but the error is also with other scheduled tasks hourly/daily/weekly ..


The sync is failing badly with this error message:

Code:

[2017-04-27T15:09:12] rsync: fstat failed: Too many open files (24)
[2017-04-27T15:09:12] rsync error: error in file IO (code 11) at sender.c(365) [sender=3.1.2]
[2017-04-27T15:09:12] rsync: [sender] write error: Broken pipe (32)
[2017-04-27T15:09:12] /usr/local/bin/rsnapshot sync: ERROR: /usr/local/bin/rsync returned 11 while processing /mnt/server1/

Jail Limits:

Code:

root@rsnap-jail:/ # sysctl kern.maxfiles kern.maxfilesperproc kern.openfiles
kern.maxfiles: 391171
kern.maxfilesperproc: 352053
kern.openfiles: 334
root@rsnapshot-jail:/ #

From /var/log/samba4/log.smbd dal FreeNAS:

Code:

 ../source3/smbd/files.c:218(file_init_global)   file_init_global: Information only: requested 352028 open files, 59392 are available.

The search in the forums the only related post found was a reference for a bug with inotify, but the current versions have it already disabled.

Any ideas ?

P.S. I also tried mounting in a JAIL a SMB share, but that FUSE was completely different not working story, which was considered to be resolved by the 10.x branch..

 

[Space]

I see also an error of "too many files open" in the logs of Freenas outside of the Jail.
As the actual mapping is happening on the Freenas, and not in the Jail (if somebody can tell me how to mount a SMB in jail would be greet), is it possibly this a bug of the upstream SMB ?!?

Forgat to mention: The content is rather diverse on the mapped shares: from documents, pictures, some misc size 7zips ecc (years of dumps :( )

 

[Space]

Is it Samba updated/reconfigured in the 11 Train ?
If I jump to 11 Nightlies, is it easy as selecting the 9.10.2-U3 boot ?

 

[dlavigne]

Yes, Samba has been updated to 4.6.3 for 11-RC1. Since RC1 was just released, you should switch trains to 11-stable if you would like to try it.

 

[Space]

FreeNAS-11.0-RC (ad90a865b)

[jail]
/var/log/rsnapshot.log
[2017-05-05T10:33:30] rsync: fstat failed: Too many open files (24)
[2017-05-05T10:33:30] rsync error: error in file IO (code 11) at sender.c(365) [sender=3.1.2]
[2017-05-05T10:33:30] rsync: [sender] write error: Broken pipe (32)

 

[dlavigne]

Does bumping up the jail's kern.maxfilesperproc to 1048576 resolve it?

 

[Space]

On FreeNAS System - Tunables - Sysctl type set:

• kern.maxfiles: 2097152
• kern.maxfilesperproc: 2097152

The settings is under FreeNas and Jail, I checked.

Tried 1048576 and 2097152, the same error occurs:

Code:

[2017-05-06T22:33:40] rsync: fstat failed: Too many open files (24)
[2017-05-06T22:33:40] rsync error: error in file IO (code 11) at sender.c(365) [sender=3.1.2]
[2017-05-06T22:33:40] rsync: [sender] write error: Broken pipe (32)

Code:

root@rsnapshot-jail:/mnt/rsnapshot #  rsnapshot -VD sync
require Lchown
Lchown module loaded successfully
Setting locale to POSIX "C"
echo 17061 > /var/run/rsnapshot.pid
/usr/local/bin/rsync -rltv --recursive \
  --exclude-from=/usr/local/etc/rsnapshot.exclude /mnt/samba_share1/ \
  /mnt/rsnapshot/.sync/samba_share1/
sending incremental file list
rsync: readdir("/mnt/samba_share1/."): Too many open files (24)

sent 30 bytes  received 12 bytes  84.00 bytes/sec
total size is 0  speedup is 0.00
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1179) [sender=3.1.2]
WARNING: Some files and/or directories in /mnt/samba_share1/ only transferred partially during rsync operation

 

[dlavigne]

Sounds like it's worth creating a bug report at bugs.freenas.org. Include your debug (from System -> Advanced -> Save Debug) and post the issue number here.