samba process spawn runaway

[fransm]

Hi,

I have recently installed freeNAS 8.0.2 and I am having some problems. The nas is primarily used as storage for nightly backups from 5 workstations. I use Comodo backup and it creates one large file for each system. The backup is then incrementally updated. The workstations write their backups to cifs shares and a server uses an iscsi connection using microsofts iscsi initiator.

The problem is that freeNAS often fail during the backups, it becomes unresponsive both via samba and the gui. The first few times this happened I talked to my friend google and found a possible solution with kern.maxfiles. I increased it in loader.conf to 950000 and the machine settled down a bit. But today I found a new thing. The console said:

Code:

Nov 30 14:41:15 nas kernel: Approaching the limit on PV entries, consider increasing either the vm.pmap.shpgperproc or the vm.pmap.pv_entry_max sysctl.

The samba shares were unavailable. I looked at the graphs in the gui and saw that the process count was higher than usual

View attachment 458

And when I ssh:d to the machine i found about 1200 instances of smbd on ps -aux

Code:

frans      24897  0.0  0.0 53836  3908  ??  D     2:40PM   0:00.02 /usr/local/sbin/smbd -D -s /usr/local/etc/smb.conf
root       24911  0.0  0.0 53836  3696  ??  D     2:41PM   0:00.02 /usr/local/sbin/smbd -D -s /usr/local/etc/smb.conf
frans      24912  0.0  0.0 53836  4084  ??  D     2:41PM   0:00.05 /usr/local/sbin/smbd -D -s /usr/local/etc/smb.conf
frans      24924  0.0  0.0 53836  4020  ??  D     2:42PM   0:00.03 /usr/local/sbin/smbd -D -s /usr/local/etc/smb.conf
root       24927  0.0  0.0 53836  3840  ??  D     2:42PM   0:00.05 /usr/local/sbin/smbd -D -s /usr/local/etc/smb.conf
frans      24956  0.0  0.1 59596  4212  ??  D     2:43PM   0:00.03 /usr/local/sbin/smbd -D -s /usr/local/etc/smb.conf
....and so on....

Version of the kernel:

Code:

#uname -a
FreeBSD nas.local 8.2-RELEASE-p3 FreeBSD 8.2-RELEASE-p3 #7: Fri Sep 30 12:51:49 PDT 2011     jpaetzel@servant.iXsystems.com:/b/sf_freenas_build/obj.amd64/b/sf_freenas_build/FreeBSD/src/sys/FREENAS.amd64  amd64

I have the following services enabled:
CIFS
SMART
SSH
ISCSI

I have attached other graphs.

I will configure the box for remote syslog to try to get more information when this happens again.

Any suggestions would be greatly appreciated and if there is any other information that you would like me to look for please tell me. I will leave the machine in its current state for as long as it remains responsive and then reboot it.

 

[fransm]

update

hi again,

The same problem as decribed above has happened twice more. It happen when i run nightly backups that write one large file to samba (60-70GB).

If i create an ISCSI device that i mount on the workstation and write the backup there, then there is no problem.

I set up remote syslog from freenas to another freebsd machine but nothing regarding samba is written to the logs in the default configuration.

Is there any way to get useful info from the samba logs using syslog?

Is anyone interested in this problem at all?

 

[malbert]

I was looking to raise a bug and then I found yours (#1160) that you updated a few days ago.

I have exactly the same symptoms, but it appears to be happening for a totally different reason. I've been running 8.0.4 and experienced the same thing in 8.2 Beta 2, but my problems started presenting when a drive died.

I had 4 1TB drives in raidz1 which is currently degraded, and since that drive failure, the number of smbd processes has been skyrocketing at random intervals.

ps -aux | grep smbd | wc -l
1426

The annoying part is that it breaks everything. Checking zpool status in the GUI against the volume fails in an error (works in CLI). Reboots get stuck failing to stop processes, meaning it becomes a physical power off job (kinda annoying needing access to the server room so frequently). CIFS shares are obviously broken. Occasionally the left-hand menu frame in the GUI stops working.

Time between reboots varies from 30 minutes to 1.5 days. In the mean time, I'm waiting for my RMA so I can get off the backup machine.

 

[dyzhang]

My samba always halt since 8.0.3... when copy from once zfs pool to another pool.do not know why it comes this.