Samba using up most of the RAM

[Morpheus187]

Hi

Since the update to version FreeNAS-11.1-U1 ( Correction: FreeNAS-11-1-RELEASE ) I'm noticing that samba is using nearly all of the available ram. Arc is getting less and less memory and Swap is already filled to 100%

In the picture attached you can also see the memory stat since the upgrade on 21st January.

I'm using the Freenas as Backup Server with lots and lots of small files (above 100 Millions of them) cumulating over 20 TB. This wasn't a problem until the update to U1

It seems that during a file copy or backup job the memory usage is increasing more and more.
Maybe someone has a hint in what direction I can start looking?

top

Code:

66 processes:  1 running, 63 sleeping, 2 zombie
CPU:  0.9% user,  0.0% nice,  6.7% system,  0.2% interrupt, 92.2% idle
Mem: 42G Active, 57M Inact, 20G Wired, 755M Free
ARC: 13G Total, 129M MFU, 11G MRU, 5326K Anon, 399M Header, 1492M Other
	 7966M Compressed, 11G Uncompressed, 1.45:1 Ratio
Swap: 10G Total, 10G Used, K Free, 100% Inuse

  PID USERNAME	  THR PRI NICE   SIZE	RES STATE   C   TIME	WCPU COMMAND
24164 root			1  25	0   231G 42825M select  3 224:29   8.33% smbd
28923 root			1  20	0  7948K  3404K CPU1	1   0:00   0.18% top

Show : smb.conf

Code:

[global]
	encrypt passwords = yes
	dns proxy = no
	strict locking = no
	oplocks = yes
	deadtime = 15
	max log size = 51200
	max open files = 1884845
	logging = file
	load printers = no
	printing = bsd
	printcap name = /dev/null
	disable spoolss = yes
	getwd cache = yes
	guest account = nobody
	map to guest = Bad User
	obey pam restrictions = yes
	ntlm auth = no
	directory name cache size = 0
	kernel change notify = no
	panic action = /usr/local/libexec/samba/samba-backtrace
	nsupdate command = /usr/local/bin/samba-nsupdate -g
	server string = FreeNAS Server
	ea support = yes
	store dos attributes = yes
	lm announce = yes
	hostname lookups = yes
	time server = yes
	acl allow execute always = true
	dos filemode = yes
	multicast dns register = yes
	domain logons = no
	local master = yes
	idmap config *: backend = tdb
	idmap config *: range = 90000001-100000000
	server role = standalone
	netbios name = DUPLICARE2
	workgroup = WORKGROUP
	security = user
	create mask = 0666
	directory mask = 0777
	client ntlmv2 auth = yes
	dos charset = CP437
	unix charset = UTF-8
	log level = 1
	ea support = no
	store dos attributes = no
	map archive = no
	map hidden = no
	map readonly = no
	map system = no


[Backup_1]
	path = "/mnt/TANK/Backup_1"
	printable = no
	veto files = /.snapshot/.windows/.mac/.zfs/
	writeable = yes
	browseable = yes
	access based share enum = no
	shadow:snapdir = .zfs/snapshot
	shadow:sort = desc
	shadow:localtime = yes
	shadow:format = auto-%Y%m%d.%H%M-1m
	shadow:snapdirseverywhere = yes
	vfs objects = shadow_copy2 zfs_space zfsacl streams_xattr aio_pthread
	hide dot files = yes
	guest ok = no
	nfs4:mode = special
	nfs4:acedup = merge
	nfs4:chown = true
	zfsacl:acesort = dontcare


[Backup_2]
	path = "/mnt/TANK/Backup_2"
	printable = no
	veto files = /.snapshot/.windows/.mac/.zfs/
	writeable = yes
	browseable = yes
	access based share enum = no
	shadow:snapdir = .zfs/snapshot
	shadow:sort = desc
	shadow:localtime = yes
	shadow:format = auto-%Y%m%d.%H%M-1y
	shadow:snapdirseverywhere = yes
	vfs objects = shadow_copy2 zfs_space zfsacl streams_xattr aio_pthread
	hide dot files = yes
	guest ok = no
	nfs4:mode = special
	nfs4:acedup = merge
	nfs4:chown = true
	zfsacl:acesort = dontcare


[Backup_3]
	path = "/mnt/TANK/Backup_3"
	printable = no
	veto files = /.snapshot/.windows/.mac/.zfs/
	writeable = yes
	browseable = yes
	access based share enum = no
	vfs objects = zfs_space zfsacl streams_xattr aio_pthread
	hide dot files = yes
	guest ok = no
	nfs4:mode = special
	nfs4:acedup = merge
	nfs4:chown = true
	zfsacl:acesort = dontcare

 

[dlavigne]

It is probably worthwhile to create a report at bugs.freenas.org that includes your debug so a dev can help analyze the situation to see if it is a bug or a configuration issue. If you do, post the issue number here (though it will be hidden from others until the debug is removed after analysis).

 

[anodos]

What software are you using to write the backups?

 

[Morpheus187]

I'm using robocopy and copy the files over via a samba share.

 

[anodos]

If you don't mind for testing purposes, try disabling streams_xattr and see if the issue persists. Note that this will make it so that your samba server stops supporting Alternate Data Streams (which may be important depending on your environment).

 

[anodos]

You should also try removing the line ea support = no EA support is required for streams_xattr to work properly.

 

[Morpheus187]

Thanks, I will have a look how I can change that value. At the moment I have checked my memory history and noticed that strange memory and arc drops began when I updated from 11.0-U4 to 11.1-RELEASE

So I've now reverted to 11.0-U4 and started a copy job. Observation so far is that the memory usage for smbd is MUCH less.

With 11.1-RELEASE and 11.1-U1 I had smbd eating 2-6 GB of ram within 1 hour. I could count the megabytes go up.
With 11.0-U4 smbd is using 32840 KILOBytes of ram.

I think it has something to do with the samba upgrade from Version 4.6.8-GIT-7a91926 to 4.7.0?

Code:

  PID USERNAME	  THR PRI NICE   SIZE	RES STATE   C   TIME	WCPU COMMAND
 4298 root		   15  21	0   510M   152M umtxn   4   0:05   0.00% uwsgi
  237 root			7  20	0   409M   101M kqread  3   0:04   0.00% python3.6
 3843 root			1  20	0   407M 99732K kqread  0   0:04   0.00% uwsgi
 3831 root			1  52	0   293M 65336K select  6   0:07   0.00% python3.6
 4879 root			1  52	0   236M 57536K ttyin   5   0:01   0.00% python3.6
 3948 root		   12  20	0   230M 35144K nanslp  0   0:03   1.14% collectd
 6198 root			1  29	0   339M 32840K zio->i  7   1:09  15.72% smbd
 4629 root			6  52	0   104M 19700K usem	4   0:00   0.00% python3.6
 2542 root			1  20	0   284M 14832K select  1   0:00   0.00% smbd

 

[Morpheus187]

I now removed the ea support line and started a copy job from the FreeNAS to a server (I went back to version 11.1-U1 for testing)

After 5 minutes of copying smbd was using about 1234MB of memory

Code:

PID USERNAME	  THR PRI NICE   SIZE	RES STATE   C   TIME	WCPU COMMAND
7329 root			1  52	0  1333M  1234M select  7   1:11  34.64% smbd

After 20 minutes

Code:

PID USERNAME	  THR PRI NICE   SIZE	RES STATE   C   TIME	WCPU COMMAND
7329 root			1  45	0  6864M  5490M CPU7	7   6:41  33.40% smbd

I then stopped the transfer and the memory usage remained there. I also tried removing all auxiliary parameters from SMB without any effect. To me it looks like samba has a giant memory leak that gradually eats up all ram. I will now try out what happens if samba uses every last bit of ram, I'm wondering how FreeNAS and zfs reacts.

 

[BaT]

Can you, please, make a formal ticket about this issue an post it here?

 

[Morpheus187]

I already posted a formal ticket. I will link it here as soon as I can delete the debug file.
Currently my system is holding at about 15 GB ARC available, everything else is basically taken away by samba.
It seems that at least that amount of ARC is guaranteed and samba is releasing a bit of ram when it gets critical. Usually samba is around 30-40 GB of ram usage.

Current status

Code:

last pid: 57240;  load averages:  0.23,  0.29,  0.24														   up 1+15:02:25  08:19:41
55 processes:  1 running, 54 sleeping
CPU:  2.5% user,  0.0% nice,  0.6% system,  0.0% interrupt, 96.9% idle
Mem: 492M Active, 32G Inact, 760M Laundry, 21G Wired, 8274M Free
ARC: 15G Total, 2172M MFU, 11G MRU, 5596K Anon, 205M Header, 1667M Other
	 12G Compressed, 20G Uncompressed, 1.64:1 Ratio
Swap: 10G Total, 301M Used, 9939M Free, 2% Inuse

  PID USERNAME	  THR PRI NICE   SIZE	RES STATE   C   TIME	WCPU COMMAND
43203 root			1  20	0   102G 32601M select  5  98:00   0.00% smbd
55186 root			1  28	0  2133M  1156M select  0   2:37  15.95% smbd
 5162 root		   15  22	0   220M   138M umtxn   5   8:32   0.00% uwsgi
  238 root		   20  20	0   196M   118M kqread  1   5:27   0.00% python3.6

 

[fzitelli]

I have the same problem, but when swap is full, the smbd service closes and sharing goes down. Tell me if you find a solution.
I have Active Directory configured, and the only way to release the memory is save the config of Active Directory. Doing that de swap memory and de memory free, sets to normal values
Thanks!

 

[ian351c]

I haven't seen any activity on this thread in a bit, but I have this same issue. I've opened a bug ticket: https://redmine.ixsystems.com/issue...ion=4&next_issue_id=28429&prev_issue_id=28433. I'll update the thread with any results.

 

[anodos]

@Morpheus187 can you please share the robocopy command that you're using for your backup jobs?

 

[anodos]

@ian351c can you please upgrade to 11.1-U1 and try to reproduce the issue.

 

[Morpheus187]

Hi

I'm using the following command

Code:

robocopy %Source% %Target% /MIR /xo /fp /ns  >"%Logdir%\MIRROR13.txt"

Source and targets are the respective directories.

I've also created a ticket a while ago under: https://redmine.ixsystems.com/issue...on=39&next_issue_id=28024&prev_issue_id=28031

The issue is still being investigated. I'm currently restarting samba after copy intensive backups to free up some ram again. I'm thinking about writing a little script that restarts samba every day to free up ram for ARC.

I've now also installed an 128 GB L2ARC SSD to increase performance, because samba can't allocate L2ARC, so L2ARC is taking over when ARC is drained to it's minimum size.

 

[anodos]

If you have a chance, please try reproducing on 11.1-U1.

 

[Morpheus187]

Sorry, maybe my initial post wasn't entirely clear, the problem also persists on the newest version of FreeNAS-11.1-U1. It started with FreeNAS-11-1-RELEASE.

 

[anodos]

Sorry, maybe my initial post wasn't entirely clear, the problem also persists on the newest version of FreeNAS-11.1-U1. It started with FreeNAS-11-1-RELEASE.

Okay. I'm trying to reproduce this on an internal system. One more thing to try:

Unless it's required for your environment, can you try un-checking Zeroconf share discovery under services->SMB?

There were some important commits related to mdns. It's one more potential source of issues.

 

[Morpheus187]

Thanks for the advice, I unticked Zeroconf but SMB still eats ram like I eat chips in front of my tv. 20 Minutes of copying files away from Freenas -> SMB uses 3272M.

 

[m7ed]

I think that I experience the same problem, so far I have tested on 11.1, 11.1-U1, 11.1-U2.

FreeNAS is running as a Fileserver for a small company with active directory. Zeroconf is unticked. Every new SMB connection uses about 150MB of ram, but after some activity it grows to ~2GB per user.



The memory is freed again after the connection is closed, but at the end of each day there are only 20GB of the 50GB ram left for the ARC.