Samba using up most of the RAM

Morpheus187

Explorer
Joined
Mar 11, 2016
Messages
61
Hi

Since the update to version FreeNAS-11.1-U1 ( Correction: FreeNAS-11-1-RELEASE ) I'm noticing that samba is using nearly all of the available ram. Arc is getting less and less memory and Swap is already filled to 100%

In the picture attached you can also see the memory stat since the upgrade on 21st January.
Memory_status.png
I'm using the Freenas as Backup Server with lots and lots of small files (above 100 Millions of them) cumulating over 20 TB. This wasn't a problem until the update to U1

It seems that during a file copy or backup job the memory usage is increasing more and more.
Maybe someone has a hint in what direction I can start looking?

top
Code:
66 processes:  1 running, 63 sleeping, 2 zombie
CPU:  0.9% user,  0.0% nice,  6.7% system,  0.2% interrupt, 92.2% idle
Mem: 42G Active, 57M Inact, 20G Wired, 755M Free
ARC: 13G Total, 129M MFU, 11G MRU, 5326K Anon, 399M Header, 1492M Other
	 7966M Compressed, 11G Uncompressed, 1.45:1 Ratio
Swap: 10G Total, 10G Used, K Free, 100% Inuse

  PID USERNAME	  THR PRI NICE   SIZE	RES STATE   C   TIME	WCPU COMMAND
24164 root			1  25	0   231G 42825M select  3 224:29   8.33% smbd
28923 root			1  20	0  7948K  3404K CPU1	1   0:00   0.18% top



Code:
[global]
	encrypt passwords = yes
	dns proxy = no
	strict locking = no
	oplocks = yes
	deadtime = 15
	max log size = 51200
	max open files = 1884845
	logging = file
	load printers = no
	printing = bsd
	printcap name = /dev/null
	disable spoolss = yes
	getwd cache = yes
	guest account = nobody
	map to guest = Bad User
	obey pam restrictions = yes
	ntlm auth = no
	directory name cache size = 0
	kernel change notify = no
	panic action = /usr/local/libexec/samba/samba-backtrace
	nsupdate command = /usr/local/bin/samba-nsupdate -g
	server string = FreeNAS Server
	ea support = yes
	store dos attributes = yes
	lm announce = yes
	hostname lookups = yes
	time server = yes
	acl allow execute always = true
	dos filemode = yes
	multicast dns register = yes
	domain logons = no
	local master = yes
	idmap config *: backend = tdb
	idmap config *: range = 90000001-100000000
	server role = standalone
	netbios name = DUPLICARE2
	workgroup = WORKGROUP
	security = user
	create mask = 0666
	directory mask = 0777
	client ntlmv2 auth = yes
	dos charset = CP437
	unix charset = UTF-8
	log level = 1
	ea support = no
	store dos attributes = no
	map archive = no
	map hidden = no
	map readonly = no
	map system = no


[Backup_1]
	path = "/mnt/TANK/Backup_1"
	printable = no
	veto files = /.snapshot/.windows/.mac/.zfs/
	writeable = yes
	browseable = yes
	access based share enum = no
	shadow:snapdir = .zfs/snapshot
	shadow:sort = desc
	shadow:localtime = yes
	shadow:format = auto-%Y%m%d.%H%M-1m
	shadow:snapdirseverywhere = yes
	vfs objects = shadow_copy2 zfs_space zfsacl streams_xattr aio_pthread
	hide dot files = yes
	guest ok = no
	nfs4:mode = special
	nfs4:acedup = merge
	nfs4:chown = true
	zfsacl:acesort = dontcare


[Backup_2]
	path = "/mnt/TANK/Backup_2"
	printable = no
	veto files = /.snapshot/.windows/.mac/.zfs/
	writeable = yes
	browseable = yes
	access based share enum = no
	shadow:snapdir = .zfs/snapshot
	shadow:sort = desc
	shadow:localtime = yes
	shadow:format = auto-%Y%m%d.%H%M-1y
	shadow:snapdirseverywhere = yes
	vfs objects = shadow_copy2 zfs_space zfsacl streams_xattr aio_pthread
	hide dot files = yes
	guest ok = no
	nfs4:mode = special
	nfs4:acedup = merge
	nfs4:chown = true
	zfsacl:acesort = dontcare


[Backup_3]
	path = "/mnt/TANK/Backup_3"
	printable = no
	veto files = /.snapshot/.windows/.mac/.zfs/
	writeable = yes
	browseable = yes
	access based share enum = no
	vfs objects = zfs_space zfsacl streams_xattr aio_pthread
	hide dot files = yes
	guest ok = no
	nfs4:mode = special
	nfs4:acedup = merge
	nfs4:chown = true
	zfsacl:acesort = dontcare

 
Last edited by a moderator:
D

dlavigne

Guest
It is probably worthwhile to create a report at bugs.freenas.org that includes your debug so a dev can help analyze the situation to see if it is a bug or a configuration issue. If you do, post the issue number here (though it will be hidden from others until the debug is removed after analysis).
 
Last edited by a moderator:

Morpheus187

Explorer
Joined
Mar 11, 2016
Messages
61
I'm using robocopy and copy the files over via a samba share.
 

anodos

Sambassador
iXsystems
Joined
Mar 6, 2014
Messages
9,554
If you don't mind for testing purposes, try disabling streams_xattr and see if the issue persists. Note that this will make it so that your samba server stops supporting Alternate Data Streams (which may be important depending on your environment).
 

Morpheus187

Explorer
Joined
Mar 11, 2016
Messages
61
Thanks, I will have a look how I can change that value. At the moment I have checked my memory history and noticed that strange memory and arc drops began when I updated from 11.0-U4 to 11.1-RELEASE

So I've now reverted to 11.0-U4 and started a copy job. Observation so far is that the memory usage for smbd is MUCH less.

With 11.1-RELEASE and 11.1-U1 I had smbd eating 2-6 GB of ram within 1 hour. I could count the megabytes go up.
With 11.0-U4 smbd is using 32840 KILOBytes of ram.

I think it has something to do with the samba upgrade from Version 4.6.8-GIT-7a91926 to 4.7.0?
Code:
  PID USERNAME	  THR PRI NICE   SIZE	RES STATE   C   TIME	WCPU COMMAND
 4298 root		   15  21	0   510M   152M umtxn   4   0:05   0.00% uwsgi
  237 root			7  20	0   409M   101M kqread  3   0:04   0.00% python3.6
 3843 root			1  20	0   407M 99732K kqread  0   0:04   0.00% uwsgi
 3831 root			1  52	0   293M 65336K select  6   0:07   0.00% python3.6
 4879 root			1  52	0   236M 57536K ttyin   5   0:01   0.00% python3.6
 3948 root		   12  20	0   230M 35144K nanslp  0   0:03   1.14% collectd
 6198 root			1  29	0   339M 32840K zio->i  7   1:09  15.72% smbd
 4629 root			6  52	0   104M 19700K usem	4   0:00   0.00% python3.6
 2542 root			1  20	0   284M 14832K select  1   0:00   0.00% smbd
 
Last edited:

Morpheus187

Explorer
Joined
Mar 11, 2016
Messages
61
I now removed the ea support line and started a copy job from the FreeNAS to a server (I went back to version 11.1-U1 for testing)

After 5 minutes of copying smbd was using about 1234MB of memory
Code:
PID USERNAME	  THR PRI NICE   SIZE	RES STATE   C   TIME	WCPU COMMAND
7329 root			1  52	0  1333M  1234M select  7   1:11  34.64% smbd


After 20 minutes
Code:
PID USERNAME	  THR PRI NICE   SIZE	RES STATE   C   TIME	WCPU COMMAND
7329 root			1  45	0  6864M  5490M CPU7	7   6:41  33.40% smbd


I then stopped the transfer and the memory usage remained there. I also tried removing all auxiliary parameters from SMB without any effect. To me it looks like samba has a giant memory leak that gradually eats up all ram. I will now try out what happens if samba uses every last bit of ram, I'm wondering how FreeNAS and zfs reacts.
 
Last edited by a moderator:

BaT

Explorer
Joined
Jun 16, 2017
Messages
62
Can you, please, make a formal ticket about this issue an post it here?
 

Morpheus187

Explorer
Joined
Mar 11, 2016
Messages
61
I already posted a formal ticket. I will link it here as soon as I can delete the debug file.
Currently my system is holding at about 15 GB ARC available, everything else is basically taken away by samba.
It seems that at least that amount of ARC is guaranteed and samba is releasing a bit of ram when it gets critical. Usually samba is around 30-40 GB of ram usage.

Current status
Code:
last pid: 57240;  load averages:  0.23,  0.29,  0.24														   up 1+15:02:25  08:19:41
55 processes:  1 running, 54 sleeping
CPU:  2.5% user,  0.0% nice,  0.6% system,  0.0% interrupt, 96.9% idle
Mem: 492M Active, 32G Inact, 760M Laundry, 21G Wired, 8274M Free
ARC: 15G Total, 2172M MFU, 11G MRU, 5596K Anon, 205M Header, 1667M Other
	 12G Compressed, 20G Uncompressed, 1.64:1 Ratio
Swap: 10G Total, 301M Used, 9939M Free, 2% Inuse

  PID USERNAME	  THR PRI NICE   SIZE	RES STATE   C   TIME	WCPU COMMAND
43203 root			1  20	0   102G 32601M select  5  98:00   0.00% smbd
55186 root			1  28	0  2133M  1156M select  0   2:37  15.95% smbd
 5162 root		   15  22	0   220M   138M umtxn   5   8:32   0.00% uwsgi
  238 root		   20  20	0   196M   118M kqread  1   5:27   0.00% python3.6

 

fzitelli

Cadet
Joined
Dec 30, 2014
Messages
5
I have the same problem, but when swap is full, the smbd service closes and sharing goes down. Tell me if you find a solution.
I have Active Directory configured, and the only way to release the memory is save the config of Active Directory. Doing that de swap memory and de memory free, sets to normal values
Thanks!
 
Last edited by a moderator:

Morpheus187

Explorer
Joined
Mar 11, 2016
Messages
61
Hi

I'm using the following command
Code:
robocopy %Source% %Target% /MIR /xo /fp /ns  >"%Logdir%\MIRROR13.txt"


Source and targets are the respective directories.

I've also created a ticket a while ago under: https://redmine.ixsystems.com/issue...on=39&next_issue_id=28024&prev_issue_id=28031

The issue is still being investigated. I'm currently restarting samba after copy intensive backups to free up some ram again. I'm thinking about writing a little script that restarts samba every day to free up ram for ARC.

I've now also installed an 128 GB L2ARC SSD to increase performance, because samba can't allocate L2ARC, so L2ARC is taking over when ARC is drained to it's minimum size.
 

Morpheus187

Explorer
Joined
Mar 11, 2016
Messages
61
Sorry, maybe my initial post wasn't entirely clear, the problem also persists on the newest version of FreeNAS-11.1-U1. It started with FreeNAS-11-1-RELEASE.
 

anodos

Sambassador
iXsystems
Joined
Mar 6, 2014
Messages
9,554
Sorry, maybe my initial post wasn't entirely clear, the problem also persists on the newest version of FreeNAS-11.1-U1. It started with FreeNAS-11-1-RELEASE.

Okay. I'm trying to reproduce this on an internal system. One more thing to try:

Unless it's required for your environment, can you try un-checking Zeroconf share discovery under services->SMB?

There were some important commits related to mdns. It's one more potential source of issues.
 

Morpheus187

Explorer
Joined
Mar 11, 2016
Messages
61
Thanks for the advice, I unticked Zeroconf but SMB still eats ram like I eat chips in front of my tv. 20 Minutes of copying files away from Freenas -> SMB uses 3272M.
 

m7ed

Cadet
Joined
Feb 26, 2018
Messages
2
I think that I experience the same problem, so far I have tested on 11.1, 11.1-U1, 11.1-U2.

FreeNAS is running as a Fileserver for a small company with active directory. Zeroconf is unticked. Every new SMB connection uses about 150MB of ram, but after some activity it grows to ~2GB per user.
memory.png
arc.png

The memory is freed again after the connection is closed, but at the end of each day there are only 20GB of the 50GB ram left for the ARC.

arc2.png
 

Attachments

  • htop.PNG
    htop.PNG
    47.2 KB · Views: 930
Top