FreeNAS Mini XL: File transfer - AFP - fast, SMB - slow (100% CPU tilt)

Status
Not open for further replies.

damjan

Dabbler
Joined
Jan 8, 2016
Messages
11
I bought FreeNAS Mini XL a while ago along with 4x3TB hard drives, SLOG in L2ARC.

File share on AFP is as fast as expected - 110MB/s, small files do not cause a big performance drop. It is very unreliable however. Lots of crashes and random errors.

SMB on the other hand is very reliable, unfortunately, also very slow. I get ~10MB/s reads and writes on the same share. That is when I copy large files (~1GB) not small ones. If I run top on FreeNAS, I see that CPU utilisation of smbd process is always between 90% and 100% when file transfer is in progress. Listing large directories is tends to be slow too.

Doing same file transfer over AFP uses very little CPU (single digit). I use pretty much standard SMBD config. I deleted it and created it again using web wizard and I get same performance.

I have this performance issue since I acquired FreeNAS Mini XL several months ago (9.10). I have 9.10.1-U4 patch now, but no improvement.

My client is 10.11. I tried with SMB2 and SMB3 protocol and almost no difference in performance. Below is my config. ZFS filesystem uses lz4 compression, no dedup, no encyption. atime is also off.

What would cause such a large CPU consumption in smbd? If there would be anything hardware or system related (e.g. ZFS features) I would expect they would manifest also in afpd. All CPU usage is in user space, not system.





Code:
[global]
	username map = /usr/local/etc/smbusers
	server max protocol = SMB2
	interfaces = 127.0.0.1 192.168.123.148 192.168.254.13
	bind interfaces only = yes
	encrypt passwords = yes
	dns proxy = no
	strict locking = no
	oplocks = yes
	deadtime = 15
	max log size = 51200
	max open files = 941978
	logging = file
	load printers = no
	printing = bsd
	printcap name = /dev/null
	disable spoolss = yes
	getwd cache = yes
	guest account = nobody
	map to guest = Bad User
	obey pam restrictions = no
	directory name cache size = 0
	kernel change notify = no
	panic action = /usr/local/libexec/samba/samba-backtrace
	nsupdate command = /usr/local/bin/samba-nsupdate -g
	server string = Mlincek
	ea support = yes
	store dos attributes = yes
	lm announce = yes
	hostname lookups = yes
	time server = yes
	acl allow execute always = true
	dos filemode = yes
	multicast dns register = yes
	domain logons = no
	local master = yes
	idmap config *: backend = tdb
	idmap config *: range = 90000001-100000000
	server role = standalone
	netbios name = MLINCEK
	workgroup = WORKGROUP
	security = user
	pid directory = /var/run/samba
	create mask = 0666
	directory mask = 0777
	client ntlmv2 auth = yes
	dos charset = CP437
	unix charset = UTF-8
	log level = 1
   

[mlincek]
	path = /mnt/mlincek/export
	printable = no
	veto files = /.snapshot/.windows/.mac/.zfs/
	writeable = yes
	browseable = yes
	recycle:repository = .recycle/%U
	recycle:keeptree = yes
	recycle:versions = yes
	recycle:touch = yes
	recycle:directory_mode = 0777
	recycle:subdir_mode = 0700
	vfs objects = zfs_space zfsacl recycle catia fruit streams_xattr aio_pthread
	hide dot files = yes
	guest ok = yes
	nfs4:mode = special
	nfs4:acedup = merge
	nfs4:chown = true
	zfsacl:acesort = dontcare
	ea support = yes
 

damjan

Dabbler
Joined
Jan 8, 2016
Messages
11
I made a dtrace profile using /usr/share/dtrace/toolkit script (BTW: perl location points to /usr/bin/perl instead of /usr/local/bin/perl).

Here is hotuser output for SMB2 configuration:
Code:
[root@mlincek] /usr/share/dtrace/toolkit# ./hotuser -lp 2145
Sampling... Hit Ctrl-C to end.
^C
LIBRARY												 COUNT   PCNT
libtime-basic-samba4.so									 1   0.0%
libiov-buf-samba4.so										1   0.0%
libndr-standard.so.0										2   0.0%
libutil-tdb-samba4.so									   2   0.0%
libsamba-hostconfig.so.0									3   0.0%
libsamba-passdb.so.0										3   0.0%
libreplace-samba4.so										4   0.0%
libsmbregistry-samba4.so									4   0.0%
streams_xattr.so											5   0.0%
libsunacl.so.1											  5   0.0%
catia.so													5   0.0%
libcli-smb-common-samba4.so								 6   0.0%
libndr-samba-samba4.so									  6   0.0%
libthr.so.3												 7   0.1%
zfsacl.so												   8   0.1%
libsamba-security-samba4.so								 9   0.1%
ld-elf.so.1												10   0.1%
libtevent.so.0											 11   0.1%
libdbwrap-samba4.so										13   0.1%
fruit.so												   13   0.1%
libtdb.so.1												14   0.1%
libndr.so.0												22   0.2%
libsmbconf.so.0											36   0.3%
libtalloc.so.2											 69   0.5%
libsamba-util.so.0										 86   0.7%
libsmbd-base-samba4.so									152   1.2%
libc.so.7												 299   2.3%
libgenrand-samba4.so									12335  93.9%


And function calls:
Code:
libgenrand-samba4.so`0x80165e986						  206   0.7%
libgenrand-samba4.so`0x80165e980						  208   0.7%
libgenrand-samba4.so`0x80165e97d						  208   0.7%
libgenrand-samba4.so`0x80165e910						  210   0.7%
libgenrand-samba4.so`0x80165e959						  214   0.7%
libgenrand-samba4.so`0x80165e96e						  221   0.7%
libgenrand-samba4.so`0x80165e6f9						  226   0.7%
libgenrand-samba4.so`0x80165e792						  233   0.8%
libc.so.7`memcpy										  285   0.9%
libgenrand-samba4.so`0x80165e6d7						  285   0.9%
libgenrand-samba4.so`0x80165e656						  286   0.9%
libgenrand-samba4.so`0x80165ed18						  288   0.9%
libgenrand-samba4.so`0x80165e82d						  296   1.0%
libgenrand-samba4.so`0x80165e907						  329   1.1%
libgenrand-samba4.so`0x80165e843						  349   1.1%
libgenrand-samba4.so`0x80165e860						  425   1.4%
libgenrand-samba4.so`0x80165e772						  490   1.6%
libgenrand-samba4.so`0x80165e7a8						  501   1.6%
libgenrand-samba4.so`0x80165e73e						  504   1.7%
libgenrand-samba4.so`0x80165e7f2						  536   1.8%
libgenrand-samba4.so`0x80165e7c3						  538   1.8%
libgenrand-samba4.so`0x80165e80d						  549   1.8%
libgenrand-samba4.so`0x80165e826						  563   1.8%
libgenrand-samba4.so`0x80165e86e						  583   1.9%
libgenrand-samba4.so`0x80165e7d9						  587   1.9%
libgenrand-samba4.so`0x80165e757						  606   2.0%
libgenrand-samba4.so`0x80165e78b						  648   2.1%
libgenrand-samba4.so`samba_SHA256_Update				  753   2.5%
libgenrand-samba4.so`0x80165e723						  854   2.8%
libgenrand-samba4.so`0x80165e898						 1078   3.5


SMB3 differs only in function calls:
Code:
libgenrand-samba4.so`0x801661e12							2   0.1%
libgenrand-samba4.so`0x801661e5d							2   0.1%
ld-elf.so.1`__tls_get_addr								  2   0.1%
libc.so.7`__error										   2   0.1%
libgenrand-samba4.so`0x801661e23							2   0.1%
libsmbconf.so.0`is_in_path								  2   0.1%
libgenrand-samba4.so`0x801661df4							2   0.1%
libgenrand-samba4.so`mdfour								 2   0.1%
libgenrand-samba4.so`0x801661e0f							2   0.1%
libtalloc.so.2`0x804859d7d								  2   0.1%
libgenrand-samba4.so`0x801661e55							3   0.1%
libgenrand-samba4.so`0x801661e59							3   0.1%
libgenrand-samba4.so`0x801661ebf							3   0.1%
libgenrand-samba4.so`0x801661e25							3   0.1%
libc.so.7`free											  3   0.1%
libc.so.7`memcpy											4   0.1%
libgenrand-samba4.so`0x801661e69							4   0.1%
libc.so.7`__malloc										  5   0.2%
libgenrand-samba4.so`0x801661e4a							5   0.2%
libc.so.7`strlen											6   0.2%
libgenrand-samba4.so`0x801661e61							7   0.2%
libc.so.7`memset											7   0.2%
libgenrand-samba4.so`samba_AES_encrypt					 23   0.7%
libgenrand-samba4.so`aes_cmac_128_update				   67   2.1%
libgenrand-samba4.so`_samba_rijndaelEncrypt			  2917  89.3%


I put smb encrypt = disabled into [global] and [share] section, but it seems it has no effect.

Questions:
1. How to disable encryption?
2. Shouldn't AES compression in SMB3 protocol perform better given that FreeNAS Mini XL can do some AES in hardware?
 

Ericloewe

Server Wrangler
Moderator
Joined
Feb 15, 2014
Messages
20,175
This is probably caused by some weird OS X behavior. SMB works fine with Windows and Samba clients.
 

damjan

Dabbler
Joined
Jan 8, 2016
Messages
11
As I see encryption support is still announced to clients:
Code:
sajba:~ bager$ smbutil statshares -a

==================================================================================================
SHARE						 ATTRIBUTE TYPE				VALUE
==================================================================================================
mlincek					  
							  SERVER_NAME				   mlincek
							  USER_ID					   503
							  SMB_NEGOTIATE				 AUTO_NEGOTIATE
							  SMB_VERSION				   SMB_3.02
							  SMB_SHARE_TYPE				DISK
							  SIGNING_SUPPORTED			 TRUE
							  EXTENDED_SECURITY_SUPPORTED   TRUE
							  UNIX_SUPPORT				  TRUE
							  LARGE_FILE_SUPPORTED		  TRUE
							  OS_X_SERVER				   TRUE
							  CLIENT_REQUIRES_SIGNING	   TRUE
							  FILE_IDS_SUPPORTED			TRUE
							  DFS_SUPPORTED				 TRUE
							  MULTI_CREDIT_SUPPORTED		TRUE
							  ENCRYPTION_SUPPORTED		  TRUE
							  SIGNING_ON					TRUE


and in my share I have smb encrypt = disabled.

Code:
	zfsacl:acesort = dontcare
	ea support = yes
	smb encrypt = disabled
 

damjan

Dabbler
Joined
Jan 8, 2016
Messages
11
I found a workaround for this performance issue on Mac OSX 10.11.5.
I disabled signing on the client.

Create a file /etc/nsmb.conf with the following content:
Code:
[default] 
	signing_required=no


Reboot the system and performance should be back to normal.

As for following comment:
2. Shouldn't AES compression in SMB3 protocol perform better given that FreeNAS Mini XL can do some AES in hardware?
Samba ships with its own crypto library that is not hardware accelerated. There are some patches around that enable use of third party libraries (GnuTLS/OpenSSL, ...) with have support for hardware crypto acceleration, but they are not yet in the main trunk.
 
Last edited by a moderator:
Status
Not open for further replies.
Top