Hi Everyone
Processor:
Intel(R) Xeon(R) CPU E5-2643 v3 @ 3.40GHz (12 cores)
Memory:
64 GiB
HDD:
OS: 2x Samsung 883 DCT Series - 240GB SATA 2.5" MZ-7LH240NE
Data: 4x Samsung 883 DCT Series - 3.84TB SATA 2.5" MZ-7LH3T8NE
Raid:
Z1
NIC:
Solareflare Flareon SFN7002F Dual-Port 10GbE
I am having issues with performance on FreeNAS-11.2-U5 particularly with cifs and small files. I am looking to migrate a windows file share server based that hosts small files and folders. Primarily csv files approx. 29.5 KB sample folder I was testing with has approx. 38,500 files and is 1.8GB. The issue I am having is related to a much larger folder with millions of files but I was using this sample set to test the performance of smb on Freenas vs a windows share. I get good performance for large files but not as good as a windows share. Below are the times I am seeing using robocopy. I know samba slows down when it uses small files but I did not expect for performance to be this bad. I have also included a copy of the smb.conf. Ive tried multichannel but it doesn't seem to be working with RSS Receive Side Scaling. If you see the output from the client server using Get-SmbMultichannelConnection, I see its making a multichannel connection, but Client RSS Capable returns False even though the Solarflare card supports RSS. The Windows file share returns True. Furthermore when transferring I also run vmstat -i to see the interrupts and I don't see the interrupts being spread accross the tx/rx queues. Running htop shown smb pegged at 100%. I have scoured the forums for answers and tested tons of parameters to no avail. Am I missing something or is this the type of performance that is expected? Please let me know if I missed any details. Thanks in advance!
Processor:
Intel(R) Xeon(R) CPU E5-2643 v3 @ 3.40GHz (12 cores)
Memory:
64 GiB
HDD:
OS: 2x Samsung 883 DCT Series - 240GB SATA 2.5" MZ-7LH240NE
Data: 4x Samsung 883 DCT Series - 3.84TB SATA 2.5" MZ-7LH3T8NE
Raid:
Z1
NIC:
Solareflare Flareon SFN7002F Dual-Port 10GbE
I am having issues with performance on FreeNAS-11.2-U5 particularly with cifs and small files. I am looking to migrate a windows file share server based that hosts small files and folders. Primarily csv files approx. 29.5 KB sample folder I was testing with has approx. 38,500 files and is 1.8GB. The issue I am having is related to a much larger folder with millions of files but I was using this sample set to test the performance of smb on Freenas vs a windows share. I get good performance for large files but not as good as a windows share. Below are the times I am seeing using robocopy. I know samba slows down when it uses small files but I did not expect for performance to be this bad. I have also included a copy of the smb.conf. Ive tried multichannel but it doesn't seem to be working with RSS Receive Side Scaling. If you see the output from the client server using Get-SmbMultichannelConnection, I see its making a multichannel connection, but Client RSS Capable returns False even though the Solarflare card supports RSS. The Windows file share returns True. Furthermore when transferring I also run vmstat -i to see the interrupts and I don't see the interrupts being spread accross the tx/rx queues. Running htop shown smb pegged at 100%. I have scoured the forums for answers and tested tons of parameters to no avail. Am I missing something or is this the type of performance that is expected? Please let me know if I missed any details. Thanks in advance!
Code:
COPY FROM WINDOWS 2012 Host to Freenas SMB SHARE: 1 file (5.6GB)
robocopy source destination /mir /mt:128 /sec /r:0 /w:0 /ns /nc /nfl /ndl /log:c:\freenas.txt
Total Copied Skipped Mismatch FAILED Extras
Dirs : 1 1 0 0 0 0
Files : 2 1 1 0 0 0
Bytes : 5.662 g 5.662 g 26.0 k 0 0 0
Times : 0:00:41 0:00:10 0:00:00 0:00:10
Ended : Tuesday, August 27, 2019 9:18:02 AM
COPY FROM WINDOWS 2012 HOST > Windows 2012 RAID 5 SSD SHARE: 1 file (5.6GB)
robocopy source destination /mir /mt:128 /sec /r:0 /w:0 /ns /nc /nfl /ndl /log:c:\freenas.txt
Total Copied Skipped Mismatch FAILED Extras
Dirs : 1 1 0 0 0 0
Files : 2 2 0 0 0 0
Bytes : 5.662 g 5.662 g 0 0 0 0
Times : 0:00:10 0:00:02 0:00:00 0:00:02
Ended : Tuesday, August 27, 2019 9:19:37 AM
COPY FROM WINDOWS 2012 Host to Freenas SMB SHARE: (38477 files 1.8GB)
robocopy source destination /mir /mt:128 /sec /r:0 /w:0 /ns /nc /nfl /ndl /log:c:\freenas.txt
Total Copied Skipped Mismatch FAILED Extras
Dirs : 333 333 332 0 0 0
Files : 38477 38476 1 0 0 0
Bytes : 1.814 g 1.814 g 26.0 k 0 0 0
Times : 4:30:01 0:02:06 0:00:00 0:00:01
Ended : Tuesday, August 27, 2019 9:33:53 AM
COPY FROM WINDOWS 2012 Host to Freenas SMB SHARE: (38477 files 1.8GB)
robocopy source destination /mir /mt:128 /sec /r:0 /w:0 /ns /nc /nfl /ndl /log:c:\freenas.txt
Total Copied Skipped Mismatch FAILED Extras
Dirs : 333 333 332 0 0 0
Files : 38477 38476 1 0 0 1
Bytes : 1.814 g 1.814 g 26.0 k 0 0 5.662 g
Times : 0:17:31 0:00:08 0:00:00 0:00:00
Ended : Tuesday, August 27, 2019 9:25:37 AM
OUTPUT FROM CLIENT: Get-SmbMultichannelConnection
Server Name Selected Client IP Server IP Client Interface Index Server Interface Index Client RSS Capable Client RDMA Capable
----------- -------- --------- --------- ---------------------- ---------------------- ------------------ -------------------
FREENAS True 10.24.14.70 10.24.14.211 17 8 False False
WINDOWS FILE SHARE True 10.24.14.70 10.24.14.164 17 19 True False
smb.conf
[global]
aio max threads = 2
allow trusted domains = No
client ldap sasl wrapping = plain
deadtime = 15
disable spoolss = Yes
dns proxy = No
domain master = No
dos charset = CP437
hostname lookups = Yes
kernel change notify = No
lm announce = Yes
load printers = No
local master = No
logging = file
max log size = 51200
max open files = 1883558
nsupdate command = /usr/local/bin/samba-nsupdate -g
obey pam restrictions = Yes
preferred master = No
printcap name = /dev/null
realm = DOMAIN.COM
security = ADS
server min protocol = SMB2_02
server multi channel support = Yes
server role = member server
server string = FreeNAS Server
template shell = /bin/sh
winbind cache time = 7200
winbind enum groups = Yes
winbind enum users = Yes
winbind offline logon = Yes
winbind refresh tickets = Yes
workgroup = DOMAIN
idmap config domain: range = 20000-90000000
idmap config domain: backend = rid
idmap config *: range = 90000001-100000000
idmap config * : backend = tdb
acl allow execute always = Yes
create mask = 0666
directory mask = 0777
directory name cache size = 0
dos filemode = Yes
strict locking = No
[root_shares]
aio write size = 0
path = "/mnt/pool1/root_shares"
read only = No
veto files = /.snapshot/.windows/.mac/.zfs/
vfs objects = zfs_space zfsacl streams_xattr
zfsacl:acesort = dontcare
nfs4:chown = true
nfs4:acedup = merge
nfs4:mode = special
