I have been trying to troubleshoot this for the past weekend and would like to hear your experience/advice
I got a relatively powerful FN box (spec in signature) and a ESXi box (dual 2667 v2, 128G RAM, a Dell 10GB NIC made by boardcom) connected by the a 10G switch. The FN box servers ESXi host as an iSCSI target, which is backed by the mirror pool. There is another pool of a 8 disk raid Z2 vdev, that backs SMB shares. The iSCSI part works great (~800MB/s seq, ~40MB/s 4K @QD1, ~400MB/s 4K @QD32), however the SMB share does not work very well with my Ubuntu VMs:
1. Server side copy never seems to work. According to Samba wiki this is suppose to work on client Linux OS as well windows ones, but not sure how relevant it is to FN. Just wondering if any of you have been able to get it to work on a Linux client...
2. Even without the server copy offload, SMB shares are still unreasonably slow on Ubuntu VMs. Usually about ~70-200MB/s read/write (measured by using coping ~10GB files using rsync), while on windows(10 or 2016) VMs it can range from 300MB/s to 1000MB/s read/write/server side copy (copying same files using windows explorer).
Here are something I have tried:
1. Iperf test shows >9 Gbit/s both ways between FN box and Ubuntu VM
2. Ran iozone test on Ubuntu VM against the SMB mount point. The result shows almost 1000MB/s read/write, this clearly results from highly compressible test data being used. But also suggests that the bottleneck is somewhere is FN the box? unless Ubuntu is also doing compression on the client side.
3. During testing the other VMs are relatively idle, only a few windows server VM providing active directory service. Both pools are almost empty (less than 5% used)
Here is my fstab
My smb4.conf
I got a relatively powerful FN box (spec in signature) and a ESXi box (dual 2667 v2, 128G RAM, a Dell 10GB NIC made by boardcom) connected by the a 10G switch. The FN box servers ESXi host as an iSCSI target, which is backed by the mirror pool. There is another pool of a 8 disk raid Z2 vdev, that backs SMB shares. The iSCSI part works great (~800MB/s seq, ~40MB/s 4K @QD1, ~400MB/s 4K @QD32), however the SMB share does not work very well with my Ubuntu VMs:
1. Server side copy never seems to work. According to Samba wiki this is suppose to work on client Linux OS as well windows ones, but not sure how relevant it is to FN. Just wondering if any of you have been able to get it to work on a Linux client...
2. Even without the server copy offload, SMB shares are still unreasonably slow on Ubuntu VMs. Usually about ~70-200MB/s read/write (measured by using coping ~10GB files using rsync), while on windows(10 or 2016) VMs it can range from 300MB/s to 1000MB/s read/write/server side copy (copying same files using windows explorer).
Here are something I have tried:
1. Iperf test shows >9 Gbit/s both ways between FN box and Ubuntu VM
2. Ran iozone test on Ubuntu VM against the SMB mount point. The result shows almost 1000MB/s read/write, this clearly results from highly compressible test data being used. But also suggests that the bottleneck is somewhere is FN the box? unless Ubuntu is also doing compression on the client side.
3. During testing the other VMs are relatively idle, only a few windows server VM providing active directory service. Both pools are almost empty (less than 5% used)
Here is my fstab
Code:
//freenasFQDM/Download-Temp /mnt/Download-Temp cifs credentials=/path/.smbcredentials,iocharset=utf8,uid=user,gid=group 0 0
My smb4.conf
Code:
[global]
server min protocol = SMB2_02
server max protocol = SMB3
interfaces = 127.0.0.1 192.168.21.25
bind interfaces only = yes
encrypt passwords = yes
dns proxy = no
strict locking = no
aio max threads = 2
oplocks = yes
deadtime = 15
max log size = 51200
private dir = /var/db/samba4/private
max open files = 3772379
logging = file
load printers = no
printing = bsd
printcap name = /dev/null
disable spoolss = yes
getwd cache = yes
guest account = nobody
map to guest = Bad User
obey pam restrictions = yes
ntlm auth = no
directory name cache size = 0
kernel change notify = no
nsupdate command = /usr/local/bin/samba-nsupdate -g
server string = FreeNAS Server
ea support = yes
store dos attributes = yes
lm announce = yes
hostname lookups = yes
acl allow execute always = false
dos filemode = yes
multicast dns register = yes
domain logons = no
idmap config *: backend = tdb
idmap config *: range = 90000001-100000000
server role = member server
workgroup = MYITLAB
realm = AD.MYITLAB.local
security = ADS
client use spnego = yes
local master = no
domain master = no
preferred master = no
ads dns update = no
winbind cache time = 7200
winbind offline logon = yes
winbind enum users = yes
winbind enum groups = yes
winbind nested groups = yes
winbind use default domain = yes
winbind refresh tickets = yes
idmap config MYITLAB: backend = rid
idmap config MYITLAB: range = 20000-90000000
allow trusted domains = yes
client ldap sasl wrapping = seal
template shell = /bin/sh
template homedir = /mnt/Z2/HomefolderShare/%D/%U
netbios name = FREENAS
create mask = 0666
directory mask = 0777
client ntlmv2 auth = yes
dos charset = CP437
unix charset = UTF-8
log level = 1
[Download-Complete]
path = "/mnt/Z2/Download-Complete"
printable = no
aio write size = 0
veto files = /.snapshot/.windows/.mac/.zfs/
writeable = yes
browseable = yes
access based share enum = yes
vfs objects = shadow_copy_zfs zfs_space zfsacl streams_xattr
hide dot files = yes
hosts allow = 192.168.21.0/24 192.168.30.0/24 192.168.50.30 192.168.50.0/24
guest ok = yes
nfs4:mode = special
nfs4:acedup = merge
nfs4:chown = true
zfsacl:acesort = dontcare
[Download-Temp]
path = "/mnt/Mirrors/Download-Temp"
printable = no
aio write size = 0
veto files = /.snapshot/.windows/.mac/.zfs/
writeable = yes
browseable = yes
access based share enum = yes
vfs objects = shadow_copy_zfs zfs_space zfsacl streams_xattr
hide dot files = yes
hosts allow = 192.168.21.0/24 192.168.30.0/24 192.168.50.30 192.168.50.0/24
guest ok = yes
nfs4:mode = special
nfs4:acedup = merge
nfs4:chown = true
zfsacl:acesort = dontcare
[homes]
valid users = %U
path = "/mnt/Z2/HomefolderShare/%D/%U"
comment = Home Directories
printable = no
aio write size = 0
veto files = /.snapshot/.windows/.mac/.zfs/
writeable = yes
browseable = no
access based share enum = yes
recycle:repository = .recycle/%U
recycle:keeptree = yes
recycle:versions = yes
recycle:touch = yes
recycle:directory_mode = 0777
recycle:subdir_mode = 0700
vfs objects = shadow_copy_zfs zfs_space zfsacl streams_xattr recycle crossrename
hide dot files = yes
guest ok = no
nfs4:mode = special
nfs4:acedup = merge
nfs4:chown = true
zfsacl:acesort = dontcare
[Z2 shares]
path = "/mnt/Z2/Shares"
printable = no
aio write size = 0
veto files = /.snapshot/.windows/.mac/.zfs/
writeable = yes
browseable = yes
access based share enum = yes
vfs objects = shadow_copy_zfs zfs_space zfsacl streams_xattr
hide dot files = yes
guest ok = no
nfs4:mode = special
nfs4:acedup = merge
nfs4:chown = true
zfsacl:acesort = dontcare
