SMB share is significantly slower on Linux clients

[Ender117]

I have been trying to troubleshoot this for the past weekend and would like to hear your experience/advice

I got a relatively powerful FN box (spec in signature) and a ESXi box (dual 2667 v2, 128G RAM, a Dell 10GB NIC made by boardcom) connected by the a 10G switch. The FN box servers ESXi host as an iSCSI target, which is backed by the mirror pool. There is another pool of a 8 disk raid Z2 vdev, that backs SMB shares. The iSCSI part works great (~800MB/s seq, ~40MB/s 4K @QD1, ~400MB/s 4K @QD32), however the SMB share does not work very well with my Ubuntu VMs:

1. Server side copy never seems to work. According to Samba wiki this is suppose to work on client Linux OS as well windows ones, but not sure how relevant it is to FN. Just wondering if any of you have been able to get it to work on a Linux client...

2. Even without the server copy offload, SMB shares are still unreasonably slow on Ubuntu VMs. Usually about ~70-200MB/s read/write (measured by using coping ~10GB files using rsync), while on windows(10 or 2016) VMs it can range from 300MB/s to 1000MB/s read/write/server side copy (copying same files using windows explorer).

Here are something I have tried:
1. Iperf test shows >9 Gbit/s both ways between FN box and Ubuntu VM
2. Ran iozone test on Ubuntu VM against the SMB mount point. The result shows almost 1000MB/s read/write, this clearly results from highly compressible test data being used. But also suggests that the bottleneck is somewhere is FN the box? unless Ubuntu is also doing compression on the client side.
3. During testing the other VMs are relatively idle, only a few windows server VM providing active directory service. Both pools are almost empty (less than 5% used)


Here is my fstab

Code:

//freenasFQDM/Download-Temp /mnt/Download-Temp cifs credentials=/path/.smbcredentials,iocharset=utf8,uid=user,gid=group  0 0

My smb4.conf

Code:

[global]
    server min protocol = SMB2_02
    server max protocol = SMB3
    interfaces = 127.0.0.1 192.168.21.25
    bind interfaces only = yes
    encrypt passwords = yes
    dns proxy = no
    strict locking = no
    aio max threads = 2
    oplocks = yes
    deadtime = 15
    max log size = 51200
    private dir = /var/db/samba4/private
    max open files = 3772379
    logging = file
    load printers = no
    printing = bsd
    printcap name = /dev/null
    disable spoolss = yes
    getwd cache = yes
    guest account = nobody
    map to guest = Bad User
    obey pam restrictions = yes
    ntlm auth = no
    directory name cache size = 0
    kernel change notify = no
    nsupdate command = /usr/local/bin/samba-nsupdate -g
    server string = FreeNAS Server
    ea support = yes
    store dos attributes = yes
    lm announce = yes
    hostname lookups = yes
    acl allow execute always = false
    dos filemode = yes
    multicast dns register = yes
    domain logons = no
    idmap config *: backend = tdb
    idmap config *: range = 90000001-100000000
    server role = member server
    workgroup = MYITLAB
    realm = AD.MYITLAB.local
    security = ADS
    client use spnego = yes
    local master = no
    domain master = no
    preferred master = no
    ads dns update = no
    winbind cache time = 7200
    winbind offline logon = yes
    winbind enum users = yes
    winbind enum groups = yes
    winbind nested groups = yes
    winbind use default domain = yes
    winbind refresh tickets = yes
    idmap config MYITLAB: backend = rid
    idmap config MYITLAB: range = 20000-90000000
    allow trusted domains = yes
    client ldap sasl wrapping = seal
    template shell = /bin/sh
    template homedir = /mnt/Z2/HomefolderShare/%D/%U
    netbios name = FREENAS
    create mask = 0666
    directory mask = 0777
    client ntlmv2 auth = yes
    dos charset = CP437
    unix charset = UTF-8
    log level = 1
   

[Download-Complete]
    path = "/mnt/Z2/Download-Complete"
    printable = no
    aio write size = 0
    veto files = /.snapshot/.windows/.mac/.zfs/
    writeable = yes
    browseable = yes
    access based share enum = yes
    vfs objects = shadow_copy_zfs zfs_space zfsacl streams_xattr
    hide dot files = yes
    hosts allow = 192.168.21.0/24 192.168.30.0/24 192.168.50.30 192.168.50.0/24
    guest ok = yes
    nfs4:mode = special
    nfs4:acedup = merge
    nfs4:chown = true
    zfsacl:acesort = dontcare
   

[Download-Temp]
    path = "/mnt/Mirrors/Download-Temp"
    printable = no
    aio write size = 0
    veto files = /.snapshot/.windows/.mac/.zfs/
    writeable = yes
    browseable = yes
    access based share enum = yes
    vfs objects = shadow_copy_zfs zfs_space zfsacl streams_xattr
    hide dot files = yes
    hosts allow = 192.168.21.0/24 192.168.30.0/24 192.168.50.30 192.168.50.0/24
    guest ok = yes
    nfs4:mode = special
    nfs4:acedup = merge
    nfs4:chown = true
    zfsacl:acesort = dontcare
   

[homes]
    valid users = %U
    path = "/mnt/Z2/HomefolderShare/%D/%U"
    comment = Home Directories
    printable = no
    aio write size = 0
    veto files = /.snapshot/.windows/.mac/.zfs/
    writeable = yes
    browseable = no
    access based share enum = yes
    recycle:repository = .recycle/%U
    recycle:keeptree = yes
    recycle:versions = yes
    recycle:touch = yes
    recycle:directory_mode = 0777
    recycle:subdir_mode = 0700
    vfs objects = shadow_copy_zfs zfs_space zfsacl streams_xattr recycle crossrename
    hide dot files = yes
    guest ok = no
    nfs4:mode = special
    nfs4:acedup = merge
    nfs4:chown = true
    zfsacl:acesort = dontcare
   

[Z2 shares]
    path = "/mnt/Z2/Shares"
    printable = no
    aio write size = 0
    veto files = /.snapshot/.windows/.mac/.zfs/
    writeable = yes
    browseable = yes
    access based share enum = yes
    vfs objects = shadow_copy_zfs zfs_space zfsacl streams_xattr
    hide dot files = yes
    guest ok = no
    nfs4:mode = special
    nfs4:acedup = merge
    nfs4:chown = true
    zfsacl:acesort = dontcare

 

[dlavigne]

Were you able to narrow down the reason for this?

 

[Ender117]

Were you able to narrow down the reason for this?

I have been busy recently and don't have time to troubleshoot. What I have in mind was test with different linux distro and testing on physical machine vs VM. If you got any more ideas I am open to try it