Samba performance and reliability issues

[dealy663]

Hi

I'm on TrueNAS-SCALE-22.02-RC.2 and have always had really strange unreliable and frequently slow behavior when trying to copy files from my Macs to the new NAS system I've built up. My pools are simple mirrors and I'm sharing with SMB. I started out with standard 1Gbe and slowly moved up to 2.5Gbe and now have a 10Gbe link to my main MacBook Pro. No matter which speed of link copying files via the finder or on the command line proved difficult. With things usually starting out well seeing flashes of 70-80MB/sec throughput, but then within 5 min or so falling lower and lower until finally hours later dropping down to a few KB/sec. So I never really got all of my data on to the NAS. I'm sure having hundreds of thousands of small file plays a big role in the avg throughput numbers. But it was taking 6+hours, sometimes more than 8 to copy less than 2 TB of data

Eventually I went to rsync on the command line and discovered that the link would break (broken pipe error) every time. This was still copying over SMB via the mounted drives. Today it finally occurred to me to setup an Rsync Module on truenas. Once I got that working I did a ~100GB transfer without dropping any broken pipe error messages and averaged ~95MB/sec transfer rate. This was over my 10Gbe connection but I'll take what I can get (and blame the rest on the spinning HD inside my server). Ok much much much better.

But what is going on here? Why was using my SMB configuration so terrible? I've been working with SMB protocol transfers for what 15-20 years now? Are there some liabilities with transferring large numbers of files in this version of TrueNAS or is there some aspect of the SMB config I have wrong that is contributing to this?

 

[morganL]

Its best to do specific performance tests like FIO to understand performance..document it and your setup.
Not enough info hesr to see if network, drives, zfs, or CPU are causing issues.
Copying files is always dependent on latency/performance of both machines....

 

[nickspacemonkey]

Do you know if the drives you are using are SMR drives? They tend to not play so nice with ZFS, in particular when writing to the pool.

 

[dealy663]

I'm not sure I they are SMR, but here is a link to the drives in that pool. Just noticed in one of the reviews of that drive someone states that all Ironwolf drives are SMR free.

So is there some aspect of the ZFS Pool config that could be causing problems over SMB, but then working more normally when bypassing it over rsync? Watching the transfer continue last night I saw bursts of up to 500+MB/sec transfer rates on some files over rsync. But that would last for just a second or so. The averages were always around 87-95 MB/sec. iPerf3 almost always shows a solid 9.7+ Gb/sec rate, I've let that run for up to 10 min with no problems. The MacBook fan eventually kicks in as it works hard to keep the data rate up. So the network link feels like it is solid. Same thing when working at a more normal 1Gbe. My 2.5Gbe dongle is not able to maintain its rates though, frequently falling to 1.7 Gb/sec.

Looking at my SMB share for the pool there isn't much in the config:
Enable ACL = true
Browseable to Network Clients = true
Enable Shadow Copies = true
Enable Alternate Data Streams = true (currently the only Windows systems is one VM hosted on TrueNAS, maybe this isn't necessary)
Enable Durable Handles = true


This is the share config out of smb4_share.conf (is there any other location I should be looking at?):
[PhotoMirror-DS]
vfs objects = {'parsed': ['fruit', 'streams_xattr', 'shadow_copy_zfs', 'nfs4acl_xattr', 'zfs_core', 'io_uring'], 'raw': 'fruit streams_xattr shadow_copy_zfs nfs4acl_xattr zfs_core io_uring'}
ea support = {'parsed': False, 'raw': 'False'}
tn:purpose = {'parsed': 'NO_PRESET', 'raw': 'NO_PRESET'}
path = {'parsed': '/mnt/PhotoMirror-Pool/PhotoMirror-DS', 'raw': '/mnt/PhotoMirror-Pool/PhotoMirror-DS'}
tn:path_suffix = {'parsed': '', 'raw': ''}
tn:home = {'parsed': False, 'raw': 'False'}
comment = {'parsed': '', 'raw': ''}
read only = {'parsed': False, 'raw': 'False'}
browseable = {'parsed': True, 'raw': 'True'}
guest ok = {'parsed': False, 'raw': 'False'}
hosts allow = {'parsed': [], 'raw': ''}
hosts deny = {'parsed': [], 'raw': ''}
access based share enum = {'parsed': False, 'raw': 'False'}
nfs4acl_xattr:nfs4_id_numeric = {'parsed': True, 'raw': 'True'}
nfs4acl_xattr:validate_mode = {'parsed': False, 'raw': 'False'}
nfs4acl_xattr:xattr_name = {'parsed': 'system.nfs4_acl_xdr', 'raw': 'system.nfs4_acl_xdr'}
nfs4acl_xattr:encoding = {'parsed': 'xdr', 'raw': 'xdr'}
nfs4:chown = {'parsed': True, 'raw': 'True'}
posix locking = {'parsed': False, 'raw': 'False'}
kernel share modes = {'parsed': False, 'raw': 'False'}
kernel oplocks = {'parsed': False, 'raw': 'False'}
smbd max xattr size = {'parsed': 2097152, 'raw': '2097152'}
fruit:metadata = {'parsed': 'stream', 'raw': 'stream'}
fruit:resource = {'parsed': 'stream', 'raw': 'stream'}
fruit:time machine = {'parsed': False, 'raw': 'False'}
fruit:time machine max size = {'parsed': 0, 'raw': '0'}
tn:vuid = {'parsed': '', 'raw': ''}

There are a number of things in the above config I'm not too familiar with.