FreeNAS Mini XL - Windows 11 File system operations slow over SMB

[damjan]

I get good file copy performance to and from server over SMB protocol at around 110MB/s. However, tasks that require quick file system operations are painfully slow. I've seen lots of TrueNAS benchmarks, but none focusing on testing performance of file system operations, so I don't know if this is sort of expected performance (hopefully not).

I observe 100% CPU usage by smbd (single core) whenever lots of files are involved. NFS is much faster for that case, but with NFS on Windows I get only 70MB/s, but that is not the biggest issue. My issue with NFS on Windows is that it NFS share hangs after a while and the only way to resolve it to reboot my Windows PC (yuck).

To peform my own file system benchmark, I used the Linux kernel source in gz format which contains about 81k files and is 1.2GB large when uncompressed.

Client:
Windows 11, with Intel 1Gbit NIC , Intel i7, 32GB Ram and Samsung 970 EVO SSD, making sure the client CPU and SSD were not a bottleneck.

SMB:

Code:

PS > Get-SmbConnection

ServerName ShareName UserName       Credential                            Dialect NumOpens
---------- --------- --------       ----------                            ------- --------
server     share                                                          3.1.1   1

PS > Get-SmbClientConfiguration

CompressibilitySamplingSize           : 524288000
CompressibleThreshold                 : 104857600
ConnectionCountPerRssNetworkInterface : 4
DirectoryCacheEntriesMax              : 16
DirectoryCacheEntrySizeMax            : 65536
DirectoryCacheLifetime                : 10
DisableCompression                    : False
DormantFileLimit                      : 1023
EnableBandwidthThrottling             : True
EnableByteRangeLockingOnReadOnlyFiles : True
EnableCompressibilitySampling         : False
EnableInsecureGuestLogons             : True
EnableLargeMtu                        : True
EnableLoadBalanceScaleOut             : True
EnableMultiChannel                    : True
EnableSecuritySignature               : True
EncryptionCiphers                     : AES_128_GCM, AES_128_CCM, AES_256_GCM, AES_256_CCM
ExtendedSessionTimeout                : 1000
FileInfoCacheEntriesMax               : 64
FileInfoCacheLifetime                 : 10
FileNotFoundCacheEntriesMax           : 128
FileNotFoundCacheLifetime             : 5
ForceSMBEncryptionOverQuic            : False
KeepConn                              : 600
MaxCmds                               : 50
MaximumConnectionCountPerServer       : 32
OplocksDisabled                       : False
RequestCompression                    : False
RequireSecuritySignature              : False
SessionTimeout                        : 60
SkipCertificateCheck                  : False
UseOpportunisticLocking               : True
WindowSizeThreshold                   : 8

NFS:

Code:

mount -o rsize=128 -o wsize=128 -o anon -o nolock -o casesensitive=no \\server\mnt\nodename\share m:

Local    Remote                                 Properties
-------------------------------------------------------------------------------
m:       \\192.168.254.13\mnt\nodename\share    UID=-2, GID=-2
                                                rsize=131072, wsize=131072
                                                mount=soft, timeout=6.4
                                                retry=1, locking=no
                                                fileaccess=755, lang=ANSI
                                                casesensitive=no
                                                sec=sys

Server share:
TrueNAS 13.0-U1.1, case insensitive RAIDZ ZFS share, with sync disabled, lz4 compression, L2ARC and ZIL cache. I run this test with default settings on SMB share, i.e. without any auxilary parameters on share. Global Samba settings as follows:

Code:

max stat cache size = 0
security = user
username map = /usr/local/etc/smbuser
smb encrypt = disabled

Code:

smbstatus -b

Samba version 4.15.7
PID     Username     Group        Machine                                   Protocol Version  Encryption           Signing
----------------------------------------------------------------------------------------------------------------------------------------
21485   username        wheel        192.168.254.252 (ipv4:192.168.254.252:51017) SMB3_11           -                    partial(AES-128-GMAC)

General observation during the benchmark:
Whenever something was slow during the benchmark, smbd was using 100% of CPU (1 core). IO was pretty much idle for all operations; probably cached as testing dataset it relatively small. The only exception is large file copy to and from server where network was the bottleneck.


Results:

Task	FreeNAS XL (local)	Units/s	SMB Elapsed Time	SMB Units/s	NFS Elapsed Time	NFS Units/S
1. Uncompress Linux kernel	0h 0m 35.7s​	32.4 MB/s	1h 0m 35.0s​	0.3 MB/s	0h 3m 38.9s​	5.3 MB/s
2. Count files	0h 0m 00.5s​	166591 files/s	0h 0m 41.2s​	1989 files/s	0h 0m 05.3s​	15368 files/s
3. Zip Linux Kernel (no-comp)	0h 0m 12.3s​	94.3 MB/s	0h 16m 49.8s​	1.2 MB/s	0h 7m 03.0s​	2.8 MB/s
4. Large file copy Client to Server	NA		0h 0m 10.4s​	112.3 MB/s	0h 0m 17.3s​	67.2 MB/s
5. Large file copy Server to Client	NA		0h 0m 10.6s​	110.0 MB/s	0h 0m 17.8s​	64.7 MB/s
6. Delete files	0h 0m 07.0s​	11694 files/s	0h 30m 01.7s​	45 files/s	0h 1m 50.1s​	745 files/s

Notes and observations to each task:

Task	FreeNAS notes	FreeNAS cmd	SMB Notes	NFS Notes	Win cmd
1. Uncompress Linux kernel	bsdtar 100% CPU single core	time tar xf linux-5.19.4.tar.gz	smbd 100% one core, iostat 1-2% utilisation	nfsd 25% CPU of single core, lots of idle IO	Measure-Command {cd M:\test; tar xf C:\linux-5.19.4.tar.gz}
2. Count files	instant	time find . -print | wc -l	smbd 100% one core, iostat 1-2% utilisation	nfsd 25% CPU of single core, lots of idle IO	Measure-Command {Get-ChildItem -Recurse -Path M:\test}
3. Zip Linux Kernel (no-comp)	bsdtar 100%	time tar cf /dev/null linux-5.19.4/*	smbd 30%, iostat 1-2% utilisation	no bottleneck identified	Measure-Command {Compress-Archive -CompressionLevel NoCompression -Path M:\test\* -DestinationPath E:\linuxkernel.zip}
4. Large file copy Client to Server	NA	NA	Works well	Only 70MB/s	Measure-Command {Copy-Item E:\linuxkernel.zip M:\test}
5. Large file copy Server to Client			Works well	Only 70Mb/s	Measure-Command {Copy-Item M:\test\linuxkernel.zip E:\linuxkernel2.zip}
6. Delete files			smbd 100% one core, iostat 1-2% utilisation	nfsd 25% CPU of single core, lots of idle IO	Measure-Command {Remove-Item M:\test\* -Recurse -Force}

I know that Avoton in FreeNAS XL is not the most powerful CPU and that Samba is not the fastest file server around, however, I did expect better performance. One hour to 81k files and 1.2GB of data is just too slow to be usable.

I used truss and dtrace to investigate smbd CPU usage and I see there are lots and lots of calls for every file accessed. It seems deeper in the directory the file is, more syscalls are being generated.

Is this expected performance of FreeNAS XL Mini and TrueNAS Core? Does TrueNAS has any unit testing for file system performance? Are file system operations also so slow in your tests? Any "go fast" setting to recommend? Is anyone else observing so slow performance when lots of files are involved?

 

[damjan]

From Samba 4.17 Release Notes:

SMB Server performance improvements​

The security improvements in recent releases (4.13, 4.14, 4.15, 4.16), mainly as protection against symlink races, caused performance regressions for meta data heavy workloads.

With 4.17 the situation improved a lot again:

• Pathnames given by a client are devided into dirname and basename. The amount of syscalls to validate dirnames is reduced to 2 syscalls (openat, close) per component. On modern Linux kernels (>= 5.6) smbd makes use of the openat2() syscall with SOLVE_NO_SYMLINKS, in order to just use 2 syscalls (openat2, close) for the whole dirname.

• Contended path based operations used to generate a lot of unsolicited wakeup events causing thundering herd problems, which lead to masive latencies for some clients. These events are now avoided in order to provide stable latencies and much higher throughput of open/close operations.

Is that what is causing the slowness in my case? Does anyone have metrics how much were meta data heavy workloads slowed down in 4.13 to 4.16? Or how much has performance improved in 4.17rc?

Is Samba release upgraded only for TrueNAS version upgrades, ie 13.0->13.1 or it can come with one of minor updates, e.g. 13.0-U4?

 

[Samuel Tai]

@anodos, can you comment on the metadata regressions in 4.15?

 

[anodos]

@anodos, can you comment on the metadata regressions in 4.15?

Yes, there is an impact for certain workloads due to the symlink race condition fixes. I have a WIP branch where I mitigated for 4.15 (but needs more baking). SCALE BlueFin will have Samba 4.17 (currently RC4), but TrueNAS 13 will stay on 4.15 since it's current samba old stable.

 

[damjan]

@anodos, from your observations, is 1h to unpack linux kernel over SMB expected performance on FreeNAS XL with Samba 4.15 (vs 3m 38s for NFS and 36s locally)? Or there might be some other misconfiguration that it further slowing it down?

 

[NugentS]

@damjan Can you post your exact hardware please (as per forum rules), in particular pool design, drive models etc

 

[damjan]

Sure. It's a FreeNAS Mini XL model, predecessor of TrueNAS Mini. But as said in the thread, I do not see any I/O bottleneck in iostat. It's just CPU, I/O is idle most of the time.

Code:

Motherboard: C2750D4I
CPU: Intel(R) Atom(TM) CPU  C2750  @ 2.40GHz
OS disk: 16GB SATA DOM
Data Disks: 4 x Hitachi HDN724030ALE640 3TB in RAIDZ
Intel SSD 335 for ZIL and L2ARC
Memory: 32GB
Network: 2 x Intel(R) I210 (Copper)

zpool config:
        NAME                                          STATE     READ WRITE CKSUM
        mlincek                                       ONLINE       0     0     0
          raidz1-0                                    ONLINE       0     0     0
            ada6                                      ONLINE       0     0     0
            ada5                                      ONLINE       0     0     0
            ada0                                      ONLINE       0     0     0
            ada4                                      ONLINE       0     0     0
        logs
          gptid/36b3e90e-440f-11e6-b5ae-d05099c0fe8f  ONLINE       0     0     0
        cache
          gptid/b7e3738f-b788-11eb-b2b5-d0509964a7e4  ONLINE       0     0     0

 

[damjan]

@anodos Is there any JIRA issue for this performance mitigation in 4.15, so that I can follow it and retest it when patch is released?

 

[damjan]

I further investigated this issue. Checking /var/log/samba4/log.smbd file, I noticed the following logs for every file created on the share:

[2022/10/02 22:49:46.198832, 1] ../../source3/smbd/service.c:357(create_connection_session_info)
create_connection_session_info: guest user (from session setup) not permitted to access this share (share)
[2022/10/02 22:49:46.198931, 1] ../../source3/smbd/service.c:546(make_connection_snum)
create_connection_session_info failed: NT_STATUS_ACCESS_DENIED
[2022/10/02 22:49:46.214680, 1] ../../source3/smbd/service.c:357(create_connection_session_info)
create_connection_session_info: guest user (from session setup) not permitted to access this share (share)
[2022/10/02 22:49:46.214786, 1] ../../source3/smbd/service.c:546(make_connection_snum)
create_connection_session_info failed: NT_STATUS_ACCESS_DENIED

Linux kernel source has lots of files, so lots of failed attempts. It took me hours to find the root cause of this. I checked every setting and I was banging my head to the wall. The culprit? Windows Defender Real Time detection, kid you not. This happens with Windows 11 21H2 (22000.1042) and 22H2 (22621.525) and it is repeatable. Real-time detection on - logs as above, switch off, logs are gone. I am not the first one to experience this issue see Edit 2.
Windows Defender impacts the performance of uncompressing kernel source and it dropped from 1h to 20min.

While uncompressing Linux kernel over SMB is now faster, it is still very slow compared to 3min for NFS and 0.5min locally.

Other filesystem operations remain as slow as before.