SOLVED Help understanding SAMBA high single core CPU usage for my workload

[tarian]

Hi,
I am running into a situation where a kopia backup that is run against a smb dataset with around a million files is causing high single core CPU (70-100%) usage.

How my system is laid out -
Windows 11 Backup PC running Kopia with TrueNAS core dataset connected via SMB.
This is over a 10Gbit link. Transfer speeds reach 10Gbit without issues.
TrueNAS core system - 2x E5-2667 v2 @3.30GHz w/ 384GB RAM (was using 2x E5-2650Lv2 which is half the per core performance)
Specific dataset is ~6TB ~1.35 million files on a RaidZ10 (4 RAIDZ1, 3 wide SATA HDD based vdevs striped) pool

Kopia spawns concurrent reads (24 because I have 24 logical CPUs on my Windows 11 Backup PC) against the dataset which leads to the smb client connection to use up it's 1 core.

I would like help understanding why this workload causes high cpu usage. I expected IOPS to be an issue rather than CPU for a workload like this.

SMB config:

Code:

hide special files = yes
server multi channel support = yes
interfaces = "192.168.88.81;capability=RSS,speed=10000000000"
aio read size = 1
aio write size = 1
# For connecting multiple clients on Windows
netbios aliases = jellyfin readonly

Multichannel is working -

Code:

ServerName           : 192.168.88.81
Selected             : True
Failed               : False
FailureCount         : 0
ClientInterfaceIndex : 45
ClientRSSCapable     : True
ClientRdmaCapable    : False
ClientLinkSpeed      : 10 Gbps
ClientIpAddress      : 192.168.88.109
ServerInterfaceIndex : 7
ServerRSSCapable     : True
ServerRdmaCapable    : False
ServerLinkSpeed      : 10 Gbps
ServerIpAddress      : 192.168.88.81
MaxChannels          : 4
CurrentChannels      : 4

 

[Patrick M. Hausen]

Samba uses a single thread per client. So that's expected. If you hammer at the server with 24 concurrent threads from your client ...

 

[tarian]

Samba uses a single thread per client. So that's expected. If you hammer at the server with 24 concurrent threads from your client ...

I restricted Kopia to 1 thread and I am still seeing high CPU. From what I can tell Kopia is just checking the file modified time and traversing the directory looking for changes (it is an up to date snapshot so no files are being transferred). So why is reading the file metadata and traversing the directories causing high CPU? Is there a way to see what samba is spending all that CPU time on?

 

[anodos]

I restricted Kopia to 1 thread and I am still seeing high CPU. From what I can tell Kopia is just checking the file modified time and traversing the directory looking for changes (it is an up to date snapshot so no files are being transferred). So why is reading the file metadata and traversing the directories causing high CPU? Is there a way to see what samba is spending all that CPU time on?

It's probably spending CPU reading listing the dirs and reading metadata. This is of course single-threaded. Doing a recursive ls -l over millions of files would also probably use a lot of CPU.

Another way of looking at it, for every file your application is "checking", the smbd has to at a minimum open the file, read the ACL, read the DOSATTRIB xattr, and close the file to get enough information for the SMB protocol response. This is more expensive than a simple getdirentries() (e.g. more akin to ls -l as opposed to plain ls).

 

[tarian]

It's probably spending CPU reading listing the dirs and reading metadata. This is of course single-threaded. Doing a recursive ls -l over millions of files would also probably use a lot of CPU.

Ya, running ls -Rl on that directory does use around 2-10% CPU (spread over the 32 threads). I just wanted to make sure I didn't have anything misconfigured. The netbios aliases let's me split my NAS usage over multiple clients so when the backup is running everything doesn't grind to a halt.