Pushing SMB large multi-gigabyte files TO the NAS causes a large transfer to slow and then fail.

SpiritWolf

Dabbler
Joined
Oct 21, 2022
Messages
29
Well, SMB is single-threaded.
I'm learning!

I've been doing tests:
Code:
iperf3 -c 192.168.1.250 -R
Connecting to host 192.168.1.250, port 5201
Reverse mode, remote host 192.168.1.250 is sending
[  5] local 192.168.1.107 port 61944 connected to 192.168.1.250 port 5201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec   229 MBytes  1.92 Gbits/sec               
[  5]   1.00-2.00   sec   230 MBytes  1.93 Gbits/sec                  
[  5]   2.00-3.00   sec   230 MBytes  1.93 Gbits/sec                  
[  5]   3.00-4.00   sec   232 MBytes  1.94 Gbits/sec                  
[  5]   4.00-5.00   sec   230 MBytes  1.93 Gbits/sec                  
[  5]   5.00-6.00   sec   232 MBytes  1.95 Gbits/sec                  
[  5]   6.00-7.00   sec   232 MBytes  1.95 Gbits/sec
[  5]   7.00-8.00   sec   233 MBytes  1.96 Gbits/sec                  
[  5]   8.00-9.00   sec   175 MBytes  1.47 Gbits/sec                  
[  5]   9.00-10.00  sec   230 MBytes  1.93 Gbits/sec                  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  2.20 GBytes  1.89 Gbits/sec  5015             sender
[  5]   0.00-10.00  sec  2.20 GBytes  1.89 Gbits/sec                  receiver


iperf3 -c 192.168.1.250   
Connecting to host 192.168.1.250, port 5201
[  5] local 192.168.1.107 port 62037 connected to 192.168.1.250 port 5201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec   252 MBytes  2.12 Gbits/sec                  
[  5]   1.00-2.00   sec   256 MBytes  2.15 Gbits/sec            
[  5]   2.00-3.00   sec   255 MBytes  2.14 Gbits/sec                  
[  5]   3.00-4.01   sec   242 MBytes  2.02 Gbits/sec                  
[  5]   4.01-5.01   sec  0.00 Bytes  0.00 bits/sec                  
[  5]   5.01-6.00   sec  0.00 Bytes  0.00 bits/sec                  
[  5]   6.00-7.00   sec   146 MBytes  1.23 Gbits/sec                  
[  5]   7.00-8.00   sec   257 MBytes  2.15 Gbits/sec                  
[  5]   8.00-9.00   sec   254 MBytes  2.13 Gbits/sec                  
[  5]   9.00-10.00  sec   253 MBytes  2.12 Gbits/sec                  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.00  sec  1.87 GBytes  1.61 Gbits/sec                  send
[  5]   0.00-10.00  sec  1.87 GBytes  1.60 Gbits/sec                  receiver


...and there are odd asymmetries to transmit/receive via TCP/IP.

While I was perusing some test code, I saw a few lines that seemed interesting:
Code:
root@truenas[~]# dd if=/dev/zero of=/mnt/WolfTruePool/test/test.dat bs=2048k count=10000


10000+0 records in
10000+0 records out
20971520000 bytes transferred in 3.821498 secs (5487774807 bytes/sec)


root@truenas[~]# dd of=/dev/zero if=/mnt/WolfTruePool/test/test.dat bs=2048k count=10000
10000+0 records in
10000+0 records out
20971520000 bytes transferred in 0.753427 secs (27834838083 bytes/sec)

root@truenas[~]# dd if=/dev/zero of=/mnt/WolfTruePool/test/test.dat bs=2048k count=10000
10000+0 records in
10000+0 records out


With all the tests I'm coding, I think I need to record all this so I don't forget.
 

Davvo

MVP
Joined
Jul 12, 2022
Messages
3,147

anodos

Sambassador
iXsystems
Joined
Mar 6, 2014
Messages
9,545
Well, SMB is single-threaded.

See libpthreadpool

So... not exactly. Though in case of SCALE we're using io_uring for reads / writes (rather than the pthreadpool). Core / Enterprise uses FreeBSD's kernel AIO (kernel threads) for reads / writes.

The process model for Samba is that there's a main smbd process that forks a child process per-client. So it's also not like there's this monolithic single-threaded application sitting there.
 

Davvo

MVP
Joined
Jul 12, 2022
Messages
3,147

See libpthreadpool

So... not exactly. Though in case of SCALE we're using io_uring for reads / writes (rather than the pthreadpool). Core / Enterprise uses FreeBSD's kernel AIO (kernel threads) for reads / writes.

The process model for Samba is that there's a main smbd process that forks a child process per-client. So it's also not like there's this monolithic single-threaded application sitting there.
Honestly the links you posted are beyond my understanding. From what I know if I copy a (single) large file to my SMB share I don't see multi-core usage, and a brief search on google told me the reason is that SMB is single-threaded.

Getting answers regarding SMB is pretty difficult in my experience since everything I get is Multichannel infos (which doesn't look like using multiple cores for the same... process I guess?) or how SMB is single-threaded; guess I googled the wrong words.

Sorry if I shared wrong informations.
 

Whattteva

Wizard
Joined
Mar 5, 2013
Messages
1,824
The process model for Samba is that there's a main smbd process that forks a child process per-client. So it's also not like there's this monolithic single-threaded application sitting there.
That does still mean that it's limited to one process per client, I'm assuming? Though technically, a process can spawn its own threads, but is that being done here?

I'm not familiar with pthreadpool, but sounds somewhat similar in concept to Apple's Grand Central Dispatch (which I've used quite extensively) in that you don't manage the threads yourself, but rather the system manages it internally and you're just sending off async tasks to the system and let it handle the threading?
 
Top