Multi-gigabit SMB throughput trouble-shooting approach (all-SSD pool)

ByteMan

Dabbler
Joined
Nov 10, 2021
Messages
32
Are there recommendations or specific resources that lay out a general step-by-step approach for troubleshooting poor Multi-gigabit (SMB) throughput performance with SSD media?

I found some scattered information but I am not sure where to continue in my case.

My specific issue:
Slow SMB file transfer speed (using 70GB test file)

Hardware:
TrueNAS Mini X+ with several 500MB/s+ (Read+Write) SSD drives reporting 512byte sector size. 10GbE NIC.

Test-Network:
TrueNAS Mini X+ <----10GbE Link----> 10GbE Switch <-----5GbE Link ----> 5GbE Client

Connection throughput test using iperf -d:
------------------------------------------------------------
Client connecting to 10.10.10.111, TCP port 5001
TCP window size: 1.24 MByte (default)
------------------------------------------------------------
[ 3] local 10.10.10.11 port 39574 connected with 10.10.10.111 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-60.0 sec 23.8 GBytes 3.41 Gbits/sec
[ 3] MSS size 1448 bytes (MTU 1500 bytes, ethernet)
*This speed is in line with what others reported for the 5GbE adapter I am using (Sabrent NT-SS5G)

I would be happy with this speed (+400MB/s) but bi-directional SMB file transfers run only at 180-230 MB/sec max:
- Single file transfers of 70GB test file run at around 180-190MB/sec.
- The highest speed of 230MB/sec can be reached with two parallel SMB file transfers.

Notes:
- CPU: 8-core Mini X+ CPU is far from performance limit (short +60% peaks with iperf)
- I started out testing speeds with an all-SSD zpool consisting of several drives. To rule out some raidz level limitations I also ran tests using single drive pools.

Pool level changes I attempted so far:
Compression (Lz4): ON+OFF
Recordsize: 128KiB + 1MiB
atime ON+OFF
(I have not touched ashift - currently at default)

Changing the settings above affected the speeds slightly but I am far of target here, correct? What speed should I be expecting? What could I try next?
 

Samuel Tai

Never underestimate your own stupidity
Moderator
Joined
Apr 24, 2020
Messages
5,398

ByteMan

Dabbler
Joined
Nov 10, 2021
Messages
32
I updated the config and restarted the SMB service. Speeds are the same. Did I miss a step?
server multi channel support = yes
interfaces = "10.10.10.111;capability=RSS,speed=10000000000"
 

Samuel Tai

Never underestimate your own stupidity
Moderator
Joined
Apr 24, 2020
Messages
5,398
I'm pretty certain the 10G NIC on the Mini X+ supports RSS, but I don't know if your 5G Sabrent NIC on your client also supports RSS.
 

ByteMan

Dabbler
Joined
Nov 10, 2021
Messages
32
Let me confirm, RSS in this case refers to "Receive Side Scaling"? If so, the purpose seems to be distributing network traffic related CPU load across multiple cores?

This is what I observe in regards to CPU load:

SMB transfer (bottle-necking issue): On the Mini X+ side I can see that the CPU load is distributed across cores. On the client side the load is minimal (plenty of CPU horsepower)
iperf (no bottle-necking): Interestingly, iperf does not distribute the load across CPU cores and a single core will see loads up to 95%.

In summary:
RSS seems to be working on the TrueNAS Mini X+ side, as the CPU load is distributed
Unclear if RSS is supported client-side, but the CPU load is far from max, so no issue?
 

mvipe01

Dabbler
Joined
Feb 1, 2022
Messages
18
I updated the config and restarted the SMB service. Speeds are the same. Did I miss a step?
How did you get this to work? I cannot add the interface line I get the error:
interfaces is a blacklisted auxiliary parameter. Changes to this parameter are not permitted.
 

ByteMan

Dabbler
Joined
Nov 10, 2021
Messages
32
How did you get this to work? I cannot add the interface line I get the error:
interfaces is a blacklisted auxiliary parameter. Changes to this parameter are not permitted.
All I did was add the two lines in the 'Auxillary Parameters' section under Services / SMB / Advanced Options:
server multi channel support = yes
interfaces = "10.10.10.111;capability=RSS,speed=10000000000"

I am still not clear what RSS (Receive side scaling to ensure CPU load is distributed across cores?) is supposed to do in regards to my bottle-necking issue, since I can achieve the desired speed using iperf. I noticed that network related CPU load was nicely spread across all 8 cores - this was the case WITH and WITHOUT the above references SMB aux parameter configuration related to RSS. This seems to confirm that RSS is auto-enabled(?)

I also ran diskinfo -tv /dev/adaX to confirm the Mini X+ can handle the SSDs at rated speed and they exceed 400MB/sec.
I also tried MTU size 9000 in addition to 1500 without success.
Are there other steps/commands/etc. that could be executed to identify the location of the bottleneck?
 

TrumanHW

Contributor
Joined
Apr 17, 2018
Messages
197
All I did was add the two lines in the 'Auxillary Parameters' section under Services / SMB / Advanced Options:


I am still not clear what RSS (Receive side scaling to ensure CPU load is distributed across cores?) is supposed to do in regards to my bottle-necking issue, since I can achieve the desired speed using iperf. I noticed that network related CPU load was nicely spread across all 8 cores - this was the case WITH and WITHOUT the above references SMB aux parameter configuration related to RSS. This seems to confirm that RSS is auto-enabled(?)

I also ran diskinfo -tv /dev/adaX to confirm the Mini X+ can handle the SSDs at rated speed and they exceed 400MB/sec.
I also tried MTU size 9000 in addition to 1500 without success.
Are there other steps/commands/etc. that could be executed to identify the location of the bottleneck?

I think you know more than I so I hope this isn't condescending ...
But if I'm not mistaken any custom modifications will be lost whenever you reboot. (please correct me if I'm wrong).
Thanks
 

mvipe01

Dabbler
Joined
Feb 1, 2022
Messages
18
I think you know more than I so I hope this isn't condescending ...
But if I'm not mistaken any custom modifications will be lost whenever you reboot. (please correct me if I'm wrong).
Thanks
As long as the modifications are done through the GUI they persist through reboot.
 
Top