TrueNAS-13.0-U2 SMB/NFS 200MB/s on a 10GB network with 600MB/s disk performance

[hugofromboss]

Hello,

I am new to TrueNAS. However, I've read and search through the forums for support on this and I'll cover everything I've tried including my current configuration.

Problem:
I can achieve local disk performance of 600-700MB/s when running dd or fio. However, over the network SMB on Windows achieves about 200MB/s and SMB on Linux achieves about 150MB/s. When performing a SMB copy on Windows I get the expected 700MB/s, going as high as 1.2GB/s. I get similarly poor speeds on Linux for NFS and ISCSI. I am expecting to have close or similar performance of 600-700MB/s over the network. The test is a single large 10GB file.

Hardware:
I'm running TrueNAS-13.0-U2 on a DELL r630 bare metal (64GB RAM E5-2640 v3). With a 40G Mellanox ConnectX-3 Pro. Dell SAS LSI 6g card connected to a DELL SC280. BIOS/IDRAC/HDDs/NIC/SAS controller everything has been updated and is at the latest revision. The Dell life cycle controller also confirms no underlying hardware issues.

ZFS Pool:
I have 3 vdevs each with 7 4TB disks in zRaidz2. I also have 3 (Samsung 970 250GB) cache SSD drives.
This is a newly created pool with < 5% usage. As I'm testing TrueNAS.
Zpool is xattr=sa,compression=off,encryption=off,dedup=off,atime=off,checksum=on

What I've already tried:
Iperf3 reports a healthy sec 1.04 GBytes 8.93 Gbits/sec. (The Windows / Linux clients have a 10G nic)
MTU is 9000 on the server, client and network.
NFS server and client are connected via a single 40G switch.
Auto tune is enabled though had little effect.
SMB has the following Auxiliary parameters:

server multi channel support = yes
interfaces = "192.168.1.3;capability=RSS,speed=10000000000"

Disks are new and report 0 issues. SMART reports no issues.
netstat -s shows healthy network performance (no frames lost etc)
Linux mount parameters are:

sudo mount -o username=hugo,domain=corp,uid=1000,gid=1000,vers=3.0 -t cifs '\\192.168.1.3\test' /mnt/test

Windows Get-SMBMultiChannelConnection:

Server Name Selected Client IP Server IP Client Interface Index Server Interface Index Client RSS Capable Client
RDMA
Capable
----------- -------- --------- --------- ---------------------- ---------------------- ------------------ -------
192.168.1.3 True 192.168.1.112 192.168.1.3 22 8 True False

Test results:
Iperf3:

[SUM] 0.00-8.42 sec 9.43 GBytes 9.62 Gbits/sec sender
[SUM] 0.00-8.42 sec 0.00 Bytes 0.00 bits/sec receiver

FIO:
Command:

fio --name=randrw \
--bs=128k \
--direct=1 \
--directory=/mnt/pool/test \
--ioengine=posixaio \
--iodepth=32 \
--group_reporting \
--numjobs=12 \
--ramp_time=10 \
--runtime=60 \
--rw=randrw \
--size=256MGB \
--time_based

Results:

Jobs: 12 (f=12): [m(12)][100.0%][r=1242MiB/s,w=1238MiB/s][r=9932,w=9902 IOPS][eta 00m:00s]
randrw: (groupid=0, jobs=12): err= 0: pid=5236: Mon Sep 5 10:05:43 2022
read: IOPS=12.0k, BW=1506MiB/s (1579MB/s)(88.2GiB/60015msec)
slat (nsec): min=1170, max=1679.7k, avg=2150.70, stdev=5738.54
clat (usec): min=64, max=86811, avg=10654.82, stdev=3758.73
lat (usec): min=126, max=86814, avg=10656.97, stdev=3758.69
clat percentiles (usec):
| 1.00th=[ 5604], 5.00th=[ 6325], 10.00th=[ 6718], 20.00th=[ 7373],
| 30.00th=[ 8029], 40.00th=[ 8848], 50.00th=[10028], 60.00th=[11338],
| 70.00th=[12911], 80.00th=[13960], 90.00th=[15008], 95.00th=[16188],
| 99.00th=[20841], 99.50th=[22938], 99.90th=[39060], 99.95th=[48497],
| 99.99th=[73925]
bw ( MiB/s): min= 911, max= 2477, per=99.92%, avg=1504.53, stdev=34.05, samples=1428
iops : min= 7287, max=19814, avg=12032.63, stdev=272.40, samples=1428
write: IOPS=12.0k, BW=1506MiB/s (1579MB/s)(88.3GiB/60015msec); 0 zone resets
slat (usec): min=3, max=8439, avg= 7.53, stdev=16.22
clat (usec): min=376, max=99270, avg=21172.64, stdev=6694.30
lat (usec): min=527, max=99278, avg=21180.17, stdev=6694.27
clat percentiles (usec):
| 1.00th=[11994], 5.00th=[13042], 10.00th=[13698], 20.00th=[15008],
| 30.00th=[16057], 40.00th=[17433], 50.00th=[19792], 60.00th=[22938],
| 70.00th=[25822], 80.00th=[27919], 90.00th=[29754], 95.00th=[31065],
| 99.00th=[35914], 99.50th=[39584], 99.90th=[63177], 99.95th=[69731],
| 99.99th=[86508]
bw ( MiB/s): min= 968, max= 2447, per=99.91%, avg=1504.58, stdev=33.71, samples=1428
iops : min= 7742, max=19574, avg=12032.96, stdev=269.66, samples=1428
lat (usec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
lat (msec) : 2=0.01%, 4=0.08%, 10=25.17%, 20=49.70%, 50=24.89%
lat (msec) : 100=0.15%
cpu : usr=2.39%, sys=3.58%, ctx=1022869, majf=0, minf=1
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.2%, 16=71.0%, 32=28.8%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=94.1%, 8=4.1%, 16=1.7%, 32=0.1%, 64=0.0%, >=64=0.0%
issued rwts: total=722806,722753,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
READ: bw=1506MiB/s (1579MB/s), 1506MiB/s-1506MiB/s (1579MB/s-1579MB/s), io=88.2GiB (94.8GB), run=60015-60015msec
WRITE: bw=1506MiB/s (1579MB/s), 1506MiB/s-1506MiB/s (1579MB/s-1579MB/s), io=88.3GiB (94.8GB), run=60015-60015msec

DD (cd /mnt/pool/test && dd if=/dev/zero of=file bs=1048576):

96702+0 records in
96701+0 records out
101398347776 bytes transferred in 154.331941 secs (657014662 bytes/sec)

Which is 657MB/s for about 100GB in 154 seconds.

Windows file transfer:

To show the copy isn't limited by the reads on the client machine:

Windows Local File copy:

Thank you in advance for any help you're able to provide.

Thanks

H

 

[Alex_K]

Hi
To begin with,

When performing a SMB copy on Windows I get the expected 700MB

What is this backed by? I mean, is Windows a separate metal machine and what's it copying from and to?

On Iperf3, was it 1 threaded test or multi? Can you post iperf3 parameters?
What's disk busy in TrueNAS when you do 200 MBps copy, whats single core CPU load?
Is 200 a limit for read, write, both?

 

[morganL]

I think you are getting good bandwidths with fio and dd..... so TrueNAS system is performing. Can you confirm.

You are getting poor performance with windows file copy..... probably because windows file copy is not very parallel in its process and hence very latency sensitive.

 

[hugofromboss]

Hi
To begin with,

What is this backed by? I mean, is Windows a separate metal machine and what's it copying from and to?

On Iperf3, was it 1 threaded test or multi? Can you post iperf3 parameters?
What's disk busy in TrueNAS when you do 200 MBps copy, whats single core CPU load?
Is 200 a limit for read, write, both?

The Windows copy I was referring is copying and pasting a file within the same SMB share.

Ipef3 params:

iperf3.exe -c 192.168.1.3 -P 2

Without the -P2 parameters. So one connection it does 6gbit/s

There is no CPU load on the TrueNAS box. Nor the client machine. Before running the tests. The setup is as ideal as it can be as I'm trying to test out TrueNAS.

The 200MB/s is for read and write.

One thing I did observe yesterday is that writes were being sent to another interface on the trueNAS box. Despite me configuring it to the 40G port. I'm looking into why this is the case at the moment.

Best,

H

 

[hugofromboss]

I think you are getting good bandwidths with fio and dd..... so TrueNAS system is performing. Can you confirm.

You are getting poor performance with windows file copy..... probably because windows file copy is not very parallel in its process and hence very latency sensitive.

Yes, this is correct. I also get slow (12MB/s - 200MB/s) performance on NFS too for Linux.

 

[morganL]

Yes, this is correct. I also get slow (12MB/s - 200MB/s) performance on NFS too for Linux.

The numbers are meaningless unless you describe how the testing was done...

Performance is as much about the client and its software.. as it is the NAS and its hardware and configuration.

 

[bmihajlovic]

I'm planing to upgrade my setup to new 13 U3, very similar setup, 40gb nic, 12 drives 4Vdevs sas3 8TB drives, lsi9300 flashed with ixsystem firmware, sas3 supermicro backplane, metadata 2x samsung 980pro 2tb and 512GB ARC

currently running 12 u8.1 without cashing. did you get some good results after this post?

Any updates?

 

[hugofromboss]

Hi, yes.I got it working. Apollogies for not following up. Despite defining the interface the SMB and NFS traffic was partially being sent to a 1G interface for some reason (didn't get to the bottom of it). There was another issue with the network configuration that componded the speed issue. I just removed the 1G management NIC in the end - so everything runs over the 40G. Works fine now.