Poor SMB performance with new server (read & write)

[Martin_Walter]

I'm evaluating TrueNAS for a new server (general SMB server, lots of big movie files, lots of smaller files).

Here are my server specs:
CPU: AMD Epyc 7313P
MB: Supermicro H12SSL-NT
RAM: 128GB DDR4 3200 regECC
HBA: LSI 9300-16i
Storage: 24x 18TB Seagate Exos X X18 (3x8 Raidz2)
4x Intel Optane P1600X 118 GB NVMe (2x Mirror Special Dev)
2x 1TB Crucial MX500 SATA (Cache)
2x Intel Optane M10 64GB NVMe (Boot)

Network speed is good (checked with iperf), but file copy is quite slow. It is a direct connection.
When I copy a test file (~4GB) from my Windows 10 machine to the server I get ~350MB/s at the start, but it slows down to 250MB/s after a few seconds, and then fluctuates between these speeds. Sometimes it even stops for a moment, which is not visible in the graph.
When I copy the same file back to my machine it starts fast at ~1GB/s, but after a short time it declines steadily and then settles at 180MB/s after a few seconds till the end.

I was expecting at least 500MB/s from this setup, and even more when the system is still empty.

 

[Arwen]

You don't list the TrueNAS version, (though we can guess family is CORE because this is the TrueNAS CORE sub-forum).

There is a bug in one version of OpenZFS for ARC. That might be affecting your read performance. But, it any case, version you are using always helpful.

If you could supply the output of zpool status that would help confirm your pool layout is as you expect it to be.

Next, you don't specify NIC connection speed, though we can guess it is 10Gbps, on both sides.

Last, do you have dataset compression enabled?

 

[Martin_Walter]

Sorry, always missing something.
I'm using TrueNAS-13.0-U4, the network speed is 10GBps (Broadcom on the server, Mellanox on the client)

Here is my zpool status:

pool: Pool
state: ONLINE
config:

NAME STATE READ WRITE CKSUM
Pool ONLINE 0 0 0
raidz2-0 ONLINE 0 0 0
gptid/2014690e-c629-11ed-8049-3cecef9adbbe ONLINE 0 0 0
gptid/20542b4c-c629-11ed-8049-3cecef9adbbe ONLINE 0 0 0
gptid/212ae836-c629-11ed-8049-3cecef9adbbe ONLINE 0 0 0
gptid/2117ebe1-c629-11ed-8049-3cecef9adbbe ONLINE 0 0 0
gptid/2100097d-c629-11ed-8049-3cecef9adbbe ONLINE 0 0 0
gptid/2209e9a3-c629-11ed-8049-3cecef9adbbe ONLINE 0 0 0
gptid/204b86c2-c629-11ed-8049-3cecef9adbbe ONLINE 0 0 0
gptid/200bef2e-c629-11ed-8049-3cecef9adbbe ONLINE 0 0 0
raidz2-1 ONLINE 0 0 0
gptid/20264545-c629-11ed-8049-3cecef9adbbe ONLINE 0 0 0
gptid/205e2d31-c629-11ed-8049-3cecef9adbbe ONLINE 0 0 0
gptid/202f8f9a-c629-11ed-8049-3cecef9adbbe ONLINE 0 0 0
gptid/2120fab6-c629-11ed-8049-3cecef9adbbe ONLINE 0 0 0
gptid/2042bd8a-c629-11ed-8049-3cecef9adbbe ONLINE 0 0 0
gptid/200219e3-c629-11ed-8049-3cecef9adbbe ONLINE 0 0 0
gptid/20399eda-c629-11ed-8049-3cecef9adbbe ONLINE 0 0 0
gptid/210da2b3-c629-11ed-8049-3cecef9adbbe ONLINE 0 0 0
raidz2-2 ONLINE 0 0 0
gptid/2487c837-c629-11ed-8049-3cecef9adbbe ONLINE 0 0 0
gptid/2474cad9-c629-11ed-8049-3cecef9adbbe ONLINE 0 0 0
gptid/249069ac-c629-11ed-8049-3cecef9adbbe ONLINE 0 0 0
gptid/24fad3a4-c629-11ed-8049-3cecef9adbbe ONLINE 0 0 0
gptid/2499b6e5-c629-11ed-8049-3cecef9adbbe ONLINE 0 0 0
gptid/247e8f90-c629-11ed-8049-3cecef9adbbe ONLINE 0 0 0
gptid/2455faf4-c629-11ed-8049-3cecef9adbbe ONLINE 0 0 0
gptid/24f1c529-c629-11ed-8049-3cecef9adbbe ONLINE 0 0 0
special
mirror-3 ONLINE 0 0 0
gptid/27d7f94a-c62b-11ed-8049-3cecef9adbbe ONLINE 0 0 0
gptid/27d6b3e7-c62b-11ed-8049-3cecef9adbbe ONLINE 0 0 0
mirror-4 ONLINE 0 0 0
gptid/311c0aa8-c62b-11ed-8049-3cecef9adbbe ONLINE 0 0 0
gptid/311cd78e-c62b-11ed-8049-3cecef9adbbe ONLINE 0 0 0
cache
gptid/230d4e51-c629-11ed-8049-3cecef9adbbe ONLINE 0 0 0
gptid/2301f795-c629-11ed-8049-3cecef9adbbe ONLINE 0 0 0

errors: No known data errors

 

[Arwen]

Version TrueNAS-13.0-U4 is the affected release, though it does not affect data reliability. Just ARC, (Adaptive Read Cache), usage.

In general, SMB is single threaded on the NAS side. Checking your Epyc CPU, it does seem reasonable as far as base & boost speeds, 3.0GHz & 3.7GHz respectively.

You still did not mention if you have compression enabled. Nor which type.

I don't have any further suggestions, but perhaps someone else will.

 

[Martin_Walter]

Compression is lz4.

Is there a workaround for the bug, or should I disable ARC altogether?

 

[jgreco]

Broadcom on the server

Please consider using a recommended ethernet adapter; see the 10 Gig Networking Primer. The Broadcoms are very hit-or-miss leaning heavily towards "miss".