Poor SMB write performance

[Zeff]

Config:
AMD Ryzen 3
16GB RAM
10G Network
RAIDZ2

I am using 13.0-U6.1 and am having extremely poor SMB write performance.

Pool is 63% used.
zpool status pool: Aphrodite state: ONLINE scan: scrub repaired 0B in 19:42:46 with 0 errors on Sun Mar 10 20:42:51 2024 config: NAME STATE READ WRITE CKSUM Aphrodite ONLINE 0 0 0 raidz2-0 ONLINE 0 0 0 gptid/e5fb0df5-72c0-11ea-9935-a85e455542ae.eli ONLINE 0 0 0 gptid/dac2c3ac-6369-11ea-8a1a-a85e455542ae.eli ONLINE 0 0 0 gptid/dc3b8a4b-6369-11ea-8a1a-a85e455542ae.eli ONLINE 0 0 0 gptid/17469227-59f4-11ed-a9c9-6805cac843fa.eli ONLINE 0 0 0 gptid/dc8dfd23-6369-11ea-8a1a-a85e455542ae.eli ONLINE 0 0 0 errors: No known data errors


10G network

iperf results
------------------------------------------------------------ Client connecting to 10.0.1.88, TCP port 5001 TCP window size: 128 KByte (default) ------------------------------------------------------------ [ 1] local 10.0.1.8 port 50884 connected with 10.0.1.88 port 5001 (icwnd/mss/irtt=69/8948/1000) [ ID] Interval Transfer Bandwidth [ 1] 0.00-10.01 sec 11.3 GBytes 9.71 Gbits/sec

ioZone test results

Command line used: iozone -i 0 -t 2 Output is in kBytes/sec Time Resolution = 0.000001 seconds. Processor cache size set to 1024 kBytes. Processor cache line size set to 32 bytes. File stride size set to 17 * record size. Throughput test with 2 processes Each process writes a 512 kByte file in 4 kByte records Children see throughput for 2 initial writers = 635.39 kB/sec Parent sees throughput for 2 initial writers = 634.80 kB/sec Min throughput per process = 316.45 kB/sec Max throughput per process = 318.94 kB/sec Avg throughput per process = 317.69 kB/sec Min xfer = 508.00 kB Children see throughput for 2 rewriters = 457.20 kB/sec Parent sees throughput for 2 rewriters = 456.95 kB/sec Min throughput per process = 228.60 kB/sec Max throughput per process = 228.60 kB/sec Avg throughput per process = 228.60 kB/sec Min xfer = 512.00 kB

 

[somethingweird]

Harddrive Model?

 

[Zeff]

Not surethat this would affect the outcome, but, 6 4TB drives:

2 HGST
4 Seagate Ironwolf

 

[somethingweird]

Not surethat this would affect the outcome, but, 6 4TB drives:

To determine if the HD are CMR or SMR. If the HD are SMR then that would be the reason. Otherwise we move on to other possibility.

 

[Zeff]

OK, but we are talking a 33x performance difference between write and read.

 

[asap2go]

Not surethat this would affect the outcome, but, 6 4TB drives:

2 HGST
4 Seagate Ironwolf

Some drives use a technology called SMR which works very poorly with zfs. You can check the datasheet.
The high reads could be due to ARC.
Did you use a specific program/OS/samba/smb version?
Also: How full is your pool?
Did you turn on sync writes or atime on the respective dataset?

 

[asap2go]

OK, but we are talking a 33x performance difference between write and read.

Reads are probably due to caching (ARC).
You can check if you go to settings -> shell and type "zpool iostat -v 2" while running your benchmark. If you see little to no read operations on the pool then the data is read from RAM which is orders of magnitude faster than your network.

 

[Zeff]

SMB 3
Pool 63% utilised as stated
Tried sync and no sync - made no difference

 

[Zeff]

zpool iostat -v 2:



This was reading a 5.5GB iso file:

 

[Stux]

Testing the benefits of SLOG using a RAM disk!

Say you were considering getting a stupid fast SLOG... because your VMs are too slow... You could create a SLOG on a RAM disk. This is an incredibly stupid idea, but it does demonstrate the maximum performance gains to be had from using the absolutely fastest SLOG you could possibly get...

www.truenas.com

 

[Zeff]

@Stux, I'm not sure what SLOG can do to close the gap. Can you elaborate, as I'm relatively new to TrueNAS.

I just ran a test on a locally connected (USB 3.1 gen 2) IronWolf of the same model, using Blackmagic.
This connection is 10Gbps, and my network is 10G.
Writes: 152 MB/s
My TrueNAS using a RAIDZ2 is 5 times slower.

 

[asap2go]

@Stux, I'm not sure what SLOG can do to close the gap. Can you elaborate, as I'm relatively new to TrueNAS.

I just ran a test on a locally connected (USB 3.1 gen 2) IronWolf of the same model, using Blackmagic.
This connection is 10Gbps, and my network is 10G.
Writes: 152 MB/s
My TrueNAS using a RAIDZ2 is 5 times slower.

Can you give use the serial numbers or precise model numbers of the hard drives?

 

[Stux]

Black magic should be doing a non random writes.

The results only make sense to me if you messed up disabling sync on the dataset when you tested it.

I’d suggest testing with a fake slog to see if your results improve.

And if they do, then you have confirmed the cause and know how to fix.

If not. Well, we’re still where we were.

 

[Zeff]

@Stux unfortunately I do not know how to test with a fake SLOG—can you point me in the right direction please?

@asap2go, I will take the system down later and get precise model numbers.

 

[NickF]

Can you re-run zpool iostat with some additional flags while running a backup?

Zpool iostat -vvyl 60 1

 

[Zeff]

@NickF, here you go: