Slow Disk Performance

[Phase]

Hi everyone, creating a new thread to focus on the disk part of the problem...

- All replications are slow: 0.7 to 1.4 gigabits/s
- Same server replication from SSD to SSD (A) and (D) is SLOWER than SSD to HDD (B) and (E)
- Cross server replication from SSD to SSD (C) shows same results as same server replication discounting the explanation that we are reading a writing to the same SSDs in (A) and (D)

Notes:

• Datasets have no compression
• All replication was done via TrueNAS replication feature and consisted of a initial snapshot 35GB in size
• Server 1 is a ASUS Z590-A PRIME with an i7-11700K and 32 GB
• Server 2 is a ASUS P8Z77-V Rev 1.xx with an i7-3770K and 32 GB (though of replacing it, but the old mobo is giving the same results as the Z590)
• Server 1 HDDs are 14 GB MG07ACA14TE (248 MiB/s) 9 units in Z3
• Server 2 HDDs are 16 GB MG08ACA16TE (262 MiB/s) 5 units in Z2
• The HDDs use LSI SAS9211-8i card(s)
• The SSDs use motherboard SATA connectors
• Servers are linked through a 40GbE network -- only relevant to test (C), confirming the poor performance seen in (A) and (D) in a across vdev setting

Thoughts?

 

[sretalla]

Same server replication from SSD to SSD (A) and (D) is SLOWER than SSD to HDD (B) and (E)

The SSDs use motherboard SATA connectors

Maybe you're overwhelming the onboard SATA controller... how about connecting them to the LSI?

 

[Phase]

Maybe you're overwhelming the onboard SATA controller... how about connecting them to the LSI?

Good call. Some progress!

There are obviously many variables, but do we have guidelines to define what a "good" throughput is?

The New Results

I change the SSDs on the target server to the LSI, leaving the source server SSDs attached to the mobo. We got some improvement, mostly on the cross server replication. The replication size is only 32GB and this server has 24GB assigned to cache, but it is still weird that the HDD performance still beats SSD. However the SSD pool finished the test 84% full, so that may have messed up the results on that run.

Item (C) Replication cross server is over twice as fast as before, but still at 403 MB/s, which is 1/10 of the network speed.

Item (D) Server 2 SSD to SSD perhaps 25% better. The first half is just under 200 MB/s but drops to 112 MB/s on the second half. That pool is getting full, and finished at ___% used.

Item (E) Server 2 SSD to HDD perhaps 30% better at 220 MB/s.