Greetings,
My servers' hardware configuration
Type: SuperMicro SYS-6029U-E1CR25M
Processor: Intel(R) Xeon(R) Gold 6226R CPU @ 2.90GHz
Memory: 12 x 32GB Samsung M393A4K40DB2-CVF
Storage:
2 x 256GB NVMe SSD, INTEL SSDPEKKA256GB
12 x 16TB HDD, Seagate ST16000NM001G-2KK103
TrueNAS-12.0-U3.1
Server 1
- Boot: 1 x 2-way mirror (SSD)
- Pool: 2 x 6-wide RAIDZ2 data VDEVs
Server 2
- Boot: 1 x 2-way mirror (SSD)
- Pool: 1 x 12-wide RAIDZ3 data VDEV
Each server has a 2 x 25G lagg0 interface for data traffic.
Spec of the HDDs
Spindle Speed: 7200RPM
interface Access Speed (Gb/s): 6.0, 3.0
Max. Sustained Transfer Rate OD (MB/s, MiB/s): 261, 249
Random Read/Write 4K QD 16 WCD (IOPS): 170/440
Interface Ports: Single
As per reference https://www.ixsystems.com/blog/zfs-pool-performance-2/, the expected performance of server 1 vs server 2
read IOPS: 340 vs 170
streaming read: 2088 MB/s, 1992 MiB/s vs 2349 MB/s, 2241 MiB/s
I created NFS shares on the two servers, mounted on Linux clients (bare metal servers and VMs) and run fio tests from these clients.
fio --name=test --filename=/mnt/bk1/test40G --size=40g --direct=1 --rw=<rw> --ioengine=libaio --fallocate=none --group_reporting --numjobs=<n> --bs=4k --iodepth=16 --ramp_time=10 --runtime=50
rw=read, randread; n=1, 10
fio --name=test --filename=/mnt/bk1/test40G --size=40g --direct=1 --rw=<rw> --ioengine=libaio --fallocate=none --group_reporting --numjobs=<n> --bs=4M --iodepth=64 --ramp_time=10 --runtime=50
rw=read, randread; n=1, 10
When compression is LZ4
sequential read IOPS for server 1 varies from 2.7k to 56.5k
sequential read IOPS for server 2 varies from 2.0k to 37.7k
random read IOPS for server 1 varies between 353 and 659 from bare metal servers, between 329 and 54.1k from VMs
random read IOPS for server 2 varies between 206 and 2.4k from bare metal servers, between 2.1k and 50.2k from VMs
For read throughput, best performance gets from tests with block size of 4M, while sequential read and random read have similar performance, the bests are 420 MiB/s for server 1 and 415 MiB/s for server 2.
I'd much appreciate if you could help me to understand the test results
read IOPS, expected 340 vs 170, test results vary from hundreds to >10k on both servers
- why read IOPS test result numbers are much larger than expected?
read throughput, expected 1992 MiB/s vs 2241 MiB/s, test results 420 MiB/s vs 415 MiB/s
- any chances to improve read throughput?
Thanks very much!
My servers' hardware configuration
Type: SuperMicro SYS-6029U-E1CR25M
Processor: Intel(R) Xeon(R) Gold 6226R CPU @ 2.90GHz
Memory: 12 x 32GB Samsung M393A4K40DB2-CVF
Storage:
2 x 256GB NVMe SSD, INTEL SSDPEKKA256GB
12 x 16TB HDD, Seagate ST16000NM001G-2KK103
TrueNAS-12.0-U3.1
Server 1
- Boot: 1 x 2-way mirror (SSD)
- Pool: 2 x 6-wide RAIDZ2 data VDEVs
Server 2
- Boot: 1 x 2-way mirror (SSD)
- Pool: 1 x 12-wide RAIDZ3 data VDEV
Each server has a 2 x 25G lagg0 interface for data traffic.
Spec of the HDDs
Spindle Speed: 7200RPM
interface Access Speed (Gb/s): 6.0, 3.0
Max. Sustained Transfer Rate OD (MB/s, MiB/s): 261, 249
Random Read/Write 4K QD 16 WCD (IOPS): 170/440
Interface Ports: Single
As per reference https://www.ixsystems.com/blog/zfs-pool-performance-2/, the expected performance of server 1 vs server 2
read IOPS: 340 vs 170
streaming read: 2088 MB/s, 1992 MiB/s vs 2349 MB/s, 2241 MiB/s
I created NFS shares on the two servers, mounted on Linux clients (bare metal servers and VMs) and run fio tests from these clients.
fio --name=test --filename=/mnt/bk1/test40G --size=40g --direct=1 --rw=<rw> --ioengine=libaio --fallocate=none --group_reporting --numjobs=<n> --bs=4k --iodepth=16 --ramp_time=10 --runtime=50
rw=read, randread; n=1, 10
fio --name=test --filename=/mnt/bk1/test40G --size=40g --direct=1 --rw=<rw> --ioengine=libaio --fallocate=none --group_reporting --numjobs=<n> --bs=4M --iodepth=64 --ramp_time=10 --runtime=50
rw=read, randread; n=1, 10
When compression is LZ4
sequential read IOPS for server 1 varies from 2.7k to 56.5k
sequential read IOPS for server 2 varies from 2.0k to 37.7k
random read IOPS for server 1 varies between 353 and 659 from bare metal servers, between 329 and 54.1k from VMs
random read IOPS for server 2 varies between 206 and 2.4k from bare metal servers, between 2.1k and 50.2k from VMs
For read throughput, best performance gets from tests with block size of 4M, while sequential read and random read have similar performance, the bests are 420 MiB/s for server 1 and 415 MiB/s for server 2.
I'd much appreciate if you could help me to understand the test results
read IOPS, expected 340 vs 170, test results vary from hundreds to >10k on both servers
- why read IOPS test result numbers are much larger than expected?
read throughput, expected 1992 MiB/s vs 2241 MiB/s, test results 420 MiB/s vs 415 MiB/s
- any chances to improve read throughput?
Thanks very much!
