Greetings,
My servers' hardware configuration
Type: SuperMicro SYS-6029U-E1CR25M
Processor: Intel(R) Xeon(R) Gold 6226R CPU @ 2.90GHz
Memory: 12 x 32GB Samsung M393A4K40DB2-CVF
Storage:
2 x 256GB NVMe SSD, INTEL SSDPEKKA256GB
12 x 16TB HDD, Seagate ST16000NM001G-2KK103
TrueNAS-12.0-U3.1
Server 1
- Boot: 1 x 2-way mirror (SSD)
- Pool: 2 x 6-wide RAIDZ2 data VDEVs
Server 2
- Boot: 1 x 2-way mirror (SSD)
- Pool: 1 x 12-wide RAIDZ3 data VDEV
Each server has a 2 x 25G lagg0 interface for data traffic.
Spec of the HDDs
Spindle Speed: 7200RPM
interface Access Speed (Gb/s): 6.0, 3.0
Max. Sustained Transfer Rate OD (MB/s, MiB/s): 261, 249
Random Read/Write 4K QD 16 WCD (IOPS): 170/440
Interface Ports: Single
As per reference https://www.ixsystems.com/blog/zfs-pool-performance-2/, the expected performance of server 1 vs server 2
write IOPS: 880 vs 440
streaming write: 2088 MB/s, 1992 MiB/s vs 2349 MB/s, 2241 MiB/s
I created NFS shares on the two servers, mounted on Linux clients (bare metal servers and VMs) and run fio tests from these clients.
fio --name=test --filename=/mnt/bk1/test40G --size=40g --direct=1 --rw=<rw> --ioengine=libaio --fallocate=none --group_reporting --numjobs=<n> --bs=4k --iodepth=16 --ramp_time=10 --runtime=50
rw=write, randwrite; n=1, 10
Best write IOPS of server 1 vs server 2: 350 vs 320
fio --name=test --filename=/mnt/bk1/test40G --size=40g --direct=1 --rw=<rw> --ioengine=libaio --fallocate=none --group_reporting --numjobs=<n> --bs=4M --iodepth=64 --ramp_time=10 --runtime=50
rw=write, randwrite; n=1, 10
Best write throughput of server 1 vs server 2: 156MiB/s vs 128MiB/s
I'd much appreciate if you could help me to understand the test results
write IOPS, expected 880 vs 440, test results 350 vs 320
write throughput, expected 1992 MiB/s vs 2241 MiB/s, test results 156MiB/s vs 128MiB/s
Thanks very much!
My servers' hardware configuration
Type: SuperMicro SYS-6029U-E1CR25M
Processor: Intel(R) Xeon(R) Gold 6226R CPU @ 2.90GHz
Memory: 12 x 32GB Samsung M393A4K40DB2-CVF
Storage:
2 x 256GB NVMe SSD, INTEL SSDPEKKA256GB
12 x 16TB HDD, Seagate ST16000NM001G-2KK103
TrueNAS-12.0-U3.1
Server 1
- Boot: 1 x 2-way mirror (SSD)
- Pool: 2 x 6-wide RAIDZ2 data VDEVs
Server 2
- Boot: 1 x 2-way mirror (SSD)
- Pool: 1 x 12-wide RAIDZ3 data VDEV
Each server has a 2 x 25G lagg0 interface for data traffic.
Spec of the HDDs
Spindle Speed: 7200RPM
interface Access Speed (Gb/s): 6.0, 3.0
Max. Sustained Transfer Rate OD (MB/s, MiB/s): 261, 249
Random Read/Write 4K QD 16 WCD (IOPS): 170/440
Interface Ports: Single
As per reference https://www.ixsystems.com/blog/zfs-pool-performance-2/, the expected performance of server 1 vs server 2
write IOPS: 880 vs 440
streaming write: 2088 MB/s, 1992 MiB/s vs 2349 MB/s, 2241 MiB/s
I created NFS shares on the two servers, mounted on Linux clients (bare metal servers and VMs) and run fio tests from these clients.
fio --name=test --filename=/mnt/bk1/test40G --size=40g --direct=1 --rw=<rw> --ioengine=libaio --fallocate=none --group_reporting --numjobs=<n> --bs=4k --iodepth=16 --ramp_time=10 --runtime=50
rw=write, randwrite; n=1, 10
Best write IOPS of server 1 vs server 2: 350 vs 320
fio --name=test --filename=/mnt/bk1/test40G --size=40g --direct=1 --rw=<rw> --ioengine=libaio --fallocate=none --group_reporting --numjobs=<n> --bs=4M --iodepth=64 --ramp_time=10 --runtime=50
rw=write, randwrite; n=1, 10
Best write throughput of server 1 vs server 2: 156MiB/s vs 128MiB/s
I'd much appreciate if you could help me to understand the test results
write IOPS, expected 880 vs 440, test results 350 vs 320
write throughput, expected 1992 MiB/s vs 2241 MiB/s, test results 156MiB/s vs 128MiB/s
Thanks very much!