We have 2 servers adjacent to one another which was handy for performance testing. I will describe them as A and B (because that's literally their short hostname)
A:
iperf3 looks good - 0 retries in both directions and an average of about 9.8 Gbits/sec each way.
Writing locally to the zpool cranks out an easy 400MB/s.
Problem:
NFS write performance is low for both systems given their internal configuration and connection to 10g networking. NFS writes are much slower on A than on B.
For testing I created a 10G file from /dev/random, attached an NFS mount to the pool of the other host, and used rsync to copy the test file across. This is what it looked like:
During the file copy I used gstats and the ZIL for A was barely even hitting 4% busy, mostly closer to 2%. When the testfile was being copied to B I was seeing the ZIL top out at 10%.
Questions:
A:
Supermicro X10DRi-T4+
Xeon E5-2620 v4
132G RAM
Intel 2-port X520 NIC
SLOG: dual Samsung SSD 970 PRO 512GB
SAS3008 HBA
46 4TB HGST ultrastars in a zpool of 2-disk mirrors (+2 spares)
SC846BE1C-R1K03JBOD chassis contains half the disks
B:Supermicro X11SPH-nCTF
Xeon Silver 4114
132G RAM
Chesio T520-SO NIC
SLOG: dual INTEL SSDPE21D280GA
SAS3008 HBA
30 6TB HGST ultrastars in a zpool of 2-disk mirrors (+2 spares)
Both servers are wired via DACs to QFX switches, trunked, storage VLAN set to mtu 9000. I can "ping -D -s 8973" in both directions so connectivity and MTU seem fine.iperf3 looks good - 0 retries in both directions and an average of about 9.8 Gbits/sec each way.
Writing locally to the zpool cranks out an easy 400MB/s.
Problem:
NFS write performance is low for both systems given their internal configuration and connection to 10g networking. NFS writes are much slower on A than on B.
For testing I created a 10G file from /dev/random, attached an NFS mount to the pool of the other host, and used rsync to copy the test file across. This is what it looked like:
Code:
a# rsync --info=progress2 testfile /var/tmp/test/ 10,485,760,000 100% 143.72MB/s 0:01:09 (xfr#1, to-chk=0/1) b# rsync --info=progress2 testfile /var/tmp/test/ 10,485,760,000 100% 35.86MB/s 0:04:38 (xfr#1, to-chk=0/1)
During the file copy I used gstats and the ZIL for A was barely even hitting 4% busy, mostly closer to 2%. When the testfile was being copied to B I was seeing the ZIL top out at 10%.
Questions:
- What kind of NFS write speed would you expect to see with the above hardware?
- What am I missing in the test setup above?
- Why is performance so low?
- What can I look for to explain the lower performance on the system with a higher spindle count?