Advice on Increasing System-Specific Performance

[Chris Tobey]

Hi everyone,

I am looking to increase the performance of my system which has some decent specs, but is starting to not be able to keep up.

Setup:

• 12 x 20 TB Seagate Exos X20 SAS drives in a 6 x 2-drive mirror vdev configuration.
• 2 x 1.6 TB Intel P5800X
• LSI 9300-8i with SuperMicro SAS backplane
• Dual Intel Xeon E5-2620 v3 @ 2.40 GHz
• 24 x 16 GB DDR4 ECC RAM
• 1 x 10 GE

ZFS Configuration:

• 6 vdevs in a 2-drive mirror (Seagate 20 TB HDDs)
• 2 caches (Intel P5800X)
• Current usage at 75%.
• Compression: LZ4
• Encryption: Off
• Dedup: Off
• Sync: Disabled
• Enable Atime: Off

Workload:

• About 100 servers accessing pool over NFS.
• Each server has 1-20 git repos being cloned and compiled on the mounted NFS share at any given time.
• Each git repo has thousands of very small source files (< 10 kB) and generates hundreds of very small (< 10 kB) to small (< 10 mB) files.
• Disk usage largely varies due to the number of builds running at a given time, but storage is only growing a slow pace on average (maybe 100 GB per week?)

I recently changed the drive configuration from 2 vdevs in a 6-drive RAID-Z2 to the current 6 vdevs in a 2-drive mirror and was hoping that would increase the performance, but do not see much difference.
I do not believe I am network limited as the NIC is not reporting even 1 Gbit, let alone 10 Gbit.
I had hoped the caches would help with the performance, but I am not sure how to measure their effect.