My apologies, still a bit of a n00b to this board but I will remember :) Specs below:
Supermicro X9DRH-7TF
2 x Xeon CPU E5-2620 (6 Core per proc)
128 GB RAM
5xLSI-2008 HBA's
FreeNAS-9.10.1 (d989edd)
Note: In interest of full disclosure, this *is* an ESXi host with PCI-Passthrough enabled to all cards. It is all server hardware so should eliminate most of the concerns.
zPool in questions is:
2 vDev, RAID-Z1, each vDev 5 x HGST/Hitachi Deskstar 7K3000 HDS723020ALA640 2TB SATAIII (enterprise class)
L2ARC: 2 x STEC ZeusIOPS SLC SAS SSD 100GB Model: Z16IZF2E-100UCU (not in a mirror of course)
L2 ARC Summary: (HEALTHY)
Passed Headroom: 843.44k
Tried Lock Failures: 38.91k
IO In Progress: 3
Low Memory Aborts: 77
Free on Write: 3.43k
Writes While Full: 6.22k
R/W Clashes: 0
Bad Checksums: 0
IO Errors: 0
SPA Mismatch: 598.47m
L2 ARC Size: (Adaptive) 94.87 GiB
Header Size: 0.07% 71.21 MiB
L2 ARC Evicts:
Lock Retries: 8
Upon Reading: 0
L2 ARC Breakdown: 21.63m
Hit Ratio: 1.41% 305.21k
Miss Ratio: 98.59% 21.33m
Feeds: 268.98k
L2 ARC Buffer:
Bytes Scanned: 28.87 TiB
Buffer Iterations: 268.98k
List Iterations: 1.07m
NULL List Iterations: 13.91k
L2 ARC Writes:
Writes Sent: 100.00% 104.95k
As it seems, I'm not evicting hardly at all and the miss rotation is darn near 100%. I've worked professional with Oracle ZFS (garbage sense they killed off and went closed source, lol) and I know we had some tunes in to improve performance with L2ARC (plus getting rid of the nasty bug in 11.1 SRU 15 and lower that caused the whole pool to fail when tickling a funky transaction group) that would great help perforamce. But I can't recall of the top of my head what they were or if they would apply since OpenZFS has forked totally into their own system.