jgreco
Resident Grinch
- Joined
- May 29, 2011
- Messages
- 18,680
Weird, why would I see such a difference in the responsiveness of my VM's with an L2ARC (even without a SLOG)?
The SLOG has nothing to do with it.
Since you're maxxed out on RAM, you can see some benefit from L2ARC, but the problem is that it will tend to be somewhat unpredictable and variable. iSCSI is the worst sort of workload. You can read up on some of that at this link here, which will help explain some of why iSCSI typically requires more resources for what seems like a simple task.
As @HoneyBadger noted, there's a rule of thumb that suggests about 1:4 or 1:5 ARC:L2ARC, and we often trivialize that by saying it's RAM:L2ARC, but that fails at small quantities of RAM. On a warmed-up busy 16GB system here, for example, the ARC size is only 8GB. The FreeNAS middleware and other resource requirements can have a tendency to push down the ARC size a bit more than might be expected (I was expecting to find it around 10GB!) and as noted, ARC meta is limited to 1/4th that. So doing @HoneyBadger 's math again, we come up with something more like 40GB L2ARC. This might vary for non-iSCSI uses.
And that'll be very likely to work and take some load off the pool. However, the rate at which data is being evicted from ARC is going to be relatively high, and this will cause it to be difficult for ZFS to identify what would be good candidates for L2ARC. The window into your dataset is relatively tiny, and that means ZFS won't be making the best selections for eviction to L2ARC. It is sending to L2ARC the blocks that have the most activity that it can no longer keep in ARC, but, if you have a 30GB file that you're sequentially reading totally hundreds of times a day, all the blocks appear to have approximately equal levels of activity. It won't easily identify that the whole thing should be sent to L2ARC...
But overall, L2ARC helps bigger busier systems a lot more.