How to use/optimize sata ssd or pci ssd for Cache?

[guysmilez]

I built a freenas from a old computer

Everything seems to be working and I'm still testing and playing with everything before i finish it up. Just to play with speed I striped 8 - 16tb drives together but when i goto do a speed test i get a burst at 730 MB/s write but then drops to an average of 320 MB/s and then Read average 438 MB/s. It looks like my 8gb ram was holding me back and i uped it to 16gb which is the max of my mobo. but still having bottleneck issues.

So im trying a 1tb sata ssd and recreated the pool with the ssd as a cache. I didnt see much of a difference. I was reading that the freenas default settings were not ideal for ssd caches. Is there a guide/tutorial on how to configure the freenas settings to utilize the ssd properly?
here is a video where i saw someone talking about optimizing freenas to use the ssd but looks like an older version of freenas https://www.youtube.com/watch?v=oDbGj4YJXDw&t=1s

Anyone configure caches using a ssd either pci or sata? what did you do and preferably why so i can try to wrap my head around it?

Thanks ahead of time

Specs:
cpu: Intel Core i7-875K Processor 2.93 GHz
ram: 16gb ddr3
mobo: gigabyte p55a-ud3
network card: Mellanox Connectx-2 PCI-E 10GBe
video card: nvidia geforce 2 mx
Hard drive expansion card: 2x MZHOU PCIe SATA Card 8 Port, PCIe to SATA Controller, Marvell 88SE9215 + JMicron JMB5xx 8-Port Chip
Hardrives: 8x Seagate 16TB HDD Exos X16

 

[Patrick M. Hausen]

Unless you have 64G or more of RAM the RAM needed to manage your SSD cache will make your experience worse, not better.

26. ZFS Primer — FreeNAS®11.3-U1 User Guide Table of Contents

www.ixsystems.com

 

[jenksdrummer]

Couple items....

Depending on the speed test and where it's ran from, it could be that your machine is buffering the data; giving a really good number until it fills that buffer.

Your SATA SSD probably maxes out at around 300-400MB/sec under the right workload. Each spindle is 120-150MB/sec. Depending on your configuration; reads may be faster from disk than SSD under certain workloads.

Cache; or L2ARC, is only a READ cache. Data is read from disk to RAM. If L2ARC is available, once RAM is full, it RAM will shovel data that isn't as 'hot' to the L2ARC in case it needs to be read again; eventually aging that out once it's full.

When data is written, it arrives to RAM; gets organized and written to the data VDEV(s). If by chance that data needs to be read from quickly, it may still be in RAM. (Not entirely sure here if writing data; once committed if it's 'flushed' from RAM or it remains until it ages out) - point being adding an SSD as L2ARC to the mix will NOT make the writes happen faster; and it may or may not make reads happen faster.