How large L2Arc for 64GB

Status
Not open for further replies.

DaveFL

Explorer
Joined
Dec 4, 2014
Messages
68
Id like to add an L2Arc at the same time I update to 64gb ram.

What size should I go with 128GB or 256GB SSD? Also is it worth it to do NVME?

Thanks!
 

BigDave

FreeNAS Enthusiast
Joined
Oct 6, 2013
Messages
2,479
I was able to Google this for you.

Read below the relevant section from the ZFS Primer

"As a general rule of thumb, an L2ARC should not be added to a system with less than 64 GB of RAM and the size of an L2ARC should not exceed 5x the amount of RAM. In some cases, it may be more efficient to have two separate pools: one on SSDs for active data and another on hard drives for rarely used content. After adding an L2ARC, monitor its effectiveness using tools such as arcstat. If you need to increase the size of an existing L2ARC, you can stripe another cache device using Volume Manager. The GUI will always stripe L2ARC, not mirror it, as the contents of L2ARC are recreated at boot. Losing an L2ARC device will not affect the integrity of the pool, but may have an impact on read performance, depending upon the workload and the ratio of dataset size to cache size. Note that a dedicated L2ARC device can not be shared between ZFS pools."
 

DaveFL

Explorer
Joined
Dec 4, 2014
Messages
68
Thank you for sharing that. But it doesn't answer what I am asking. Is there a general sweet spot on a reasonable size to use? I've read cyber's post stating that 120GB should be the max used for 64GB. I've read others saying to use a 4:1 ratio for L2Arc:Arc and now you are posting something that says 5:1. What is the safer bet here? 2:1 or 4:1 ? Or does it not really matter?
 

BigDave

FreeNAS Enthusiast
Joined
Oct 6, 2013
Messages
2,479
Thank you for sharing that. But it doesn't answer what I am asking. Is there a general sweet spot on a reasonable size to use? I've read cyber's post stating that 120GB should be the max used for 64GB. I've read others saying to use a 4:1 ratio for L2Arc:Arc and now you are posting something that says 5:1. What is the safer bet here? 2:1 or 4:1 ? Or does it not really matter?
Yes it matters. The link I posted comes from the 9.10 documentation of freenas.org
Note the statement "rule of thumb". The sweet spot you ask about is not a concrete number regarding size
due to the many volume configurations and the type/size of files you are serving up with them.
If cyberjock is recommending a max L2ARC of 120GB being max used with 64GB of RAM, you can be sure
that he has tested this, however I don't know under what conditions this was done.
I agree with you about the wide variances, that said, my WAG is that starting with a smaller L2ARC and
some testing would be a good plan. If you find the size lacking after testing, stripe in another.
There's just to many variables (IMHO) to make a sure fire answer to your question.
 

Evi Vanoost

Explorer
Joined
Aug 4, 2016
Messages
91
I'm not sure what the reasons are for these 'rules of thumb'. Basically, depending on your record size, your L2ARC does use up a certain amount of your ARC. I think it's something along the lines of 1:20 (so 400GB of L2ARC would use up about 20GB of your ARC) if your record size is an average 32kb.

In my current system (96GB RAM) I have 960GB of L2ARC and the ARC (in-memory) cache hits for 90%, L2ARC 98%. It is stable as a rock and performance is about as good as the configuration (6G SAS SSD) will do, I get ~20k average read IOPS from the L2ARC (these SSD are about 7 years old now and they were relatively cheap back then).

In my situation (about 100 users file server) I rather have more of my most used data (~4TB gets frequently used out of the ~150TB pool which consist of billions of 8-20kb files) in the L2ARC even if it means less of it is in ARC because it gets more consistent speeds for all of my users (I really should have more spinning rust).

For most of your single user media center usage however, I'm even wondering if L2ARC is even worth the diminishing ARC and expense of SSD (streaming doesn't really get accelerated by L2ARC) over another set of relatively cheap RAM sticks.
 

tvsjr

Guru
Joined
Aug 29, 2015
Messages
959
IMO, you shouldn't pursue L2ARC until you've maximized the RAM in your system (or, at least, maximized it at an affordable level... no one's expecting 1.5TB in a dual E5 system). You should be looking at arcstat and other commands (search the forums) to determine if an L2ARC would really benefit you at all.
 

Evi Vanoost

Explorer
Joined
Aug 4, 2016
Messages
91
Regards NVMe: I don't think it's worth it if you're not plunking down big money to get an actual datacenter version of an SSD.

I've done a preliminary test on a brand new NVMe system and I'm not all that impressed by desktop versions of NVMe SSD (950Pro) - I'd rather spend a little more on a decent SATA or SAS SSD than spending more on NVMe tech.

If you however CAN afford a set of "DataCenter" SSD model (I tested a Intel DC P3700) and a machine to support it, NVMe may be worth it if you really NEED the lower latency and higher throughput but you also lose the benefits of SAS (if you ever do need to failover to a spare machine).

If you have few or slow drives, get a small yet fast SSD for ZIL, hell even using a non-NVMe PCIe SSD may be 'cheaper' for you than getting a motherboard with NVMe.
 
Status
Not open for further replies.
Top