How much NVMe based L2ARC is too much?

[HeloJunkie]

So I am in the process of upgrading my 'PlexNAS FreeNAS' server (listed below in my signature) as follows:

UPGRADES

1. Upgrade CPU to E5-2699V4
2. Upgrade RAM to 128GB ECC
3. Replace USB Boot with 2 x SM SSD SATADOM (16GB each mirrored)
4. Increase storage from 71TB Usage to 120TB Usable (4 x 6 Drive VDEVs, 12GB/s SAS)
5. Upgrade from FreeNAS-11.2-RELEASE-U1 to TrueNAS Core

So I have read a lot about L2ARC, I have taken a look at this presentation to try and get a better understanding of the use cases of NVMe-based L2ARC, but I am still unsure of the best way to move forward. As I said, I am upgrading my system to be able to support additional space and I have a Samsung EVO 500GB NVMe drive as well as several Intel 3.2TB Enterprise NVMe drives collecting dust and I was thinking if they would help increase performance in the system I would use them there, but I don't want to waste them if they will not help.

The system is mostly a Plex server but does also house our primary NextCloud instance. It is a dual 10GB connected box (bonded interface), but overall there might be 10 to 15 plex users on at any given time and maybe 20 internal machines accessing Nextcloud, but the Nextcloud traffic is very minimal due to how Nextcloud actually works. This is a 100% NFS environment, no other services run on the box. No jails, no SMB, etc. This is also the storage array for my Proxmox cluster snapshots, but again, that is not really moving much data.

I am currently using 56TB of storage with 15TB available.

So I guess I am wondering from all of you TrueNAS Core gurus, will the NVMe-based L2ARC help at all, and if so, is 3.2TB of enterprise-grade Intel NVMe better than 500GB of consumer-grade Samsung EVO space?

 

[NugentS]

Of the list of servers in your sig, which one are you talking about?
With 128GB RAM - try without any L2ARC first. After a week or so check the ARC Hit Rate. If its 90%+ then L2ARC probably won't help

Other things to consider:
1. All NFS = probaly sync writes = slow unless sync=disabled or you use a SLOG
2. A small L2ARC, set to metadata only is:
a. Not pool critical
b. Can help speed up system response times.

Generally Enterpise much better than consumer. What are the Intel NVMe drives specifically

 

[HeloJunkie]

Of the list of servers in your sig, which one are you talking about?
With 128GB RAM - try without any L2ARC first. After a week or so check the ARC Hit Rate. If its 90%+ then L2ARC probably won't help

Other things to consider:
1. All NFS = probaly sync writes = slow unless sync=disabled or you use a SLOG
2. A small L2ARC, set to metadata only is:
a. Not pool critical
b. Can help speed up system response times.

Generally Enterpise much better than consumer. What are the Intel NVMe drives specifically

Hey @NugentS

The server is my 'Plex FreeNAS Server' as listed in my signature.

The NVMe drives I have are the Intel DC P4608 Series. I have a handful of 3.2 and 6.4TB laying around that I was going to use, but only if it might help.

You mentioned a couple of things. First, if my ARC hit rate is over 90% isn't that when L2ARC would help more since it would extend my ARC and help with faster access to those items in ARC? Maybe I am misunderstanding ARC. When I read this:

To improve read performance, ZFS uses system memory as an Adaptive Replacement Cache (ARC). System memory stores the most frequent and recent ZFS pool data used. Layer 2 ARC (L2ARC) is another option to extend the ARC to dedicated disks for dramatic improvement in read speeds verses slower data disks in the pool.

I assumed that adding NVMe-backed L2ARC would be helpful since it is super fast compared to the VDEVs in the pool.

Next, while I am all NFS, I do not use NFS for my virtualization other than for snapshots and backups. All of my virtualization drives reside on high-speed Intel DC series NVMe on my servers. The NFS mounts are strictly to hold snapshots and backups of those VMs, but they run natively on the Intel drives and are also 100% ZFS-based (Proxmox cluster). My largest NFS use case is my multiple Plex servers reading and writing movies and TV show data to and from the array.

I could add a SLOG as well if you think that would help, but it looks like 16GB is recommended limit and I don't have anything that small in NVMe, and I am not sure what the benefit would be if I threw in a Samsung SSD for the SLOG.

What do you think?

Thanks for the advice!

 

[Patrick M. Hausen]

I assumed that adding NVMe-backed L2ARC would be helpful since it is super fast compared to the VDEVs in the pool.

Yes. But if you have a large(ish) memory and a >90% ARC hit rate, then >90% of your reads are already served from RAM and not from your disks. L2ARC will not improve that.

If you have a low ARC hit rate despite a large ARC in RAM, e.g because your working set (the "hot data") is huge (video editing?) then an L2ARC is an opportunity to grow your cache beyond the RAM your machine can hold. But no L2ARC will be faster than ARC in RAM.

 

[jgreco]

Please note that while you can technically cram quite a bit of L2ARC into a system, the recommended amount is around 5:1 ratio going up to maybe 10:1, so for 128GB, going much past 1TB may end up being suboptimal. There's probably no harm in adding the 500GB NVMe as L2ARC.

Note also that SLOG only ever slows down a host compared to just doing async writes, and isn't going to make a big difference on a pool that is mostly async writes. Plus you need to ensure that it has proper power loss protection.

 

[NugentS]

Samsung SSD is not SLOG material. The P4608 however is (although its not an Optane, and doesn't have a massive endurance but if you under or is it over provision it - it should last quite a while).
The issue as @jgreco says is that it will only effect sync writes which if you aren't using them then there will be no effect.

 

[winnielinnie]

With 128GB RAM - try without any L2ARC first. After a week or so check the ARC Hit Rate. If its 90%+ then L2ARC probably won't help

Yes. But if you have a large(ish) memory and a >90% ARC hit rate, then >90% of your reads are already served from RAM and not from your disks. L2ARC will not improve that.

Will second that approach.

In addition to that, if (that's a big "if") you do decide to introduce an L2ARC device in the pool, you might want to limit it to only the dataset(s) that serves streaming (Plex) media.

 

[HeloJunkie]

OK, so it would appear the best course of action would be to increase my RAM to 128GB, and watch my ARC hit ratios to see if they fall below 90% (meaning that more and more of my data is not being served from ARC). I don't mind throwing in the Intel and it has a 35PBW life (and it's never been used) so it would take years for me to use up that endurance, but from what you guys are telling me it would be a waste to use even if my ARC hit % was low just due to its large size, correct?

Is there any downside to throwing in the 500GB NVMe as L2ARC which I have the system down for upgrade as opposed to having to take it back down in the future if I decide I actually need it? Right now, it looks like I am almost 100% ARC hits and this is with 64GB RAM currently:

 

[NugentS]

If you stick the L2ARC in - it will use up some of your ARC to keep its tables in

 

[HeloJunkie]

OK, thank you guys for the information, I think I will start without L2ARC as suggested and keep an eye on it for awhile and see if it is needed. Thanks again for the advice!

 

[Elliot Dierksen]

This isn't an exact parallel, but I have 256G of RAM in my E5-2637 V4 system and there are just short of 20 VM's (on separate ESXi server) on this NFS pool and this is what my ARC stats look like. I have an Optane SLOG and I debated using some of that for L2ARC, but I can't see a reason to do it.

I