Use cases for L2ARC

[Yorick]

I’m having a lively discussion with @microserf elsewhere in this forum, and we’re hijacking a question about “what should be my first motherboard”. I’m taking this discussion here.

The two positions:

I can’t think of a home / enthusiast use case that would ever justify an L2ARC. Certainly not if the NAS feeds data through a 1Gb or 2x1Gb link.

@microserf asserts that an L2ARC can be beneficial in home setups with at least 32GB of RAM.

With that, in this post, I will attempt to refute his latest scenario.

Here’s a hypothetical he gave, and it’s off the top of his head, so no niggling on the numbers too much:

>>
First, don't conflate network link speed with ARC/L2ARC speed more than is necessary. Second, don't forget writes.

Off the top of my head :) ,

Alice is streaming a 20Mbps movie to the TV in the living room and is recording a 15Mbps OTA TV show using Plex . Bob is in his room talking to his girlfriend while his desktop is running an automated backup and streaming his playlist. Carol works from home and is updating NextCloud for the week ahead. Dave is in the backyard smoking a brisket while being recorded from two angles in HD (2 x 16Mbit/s).

The Plex streams are taxing the storage system (light to moderate, depending on fragmentation). If it's being transcoded, the CPU is awake. The backup is taxing the storage system (light to moderate) and might be using an internal db (more storage, random read/write, CPU). Carol is a wildcard but will interrupt the process of smoking brisket (sitting and drinking beer) if things get laggy. Dave is trying to hide from the multi-cam surveilllance system streaming to the NAS (moderate to...depends on stream count).

The 2 x 1Gb/s network is fine. The flying drive heads are getting a workout. Don't upset Dave.
>>

@microserf, you had linked me to the ZFS primer at https://www.ixsystems.com/documentation/freenas/11.2/zfsprimer.html . I’ll quote from that:

“When applications perform large amounts of random reads on a dataset small enough to fit into L2ARC, read performance can be increased by adding a dedicated cache device. SSD cache devices only help if the active data is larger than system RAM but small enough that a significant percentage fits on the SSD.”

Streaming by its nature doesn’t fit the description of “large amounts of random reads on a dataset”. L2ARC is not a readahead. That’s key to my argument. Once a piece of the show, some music, has been streamed, we’re not revisiting it. Only when there’s a dataset that doesn’t fit into RAM but does fit on L2ARC SSD, and we’re doing a large amount of random reads on it over and over again, does the L2ARC help me.

Nor do the writes mean anything in this situation. Sure they tax the heads, but since L2ARC isn’t a readahead, they’ll get taxed the same with or without L2ARC.

That’s the gist of my argument. L2ARC isn’t a readahead, and “large amounts of random reads over a dataset” should be taken literally, with “large” meaning a scale you’d see on a busy production server, not a home server.

Therefore, L2ARC does nothing for me in the home. It can’t.

ZIL doesn’t apply here either, none of these writes are sync writes.

I know some people with home labs ... two times ESXi, 10Gb NICs, sync writes, 38 VMs ... have seen great benefits from ZIL. That’s someone reproducing a datacenter workload in the home.

My question to the community, out of curiosity: If you implemented L2ARC because the analysis tools of FreeNAS indicated it was warranted, what was the environment? Type of workload, connectivity and throughput, measured performance increase?

 

[m0nkey_]

Therefore, L2ARC does nothing for me in the home. It can’t.

You are correct in your conclusion. Even with 32GB RAM for the average home user, L2ARC will do nothing to increase performance.

Even before considering L2ARC, you should ideally max out the RAM on the motherboard. If you ARC hit ratio is still well below average, then a L2ARC may help.

 

[Chris Moore]

@Yorick , You already know that you have the right answer. Often times it is not even useful in a working environment. I have servers at work where L2ARC was installed in an effort to make it work faster and it has not been able to deliver the results we wanted because of this size of the data and it's not repeatedly reading the same data.

The answer in a situation like you described is more vdevs. More vdevs gives the system greater ability to handle concurrent random IO.

 

[microserf]

Lol.

Streaming by its nature doesn’t fit the description of “large amounts of random reads on a dataset”.

I agree. Their inclusion was to allow consideration of two things: physical limitations in a multi-user system and the effect of Adaptive Replacement in an Adaptive Replacement Cache.

Sure they tax the heads, but since L2ARC isn’t a readahead, they’ll get taxed the same with or without L2ARC.

We can agree that the ARC+L2ARC is not a readahead cache. Whether the L2ARC hit rate reduces seeking in any meaningful way is the question. Leave ZIL out of it.

I'd like to ask the two forum regulars that have chimed in a question or two.

 

[microserf]

Even with 32GB RAM for the average home user,...

What's an average home user?

Even before considering L2ARC, you should ideally max out the RAM on the motherboard.

I did say minimum of 32GB of RAM and I believe the boards in question max out at 64 but, anecdotally, 16-32GB seems to be the norm.

 

[microserf]

Often times it is not even useful in a working environment.

What hit hate do you consider useful?

I have servers at work where L2ARC was installed in an effort to make it work faster and it has not been able to deliver the results we wanted because of this size of the data and it's not repeatedly reading the same data.

Over what time period? What did the MRU and MFU look like?

More vdevs gives the system greater ability to handle concurrent random IO.

Home system! Home system! Single pool with 6-8 drives maximum, very possibly only 2-4.

 

[Chris Moore]

I'd like to ask the two forum regulars that have chimed in a question or two.

Here is a good demo video about L2ARC:
https://www.youtube.com/watch?v=oDbGj4YJXDw&t

Home system! Home system! Single pool with 6-8 drives maximum, very possibly only 2-4.

It being a home system with only 1Gb networking makes the usefulness of L2ARC even less. It would be better to add more vdevs because each vdev gives the IOPS (roughly) of a single drive. That means, in a home system with six drives in a single vdev, you have the effective performance of a single drive, the slowest drive in the vdev. In my home NAS, I doubled the performance by adding a second vdev to the system. I have systems at work where I have ten vdevs of six drives each and it gives greater speed for random IO. The reason that mirror vdevs are suggested for use with VMs is to allow less drives to make more vdevs.

 

[Chris Moore]

I did say minimum of 32GB of RAM and I believe the boards in question max out at 64 but, anecdotally, 16-32GB seems to be the norm.

PS. I don't want to say the amount of memory doesn't matter, but it isn't going to make a difference in this case. Any kind of cache, RAM or L2, is not going to help if you are streaming audio or video files or if you are always accessing different files. The only time a cache helps is if you are repeatedly accessing the SAME data.

 

[microserf]

Here is a good demo video about L2ARC:
https://www.youtube.com/watch?v=oDbGj4YJXDw&t

I've seen the video but thanks for the link. I watched most of it again. Did you notice the corner case being addressed? They're using an L2ARC to increase frequent sequential read performance of the same data on a 10Gb link. As we're out of the home environment and into a business case here, I'll quote you:

It would be better to add more vdevs because each vdev gives the IOPS (roughly) of a single drive.

A small, cheap, SSD pool would solve the problem being addressed nicely. No need for an L2ARC. Watch 10:18-10:48 of the clip.

It being a home system with only 1Gb networking makes the usefulness of L2ARC even less.

This is one of the variables.

That means, in a home system with six drives in a single vdev, you have the effective performance of a single drive, the slowest drive in the vdev.

Exactly.

I have systems at work where I have ten vdevs of six drives each and it gives greater speed for random IO.

I don't doubt you...but @Yorick said we can't do that.

Any kind of cache, RAM or L2, is not going to help if you are streaming audio or video files or if you are always accessing different files.

I didn't say that it would. At least, not directly.

The only time a cache helps is if you are repeatedly accessing the SAME data.

Yep. At least twice. Sequential read bursts don't normally evict pages from the L2ARC.
https://github.com/zfsonlinux/zfs/blob/master/module/zfs/arc.c
https://github.com/freebsd/freebsd/blob/master/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c

 

[Ender117]

While I agree that L2ARC is not useful in most cases. I just wanted to add that (IIRC) there are tuneables that could make it a readahead cache. It might be (hypothetically) for...... ehh, loading ridiculously large folders?

 

[microserf]

I just wanted to add that (IIRC) there are tuneables that could make it a readahead cache.

vfs.zfs.l2arc_noprefetch=0
After looking at ARC:
vfs.zfs.prefetch_disable
And vdev LRU:
vfs.zfs.vdev.cache.size

 

[Chris Moore]

I know it isn't quite 10Gb, but you can get plenty of speed out of a system that has no SLOG or L2ARC.
Those things are only useful in some very limited cases. I suggest not wasting time on it.

 

[Chris Moore]

PS. While doing that, CPU utilization is actually fairly low:

I have a feeling that the system is able to do more, but the network interface is the limitation.

 

[Yorick]

As I understand it, those tunables mean that prefetched data ... which ZFS does do as part of its normal operation .. are eligible for L2ARC. Even more niche.

There is a recent-ish discussions about a use case that can saturate IO, some running a torrent server at home. Conclusion was that even in that case, with data being read repeatedly, L2ARC would give no to marginal benefits. That’s at https://www.reddit.com/r/zfs/comments/9iledr/should_i_use_l2arc/?st=JR0GMI4A&sh=fbcc4bea. Edit: They’re running mirrored vdevs with 16GB RAM ... okay they just need 32 or 64 instead. Still a good discussion of all the variables in L2ARC and how it differs between implementations.

I agree wholeheartedly with “don’t waste time and money on L2ARC”. It’s that consensus I created this thread for, so people curious about nas and wanting to build a home freenas know what they don’t need to consider in their builds. There’s value in that.

I did learn a lot, thank you. I didn’t know that more disks in a raidz didn’t increase random io throughput. I was under the impression it would.

 

[microserf]

I learned something as well. When I created an account on this forum, I had no idea that I was entering a church.

As I understand it, those tunables mean that prefetched data ... which ZFS does do as part of its normal operation .. are eligible for L2ARC. Even more niche.

Of course, minuscule niche: media server implementations with more than one client watching the same video and other sequential read biased uses where the data is larger than the pool cache.

So, you whip out a measuring tape:

With that, in this post, I will attempt to refute his latest scenario.

To have this vivisected:

L2ARC is not a readahead. That’s key to my argument.

And hide until closing with this gem:

I agree wholeheartedly with “don’t waste time and money on L2ARC”. It’s that consensus I created this thread for, so people curious about nas and wanting to build a home freenas know what they don’t need to consider in their builds.

Good luck on reddit,

https://www.reddit.com/r/zfs/comments/9iledr/should_i_use_l2arc/?st=JR0GMI4A&sh=fbcc4bea

But if I may, should someone respond, ask them why an application performing its own filesystem caching would benefit from something other than metadata in the L2ARC.

For anyone wishing to start at the beginning:
https://forums.freenas.org/index.ph...ich-supermicro-mainboard-should-i-take.72806/

 

[Constantin]

I agree wholeheartedly with “don’t waste time and money on L2ARC”. It’s that consensus I created this thread for, so people curious about nas and wanting to build a home freenas know what they don’t need to consider in their builds. There’s value in that.

I'm just a home user and I have a mere 500GB L2ARC but I already maxed out the RAM to 64GB. One viable use for L2ARC is to store vast amounts of metadata that we may have to enhance directory browsing speeds.

Informal testing on my part suggests a 2x improvement re: a typical rsync I will perform with an external RAID backup. Other forum users have suggested that the speedup may be related to how rsync traverses directories and how it is a one-task-at-a-time backup system. I like reducing latency for directory browsing - why else have a 10GBe connection to the server?

So, this is my primary use for L2ARC - speeding up directory browsing (which is great for trying to find a photo, etc) and reducing the time it takes rsync to execute. My testing to date is limited, however, and I also want to experiment with rsyncs using "secondary = metdata only" vs. standard L2ARC vs. playing with the fill parameters. That will take time.

 

[Constantin]

I learned something as well. When I created an account on this forum, I had no idea that I was entering a church.

You may not agree with what the small subset of people here are discussing with you but painting the whole forum with a broad brush like that is not helpful.

... Even with 32GB RAM for the average home user, L2ARC will do nothing to increase performance....Even before considering L2ARC, you should ideally max out the RAM on the motherboard.

Defining what an average user is or is not, will likely end without resolution. As discussed in the article I linked to below, L2ARC does show a benefit depending on how the system is being used. Yes, I'd max out RAM to max out the ARC cache first, if possible, but sometimes RAM is hard to come by (i.e. the unicorn-like 16GB DDR3 ECC UDIMM RAM sticks used on boards featuring the C2xxx Intel CPU series). Nothing wrong with experimenting based on your own use case. Just make a backup first. :)

L2ARCs are inexpensive to try out (i.e. any used SSD will do, power loss protection is not needed like on the SLOG side of the aisle) and if they don't show a benefit, they are easily removed from a pool. If the drive fails, no data is lost and the system moves on. Trying out a L2ARC is unlikely to harm a pool, other than potentially slowing it down - just make sure there is enough RAM to allow the system to run well in the first place. Because L2ARC eats some RAM, you don't want to trigger system instability by pairing a huge L2ARC with a system that has little RAM to start with.

My very limited experimentation suggests that a L2ARC has benefits for my use applications. That's what I love about FreeNAS. It's an incredibly flexible system that allows all sorts of users to tailor the system to their specific needs while performing a series of really important file-server infrastructure tasks. So it's entirely possible that L2ARC is not helpful for the vast majority of users, but for those who can benefit from it, it's great that FreeNAS gives us the option to enable and use it.

FWIW, I was inspired by the extensive work that @cyberjock, @anodos and others have done to help folk with improving the performance of FreeNAS. If you want to delve a bit more into why I added L2ARC, see this post from @cyberjock discussing how to speed up SMB performance.

 

[Chris Moore]

I learned something as well. When I created an account on this forum, I had no idea that I was entering a church.

Not a church. Nothing like it. I am not even sure what you are trying to suggest.

 

[Ericloewe]

I'm not sure why this is so controversial for some people. L2ARC is useful if the following criteria are met:

• ARC hit rate is lower than desired
• Performance is lower than desired due to disks running out of IOPS
• The ARC ghost lists are getting a substantial amount of hits
• Adding more RAM is not viable or too expensive compared to L2ARC

All these criteria are necessary and objectively measurable. There is little room for guesswork, as long as the L2ARC is correctly sized and backed by good-enough SSDs.

 

[Yorick]

@Constantin
Thanks for that. The 10Gb connection may be key to your use case here. The video that @Chris Moore linked also speaks of maxing out a 10Gb link.

@Ericloewe Can you elaborate or link? What's a "substantial amount of hits" for ARC ghost? I am searching for an article that explains those criteria you list, and how to measure them and come to an objective conclusion, and I am coming up empty.

My best guess as to why this topic may be controversial is that a method of measuring objectively is not known to everyone, or not widely (enough) known.

The scenario that started this discussion was a home media server with a single Gb link, max two, and a handful of users streaming and doing backups, maybe some home cloud.

@microserf We have a fundamental misunderstanding about intent. I'll PM you to try and clear the air. I'm keen to learn more about objective measurement. A clear and objective methodology exists for determining whether L2ARC is beneficial; now I just need to learn how that works in detail.