Use cases for L2ARC

[Constantin]

@Constantin
Thanks for that. The 10Gb connection may be key to your use case here. The video that @Chris Moore linked also speaks of maxing out a 10Gb link.

In my experience, the benefit of L2ARC is use-specific and the connection speed is just one factor.

For example, when I look at rsync while its running its synchronization process, it's network data stream consumes less than 10 MB/s as the program collects file data and compares files from the server & backup. Yet, adding a L2ARC cut the time it took my system to compare / synchronize a backup and the server in half (from six to three hours) once the L2ARC cache had been filled by doing it once. The actual data transferred was miniscule, i.e. less than 100MB, so that wasn't a factor.

The only time I see a huge benefit of the 10GBe connection to my server is whenever very large continuous files are concerned. Lots of little files take their time to get transferred, SLOG/L2ARC or not. That's a price I pay for having a single Z3 VDEV.

 

[jgreco]

In my experience, the benefit of L2ARC is use-specific and the connection speed is just one factor.

For example, when I look at rsync while its running its synchronization process, it's network data stream consumes less than 10 MB/s as the program collects file data and compares files from the server & backup. Yet, adding a L2ARC cut the time it took my system to compare / synchronize a backup and the server in half (from six to three hours) once the L2ARC cache had been filled by doing it once. The actual data transferred was miniscule, i.e. less than 100MB, so that wasn't a factor.

The only time I see a huge benefit of the 10GBe connection to my server is whenever very large continuous files are concerned. Lots of little files take their time to get transferred, SLOG/L2ARC or not. That's a price I pay for having a single Z3 VDEV.

That's correct. Basically L2ARC is only going to help if there's a lot of data that had been read that's been flushed out of the ARC in the meantime. L2ARC isn't going to make your server magically hit 10Gbps. You can get great sequential speeds WITHOUT L2ARC if you have low pool occupancy rates and lots of sequential data to read/write. You can also get great speeds if you have sufficient ARC/L2ARC to cache all the data that's going to be read.

 

[Ericloewe]

What's a "substantial amount of hits" for ARC ghost?

Depends on the situation, but if you see that, say, 50% of your ARC misses were previously in ARC, that's a clear indicator that a somewhat larger ARC/L2ARC would allow you to capture those hits.

I am searching for an article that explains those criteria you list, and how to measure them and come to an objective conclusion, and I am coming up empty.

As usual, the best resource around are the ZFS books by Lucas and Jude: https://mwl.io/nonfiction/os#fmzfs

 

[Yorick]

L2ARCs are inexpensive to try out (i.e. any used SSD will do, power loss protection is not needed like on the SLOG side of the aisle).

That comes as a surprise to me, and it's awesome to see that's so. I'd have thought a read cache that delivers wrong data might be rather devastating. According to Advanced ZFS, "ZFS gracefully manages a dying or dead L2ARC". Good to know. It's still annoying to have to replace it, but in a non-production setting, fair enough.

Thank you as well for that link to the CIFS performance tests. I was wondering whether tuning ARC to use more space for metadata might have helped your rsync use case.

@Ericloewe I got myself the Advanced ZFS book. The chapter on ARC and L2ARC is short, and I understand them better now. That was 9.99 well spent. Thank you.

 

[Ericloewe]

I'd have thought a read cache that delivers wrong data might be rather devastating.

That's what checksums are for.

 

[marcevan]

oh what to do with an M.2 NVME 480GB stick then? Zdev or L2ARC?

 

[Chris Moore]

L2ARC

I have found (I think) that L2ARC is not very useful unless you are often reading the same data.

 

[marcevan]

My "Media" Pool is housing the typical data we all have but wondering if I can reshuffle:

1. Documents - low write, some read, some re-read (e.g., I do taxes for people and businesses so have to refer back to those docs)
2. Photos - mostly write, takes up good pct of pool given volume and stored by year folders
3. Movies - Blu-Ray rips, home shot movies, iphone movies, like documents low write, some read, some re-read if kids keep watching same movies
4. TV Shows - OTA rips written too daily/weekly; kids like to re-watch some shows using Plex over and over
5. Music - mostly read for in-home streaming; some writing but not much (used to work in Music so have large catalog, now not much new)

I will be starting an app dev for some ideas I want so one jail will be created with FAMP but it won't be a ton of data there.

So with the NVME M.2 stick... I can obviously attach it and TrueNAS will let me mount it or use it in pool, or L2ARC it, but I likely don't have too much being re-read for that to do much given 32GB RAM and new Xeon Kaby Lake CPU.

Perhaps just let it be for now and see what the future holds?

 

[Constantin]

Try it out on a metadata only basis and see if that helps responsiveness over time. You can make the L2ARC persistent as of TrueNAS, so it doesn't have to be reheated after a reboot.

 

[Chris Moore]

Try it out on a metadata only basis and see if that helps responsiveness over time.

You might want to point to some instructions.

 

[Constantin]

Good point, @Chris Moore I was on my mobile though and that is a lot of work in iOS. Anyhow,

1. How to make L2ARC persistent in TrueNAS (not possible in FreeNAS)
2. How to configure your L2ARC to be dedicated to metadata only (works in FreeNAS and TrueNAS)
   1. Drop into the command line and type: zfs set secondarycache=metadata <Poolname>
   2. To revert L2ARC to caching whatever ZFS wants to: zfs set secondarycache=none <Poolname>
3. Here is some neat info for how to look at the metadata.
4. Once you've been using the machine for a while, see what entering /usr/local/www/freenasUI/tools/arc_summary.py on the command line spits out. With any luck, the results will please you.