L2ARC setup guide

[ECC]

Hello,
I have a server with a 8x 18TB Raidz2 pool and 64gb ram. This is my usecase:
- mainly 1 client access
- server runs only occasionally. Most of the time, the server is offline
- workload is mostly read&write big files. During update writing, there are also a lot of small files, which affect performance.

To improve small file reading performance, i was thinking about adding l2arc. It will also be persitent cache, which is important because the server never runs longer than a few hours.

I have a 2TB enterprise nvme laying around, which could be use for this purpose.

How would you setup this l2arc or what would be your approach to improve the Performance?

Thanks.

 

[sretalla]

Start by looking at arc_sumary to understand if you're going to benefit from L2ARC or not.

If you're getting high enough ARC hits, you may get nothing out of L2ARC and it will eat some of your ARC to store its metadata, so may even make performance worse.

It will also be persitent cache, which is important because the server never runs longer than a few hours.

Warming the persistent cache on boot can take hours if it's large... I don't think your use-case is sounding very sensible.

 

[sretalla]

Setting it up as metadata only may make some sense to help with small file access (metadata heavy activity) and will also help with warming times as not much actual data is cached.

 

[danb35]

How would you setup this l2arc

Easy, just add it to the pool as a L2ARC device. Whether it would benefit you is a separate question, and questionable for the reasons already mentioned.

Setting it up as metadata only may make some sense to help with small file access

...but while this may be more beneficial, it's also essential to the pool--if a metadata vdev fails, the whole pool is toast, and I'm not sure it's possible to remove these later.

 

[sretalla]

...but while this may be more beneficial, it's also essential to the pool--if a metadata vdev fails, the whole pool is toast, and I'm not sure it's possible to remove these later.

Not what I suggested... I was talking about a metadata only L2ARC...

L2ARC impact on rsync performance for largely dormant data

I've been doing some more testing and the evidence suggests that using a L2ARC with a metadata only flag has a potential substantial positive impact on rsync performance over AFP. For largely dormant data, the combination of L2ARC dedicated to metadata seems to benefit rsync performance a lot...

www.truenas.com

 

[danb35]

I was talking about a metadata only L2ARC

Ah, hadn't been tracking that.

 

[ECC]

Setting it up as metadata only may make some sense to help with small file access

That makes sense. So if i would do that, how would you set up l2arc (e.g. size) and which parameters for arc & l2arc would you tune, to improve performance? I setup blocksize to 1M, because the pool mainly holds big (media) files

 

[sretalla]

So if i would do that, how would you set up l2arc (e.g. size) and which parameters for arc & l2arc would you tune, to improve performance?

For each dataset you want it to apply to (and I would advise to do it for all datasets in the pool you add the L2ARC to), you need to zfs set secondarycache=metadata pool/dataset (you can't do that in the GUI)

Then add the L2Arc... you don't need it to be massive... 100GB would already be a lot, but adding something larger won't hurt as the excess capacity just isn't used.

No other tuning is advised (at least initially).

Read this thread if you want some ideas about other things you might do:

ZFS "ARC" doesn't seem that smart...

Without yet resorting to adding an L2ARC, is there a "tuneable" that I can test which instructs the ARC to prioritize metadata? I upgraded from 16GB to 32GB ECC RAM. Yet there is zero change in this behavior. This keeps happening: I run regular rsync tasks from a few local clients, which is...

www.truenas.com

I setup blocksize to 1M, because the pool mainly holds big (media) files

Good to know and probably the right thing to do... but you can set it per dataset and for the smaller files you mention, maybe wise to have it smaller.

 

[winnielinnie]

Read this thread if you want some ideas about other things you might do:

ZFS "ARC" doesn't seem that smart...

Without yet resorting to adding an L2ARC, is there a "tuneable" that I can test which instructs the ARC to prioritize metadata? I upgraded from 16GB to 32GB ECC RAM. Yet there is zero change in this behavior. This keeps happening: I run regular rsync tasks from a few local clients, which is...

www.truenas.com

Once ZFS 2.2 lands, things will not only be improved, but also be simplified and streamlined:

ZFS "ARC" is getting smarter with version 2.2+ :smiley_face_emoji:

This is a continuation and update of this thread: https://www.truenas.com/community/threads/zfs-arc-doesnt-seem-that-smart.100423/ Looks like OpenZFS 2.2+ introduces some cleaned up and rewritten code¹ to more intelligently and gracefully handle ARC data / metadata eviction from RAM. It's such...

www.truenas.com

Really excited for this!