L2 Arc Expiry?

[kspare]

What methods are there to control how long data takes to expire from the l2arc?

We have 2 storage servers with about 40 terminal servers on each. after the move the full 2tb is used, but after a week or so its dropped to 826gb on one server and 1300 on the other. Seems odd?

 

[Ericloewe]

It doesn't expire. Blocks are evicted by pressure from new ones or if they're freed.

 

[kspare]

Thats what I thought....so why isn't my cache staying full?

 

[sretalla]

so why isn't my cache staying full?

After a reboot? or are you seeing the ARC and/or L2ARC stats actually decrease while the system is running?

 

[kspare]

Like I said above...after a week or 2 it will drop from being full (2tb) to about 800gb

 

[Ericloewe]

Most likely blocks being freed and not replaced in cache.

 

[kspare]

I get that....back to my question...are there options to manage this?

 

[HoneyBadger]

I'd have to look at this as likely different utilization patterns on the terminal servers residing there. As @Ericloewe mentions, FREEs (deletes) from upstream will cause the records to be kicked out of L2ARC. If you've just migrated you might not have generated enough active traffic to really churn the hot and warm data into the right spots.

That said, what are your current ARC stats regarding capacity and hit rate %, as well as did you put any tunables in to change the L2ARC fill rate?

 

[kspare]

I'd have to look at this as likely different utilization patterns on the terminal servers residing there. As @Ericloewe mentions, FREEs (deletes) from upstream will cause the records to be kicked out of L2ARC. If you've just migrated you might not have generated enough active traffic to really churn the hot and warm data into the right spots.

That said, what are your current ARC stats regarding capacity and hit rate %, as well as did you put any tunables in to change the L2ARC fill rate?

I'll circle back to this in a few days...we just moved a bunch of vm's around for another test..basically 32kb vs 64kb block size. So I need to let it all settle for a week.

 

[kspare]

These are my tunes. I have a 2tb Intel P3700 configured for 4K

sysctl vfs.zfs.l2arc_headroom=2
sysctl vfs.zfs.l2arc_noprefetch=1
sysctl vfs.zfs.l2arc_norw=0
sysctl vfs.zfs.l2arc_write_boost=536870912
sysctl vfs.zfs.l2arc_write_max=536870912

 

[kspare]

I'd have to look at this as likely different utilization patterns on the terminal servers residing there. As @Ericloewe mentions, FREEs (deletes) from upstream will cause the records to be kicked out of L2ARC. If you've just migrated you might not have generated enough active traffic to really churn the hot and warm data into the right spots.

That said, what are your current ARC stats regarding capacity and hit rate %, as well as did you put any tunables in to change the L2ARC fill rate?

What stats would you like to see? The cache seems to have warmed up pretty decent now.

 

[HoneyBadger]

What stats would you like to see? The cache seems to have warmed up pretty decent now.

Full output of arc_summary.py is a good place to start.

The tunables don't look bad although 512MB/s as a write limit might be aggressive, it doesn't consider recordsize there so you could easily swamp it if you're sending smaller records at the drive. There's also the read latency penalty during writing to consider from norw=0. A P3700 will be good at simultaneous R/W, but there's still an impact.

 

[kspare]

Full output of arc_summary.py is a good place to start.

The tunables don't look bad although 512MB/s as a write limit might be aggressive, it doesn't consider recordsize there so you could easily swamp it if you're sending smaller records at the drive. There's also the read latency penalty during writing to consider from norw=0. A P3700 will be good at simultaneous R/W, but there's still an impact.

Where is that script on truenas?

 

[HoneyBadger]

Should be in /usr/local/bin/

 

[kspare]

I just get an error...

./arc_summary
Traceback (most recent call last):
File "./arc_summary", line 6, in <module>
from freenasUI.tools.arc_summary import main

 

[HoneyBadger]

Yep, seeing that here too. Guessing the name change in the middleware broke something in the dependencies.

How about grabbing some output from arcstat -a -o /path/to/an/output/file 1 100 - this will dump the stats every second for 100 seconds (adjust the last two numbers if you want different frequency/duration) and attaching the file (since it'll be super-wide text even in CODE format)

 

[kspare]

Thats broke in rc1 as well.

arcstat -a
python: can't open file '/usr/local/www/freenasUI/tools/arc_summary.py': [Errno 2] No such file or directory

• Put your data type(s) here...
• 12:31PM up 2 days, 7:34, 1 user, load averages: 0.68, 1.85, 1.76
• 10.9GiB / 119GiB (freenas-boot)
• 5.86TiB / 40.2TiB (san04)
• (MRU: , MFU: ) /
• Hit ratio -> (higher is better)
• Prefetch -> (higher is better)
• Hit MFU:MRU -> : (higher ratio is better)
• Hit MRU Ghost -> (lower is better)
• Hit MFU Ghost -> (lower is better)

 

[HoneyBadger]

Strange. Working fine here on 12.0-RC1 - I get a Python deprecation warning for use of implicit __int__ conversion but the output generates.

Did you clean-install TrueNAS or do an upgrade?

 

[kspare]

upgrade

 

[HoneyBadger]

Got me puzzled then. Looking like we went through the same path and yet I have a working arcstat script where you don't.

sysctl kstat.zfs.misc.arcstats ?