How can I create ARC MRU cache(Most Recently Used) test load to stress L2ARC

[Peter Jakab]

Hi All,

I am able to test MFU (Most Frequently Used) cache by simple copy same files more times. First copy from NAS slower but second works from cache. arc_summary.py show correctly how they increased as MFU:MRU ratio, global ARC cache size etc.

After I upgraded to 32GB RAM on test system and put OCZ Vertex2 60GB SSD as L2ARC cache drive (just for testing not for production - I know min. 64GB RAM needed).
I would like make load on MRU (Most Recently Used) cache in ARC and as overload could flow into L2ARC.
How I understand first file operation will load into MRU cache and response next question, but after second copy the block moved to MFU instead.
Actually my MRU cache have small ratio 6.77% after the duplicated copy operations (MFU 93.23%).

Results: Also this actually results show that not need for L2ARC for frequently re-reads due mostly ARC uses the MFU side in RAM. But before put to production I would like play with this test.

How I can make a simple traffic load to increase MFU cache and after those cache blocks goes into Ghost MRU cache (land in L2ARC) re-read to increase L2ARC hit ratio (at now 0.00% with 58 operation). And possible check difference between with and without L2ARC cache operation. I don't want install database on that.

My pool relative small so I have about 500-600GB free space to check it.

Many Thanks for ideas,
Jackson

 

[Peter Jakab]

I try to make L2ARC cache test, but not works:
Copy 300Gb of new files (new for NAS never see that before) from my PC to NAS
I check files and after few big files copied but not more than actual MRUsize+L2ARC size (due not in MRU cache any more)
so copy back same files to trigger MRU and possible L2ARC cache hit

My starting point:

Code:

ARC Summary: (HEALTHY)

        Storage pool Version:                   5000
        Filesystem Version:                     5
        Memory Throttle Count:                  0

ARC Misc:
        Deleted:                                2.20m
        Recycle Misses:                         116.20k
        Mutex Misses:                           105
        Evict Skips:                            105

ARC Size:                               75.36%  22.57   GiB
        Target Size: (Adaptive)         75.48%  22.60   GiB
        Min Size (Hard Limit):          12.50%  3.74    GiB
        Max Size (High Water):          8:1     29.94   GiB

ARC Size Breakdown:
        Recently Used Cache Size:       6.77%   1.53    GiB
        Frequently Used Cache Size:     93.23%  21.07   GiB

ARC Hash Breakdown:
        Elements Max:                           691.44k
        Elements Current:               95.94%  663.37k
        Collisions:                             187.10k
        Chain Max:                              4
        Chains:                                 47.83k
                                                                Page:  2
------------------------------------------------------------------------

ARC Total accesses:                                     14.80m
        Cache Hit Ratio:                78.86%  11.67m
        Cache Miss Ratio:               21.14%  3.13m
        Actual Hit Ratio:               78.10%  11.56m

        Data Demand Efficiency:         98.17%  7.82m
        Data Prefetch Efficiency:       0.05%   2.93m

        CACHE HITS BY CACHE LIST:
          Most Recently Used:           19.27%  2.25m
          Most Frequently Used:         79.75%  9.31m
          Most Recently Used Ghost:     1.14%   132.78k
          Most Frequently Used Ghost:   4.57%   533.71k

        CACHE HITS BY DATA TYPE:
          Demand Data:                  65.76%  7.67m
          Prefetch Data:                0.01%   1.54k
          Demand Metadata:              33.26%  3.88m
          Prefetch Metadata:            0.96%   112.41k

        CACHE MISSES BY DATA TYPE:
          Demand Data:                  4.57%   142.95k
          Prefetch Data:                93.70%  2.93m
          Demand Metadata:              1.45%   45.20k
          Prefetch Metadata:            0.28%   8.83k
                                                                Page:  3
------------------------------------------------------------------------

L2 ARC Summary: (HEALTHY)
        Passed Headroom:                        1.65m
        Tried Lock Failures:                    1.11k
        IO In Progress:                         0
        Low Memory Aborts:                      1
        Free on Write:                          107.80k
        Writes While Full:                      7.13k
        R/W Clashes:                            17
        Bad Checksums:                          0
        IO Errors:                              0
        SPA Mismatch:                           51.99m

L2 ARC Size: (Adaptive)                         55.95   GiB
        Header Size:                    0.11%   65.65   MiB

L2 ARC Evicts:
        Lock Retries:                           4
        Upon Reading:                           0

L2 ARC Breakdown:                               3.08m
        Hit Ratio:                      0.00%   58
        Miss Ratio:                     100.00% 3.08m
        Feeds:                                  28.32k

L2 ARC Buffer:
        Bytes Scanned:                          1.80    TiB
        Buffer Iterations:                      28.32k
        List Iterations:                        1.69m
        NULL List Iterations:                   445

L2 ARC Writes:
        Writes Sent:                    100.00% 10.45k

Easy visible MFU:MRU ratio very high for MFU do several times copy was before. MFU cache hit better off course Ghost is also MFU higher.

After copy done:

Code:

ARC Size:                               76.91%  23.03   GiB
        Target Size: (Adaptive)         77.02%  23.06   GiB
        Min Size (Hard Limit):          12.50%  3.74    GiB
        Max Size (High Water):          8:1     29.94   GiB

ARC Size Breakdown:
        Recently Used Cache Size:       96.20%  22.18   GiB
        Frequently Used Cache Size:     3.80%   898.09  MiB

ARC Hash Breakdown:
        Elements Max:                           861.87k
        Elements Current:               100.00% 861.87k
        Collisions:                             546.17k
        Chain Max:                              6
        Chains:                                 83.07k
                                                                Page:  2
------------------------------------------------------------------------

ARC Total accesses:                                     18.27m
        Cache Hit Ratio:                78.28%  14.30m
        Cache Miss Ratio:               21.72%  3.97m
        Actual Hit Ratio:               77.53%  14.17m

        Data Demand Efficiency:         97.72%  9.70m
        Data Prefetch Efficiency:       0.23%   3.66m

        CACHE HITS BY CACHE LIST:
          Most Recently Used:           20.86%  2.98m
          Most Frequently Used:         78.19%  11.18m
          Most Recently Used Ghost:     1.56%   223.59k
          Most Frequently Used Ghost:   3.84%   549.18k

        CACHE HITS BY DATA TYPE:
          Demand Data:                  66.25%  9.48m
          Prefetch Data:                0.06%   8.29k
          Demand Metadata:              32.79%  4.69m
          Prefetch Metadata:            0.90%   129.16k

        CACHE MISSES BY DATA TYPE:
          Demand Data:                  5.57%   220.89k
          Prefetch Data:                91.90%  3.65m
          Demand Metadata:              2.31%   91.84k
          Prefetch Metadata:            0.22%   8.90k
                                                                Page:  3
------------------------------------------------------------------------

L2 ARC Summary: (HEALTHY)
        Passed Headroom:                        2.09m
        Tried Lock Failures:                    1.53k
        IO In Progress:                         0
        Low Memory Aborts:                      3
        Free on Write:                          204.50k
        Writes While Full:                      13.44k
        R/W Clashes:                            18
        Bad Checksums:                          0
        IO Errors:                              0
        SPA Mismatch:                           67.92m

L2 ARC Size: (Adaptive)                         60.92   GiB
        Header Size:                    0.18%   110.96  MiB

L2 ARC Evicts:
        Lock Retries:                           7
        Upon Reading:                           0

L2 ARC Breakdown:                               3.92m
        Hit Ratio:                      0.02%   648
        Miss Ratio:                     99.98%  3.92m
        Feeds:                                  37.05k

L2 ARC Buffer:
        Bytes Scanned:                          2.40    TiB
        Buffer Iterations:                      37.05k
        List Iterations:                        2.15m
        NULL List Iterations:                   10.67k

L2 ARC Writes:
        Writes Sent:                    100.00% 17.20k

As you can see MRU adapting them self to different usage as before.
MRU Size 96.20% as 22.18GB compared to MFU size 3.8% as 898.09MB

MRU usage goes higher as ratio 20.86% and count 2.98million
meanwhile MFU usage not to much changed 78.19% 11.18m

But the two ghost not changed drastically:
Most Recently Used Ghost: 1.56% 223.59k
Most Frequently Used Ghost: 3.84% 549.18k

And L2ARC also not used as cache hit:
Hit Ratio: 0.02% 648
Miss Ratio: 99.98% 3.92m

So how to make more better?

Bye,
Jackson

 

[dlavigne]

Were you able to figure this out?

 

[Peter Jakab]

Were you able to figure this out?

This is build in command:

Code:

arc_summary.py

Show ARC stats since boot-up. Off course more details inside the command results, this is just my cropped version focus on important elements.

Another usefully command to actual status:

Code:

arcstat.py 1 5

Bye,
Jackson

 

[Peter Jakab]

What was my best in my L2ARC stress test? When I remove L2ARC by command: :)

Code:

zpool status
   copy out cache drive gptid
zpool remove z2test gptid/xxxxx
    xxxxx of the cache drive above
zpool status
    To check cache drive removed correctly

My test results:
- L2ARC extremely slow down the shutdown and boot times.
- How I see that L2ARC just used for MRU caching I write one times and read 10-15 time larger data than is ARC MFU size and not read from L2ARC so L2ARC just used for MRU.
- I found this info on web (sorry source URL missing just the text I copied out): The "Most Recently Used Ghosts" entries show what might have gone to L2ARC, given your current RAM
- The best MFU/MRU graphic description I found:
https://pthree.org/2012/12/07/zfs-administration-part-iv-the-adjustable-replacement-cache/
Could be clone of that ? http://www.c0t0d0s0.org/archives/5329-Some-insight-into-the-read-cache-of-ZFS-or-The-ARC.html

All "use L2ARC" fan check your stats:
- In GUI Reporting/ZFS If L2ARC Hit ratio below 1% you NOT use it (my max was 0.02% under the test). Results= You not needed L2ARC
By CLI check arc_summary.py command output also:
- If MFU:MRU ratio mean you have less then 5-10% of MRU and rest of them is MFU then your system not uses L2ARC. Due L2ARC is just for "MRU expansion" technology. Results= You not needed L2ARC
- If "CACHE HITS BY CACHE LIST"/"Most Recently Used Ghost" have small number on right (the percentage always small). That mean you not use L2ARC due just sometimes try access MRU which is not in RAM any more. Results= You not needed L2ARC

If you still don't believe you don't need L2ARC same facts from your system:
- "L2 ARC Breakdown"/Hit Ratio not just % but also number of cache hits here. After running same week check this number and compare with ARC hit ratio number. You wee see how many times was usefully to read from L2ARC.
- Check your GUI Reporting/Disk and your L2ARC cache SSD disk. How many the Total Read and Total Writes (rightest value). As you can see the the write is high number (that is stress your SSD wear factor) meanwhile the Total Read is very small number. That is how many Byte read from cache.
If one of the (number of cache hit OR cache read size) number is small why you invest many into L2ARC SSD instead of buy more RAM or new mainboard?
Results= You not needed L2ARC

Show me somebody stats who really use it (1% of forum users);)
Somebody could tell me why is L2ARC not designed to use for MFU (what we frequently use see in ARC stats forum) extension cache? One times write and many times read that is the best for SSDs. Could any FreeNAS differences in original ZFS L2ARC IBM/Solaris implementation why not used for MFU cache?

Bye,
Jackson