My example of an ARC starved NAS. (informational)

[titan_rw]

I thought I would post about my experience.

I noticed a network copy to one of my nas's was a bit bursty. Nothing horrible, but i noticed it:

So I went and looked at the nas. gstat showed the regular txg's being flushed, as the drives would all write like crazy for a second, then stop. But between txg flushes, some of the drives were extremely busy with what looked like random reads. 90% + utilization doing 150 ops/sec at a total of about 500 KB /sec.

Nothing else would have been reading from the pool at that time. So I'm thinking, in between txg's it's crazy busy with random reads from the disks. Hmm, let's check arc_summary.py.

Here's what I see at first:

Code:

CACHE HITS BY CACHE LIST:
          Most Recently Used:          11.00%  3.09m
          Most Frequently Used:        88.97%  24.98m
          Most Recently Used Ghost:    2.58%  725.13k
          Most Frequently Used Ghost:  0.06%  16.46k

Hmm, 2.58% of requests to the most recently used list in the ARC used to be in the ARC, but were evicted. I checked it again a minute or so later:

Code:

CACHE HITS BY CACHE LIST:
          Most Recently Used:          10.82%  3.27m
          Most Frequently Used:        89.15%  26.90m
          Most Recently Used Ghost:    3.03%  914.25k
          Most Frequently Used Ghost:  0.05%  16.53k

Whoa, that's a huge jump in percent for only a minute or so. About 200,000 reads were forced to disk because the ARC was too small. And about 200,000 were served out of the ARC successfully. (914k - 725k and 3.27m - 3.09m respectively)

Here's the specs on the machine: (and for the record, yes, it's consumer hardware, non ecc, etc, etc. It's slated for upgrade to proper server hardware when finances allow. Until that time, I accept all risks associated with non-ecc, and smallish size ram)

Ram: 12 gigs
Processor not relevant in this case.
zpool is a total of 12 drives, 3tb. 2 vdevs of 6 drives in z2.

Memory stats in top:

Code:

Mem: 168M Active, 109M Inact, 9543M Wired, 387M Buf, 2031M Free
ARC: 8502M Total, 3698M MFU, 4710M MRU, 12M Anon, 78M Header, 4462K Other
Swap: 8191M Total, 8191M Free

I'm also deduping a very select amount of data. Total ram used by dedupe is about 1.1 gigs.

"zfs list":

Code:

root@nas2 ~ # zfs list
NAME                        USED  AVAIL  REFER  MOUNTPOINT
nas2pool                  29.6T  4.21T  9.63T  /mnt/nas2pool

Time to start cleaning up some stuff. It's getting full.

"zpool list":

Code:

root@nas2 ~ # zpool list
NAME      SIZE  ALLOC  FREE    CAP  DEDUP  HEALTH  ALTROOT
nas2pool  32.5T  25.4T  7.13T    78%  42.41x  ONLINE  /mnt

"zdb -Dv":

Code:

root@nas2 ~ # zdb -Dv nas2pool
DDT-sha256-zap-duplicate: 2924858 entries, size 1102 on disk, 178 in core
DDT-sha256-zap-unique: 1076954 entries, size 1350 on disk, 218 in core
 
DDT histogram (aggregated over all DDTs):
 
bucket              allocated                      referenced
______  ______________________________  ______________________________
refcnt  blocks  LSIZE  PSIZE  DSIZE  blocks  LSIZE  PSIZE  DSIZE
------  ------  -----  -----  -----  ------  -----  -----  -----
    1    1.03M    131G  45.3G  48.5G    1.03M    131G  45.3G  48.5G
    2    333K  41.7G  21.2G  22.1G    790K  98.8G  50.0G  52.1G
    4    210K  26.3G  16.0G  16.4G    1.04M    133G  81.1G  83.2G
    8    361K  45.2G  38.5G  38.8G    3.58M    458G    387G    390G
    16    403K  50.3G  36.9G  37.4G    9.26M  1.16T    877G    889G
    32    554K  69.2G  47.9G  48.8G    27.3M  3.41T  2.29T  2.34T
    64    991K    124G  99.9G    101G    89.2M  11.2T  9.08T  9.16T
  128    3.50K    448M    204M    215M    606K  75.7G  34.4G  36.3G
  256      201  25.1M  8.18M  8.88M    66.9K  8.36G  2.51G  2.76G
  512      15  1.75M    232K    312K    9.32K  1.10G    148M    194M
    1K        7    896K    276K    296K    10.6K  1.33G    349M    381M
    2K        1    128K      4K  7.99K    2.81K    360M  11.2M  22.5M
    4K        1    128K      4K  7.99K    6.08K    778M  24.3M  48.6M
    8K        2    256K      8K  16.0K    27.5K  3.43G    110M    219M
  16K        1    128K      4K  7.99K    18.3K  2.28G  73.1M    146M
  64K        1    128K      4K  7.99K    65.6K  8.20G    262M    524M
Total    3.82M    488G    306G    313G    133M  16.6T  12.8T  13.0T
 
dedup = 42.41, compress = 1.30, copies = 1.01, dedup * compress / copies = 54.35

So appx 300 GB of data on the platters after being deduped. Let's see how much data this actually represents if we turned off dedupe, compression, and sparse files:

Code:

root@nas2 ~ # du -Ash /mnt/nas2pool/vmbackup/
29T    /mnt/nas2pool/vmbackup/

Anyway, I thought it was cool to actually see a ram starved nas in operation. Considering it's got almost 30 TB of data on it, and I'm doing a small amount of dedupe, this box should really have 32 gigs. As a comparison, here's my other nas that does have 32 gig: (11 drives in z3, also 3tb drives)

arc_summary.py:

Code:

CACHE HITS BY CACHE LIST:
          Anonymously Used:            0.20%  1.17m
          Most Recently Used:          0.28%  1.60m
          Most Frequently Used:        99.51%  576.06m
          Most Recently Used Ghost:    0.00%  580
          Most Frequently Used Ghost:  0.01%  86.79k

Much better. Almost zero for both ghost stats. It's rarely having to go back to disk when it previously had the data in ram.