What are your ARC statistics?

DataKeeper

DataKeeper

Patron
Joined
Feb 19, 2015
Messages
223
Still chugging along like a snail on steroids o_O

~/bin# uptime
  • 2:21AM up 20 days, 22:48, 2 user, load averages: 6.73, 7.08, 6.81
~/bin# ./arc_stats

  • Put your data type(s) here...
  • 5.75GiB / 59.5GiB (freenas-boot)
  • 9.12TiB / 65.2TiB (garage)
  • 45.98GiB (MRU: 41.99GiB, MFU: 3.99GiB) / 64.00GiB
  • Hit ratio -> 91.94% (higher is better)
  • Prefetch -> 95.15% (higher is better)
  • Hit MFU:MRU -> 81.27%:3.10% (higher ratio is better)
  • Hit MRU Ghost -> 0.07% (lower is better)
  • Hit MFU Ghost -> 0.09% (lower is better)
 
JMC

JMC

Dabbler
Joined
Jun 2, 2016
Messages
11
  • Central family cloud; ZPool is 2 drives mirrored for each vdev
  • 10:33PM up 2 days, 8:34, 1 user, load averages: 1.07, 1.13, 1.08
  • 644MiB / 28.5GiB (freenas-boot)
  • 15.5GiB / 472GiB (nvme-01)
  • 744GiB / 17.2TiB (pool-01)
  • 3.37MiB / 26.5GiB (systembackup)
  • 52.26GiB (MRU: 39.08GiB, MFU: 13.18GiB) / 64.00GiB
  • Hit ratio -> 75.28% (higher is better)
  • Prefetch -> 90.86% (higher is better)
  • Hit MFU:MRU -> 95.86%:2.97% (higher ratio is better)
  • Hit MRU Ghost -> 0.01% (lower is better)
  • Hit MFU Ghost -> 0.00% (lower is better)

NB: My system has changed, but until I reach 15 posts, I can't update my signature
 
M

Morpheus187

Explorer
Joined
Mar 11, 2016
Messages
61
  • Basically my storage for all stuff, movies, tv-shows, documents, backus
  • 2:17PM up 44 days, 19:46, 1 user, load averages: 0.57, 0.57, 0.56
  • 18.3TiB / 43.5TiB (VOL1)
  • 625MiB / 14.2GiB (freenas-boot)
  • 27.39GiB (MRU: 22.04GiB, MFU: 5.37GiB) / 32.00GiB
  • Hit ratio -> 31.88% (higher is better)
  • Prefetch -> 7.68% (higher is better)
  • Hit MFU:MRU -> 73.36%:23.91% (higher ratio is better)
  • Hit MRU Ghost -> 0.11% (lower is better)
  • Hit MFU Ghost -> 0.29% (lower is better)
 
Ove Gjøran Johnsen

Ove Gjøran Johnsen

Cadet
Joined
Aug 13, 2016
Messages
2
Here are some updated stats after adding up to 100 GB ram (system is running as a Virtual machine in VMware)
Im actually surprised about how low the hit ratio is, i would have expected more. but im not really complaining about performance.

in my previous stat post, i managed to get about 100 - 150 MB/s troughput. With this post I'm experiencing 250 - 300 MB/s troughput.
so the actual performance is more or less in line With what i am/was expecting to get.

Next thing I'm considering now is to instead of running it virtually, to run it as a physical machiene. that way i would be able to dedicate 144 GB ram to the setup.

  • media
  • 12:53PM up 21:49, 1 user, load averages: 1.33, 1.56, 1.23
  • 697MiB / 7.94GiB (freenas-boot)
  • 26.4TiB / 65TiB (tank)
  • 89.99GiB (MRU: 84.35GiB, MFU: 5.65GiB) / 112.00GiB
  • Hit ratio -> 78.61% (higher is better)
  • Prefetch -> 48.29% (higher is better)
  • Hit MFU:MRU -> 42.73%:42.58% (higher ratio is better)
  • Hit MRU Ghost -> 1.11% (lower is better)
  • Hit MFU Ghost -> 1.53% (lower is better)
 
MatthewSteinhoff

MatthewSteinhoff

Guru
Joined
Feb 2, 2016
Messages
574
Primary NAS (running 9.10)...
  • Data types listed with volumes below.
  • 11:20AM up 12 days, 21:26, 1 user, load averages: 0.62, 0.73, 0.70
  • 22.7TiB / 32.5TiB (V-D2600) ~ Security Video (not replicated, 2 x (3tb x 6 disks raidz2))
  • 196GiB / 1.73TiB (V-SSD) ~ XenServer VMs (two mirrors of 960GB each on SSD)
  • 4.08TiB / 10.9TiB (V-Super) ~ Generic Office Data and Photo Storage (six mirrors of 2TB each)
  • 916MiB / 30GiB (freenas-boot)
  • 40.99GiB (MRU: 29.36GiB, MFU: 11.63GiB) / 48.00GiB
  • Hit ratio -> 94.67% (higher is better)
  • Prefetch -> 20.08% (higher is better)
  • Hit MFU:MRU -> 92.72%:5.54% (higher ratio is better)
  • Hit MRU Ghost -> 0.23% (lower is better)
  • Hit MFU Ghost -> 0.37% (lower is better)

Replication Target (running 9.3)...
  • Data types listed with volumes below.
  • 11:24AM up 26 days, 22:51, 1 user, load averages: 0.12, 0.09, 0.25
  • 5.75TiB / 7.25TiB (V-Live1000) ~ Office Data Replication Target (8 x 1tb, raidz2)
  • 292GiB / 3.78TiB (V-Live600) ~ VM Replication Target (7 x 600mb, raidz2)
  • 3.08GiB / 7.44GiB (freenas-boot)
  • 22.49GiB (MRU: 21.32GiB, MFU: 1.22GiB) / 32.00GiB
  • Hit ratio -> 93.66% (higher is better)
  • Prefetch -> 12.46% (higher is better)
  • Hit MFU:MRU -> 96.86%:2.63% (higher ratio is better)
  • Hit MRU Ghost -> 0.07% (lower is better)
  • Hit MFU Ghost -> 0.30% (lower is better)

Cheers,
Matt
 
C

CookiesLikeWhoa

Patron
Joined
Mar 22, 2016
Messages
217
  • 10:46PM up 9 days, 8:49, 1 user, load averages: 0.79, 0.57, 0.50
  • 46.7GiB / 920GiB (Jails)
  • 1.59TiB / 21.8TiB (Tank)
  • 982MiB / 14.5GiB (freenas-boot)
  • 229.33GiB (MRU: 123.57GiB, MFU: 105.76GiB) / 256.00GiB
  • Hit ratio -> 64.45% (higher is better)
  • Prefetch -> 57.53% (higher is better)
  • Hit MFU:MRU -> 80.86%:18.61% (higher ratio is better)
  • Hit MRU Ghost -> 0.00% (lower is better)
  • Hit MFU Ghost -> 0.03% (lower is better)
Well I might need to actually use that L2ARC after all, I figured 256GB's of RAM would cover it for an ESXI iSCSI target, but alas here I am. Granted the server only has an up time of 9 days, the hit ratio has actually gotten worse over the last couple of days.
 
Scharbag

Scharbag

Guru
Joined
Feb 1, 2012
Messages
620
  • Movies, Data, VMs shared over NFS to ESXi
  • 3:20PM up 2 days, 5:56, 1 user, load averages: 2.43, 1.53, 1.08
  • 19.0TiB / 32.5TiB (backuptank)
  • 24.6TiB / 43.5TiB (bigtank)
  • 179GiB / 444GiB (fasttank)
  • 1.07GiB / 15.9GiB (freenas-boot)
  • 32.70GiB (MRU: 32.34GiB, MFU: 4.48GiB) / 48.00GiB
  • Hit ratio -> 68.60% (higher is better)
  • Prefetch -> 57.68% (higher is better)
  • Hit MFU:MRU -> 82.55%:11.19% (higher ratio is better)
  • Hit MRU Ghost -> 0.22% (lower is better)
  • Hit MFU Ghost -> 0.22% (lower is better)
 
CSP-on-FN

CSP-on-FN

Dabbler
Joined
Apr 16, 2015
Messages
15
  • Hard-drive-image backups, and multiple small .DOC and .LOG (text) files.
  • 9:02PM up 19 days, 9:43, 1 user, load averages: 0.22, 0.12, 0.06
  • 1.82TiB / 3.62TiB (XEPOOL_A)
  • 966MiB / 14.4GiB (freenas-boot)
  • 48.2MiB / 26.8GiB (syslog_usb3)
  • 11.20GiB (MRU: 12.52GiB, MFU: 1.00GiB) / 16.00GiB
  • Hit ratio -> 95.77% (higher is better)
  • Prefetch -> 43.46% (higher is better)
  • Hit MFU:MRU -> 96.05%:2.82% (higher ratio is better)
  • Hit MRU Ghost -> 0.15% (lower is better)
  • Hit MFU Ghost -> 0.05% (lower is better)

My FreeNAS box is the target for incremental drive-image backups from six drives in two Windows 7 PCs, written to the server on alternate days, plus a few hundred small .DOC and .LOG (text) files. The incremental drive-image files range in size from 200MB (not MiB) to 1.5GB.

I don't run any Jails or other non-native services on it.
 
R

redbull666

Cadet
Joined
Dec 2, 2016
Messages
4
Hi, new here :)

My stats, still looking at some ways to improve performance. For example, I feel 64 GB is not enough RAM for this node.

  • Data
  • 00:24:03 up 8 days, 13:04, 2 users, load average: 4.19, 1.62, 1.19
  • 30.5TiB / 382TiB (tank)
  • 32.00GiB (MRU: 1002.71MiB, MFU: 31.02GiB) /
  • Hit ratio -> 76.93% (higher is better)
  • Prefetch -> 2.48% (higher is better)
  • Hit MFU:MRU -> 28.64%:51.18% (higher ratio is better)
  • Hit MRU Ghost -> 2.51% (lower is better)
  • Hit MFU Ghost -> 1.22% (lower is better)
 
S

SRT1975

Cadet
Joined
Nov 18, 2016
Messages
2
Only recently rebooted, due to adding and configuring a UPS.
  • Movies, Training Material and VMware Storage
  • 7:32PM up 3:22, 1 user, load averages: 0.40, 0.30, 0.21
  • 632MiB / 15.9GiB (freenas-boot)
  • 4.95TiB / 10.9TiB (zpool1)
  • 4.34GiB (MRU: 11.17GiB, MFU: 11.17GiB) / 24.00GiB
  • Hit ratio -> 94.31% (higher is better)
  • Prefetch -> 0.17% (higher is better)
  • Hit MFU:MRU -> 86.28%:13.30% (higher ratio is better)
  • Hit MRU Ghost -> 0.00% (lower is better)
  • Hit MFU Ghost -> 0.00% (lower is better)
 
T

tvsjr

Guru
Joined
Aug 29, 2015
Messages
959
Recently rebooted due to system upgrades and reconfiguration.

  • Tier2 is a 12-drive array of striped mirrors for VM storage, Tier3 is a 6-drive RAIDZ2 array for media/photos/music/etc. Tier1 will be an SSD array, someday. Tier2 also has a 200GB L2ARC attached, although it doesn't see a ton of use. 128GB RAM in this box.
  • 9:29PM up 6 days, 20:46, 1 user, load averages: 0.21, 0.42, 0.27
  • 604GiB / 16.3TiB (Tier2)
  • 15.0TiB / 21.8TiB (Tier3)
  • 3.00GiB / 37.2GiB (freenas-boot)
  • 114.43GiB (MRU: 105.98GiB, MFU: 8.46GiB) / 128.00GiB
  • Hit ratio -> 91.63% (higher is better)
  • Prefetch -> 1.64% (higher is better)
  • Hit MFU:MRU -> 74.97%:24.93% (higher ratio is better)
  • Hit MRU Ghost -> 0.01% (lower is better)
  • Hit MFU Ghost -> 0.03% (lower is better)
 
L

logan893

Dabbler
Joined
Dec 31, 2015
Messages
44
  • Current uptime 48 days
  • 321GiB / 1.16TiB (games; 4x640GB WD Black in striped mirror; iSCSI for Blizzard games, SMB share for Steam and other games)
  • 2.64TiB / 3.62TiB (data; 2x4TB Seagate NAS in mirror; all things random including VM rsync backups, personal files and photos, active downloads)
  • 1.54TiB / 4.53TiB (ext_backup; 1x5TB Seagate external with copies=2 or =3; snapshot mirror for personal files and VM backups)
  • 1.20GiB / 15.9GiB (freenas-boot)
  • 9.41GiB (MRU: 6.85GiB, MFU: 2.56GiB) / 12.00GiB
  • Hit ratio -> 82.07% (higher is better)
  • Prefetch -> 5.87% (higher is better)
  • Hit MFU:MRU -> 96.25%:1.68% (higher ratio is better)
  • Hit MRU Ghost -> 0.24% (lower is better)
  • Hit MFU Ghost -> 0.52% (lower is better)

FreeNAS 9.10 running in a VM under ESXi 6. VM storage is hosted on a hardware RAID mirror, 2x Intel DC S3700 400GB in RAID 1 via LSI 2008 HBA.
Drives for FreeNAS are connected via a second LSI 2008 HBA, configured for passthrough (VT-d).
The FreeNAS VM is allocated 2 cores from a Xeon E3-1226v3, and 12 GB out of the maximum 32 GB RAM.
My gaming/workstation PC connects directly to the server via a Mellanox Connect-X2 10 Gbps fiber link, dedicated for FreeNAS communication, and using 9k jumbo frames on this isolated network segment.

I've also experimented with using an L2ARC on the "games" pool (24 GB virtual partition on the array of Intel DC S3700 SSDs). I posited that it may help if I play the same game several times in a row, or have to reload game assets where they would not all fit in my small ARC. I haven't played the same game frequently in a while, and even if I do, the L2ARC may not provide much gain at all. The hit ratio is abysmal at only 0.11%.

Still, I'm very happy with the FreeNAS performance, both for everyday activities and for game storage.

Code:
L2 ARC Summary: (HEALTHY)
        Passed Headroom:                        14.83m
        Tried Lock Failures:                    4.38m
        IO In Progress:                         10
        Low Memory Aborts:                      258
        Free on Write:                          23.82k
        Writes While Full:                      4.25k
        R/W Clashes:                            0
        Bad Checksums:                          0
        IO Errors:                              0
        SPA Mismatch:                           26.12b

L2 ARC Size: (Adaptive)                         29.38   GiB
        Header Size:                    0.07%   22.10   MiB

L2 ARC Evicts:
        Lock Retries:                           19
        Upon Reading:                           0

L2 ARC Breakdown:                               942.42m
        Hit Ratio:                      0.11%   1.04m
        Miss Ratio:                     99.89%  941.38m
        Feeds:                                  4.21m

L2 ARC Buffer:
        Bytes Scanned:                          487.33  TiB
        Buffer Iterations:                      4.21m
        List Iterations:                        16.83m
        NULL List Iterations:                   1.82k

L2 ARC Writes:
        Writes Sent:                    100.00% 41.97k
 
X

Xyrgh

Explorer
Joined
Apr 11, 2016
Messages
69
Pretty new here :smile: This is currently based in my office, awaiting a new house to be built and will be relocated sometime in June this year, hopefully then the CPU will get a workout.

Specs:

Supermicro X11SSL-F
Xeon E3-1230v5
32GB ECC
3 x M1015
Norco 4224

Running baremetal 9.10.2. Scratch pool is just a 4TB mirror for P2P, tank is my main storage, comprised of two by six 4TB disk vdevs in raidz2.

  • Media, photos, network storage for Windows documents folders.
  • 9:15AM up 34 days, 19:44, 1 user, load averages: 0.20, 0.20, 0.17
  • 2.53GiB / 111GiB (freenas-boot)
  • 35.4GiB / 109GiB (jails)
  • 2.71TiB / 3.62TiB (scratch)
  • 28.8TiB / 43.5TiB (tank)
  • 22.01GiB (MRU: 20.64GiB, MFU: 1.38GiB) / 32.00GiB
  • Hit ratio -> 86.04% (higher is better)
  • Prefetch -> 28.51% (higher is better)
  • Hit MFU:MRU -> 50.27%:47.10% (higher ratio is better)
  • Hit MRU Ghost -> 0.22% (lower is better)
  • Hit MFU Ghost -> 0.40% (lower is better)
 
RoboKaren

RoboKaren

Contributor
Joined
Apr 8, 2014
Messages
130
  • Media, video, etc.
  • 5:29AM up 14 days, 16:38, 1 user, load averages: 1.12, 1.17, 1.16
  • 1.13TiB / 3.62TiB (v1)
  • 6.23GiB / 14.5GiB (freenas-boot)
  • 13.4TiB / 21.8TiB (v2)
  • 16.18GiB (MRU: 14.48GiB, MFU: 1.71GiB) / 32.00GiB
  • Hit ratio -> 56.70% (higher is better)
  • Prefetch -> 18.90% (higher is better)
  • Hit MFU:MRU -> 45.59%:40.67% (higher ratio is better)
  • Hit MRU Ghost -> 0.45% (lower is better)
  • Hit MFU Ghost -> 4.07% (lower is better)

I think my hit ratio is really low as the main task for the past couple of days is that I'm running a Crashplan backup in a bhyve vm.
 
S

srit

Dabbler
Joined
Nov 29, 2011
Messages
23
  • VM Datastores, Backup, Media
  • 7:38PM up 22:07, 2 users, load averages: 0.98, 1.03, 1.04
  • 14.4TiB / 43.5TiB (Pool1)
  • 45.4GiB / 7.25TiB (Pool2)
  • 474GiB / 1.35TiB (Pool3)
  • 392GiB / 880GiB (VM_T1_SSD)
  • 437GiB / 1.08TiB (VM_T3_10K_HDD)
  • 1.89GiB / 14.9GiB (freenas-boot)
  • 24.85GiB (MRU: 10.41GiB, MFU: 14.44GiB) / 32.00GiB
  • Hit ratio -> 81.09% (higher is better)
  • Prefetch -> 26.26% (higher is better)
  • Hit MFU:MRU -> 66.05%:31.81% (higher ratio is better)
  • Hit MRU Ghost -> 3.25% (lower is better)
  • Hit MFU Ghost -> 2.71% (lower is better)

Probably could use more memory to speed things up, maybe then id see less latency in my ssd based vm datastore
 
Ericloewe

Ericloewe

Server Wrangler
Moderator
Joined
Feb 15, 2014
Messages
20,194
It should be mentioned that 90% ARC hit rates were something of a FreeNAS artifact due to a certain file being constantly constantly hit. We won't be seeing many of those from now on.
 
S

srit

Dabbler
Joined
Nov 29, 2011
Messages
23
Ericloewe said:
It should be mentioned that 90% ARC hit rates were something of a FreeNAS artifact due to a certain file being constantly constantly hit. We won't be seeing many of those from now on.
Click to expand...

Good to know. guess ill dig in elsewhere as to my issues. esxtop is pointing towards FreeNAS as being the culprit. But this is not the forum for that so :)
 
RoboKaren

RoboKaren

Contributor
Joined
Apr 8, 2014
Messages
130
Ericloewe said:
It should be mentioned that 90% ARC hit rates were something of a FreeNAS artifact due to a certain file being constantly constantly hit. We won't be seeing many of those from now on.
Click to expand...

Do we have a sense of what is a reasonable hit ratio is now?

Or to put it another way, what an unreasonably low hit ratio is -- so that we can be encouraged to buy more memory?
 
Ericloewe

Ericloewe

Server Wrangler
Moderator
Joined
Feb 15, 2014
Messages
20,194
RoboKaren said:
Do we have a sense of what is a reasonable hit ratio is now?

Or to put it another way, what an unreasonably low hit ratio is -- so that we can be encouraged to buy more memory?
Click to expand...
I don't quite have that feeling yet. I'd call my setup lightly loaded, so I'll post an update after I update to 9.10.2-U1 (or whatever version), when I have the time.
 
S

srit

Dabbler
Joined
Nov 29, 2011
Messages
23
Here are my updated Stats since upgrading my memory.
I have another thread going for an issue and was hoping someone could take a look and let me know their thoughts. https://forums.freenas.org/index.php?threads/disk-latency-in-mirrored-ssd-pool.49470/#post-345838

  • Put your data type(s) here...
  • 8:42AM up 7 days, 1:27, 1 user, load averages: 0.21, 0.12, 0.09
  • 14.9TiB / 43.5TiB (Pool1)
  • 1.08TiB / 7.25TiB (Pool2)
  • 549GiB / 1.80TiB (Pool3)
  • 437GiB / 880GiB (VM_T1_SSD)
  • 436GiB / 1.08TiB (VM_T3_10K_HDD)
  • 1.89GiB / 14.9GiB (freenas-boot)
  • 218.28GiB (MRU: 56.71GiB, MFU: 161.57GiB) / 256.00GiB
  • Hit ratio -> 87.65% (higher is better)
  • Prefetch -> 34.09% (higher is better)
  • Hit MFU:MRU -> 78.66%:18.99% (higher ratio is better)
  • Hit MRU Ghost -> 1.15% (lower is better)
  • Hit MFU Ghost -> 0.53% (lower is better)
 
You must log in or register to reply here.

Important Announcement for the TrueNAS Community.

The TrueNAS Community has now been moved. This forum will now become READ-ONLY for historical purposes. Please feel free to join us on the new TrueNAS Community Forums.

Related topics on forums.truenas.com for thread: "What are your ARC statistics?"

Similar threads

D
  • Locked
ARC performance issues on 10gbe NIC
Replies
17
Views
7K
Mlovelace
Mlovelace
R
  • Locked
Reasons of poor performance
Replies
7
Views
4K
SweetAndLow
SweetAndLow
Bidule0hm
  • Locked
ARC stats questions/problems thread
2 3 4
Replies
61
Views
66K
Bidule0hm
Bidule0hm
S
  • Locked
Disk Latency in mirrored ssd pool
2
Replies
22
Views
14K
srit
S
arvdsn
  • Locked
New hardware, slow transfer speeds
Replies
10
Views
4K
Dice
Dice
Top