Hi All,
I have a system running that is exporting the pool/datasets using NFS to some ESXi server, sinds about 3 weeks my pool has lost all possible perfomance, an i would like to know why, i have been looking at all the numbers of my system but am unable to point to the real reason in this performance degradation.
The system has 64GB Memory (max), 15 WD Red Pro (Pool of 14) 1HS and was always capable of keeping up.
The only this i suspect is that my ARC/L2ARC is under scaled and that my data usage pattern has change to much for wicth the Cache cannot keep up.
While looking for proof i was surprised at the output of acrstat and arc_summary the both report a 'ARC Size' of 175GB or 310% if i understand ARC correctly this could never be more than 100% or more than system memory? (64GB) even ARC+L2ARC combine this does not match.
Could anyone tell me if i am correct or wrong.
i include my stats and pool config below, i hope someone has any suggestions.
Thanks.
Richard
edit : better formatting Hardware details.
I have a system running that is exporting the pool/datasets using NFS to some ESXi server, sinds about 3 weeks my pool has lost all possible perfomance, an i would like to know why, i have been looking at all the numbers of my system but am unable to point to the real reason in this performance degradation.
The system has 64GB Memory (max), 15 WD Red Pro (Pool of 14) 1HS and was always capable of keeping up.
The only this i suspect is that my ARC/L2ARC is under scaled and that my data usage pattern has change to much for wicth the Cache cannot keep up.
While looking for proof i was surprised at the output of acrstat and arc_summary the both report a 'ARC Size' of 175GB or 310% if i understand ARC correctly this could never be more than 100% or more than system memory? (64GB) even ARC+L2ARC combine this does not match.
Could anyone tell me if i am correct or wrong.
i include my stats and pool config below, i hope someone has any suggestions.
Thanks.
Richard
Code:
^C[rvv@freenas01] /mnt/ZPOOL1% arc_summary.py System Memory: 0.19% 117.72 MiB Active, 1.28% 804.68 MiB Inact 42.31% 25.96 GiB Wired, 0.00% 0 Bytes Cache 56.22% 34.49 GiB Free, 0.00% 0 Bytes Gap Real Installed: 64.00 GiB Real Available: 98.38% 62.97 GiB Real Managed: 97.42% 61.34 GiB Logical Total: 64.00 GiB Logical Used: 44.89% 28.73 GiB Logical Free: 55.11% 35.27 GiB Kernel Memory: 493.42 MiB Data: 94.39% 465.74 MiB Text: 5.61% 27.68 MiB Kernel Memory Map: 78.71 GiB Size: 30.22% 23.79 GiB Free: 69.78% 54.92 GiB Page: 1 ------------------------------------------------------------------------ ARC Summary: (HEALTHY) Storage pool Version: 5000 Filesystem Version: 5 Memory Throttle Count: 0 ARC Misc: Deleted: 1.02b Mutex Misses: 2.65b Evict Skips: 2.65b ARC Size: 310.46% 175.94 GiB Target Size: (Adaptive) 100.00% 56.67 GiB Min Size (Hard Limit): 12.50% 7.08 GiB Max Size (High Water): 8:1 56.67 GiB ARC Size Breakdown: Recently Used Cache Size: 21.45% 37.73 GiB Frequently Used Cache Size: 78.55% 138.21 GiB ARC Hash Breakdown: Elements Max: 3.24m Elements Current: 43.79% 1.42m Collisions: 1.71b Chain Max: 8 Chains: 107.37k Page: 2 ------------------------------------------------------------------------ ARC Total accesses: 6.90b Cache Hit Ratio: 64.23% 4.43b Cache Miss Ratio: 35.77% 2.47b Actual Hit Ratio: 55.73% 3.85b Data Demand Efficiency: 71.59% 3.88b Data Prefetch Efficiency: 40.64% 1.40b CACHE HITS BY CACHE LIST: Most Recently Used: 81.87% 3.63b Most Frequently Used: 4.90% 217.12m Most Recently Used Ghost: 9.46% 419.37m Most Frequently Used Ghost: 24.04% 1.07b CACHE HITS BY DATA TYPE: Demand Data: 62.64% 2.78b Prefetch Data: 12.83% 568.99m Demand Metadata: 22.90% 1.02b Prefetch Metadata: 1.62% 71.99m CACHE MISSES BY DATA TYPE: Demand Data: 44.65% 1.10b Prefetch Data: 33.66% 831.17m Demand Metadata: 20.32% 501.83m Prefetch Metadata: 1.36% 33.68m Page: 3 ------------------------------------------------------------------------ L2 ARC Summary: (HEALTHY) Passed Headroom: 24.01m Tried Lock Failures: 19.97m IO In Progress: 3.68m Low Memory Aborts: 55 Free on Write: 8.64m Writes While Full: 664.28k R/W Clashes: 1.44k Bad Checksums: 0 IO Errors: 0 SPA Mismatch: 159.76m L2 ARC Size: (Adaptive) 65.85 GiB Header Size: 0.07% 45.81 MiB L2 ARC Evicts: Lock Retries: 72.02k Upon Reading: 6 L2 ARC Breakdown: 2.47b Hit Ratio: 25.97% 641.21m Miss Ratio: 74.03% 1.83b Feeds: 14.41m L2 ARC Buffer: Bytes Scanned: 1008.69 TiB Buffer Iterations: 14.41m List Iterations: 57.53m NULL List Iterations: 20.61m L2 ARC Writes: Writes Sent: 100.00% 11.20m Page: 4 ------------------------------------------------------------------------ DMU Prefetch Efficiency: 41.74b Hit Ratio: 3.12% 1.30b Miss Ratio: 96.88% 40.44b Page: 5 ------------------------------------------------------------------------ Page: 6 ------------------------------------------------------------------------ ZFS Tunable (sysctl): kern.maxusers 4365 vm.kmem_size 84511883264 vm.kmem_size_scale 1 vm.kmem_size_min 0 vm.kmem_size_max 1319413950874 vfs.zfs.vol.unmap_enabled 1 vfs.zfs.vol.mode 2 vfs.zfs.sync_pass_rewrite 2 vfs.zfs.sync_pass_dont_compress 5 vfs.zfs.sync_pass_deferred_free 2 vfs.zfs.zio.dva_throttle_enabled 1 vfs.zfs.zio.exclude_metadata 0 vfs.zfs.zio.use_uma 1 vfs.zfs.zil_slog_limit 786432 vfs.zfs.cache_flush_disable 0 vfs.zfs.zil_replay_disable 0 vfs.zfs.version.zpl 5 vfs.zfs.version.spa 5000 vfs.zfs.version.acl 1 vfs.zfs.version.ioctl 7 vfs.zfs.debug 0 vfs.zfs.super_owner 0 vfs.zfs.min_auto_ashift 9 vfs.zfs.max_auto_ashift 13 vfs.zfs.vdev.queue_depth_pct 1000 vfs.zfs.vdev.write_gap_limit 4096 vfs.zfs.vdev.read_gap_limit 32768 vfs.zfs.vdev.aggregation_limit 131072 vfs.zfs.vdev.trim_max_active 64 vfs.zfs.vdev.trim_min_active 1 vfs.zfs.vdev.scrub_max_active 2 vfs.zfs.vdev.scrub_min_active 1 vfs.zfs.vdev.async_write_max_active 10 vfs.zfs.vdev.async_write_min_active 1 vfs.zfs.vdev.async_read_max_active 3 vfs.zfs.vdev.async_read_min_active 1 vfs.zfs.vdev.sync_write_max_active 10 vfs.zfs.vdev.sync_write_min_active 10 vfs.zfs.vdev.sync_read_max_active 10 vfs.zfs.vdev.sync_read_min_active 10 vfs.zfs.vdev.max_active 1000 vfs.zfs.vdev.async_write_active_max_dirty_percent60 vfs.zfs.vdev.async_write_active_min_dirty_percent30 vfs.zfs.vdev.mirror.non_rotating_seek_inc1 vfs.zfs.vdev.mirror.non_rotating_inc 0 vfs.zfs.vdev.mirror.rotating_seek_offset1048576 vfs.zfs.vdev.mirror.rotating_seek_inc 5 vfs.zfs.vdev.mirror.rotating_inc 0 vfs.zfs.vdev.trim_on_init 1 vfs.zfs.vdev.larger_ashift_minimal 0 vfs.zfs.vdev.bio_delete_disable 0 vfs.zfs.vdev.bio_flush_disable 0 vfs.zfs.vdev.cache.bshift 16 vfs.zfs.vdev.cache.size 0 vfs.zfs.vdev.cache.max 16384 vfs.zfs.vdev.metaslabs_per_vdev 200 vfs.zfs.vdev.trim_max_pending 10000 vfs.zfs.txg.timeout 5 vfs.zfs.trim.enabled 1 vfs.zfs.trim.max_interval 1 vfs.zfs.trim.timeout 30 vfs.zfs.trim.txg_delay 32 vfs.zfs.space_map_blksz 4096 vfs.zfs.spa_min_slop 134217728 vfs.zfs.spa_slop_shift 5 vfs.zfs.spa_asize_inflation 24 vfs.zfs.deadman_enabled 1 vfs.zfs.deadman_checktime_ms 5000 vfs.zfs.deadman_synctime_ms 1000000 vfs.zfs.debug_flags 0 vfs.zfs.recover 0 vfs.zfs.spa_load_verify_data 1 vfs.zfs.spa_load_verify_metadata 1 vfs.zfs.spa_load_verify_maxinflight 10000 vfs.zfs.ccw_retry_interval 300 vfs.zfs.check_hostid 1 vfs.zfs.mg_fragmentation_threshold 85 vfs.zfs.mg_noalloc_threshold 0 vfs.zfs.condense_pct 200 vfs.zfs.metaslab.bias_enabled 1 vfs.zfs.metaslab.lba_weighting_enabled 1 vfs.zfs.metaslab.fragmentation_factor_enabled1 vfs.zfs.metaslab.preload_enabled 1 vfs.zfs.metaslab.preload_limit 3 vfs.zfs.metaslab.unload_delay 8 vfs.zfs.metaslab.load_pct 50 vfs.zfs.metaslab.min_alloc_size 33554432 vfs.zfs.metaslab.df_free_pct 4 vfs.zfs.metaslab.df_alloc_threshold 131072 vfs.zfs.metaslab.debug_unload 0 vfs.zfs.metaslab.debug_load 0 vfs.zfs.metaslab.fragmentation_threshold70 vfs.zfs.metaslab.gang_bang 16777217 vfs.zfs.free_bpobj_enabled 1 vfs.zfs.free_max_blocks 18446744073709551615 vfs.zfs.no_scrub_prefetch 0 vfs.zfs.no_scrub_io 0 vfs.zfs.resilver_min_time_ms 3000 vfs.zfs.free_min_time_ms 1000 vfs.zfs.scan_min_time_ms 1000 vfs.zfs.scan_idle 50 vfs.zfs.scrub_delay 4 vfs.zfs.resilver_delay 2 vfs.zfs.top_maxinflight 32 vfs.zfs.delay_scale 500000 vfs.zfs.delay_min_dirty_percent 60 vfs.zfs.dirty_data_sync 67108864 vfs.zfs.dirty_data_max_percent 10 vfs.zfs.dirty_data_max_max 4294967296 vfs.zfs.dirty_data_max 4294967296 vfs.zfs.max_recordsize 1048576 vfs.zfs.zfetch.array_rd_sz 1048576 vfs.zfs.zfetch.max_distance 33554432 vfs.zfs.zfetch.min_sec_reap 2 vfs.zfs.zfetch.max_streams 8 vfs.zfs.prefetch_disable 0 vfs.zfs.send_holes_without_birth_time 1 vfs.zfs.mdcomp_disable 0 vfs.zfs.nopwrite_enabled 1 vfs.zfs.dedup.prefetch 1 vfs.zfs.l2c_only_size 0 vfs.zfs.mfu_ghost_data_esize 74550784 vfs.zfs.mfu_ghost_metadata_esize 59585024 vfs.zfs.mfu_ghost_size 134135808 vfs.zfs.mfu_data_esize 524288 vfs.zfs.mfu_metadata_esize 0 vfs.zfs.mfu_size 37075968 vfs.zfs.mru_ghost_data_esize 51617144320 vfs.zfs.mru_ghost_metadata_esize 9061387264 vfs.zfs.mru_ghost_size 60678531584 vfs.zfs.mru_data_esize 0 vfs.zfs.mru_metadata_esize 0 vfs.zfs.mru_size 130000384 vfs.zfs.anon_data_esize 0 vfs.zfs.anon_metadata_esize 0 vfs.zfs.anon_size 66115072 vfs.zfs.l2arc_norw 0 vfs.zfs.l2arc_feed_again 1 vfs.zfs.l2arc_noprefetch 0 vfs.zfs.l2arc_feed_min_ms 200 vfs.zfs.l2arc_feed_secs 1 vfs.zfs.l2arc_headroom 2 vfs.zfs.l2arc_write_boost 40000000 vfs.zfs.l2arc_write_max 10000000 vfs.zfs.arc_meta_limit 15212138803 vfs.zfs.arc_free_target 111529 vfs.zfs.compressed_arc_enabled 1 vfs.zfs.arc_shrink_shift 7 vfs.zfs.arc_average_blocksize 8192 vfs.zfs.arc_min 7606069401 vfs.zfs.arc_max 60848555212 Page: 7 ------------------------------------------------------------------------
Code:
% zpool list NAME SIZE ALLOC FREE EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT ZPOOL1 25.2T 7.19T 18.1T - 46% 28% 1.00x ONLINE /mnt freenas-boot 14.4G 3.34G 11.0G - - 23% 1.00x ONLINE -
Code:
% zpool status pool: ZPOOL1 state: ONLINE status: Some supported features are not enabled on the pool. The pool can still be used, but some features are unavailable. action: Enable all features using 'zpool upgrade'. Once this is done, the pool may no longer be accessible by software that does not support the features. See zpool-features(7) for details. scan: scrub repaired 0 in 20h22m with 0 errors on Sun Jul 2 20:22:56 2017 config: NAME STATE READ WRITE CKSUM ZPOOL1 ONLINE 0 0 0 raidz3-0 ONLINE 0 0 0 gptid/32849948-7477-11e4-b39b-XXXXXXXXXXXX ONLINE 0 0 0 gptid/32f8aab4-7477-11e4-b39b-XXXXXXXXXXXX ONLINE 0 0 0 gptid/3e7b3950-7478-11e4-b39b-XXXXXXXXXXXX ONLINE 0 0 0 gptid/33f6198f-7477-11e4-b39b-XXXXXXXXXXXX ONLINE 0 0 0 gptid/346a842d-7477-11e4-b39b-XXXXXXXXXXXX ONLINE 0 0 0 gptid/34db0bc8-7477-11e4-b39b-XXXXXXXXXXXX ONLINE 0 0 0 gptid/3540e891-7477-11e4-b39b-XXXXXXXXXXXX ONLINE 0 0 0 raidz3-1 ONLINE 0 0 0 gptid/35b3cbb5-7477-11e4-b39b-XXXXXXXXXXXX ONLINE 0 0 0 gptid/361ae2cb-7477-11e4-b39b-XXXXXXXXXXXX ONLINE 0 0 0 gptid/3684dcf1-7477-11e4-b39b-XXXXXXXXXXXX ONLINE 0 0 0 gptid/36f1cdc1-7477-11e4-b39b-XXXXXXXXXXXX ONLINE 0 0 0 gptid/37622848-7477-11e4-b39b-XXXXXXXXXXXX ONLINE 0 0 0 gptid/37e0d7e1-7477-11e4-b39b-XXXXXXXXXXXX ONLINE 0 0 0 gptid/385fe0ae-7477-11e4-b39b-XXXXXXXXXXXX ONLINE 0 0 0 logs gptid/fa5e0edc-795f-11e4-9348-XXXXXXXXXXXX ONLINE 0 0 0 cache gptid/f49178e0-7477-11e4-b39b-XXXXXXXXXXXX ONLINE 0 0 0 spares gptid/874a11bd-8a2d-11e5-aa09-XXXXXXXXXXXX AVAIL errors: No known data error
edit : better formatting Hardware details.
Last edited:
