Kevin Horton
Guru
- Joined
- Dec 2, 2015
- Messages
- 730
I'm pondering adding an SSD for L2ARC dedicated to metadata to hopefully speed up rsync. I use rsync to back up to two different pairs of striped disks for offsite backup (I also have two automatically replicated RAIDZ2 onsite backups). The data is mostly static, and rsync takes many hours to determine which files have changed since the last backup. @Constantin reported a very significant improvement in rsync perf, and I hope my use case is similar enough to get a worthwhile benefit.
Current HW:
Motherboard: Supermicro X10SRH-cF
CPU: Intel E5-1650v4
RAM: 32GB
Storage: main and backkup pools, each with 8 x 4TB WD Red, in RAIDZ2
Jails: Plex, homebridge and nextcloud, none of which are used heavily
Server use-case; static data storage, Apple Time Machine backup destination, run Plex, homebridge and Nextcloud servers
I'm looking for any recommendations on suitable SSDs for L2ARC. I'm also pondering whether I would need to buy more RAM. My current plan is to try the L2ARC with my current 32GB of RAM, and increase that to 64GB (or more) if I see signs that RAM has become a bottleneck.
Current HW:
Motherboard: Supermicro X10SRH-cF
CPU: Intel E5-1650v4
RAM: 32GB
Storage: main and backkup pools, each with 8 x 4TB WD Red, in RAIDZ2
Jails: Plex, homebridge and nextcloud, none of which are used heavily
Server use-case; static data storage, Apple Time Machine backup destination, run Plex, homebridge and Nextcloud servers
Code:
arc_summary.py System Memory: 1.00% 317.91 MiB Active, 14.24% 4.41 GiB Inact 81.79% 25.36 GiB Wired, 0.00% 0 Bytes Cache 2.97% 942.17 MiB Free, -0.00% -221184 Bytes Gap Real Installed: 32.00 GiB Real Available: 99.53% 31.85 GiB Real Managed: 97.34% 31.00 GiB Logical Total: 32.00 GiB Logical Used: 83.33% 26.67 GiB Logical Free: 16.67% 5.33 GiB Kernel Memory: 673.78 MiB Data: 93.72% 631.49 MiB Text: 6.28% 42.29 MiB Kernel Memory Map: 31.00 GiB Size: 7.71% 2.39 GiB Free: 92.29% 28.61 GiB Page: 1 ------------------------------------------------------------------------ ARC Summary: (HEALTHY) Storage pool Version: 5000 Filesystem Version: 5 Memory Throttle Count: 0 ARC Misc: Deleted: 29.96m Mutex Misses: 30.95k Evict Skips: 30.95k ARC Size: 56.24% 16.88 GiB Target Size: (Adaptive) 56.10% 16.83 GiB Min Size (Hard Limit): 12.50% 3.75 GiB Max Size (High Water): 8:1 30.00 GiB ARC Size Breakdown: Recently Used Cache Size: 44.81% 7.56 GiB Frequently Used Cache Size: 55.19% 9.31 GiB ARC Hash Breakdown: Elements Max: 2.25m Elements Current: 56.09% 1.26m Collisions: 14.41m Chain Max: 7 Chains: 156.31k Page: 2 ------------------------------------------------------------------------ ARC Total accesses: 2.43b Cache Hit Ratio: 94.82% 2.31b Cache Miss Ratio: 5.18% 126.04m Actual Hit Ratio: 94.58% 2.30b Data Demand Efficiency: 88.82% 51.34m Data Prefetch Efficiency: 2.56% 15.98m CACHE HITS BY CACHE LIST: Anonymously Used: 0.15% 3.55m Most Recently Used: 8.15% 188.15m Most Frequently Used: 91.59% 2.11b Most Recently Used Ghost: 0.09% 1.97m Most Frequently Used Ghost: 0.02% 461.47k CACHE HITS BY DATA TYPE: Demand Data: 1.98% 45.60m Prefetch Data: 0.02% 409.43k Demand Metadata: 97.59% 2.25b Prefetch Metadata: 0.42% 9.73m CACHE MISSES BY DATA TYPE: Demand Data: 4.55% 5.74m Prefetch Data: 12.35% 15.57m Demand Metadata: 79.42% 100.10m Prefetch Metadata: 3.67% 4.63m Page: 3 ------------------------------------------------------------------------ Page: 4 ------------------------------------------------------------------------ DMU Prefetch Efficiency: 516.21m Hit Ratio: 2.03% 10.50m Miss Ratio: 97.97% 505.71m Page: 5 ------------------------------------------------------------------------ Page: 6 ------------------------------------------------------------------------ ZFS Tunable (sysctl): kern.maxusers 2374 vm.kmem_size 33290891264 vm.kmem_size_scale 1 vm.kmem_size_min 0 vm.kmem_size_max 1319413950874 vfs.zfs.vol.immediate_write_sz 32768 vfs.zfs.vol.unmap_sync_enabled 0 vfs.zfs.vol.unmap_enabled 1 vfs.zfs.vol.recursive 0 vfs.zfs.vol.mode 2 vfs.zfs.sync_pass_rewrite 2 vfs.zfs.sync_pass_dont_compress 5 vfs.zfs.sync_pass_deferred_free 2 vfs.zfs.zio.dva_throttle_enabled 1 vfs.zfs.zio.exclude_metadata 0 vfs.zfs.zio.use_uma 1 vfs.zfs.zil_slog_bulk 786432 vfs.zfs.cache_flush_disable 0 vfs.zfs.zil_replay_disable 0 vfs.zfs.version.zpl 5 vfs.zfs.version.spa 5000 vfs.zfs.version.acl 1 vfs.zfs.version.ioctl 7 vfs.zfs.debug 0 vfs.zfs.super_owner 0 vfs.zfs.immediate_write_sz 32768 vfs.zfs.standard_sm_blksz 131072 vfs.zfs.dtl_sm_blksz 4096 vfs.zfs.min_auto_ashift 12 vfs.zfs.max_auto_ashift 13 vfs.zfs.vdev.queue_depth_pct 1000 vfs.zfs.vdev.write_gap_limit 4096 vfs.zfs.vdev.read_gap_limit 32768 vfs.zfs.vdev.aggregation_limit_non_rotating131072 vfs.zfs.vdev.aggregation_limit 1048576 vfs.zfs.vdev.trim_max_active 64 vfs.zfs.vdev.trim_min_active 1 vfs.zfs.vdev.scrub_max_active 2 vfs.zfs.vdev.scrub_min_active 1 vfs.zfs.vdev.async_write_max_active 10 vfs.zfs.vdev.async_write_min_active 1 vfs.zfs.vdev.async_read_max_active 3 vfs.zfs.vdev.async_read_min_active 1 vfs.zfs.vdev.sync_write_max_active 10 vfs.zfs.vdev.sync_write_min_active 10 vfs.zfs.vdev.sync_read_max_active 10 vfs.zfs.vdev.sync_read_min_active 10 vfs.zfs.vdev.max_active 1000 vfs.zfs.vdev.async_write_active_max_dirty_percent60 vfs.zfs.vdev.async_write_active_min_dirty_percent30 vfs.zfs.vdev.mirror.non_rotating_seek_inc1 vfs.zfs.vdev.mirror.non_rotating_inc 0 vfs.zfs.vdev.mirror.rotating_seek_offset1048576 vfs.zfs.vdev.mirror.rotating_seek_inc 5 vfs.zfs.vdev.mirror.rotating_inc 0 vfs.zfs.vdev.trim_on_init 1 vfs.zfs.vdev.bio_delete_disable 0 vfs.zfs.vdev.bio_flush_disable 0 vfs.zfs.vdev.cache.bshift 16 vfs.zfs.vdev.cache.size 0 vfs.zfs.vdev.cache.max 16384 vfs.zfs.vdev.default_ms_shift 29 vfs.zfs.vdev.min_ms_count 16 vfs.zfs.vdev.max_ms_count 200 vfs.zfs.vdev.trim_max_pending 10000 vfs.zfs.txg.timeout 5 vfs.zfs.trim.enabled 1 vfs.zfs.trim.max_interval 1 vfs.zfs.trim.timeout 30 vfs.zfs.trim.txg_delay 32 vfs.zfs.spa_min_slop 134217728 vfs.zfs.spa_slop_shift 5 vfs.zfs.spa_asize_inflation 24 vfs.zfs.deadman_enabled 1 vfs.zfs.deadman_checktime_ms 5000 vfs.zfs.deadman_synctime_ms 1000000 vfs.zfs.debug_flags 0 vfs.zfs.debugflags 0 vfs.zfs.recover 0 vfs.zfs.spa_load_verify_data 1 vfs.zfs.spa_load_verify_metadata 1 vfs.zfs.spa_load_verify_maxinflight 10000 vfs.zfs.max_missing_tvds_scan 0 vfs.zfs.max_missing_tvds_cachefile 2 vfs.zfs.max_missing_tvds 0 vfs.zfs.spa_load_print_vdev_tree 0 vfs.zfs.ccw_retry_interval 300 vfs.zfs.check_hostid 1 vfs.zfs.mg_fragmentation_threshold 85 vfs.zfs.mg_noalloc_threshold 0 vfs.zfs.condense_pct 200 vfs.zfs.metaslab_sm_blksz 4096 vfs.zfs.metaslab.bias_enabled 1 vfs.zfs.metaslab.lba_weighting_enabled 1 vfs.zfs.metaslab.fragmentation_factor_enabled1 vfs.zfs.metaslab.preload_enabled 1 vfs.zfs.metaslab.preload_limit 3 vfs.zfs.metaslab.unload_delay 8 vfs.zfs.metaslab.load_pct 50 vfs.zfs.metaslab.min_alloc_size 33554432 vfs.zfs.metaslab.df_free_pct 4 vfs.zfs.metaslab.df_alloc_threshold 131072 vfs.zfs.metaslab.debug_unload 0 vfs.zfs.metaslab.debug_load 0 vfs.zfs.metaslab.fragmentation_threshold70 vfs.zfs.metaslab.force_ganging 16777217 vfs.zfs.free_bpobj_enabled 1 vfs.zfs.free_max_blocks 18446744073709551615 vfs.zfs.zfs_scan_checkpoint_interval 7200 vfs.zfs.zfs_scan_legacy 0 vfs.zfs.no_scrub_prefetch 0 vfs.zfs.no_scrub_io 0 vfs.zfs.resilver_min_time_ms 3000 vfs.zfs.free_min_time_ms 1000 vfs.zfs.scan_min_time_ms 1000 vfs.zfs.scan_idle 50 vfs.zfs.scrub_delay 4 vfs.zfs.resilver_delay 2 vfs.zfs.top_maxinflight 32 vfs.zfs.delay_scale 500000 vfs.zfs.delay_min_dirty_percent 60 vfs.zfs.dirty_data_sync 67108864 vfs.zfs.dirty_data_max_percent 10 vfs.zfs.dirty_data_max_max 4294967296 vfs.zfs.dirty_data_max 3419890892 vfs.zfs.max_recordsize 1048576 vfs.zfs.default_ibs 15 vfs.zfs.default_bs 9 vfs.zfs.zfetch.array_rd_sz 1048576 vfs.zfs.zfetch.max_idistance 67108864 vfs.zfs.zfetch.max_distance 8388608 vfs.zfs.zfetch.min_sec_reap 2 vfs.zfs.zfetch.max_streams 8 vfs.zfs.prefetch_disable 0 vfs.zfs.send_holes_without_birth_time 1 vfs.zfs.mdcomp_disable 0 vfs.zfs.per_txg_dirty_frees_percent 30 vfs.zfs.nopwrite_enabled 1 vfs.zfs.dedup.prefetch 1 vfs.zfs.dbuf_cache_lowater_pct 10 vfs.zfs.dbuf_cache_hiwater_pct 10 vfs.zfs.dbuf_cache_shift 5 vfs.zfs.dbuf_cache_max_bytes 1006785920 vfs.zfs.arc_min_prescient_prefetch_ms 6 vfs.zfs.arc_min_prefetch_ms 1 vfs.zfs.l2c_only_size 0 vfs.zfs.mfu_ghost_data_esize 495983616 vfs.zfs.mfu_ghost_metadata_esize 4735586816 vfs.zfs.mfu_ghost_size 5231570432 vfs.zfs.mfu_data_esize 10333800448 vfs.zfs.mfu_metadata_esize 230963712 vfs.zfs.mfu_size 10951674368 vfs.zfs.mru_ghost_data_esize 418293248 vfs.zfs.mru_ghost_metadata_esize 12361377792 vfs.zfs.mru_ghost_size 12779671040 vfs.zfs.mru_data_esize 3002670080 vfs.zfs.mru_metadata_esize 426496 vfs.zfs.mru_size 5335509504 vfs.zfs.anon_data_esize 0 vfs.zfs.anon_metadata_esize 0 vfs.zfs.anon_size 132692992 vfs.zfs.l2arc_norw 1 vfs.zfs.l2arc_feed_again 1 vfs.zfs.l2arc_noprefetch 1 vfs.zfs.l2arc_feed_min_ms 200 vfs.zfs.l2arc_feed_secs 1 vfs.zfs.l2arc_headroom 2 vfs.zfs.l2arc_write_boost 8388608 vfs.zfs.l2arc_write_max 8388608 vfs.zfs.arc_meta_limit 8054287360 vfs.zfs.arc_free_target 173196 vfs.zfs.arc_kmem_cache_reap_retry_ms 1000 vfs.zfs.compressed_arc_enabled 1 vfs.zfs.arc_grow_retry 60 vfs.zfs.arc_shrink_shift 7 vfs.zfs.arc_average_blocksize 8192 vfs.zfs.arc_no_grow_shift 5 vfs.zfs.arc_min 4027143680 vfs.zfs.arc_max 32217149440 vfs.zfs.abd_chunk_size 4096 Page: 7 ------------------------------------------------------------------------
I'm looking for any recommendations on suitable SSDs for L2ARC. I'm also pondering whether I would need to buy more RAM. My current plan is to try the L2ARC with my current 32GB of RAM, and increase that to 64GB (or more) if I see signs that RAM has become a bottleneck.
Last edited: