L2ARC device recommendation?

[Kevin Horton]

I'm pondering adding an SSD for L2ARC dedicated to metadata to hopefully speed up rsync. I use rsync to back up to two different pairs of striped disks for offsite backup (I also have two automatically replicated RAIDZ2 onsite backups). The data is mostly static, and rsync takes many hours to determine which files have changed since the last backup. @Constantin reported a very significant improvement in rsync perf, and I hope my use case is similar enough to get a worthwhile benefit.

Current HW:
Motherboard: Supermicro X10SRH-cF
CPU: Intel E5-1650v4
RAM: 32GB
Storage: main and backkup pools, each with 8 x 4TB WD Red, in RAIDZ2
Jails: Plex, homebridge and nextcloud, none of which are used heavily
Server use-case; static data storage, Apple Time Machine backup destination, run Plex, homebridge and Nextcloud servers

Code:

arc_summary.py
System Memory:

        1.00%   317.91  MiB Active,     14.24%  4.41    GiB Inact
        81.79%  25.36   GiB Wired,      0.00%   0       Bytes Cache
        2.97%   942.17  MiB Free,       -0.00%  -221184 Bytes Gap

        Real Installed:                         32.00   GiB
        Real Available:                 99.53%  31.85   GiB
        Real Managed:                   97.34%  31.00   GiB

        Logical Total:                          32.00   GiB
        Logical Used:                   83.33%  26.67   GiB
        Logical Free:                   16.67%  5.33    GiB

Kernel Memory:                                  673.78  MiB
        Data:                           93.72%  631.49  MiB
        Text:                           6.28%   42.29   MiB

Kernel Memory Map:                              31.00   GiB
        Size:                           7.71%   2.39    GiB
        Free:                           92.29%  28.61   GiB
                                                                Page:  1
------------------------------------------------------------------------

ARC Summary: (HEALTHY)
        Storage pool Version:                   5000
        Filesystem Version:                     5
        Memory Throttle Count:                  0

ARC Misc:
        Deleted:                                29.96m
        Mutex Misses:                           30.95k
        Evict Skips:                            30.95k

ARC Size:                               56.24%  16.88   GiB
        Target Size: (Adaptive)         56.10%  16.83   GiB
        Min Size (Hard Limit):          12.50%  3.75    GiB
        Max Size (High Water):          8:1     30.00   GiB

ARC Size Breakdown:
        Recently Used Cache Size:       44.81%  7.56    GiB
        Frequently Used Cache Size:     55.19%  9.31    GiB

ARC Hash Breakdown:
        Elements Max:                           2.25m
        Elements Current:               56.09%  1.26m
        Collisions:                             14.41m
        Chain Max:                              7
        Chains:                                 156.31k
                                                                Page:  2
------------------------------------------------------------------------

ARC Total accesses:                                     2.43b
        Cache Hit Ratio:                94.82%  2.31b
        Cache Miss Ratio:               5.18%   126.04m
        Actual Hit Ratio:               94.58%  2.30b

        Data Demand Efficiency:         88.82%  51.34m
        Data Prefetch Efficiency:       2.56%   15.98m

        CACHE HITS BY CACHE LIST:
          Anonymously Used:             0.15%   3.55m
          Most Recently Used:           8.15%   188.15m
          Most Frequently Used:         91.59%  2.11b
          Most Recently Used Ghost:     0.09%   1.97m
          Most Frequently Used Ghost:   0.02%   461.47k

        CACHE HITS BY DATA TYPE:
          Demand Data:                  1.98%   45.60m
          Prefetch Data:                0.02%   409.43k
          Demand Metadata:              97.59%  2.25b
          Prefetch Metadata:            0.42%   9.73m

        CACHE MISSES BY DATA TYPE:
          Demand Data:                  4.55%   5.74m
          Prefetch Data:                12.35%  15.57m
          Demand Metadata:              79.42%  100.10m
          Prefetch Metadata:            3.67%   4.63m
                                                                Page:  3
------------------------------------------------------------------------

                                                                Page:  4
------------------------------------------------------------------------

DMU Prefetch Efficiency:                        516.21m
        Hit Ratio:                      2.03%   10.50m
        Miss Ratio:                     97.97%  505.71m

                                                                Page:  5
------------------------------------------------------------------------

                                                                Page:  6
------------------------------------------------------------------------

ZFS Tunable (sysctl):
        kern.maxusers                           2374
        vm.kmem_size                            33290891264
        vm.kmem_size_scale                      1
        vm.kmem_size_min                        0
        vm.kmem_size_max                        1319413950874
        vfs.zfs.vol.immediate_write_sz          32768
        vfs.zfs.vol.unmap_sync_enabled          0
        vfs.zfs.vol.unmap_enabled               1
        vfs.zfs.vol.recursive                   0
        vfs.zfs.vol.mode                        2
        vfs.zfs.sync_pass_rewrite               2
        vfs.zfs.sync_pass_dont_compress         5
        vfs.zfs.sync_pass_deferred_free         2
        vfs.zfs.zio.dva_throttle_enabled        1
        vfs.zfs.zio.exclude_metadata            0
        vfs.zfs.zio.use_uma                     1
        vfs.zfs.zil_slog_bulk                   786432
        vfs.zfs.cache_flush_disable             0
        vfs.zfs.zil_replay_disable              0
        vfs.zfs.version.zpl                     5
        vfs.zfs.version.spa                     5000
        vfs.zfs.version.acl                     1
        vfs.zfs.version.ioctl                   7
        vfs.zfs.debug                           0
        vfs.zfs.super_owner                     0
        vfs.zfs.immediate_write_sz              32768
        vfs.zfs.standard_sm_blksz               131072
        vfs.zfs.dtl_sm_blksz                    4096
        vfs.zfs.min_auto_ashift                 12
        vfs.zfs.max_auto_ashift                 13
        vfs.zfs.vdev.queue_depth_pct            1000
        vfs.zfs.vdev.write_gap_limit            4096
        vfs.zfs.vdev.read_gap_limit             32768
        vfs.zfs.vdev.aggregation_limit_non_rotating131072
        vfs.zfs.vdev.aggregation_limit          1048576
        vfs.zfs.vdev.trim_max_active            64
        vfs.zfs.vdev.trim_min_active            1
        vfs.zfs.vdev.scrub_max_active           2
        vfs.zfs.vdev.scrub_min_active           1
        vfs.zfs.vdev.async_write_max_active     10
        vfs.zfs.vdev.async_write_min_active     1
        vfs.zfs.vdev.async_read_max_active      3
        vfs.zfs.vdev.async_read_min_active      1
        vfs.zfs.vdev.sync_write_max_active      10
        vfs.zfs.vdev.sync_write_min_active      10
        vfs.zfs.vdev.sync_read_max_active       10
        vfs.zfs.vdev.sync_read_min_active       10
        vfs.zfs.vdev.max_active                 1000
        vfs.zfs.vdev.async_write_active_max_dirty_percent60
        vfs.zfs.vdev.async_write_active_min_dirty_percent30
        vfs.zfs.vdev.mirror.non_rotating_seek_inc1
        vfs.zfs.vdev.mirror.non_rotating_inc    0
        vfs.zfs.vdev.mirror.rotating_seek_offset1048576
        vfs.zfs.vdev.mirror.rotating_seek_inc   5
        vfs.zfs.vdev.mirror.rotating_inc        0
        vfs.zfs.vdev.trim_on_init               1
        vfs.zfs.vdev.bio_delete_disable         0
        vfs.zfs.vdev.bio_flush_disable          0
        vfs.zfs.vdev.cache.bshift               16
        vfs.zfs.vdev.cache.size                 0
        vfs.zfs.vdev.cache.max                  16384
        vfs.zfs.vdev.default_ms_shift           29
        vfs.zfs.vdev.min_ms_count               16
        vfs.zfs.vdev.max_ms_count               200
        vfs.zfs.vdev.trim_max_pending           10000
        vfs.zfs.txg.timeout                     5
        vfs.zfs.trim.enabled                    1
        vfs.zfs.trim.max_interval               1
        vfs.zfs.trim.timeout                    30
        vfs.zfs.trim.txg_delay                  32
        vfs.zfs.spa_min_slop                    134217728
        vfs.zfs.spa_slop_shift                  5
        vfs.zfs.spa_asize_inflation             24
        vfs.zfs.deadman_enabled                 1
        vfs.zfs.deadman_checktime_ms            5000
        vfs.zfs.deadman_synctime_ms             1000000
        vfs.zfs.debug_flags                     0
        vfs.zfs.debugflags                      0
        vfs.zfs.recover                         0
        vfs.zfs.spa_load_verify_data            1
        vfs.zfs.spa_load_verify_metadata        1
        vfs.zfs.spa_load_verify_maxinflight     10000
        vfs.zfs.max_missing_tvds_scan           0
        vfs.zfs.max_missing_tvds_cachefile      2
        vfs.zfs.max_missing_tvds                0
        vfs.zfs.spa_load_print_vdev_tree        0
        vfs.zfs.ccw_retry_interval              300
        vfs.zfs.check_hostid                    1
        vfs.zfs.mg_fragmentation_threshold      85
        vfs.zfs.mg_noalloc_threshold            0
        vfs.zfs.condense_pct                    200
        vfs.zfs.metaslab_sm_blksz               4096
        vfs.zfs.metaslab.bias_enabled           1
        vfs.zfs.metaslab.lba_weighting_enabled  1
        vfs.zfs.metaslab.fragmentation_factor_enabled1
        vfs.zfs.metaslab.preload_enabled        1
        vfs.zfs.metaslab.preload_limit          3
        vfs.zfs.metaslab.unload_delay           8
        vfs.zfs.metaslab.load_pct               50
        vfs.zfs.metaslab.min_alloc_size         33554432
        vfs.zfs.metaslab.df_free_pct            4
        vfs.zfs.metaslab.df_alloc_threshold     131072
        vfs.zfs.metaslab.debug_unload           0
        vfs.zfs.metaslab.debug_load             0
        vfs.zfs.metaslab.fragmentation_threshold70
        vfs.zfs.metaslab.force_ganging          16777217
        vfs.zfs.free_bpobj_enabled              1
        vfs.zfs.free_max_blocks                 18446744073709551615
        vfs.zfs.zfs_scan_checkpoint_interval    7200
        vfs.zfs.zfs_scan_legacy                 0
        vfs.zfs.no_scrub_prefetch               0
        vfs.zfs.no_scrub_io                     0
        vfs.zfs.resilver_min_time_ms            3000
        vfs.zfs.free_min_time_ms                1000
        vfs.zfs.scan_min_time_ms                1000
        vfs.zfs.scan_idle                       50
        vfs.zfs.scrub_delay                     4
        vfs.zfs.resilver_delay                  2
        vfs.zfs.top_maxinflight                 32
        vfs.zfs.delay_scale                     500000
        vfs.zfs.delay_min_dirty_percent         60
        vfs.zfs.dirty_data_sync                 67108864
        vfs.zfs.dirty_data_max_percent          10
        vfs.zfs.dirty_data_max_max              4294967296
        vfs.zfs.dirty_data_max                  3419890892
        vfs.zfs.max_recordsize                  1048576
        vfs.zfs.default_ibs                     15
        vfs.zfs.default_bs                      9
        vfs.zfs.zfetch.array_rd_sz              1048576
        vfs.zfs.zfetch.max_idistance            67108864
        vfs.zfs.zfetch.max_distance             8388608
        vfs.zfs.zfetch.min_sec_reap             2
        vfs.zfs.zfetch.max_streams              8
        vfs.zfs.prefetch_disable                0
        vfs.zfs.send_holes_without_birth_time   1
        vfs.zfs.mdcomp_disable                  0
        vfs.zfs.per_txg_dirty_frees_percent     30
        vfs.zfs.nopwrite_enabled                1
        vfs.zfs.dedup.prefetch                  1
        vfs.zfs.dbuf_cache_lowater_pct          10
        vfs.zfs.dbuf_cache_hiwater_pct          10
        vfs.zfs.dbuf_cache_shift                5
        vfs.zfs.dbuf_cache_max_bytes            1006785920
        vfs.zfs.arc_min_prescient_prefetch_ms   6
        vfs.zfs.arc_min_prefetch_ms             1
        vfs.zfs.l2c_only_size                   0
        vfs.zfs.mfu_ghost_data_esize            495983616
        vfs.zfs.mfu_ghost_metadata_esize        4735586816
        vfs.zfs.mfu_ghost_size                  5231570432
        vfs.zfs.mfu_data_esize                  10333800448
        vfs.zfs.mfu_metadata_esize              230963712
        vfs.zfs.mfu_size                        10951674368
        vfs.zfs.mru_ghost_data_esize            418293248
        vfs.zfs.mru_ghost_metadata_esize        12361377792
        vfs.zfs.mru_ghost_size                  12779671040
        vfs.zfs.mru_data_esize                  3002670080
        vfs.zfs.mru_metadata_esize              426496
        vfs.zfs.mru_size                        5335509504
        vfs.zfs.anon_data_esize                 0
        vfs.zfs.anon_metadata_esize             0
        vfs.zfs.anon_size                       132692992
        vfs.zfs.l2arc_norw                      1
        vfs.zfs.l2arc_feed_again                1
        vfs.zfs.l2arc_noprefetch                1
        vfs.zfs.l2arc_feed_min_ms               200
        vfs.zfs.l2arc_feed_secs                 1
        vfs.zfs.l2arc_headroom                  2
        vfs.zfs.l2arc_write_boost               8388608
        vfs.zfs.l2arc_write_max                 8388608
        vfs.zfs.arc_meta_limit                  8054287360
        vfs.zfs.arc_free_target                 173196
        vfs.zfs.arc_kmem_cache_reap_retry_ms    1000
        vfs.zfs.compressed_arc_enabled          1
        vfs.zfs.arc_grow_retry                  60
        vfs.zfs.arc_shrink_shift                7
        vfs.zfs.arc_average_blocksize           8192
        vfs.zfs.arc_no_grow_shift               5
        vfs.zfs.arc_min                         4027143680
        vfs.zfs.arc_max                         32217149440
        vfs.zfs.abd_chunk_size                  4096
                                                                Page:  7
------------------------------------------------------------------------

I'm looking for any recommendations on suitable SSDs for L2ARC. I'm also pondering whether I would need to buy more RAM. My current plan is to try the L2ARC with my current 32GB of RAM, and increase that to 64GB (or more) if I see signs that RAM has become a bottleneck.

 

[Chris Moore]

rsync takes many hours to determine which files have changed since the last backup

Perhaps I missed it, but how much data do you have in the pool? I have a pool at work that has 311 (yes three hundred) TB of data and it only takes about half an hour to scan for changes. It has a 2TB NVMe SSD for the L2ARC. I can't say how much that influences the performance.

I'm also pondering whether I would need to buy more RAM.

The historical suggestion was to max the RAM before adding L2ARC.

 

[Kevin Horton]

Perhaps I missed it, but how much data do you have in the pool? I have a pool at work that has 311 (yes three hundred) TB of data and it only takes about half an hour to scan for changes. It has a 2TB NVMe SSD for the L2ARC. I can't say how much that influences the performance.

The historical suggestion was to max the RAM before adding L2ARC.

The pool that is backed up has about 6.5 TB of data (plus another 4 TB of snapshots which are not backed up by rsync).

This board can take up to 1 TB of RAM, if I read the spec sheet correctly. I certainly won't be buying that much RAM.

I would think that for any given dollar value investment to address my issue, there is some combination of RAM and SSD that provides the best performance.

I assume I'd want a PCIe SSD to get the lowest latency. Are there any other design features I should be looking for?

 

[HoneyBadger]

L2ARC doesn't have the same requirements as an SLOG device with regards to power-loss-protection and low-latency sync writes. While it certainly won't hurt, it's not required. You could start with a smaller, slower device and set secondarycache=metadata and see if that has an impact on your hits/misses.

Your arc results certainly show a heavy focus on metadata in both the hits and misses:

Code:

        CACHE HITS BY DATA TYPE:
          Demand Data:                  1.98%   45.60m
          Prefetch Data:                0.02%   409.43k
          Demand Metadata:              97.59%  2.25b
          Prefetch Metadata:            0.42%   9.73m

        CACHE MISSES BY DATA TYPE:
          Demand Data:                  4.55%   5.74m
          Prefetch Data:                12.35%  15.57m
          Demand Metadata:              79.42%  100.10m
          Prefetch Metadata:            3.67%   4.63m

However you're only using 17GB of the available 30GB max ARC ... is there that much pressure from your jails?

 

[Kevin Horton]

However you're only using 17GB of the available 30GB max ARC ... is there that much pressure from your jails?

The server was rebooted a couple of days ago, so maybe the ARC is still settling down. The server does not experience much load as it is mostly used as a repository for infrequently used, static data.

What is the best way to identify how much RAM is used by jails?

 

[HoneyBadger]

What is the best way to identify how much RAM is used by jails?

I found this rather neat-looking one-liner from the FreeBSD forums:

Code:

ps -eo %mem -J jailid | tail +2 | awk 'BEGIN{i=0}\
{i=i+$1}\
END{printf"%f%%\n",i}

Should give you a summary of the usage of the jail with ID jailid as a percentage of main memory.

 

[Constantin]

A couple of thoughts:

• My L2ARC is set for metadata only.
• I have a mix of files, ranging from many, random little bits and bobs to large video files. As can be imagined, the biggest impact of a L2ARC that caches metadata is on the rsync performance for large numbers of little itty bitty files.
• Currently, about 248.42 GiB of L2ARC covers my metadata needs. Total storage is around 10TB ATM, IIRC. That suggests a 40:1 ratio between the size of the L2ARC metadata-only cache vs. the size of the dataset (for my use case) If your data set consists only of small files, I'd expect the ratio would be lower, and vice versa.
• Since the fill rate of L2ARC cache is leisurely under default conditions (i.e. it takes a few trials before the L2ARC cache is filled completely), you will likely have to run at least 3 sets of backups before you see the full benefit.

My biggest worry is the impact of the link speed to the offsite backup. My main use case benefited a lot from a 10GBe connection to the NAS and a dedicated USB 3.0 connection to the DAS RAID Array. In other words, a lot of bandwidth, speed, etc. to minimize the time needed to rsync each file.

In the second set of backup time measurements I made, the link speed drops down to 1GBe, the recipient is a single 2.5" HDD is hanging off a USB2 port on a Airport Extreme base station. The backup time was likely cut more than 1/2 but it still took 108 minutes vs. 5 minutes for the iTunes folder. Granted, your recipient system will be much faster than a HDD hanging off a Airport AP, but the link speed issue may yet significantly impact total rsync time.

With a maximum 16TB of storage as described by the OP, I'd start with a 500GB or larger SSD for the L2ARC. Ideally the SSD would feature a fast read rate for random small files. Reliability is not important, as the L2ARC is flushed every time the system is rebooted and if there is an error, the system reverts to using the pool. HTH.

 

[Kevin Horton]

I found this rather neat-looking one-liner from the FreeBSD forums:

Code:

ps -eo %mem -J jailid | tail +2 | awk 'BEGIN{i=0}\
{i=i+$1}\
END{printf"%f%%\n",i}

Should give you a summary of the usage of the jail with ID jailid as a percentage of main memory.

Thanks! That gives 0.8%, 0.5% and 1.0% for the three jails that are currently running. If I understand correctly, that is a total of 2.3% of 32GB RAM used by jails, or about 754 MB used by the jails - effectively negligible.