Odd ARC Memory Usage Behavior in 13.0-U4

[Volts]

Uh, it is simpler to find ZFS version than that:
Code:
root@truenas[~]# zfs --version

zfs -V

You type all of those extra characters and call yourself a sysadmin. :shakes head:

(My inner Rube Goldberg-demon is also wondering if it would be possible to launch a VM of the old BE in fewer characters ...)

---

Can anybody speak to an actual performance change related to this? Cache hit ratios would *imply* a performance change, but primary measurements would be good too.

And has anybody tried pinning a larger ZFS arc_min? I don't think that's a great idea *in general* but it might be interesting to compare the behavior.

 

[winnielinnie]

zfs -V

Amateurs! The whole lot of you...

You make an alias of "z" for "zfs --version".

z

Done. Arthritis avoided.

 

[dak180]

what are you using to get that graph? I’d love to contribute to this.

I am using netdata (pay attention to the jail creation line it needs very special access to things).

 

[Arwen]

I figured it out.

This is a ruse to get everyone to switch to SCALE.

"What's this, you say? Core has better memory management over SCALE? Not anymore, suckers!"

But what if that ZFS change made SCALE even worse at ZFS ARC?

That would mean Core is still better at memory management for ZFS.

 

[dak180]

I run 4 servers with 1tb of ram, all running U2 currently. I upgraded on server to U4.

my U2 servers over the weekend, so things being idle, are all only at about 60-75% arc utilization.
my U4 is full, but vms also migrated back on.
running iscsi.
12 800gb sas ssds for l2 arc.

My systems most significant activity is indexing tasks that get kicked off every three hours.

My current guess based my observation of my system is during times of high activity the MRU section of arc shoots up and drags the MFU up with it, then as it goes idle the MRU loses the pressure keeping it up and as things age out the MFU also lets most of its capacity go because the background access pattern is very different and much lower impact compared to the indexing runs.

 

[winnielinnie]

Out of the blue I lost 21 GiB of ARC which became 21 GiB of "Free", on my system with 32 GiB of physical RAM.

In other words: 24 GiB ARC became 3 GiB ARC, and the missing 21 GiB is now "Free".

How is this an improvement from earlier versions of ZFS?

 

[Juan Manuel Palacios]

Four days after an unscheduled reboot, and regular usage day-in-and-day-out, and my NAS is consuming even less RAM for ARC, with only 6.3 GiB dedicated to it, 14.6 to "Services", and the remainder, the whole lot of 43.0 GiB, free!

I could swear that for a bit I saw the numbers for Free and ARC swapped, with ARC consuming upwards of 40 GiB and my free RAM progressively decreasing, but it could have been my tired eyes confusing the colors on TrueNAS dashboard's Memory widget.

Still, however, I can't say I'm experiencing any problems nor any kind of degraded performance.

What's everybody else seeing after a few days? Still the same pattern? It's making me regret having shelled out the money for that last DIMM of RAM, that amazing discount I got notwithstanding.

 

[Juan Manuel Palacios]

Not that I have noticed but with my persistent L2ARC that might mask things; @Juan Manuel Palacios, not having L2ARC would be in a better position to speak to this.


Having looked through the changes (very briefly) my impression is that ARC now goes much more out of its way to maintain a 1 to 1 ratio between MFU and MRU to the point of (what I would consider) premature evictions from one or the other in order to maintain a balance.

By premature I am defining that by there being free ram that is not being used for other things.

I really can't say I'm experiencing any problems or degraded performance in any way after installing 13.0-U4, other than the frustration of having bought and installed a new RAM DIMM just weeks before 13.0-U4 came out solely for the purpose of maximizing ARC. With 13.0-U3.1, ZFS did put that extra DIMM to good use, but now it's just sitting there twiddling its thumbs waiting for someone to throw it a bone.

Still, however, no real problems nor degraded performance, truth be told… at least that I've noticed. Networking is fine (virtualized pfSense router and UniFi Controller jail), TimeMachine backups (which might be the most common and intensive disk-bound I/O here) are also fine (though I can't really say I time them), my FAMP jails also working as usual, etc.

 

[Volts]

Another interesting change was just committed to ZFS:

More adaptive ARC eviction. by amotin · Pull Request #14359 · openzfs/zfs

Traditionally ARC adaptation was limited to MRU/MFU distribution. But for years people with metadata-centric workload demanded mechanisms to also manage data/metadata distribution, that in origina...

github.com

 

[awasb]

Experienced the same with TNC 13-U4. My test machine just collects backups from 5 to 7 client machines via SMB and NFS and does some replications. Three days of that "idling" with the usual backups resulted in around 25 to 30 GB adaptive target size. Before (including 13-U3.1) it was at maximum usage after 3 to 5 days. (Depending on backup sitze.)

Once I started iSCSI-services and did some copies, ARC RAM usage went up within mins to the 112GB max. Stopping iSCSI reverts things (after attributing the former "iSCSI" ARC RAM to services) slowly.

At the moment, just the aforementioned backups enabled, things look like this (again):

Code:

root@truenas ~ # arc_summary                                                                                                                               [0]

------------------------------------------------------------------------
ZFS Subsystem Report                            Fri Mar 10 19:17:15 2023
FreeBSD 13.1-RELEASE-p7                                    zpl version 5
Machine: truenas.fritz.box (amd64)                      spa version 5000

ARC status:                                                      HEALTHY
        Memory throttle count:                                         0

ARC size (current):                                    22.3 %   28.3 GiB
        Target size (adaptive):                        88.5 %  112.3 GiB
        Min size (hard limit):                          3.1 %    4.0 GiB
        Max size (high water):                           31:1  126.9 GiB
        Most Frequently Used (MFU) cache size:         82.3 %   22.7 GiB
        Most Recently Used (MRU) cache size:           17.7 %    4.9 GiB
        Metadata cache size (hard limit):              75.0 %   95.2 GiB
        Metadata cache size (current):                  1.7 %    1.6 GiB
        Dnode cache size (hard limit):                 10.0 %    9.5 GiB
        Dnode cache size (current):                     2.2 %  217.6 MiB

ARC hash breakdown:
        Elements max:                                               2.1M
        Elements current:                              27.2 %     569.8k
        Collisions:                                               263.1k
        Chain max:                                                     5
        Chains:                                                     9.6k

ARC misc:
        Deleted:                                                    8.1k
        Mutex misses:                                                255
        Eviction skips:                                             3.4M
        Eviction skips due to L2 writes:                               0
        L2 cached evictions:                                     0 Bytes
        L2 eligible evictions:                                   2.3 GiB
        L2 eligible MFU evictions:                     69.6 %    1.6 GiB
        L2 eligible MRU evictions:                     30.4 %  700.8 MiB
        L2 ineligible evictions:                                98.3 GiB

ARC total accesses (hits + misses):                               549.6M
        Cache hit ratio:                               99.8 %     548.5M
        Cache miss ratio:                               0.2 %       1.2M
        Actual hit ratio (MFU + MRU hits):             99.7 %     548.0M
        Data demand efficiency:                        98.2 %      12.5M
        Data prefetch efficiency:                      34.0 %     880.7k

Cache hits by cache type:
        Most frequently used (MFU):                    98.2 %     538.8M
        Most recently used (MRU):                       1.7 %       9.2M
        Most frequently used (MFU) ghost:             < 0.1 %     155.8k
        Most recently used (MRU) ghost:                 0.1 %     531.4k

Cache hits by data type:
        Demand data:                                    2.2 %      12.3M
        Prefetch data:                                  0.1 %     299.5k
        Demand metadata:                               92.5 %     507.2M
        Prefetch metadata:                              5.2 %      28.7M

Cache misses by data type:
        Demand data:                                   19.7 %     229.3k
        Prefetch data:                                 50.0 %     581.2k
        Demand metadata:                               11.6 %     135.4k
        Prefetch metadata:                             18.7 %     217.2k

DMU prefetch efficiency:                                            8.0M
        Hit ratio:                                     16.5 %       1.3M
        Miss ratio:                                    83.5 %       6.6M

L2ARC not detected, skipping section

Tunables:
        abd_scatter_enabled                                            1
        abd_scatter_min_size                                        4097
        allow_redacted_dataset_mount                                   0
        anon_data_esize                                                0
        anon_metadata_esize                                            0
        anon_size                                              161264128
        arc.average_blocksize                                       8192
        arc.dnode_limit                                                0
        arc.dnode_limit_percent                                       10
        arc.dnode_reduce_percent                                      10
        arc.evict_batch_limit                                         10
        arc.eviction_pct                                             200
        arc.grow_retry                                                 0
        arc.lotsfree_percent                                          10
        arc.max                                                        0
        arc.meta_adjust_restarts                                    4096
        arc.meta_limit                                                 0
        arc.meta_limit_percent                                        75
        arc.meta_min                                                   0
        arc.meta_prune                                             10000
        arc.meta_strategy                                              1
        arc.min                                                        0
        arc.min_prefetch_ms                                            0
        arc.min_prescient_prefetch_ms                                  0
        arc.p_dampener_disable                                         1
        arc.p_min_shift                                                0
        arc.pc_percent                                                 0
        arc.prune_task_threads                                         1
        arc.shrink_shift                                               0
        arc.sys_free                                                   0
        arc_free_target                                           695855
        arc_max                                                        0
        arc_min                                                        0
        arc_no_grow_shift                                              5
        async_block_max_blocks                      18446744073709551615
        autoimport_disable                                             1
        btree_verify_intensity                                         0
        ccw_retry_interval                                           300
        checksum_events_per_second                                    20
        commit_timeout_pct                                             5
        compressed_arc_enabled                                         1
        condense.indirect_commit_entry_delay_ms                        0
        condense.indirect_obsolete_pct                                25
        condense.indirect_vdevs_enable                                 1
        condense.max_obsolete_bytes                           1073741824
        condense.min_mapping_bytes                                131072
        condense_pct                                                 200
        crypt_sessions                                           1907683
        dbgmsg_enable                                                  1
        dbgmsg_maxsize                                           4194304
        dbuf.cache_shift                                               5
        dbuf.metadata_cache_max_bytes               18446744073709551615
        dbuf.metadata_cache_shift                                      6
        dbuf_cache.hiwater_pct                                        10
        dbuf_cache.lowater_pct                                        10
        dbuf_cache.max_bytes                        18446744073709551615
        dbuf_state_index                                               0
        ddt_data_is_special                                            1
        deadman.checktime_ms                                       60000
        deadman.enabled                                                1
        deadman.failmode                                            wait
        deadman.synctime_ms                                       600000
        deadman.ziotime_ms                                        300000
        debug                                                          0
        debugflags                                                     0
        dedup.prefetch                                                 0
        default_bs                                                     9
        default_ibs                                                   15
        delay_min_dirty_percent                                       60
        delay_scale                                               500000
        dirty_data_max                                        4294967296
        dirty_data_max_max                                    4294967296
        dirty_data_max_max_percent                                    25
        dirty_data_max_percent                                        10
        dirty_data_sync_percent                                       20
        disable_ivset_guid_check                                       0
        dmu_object_alloc_chunk_shift                                   7
        dmu_offset_next_sync                                           1
        dmu_prefetch_max                                       134217728
        dtl_sm_blksz                                                4096
        embedded_slog_min_ms                                          64
        flags                                                          0
        fletcher_4_impl [fastest] scalar superscalar superscalar4 sse2 ssse3
        free_bpobj_enabled                                             1
        free_leak_on_eio                                               0
        free_min_time_ms                                            1000
        history_output_max                                       1048576
        immediate_write_sz                                         32768
        initialize_chunk_size                                    1048576
        initialize_value                            16045690984833335022
        keep_log_spacemaps_at_export                                   0
        l2arc.exclude_special                                          0
        l2arc.feed_again                                               1
        l2arc.feed_min_ms                                            200
        l2arc.feed_secs                                                1
        l2arc.headroom                                                 2
        l2arc.headroom_boost                                         200
        l2arc.meta_percent                                            33
        l2arc.mfuonly                                                  0
        l2arc.noprefetch                                               1
        l2arc.norw                                                     0
        l2arc.rebuild_blocks_min_l2size                       1073741824
        l2arc.rebuild_enabled                                          0
        l2arc.trim_ahead                                               0
        l2arc.write_boost                                        8388608
        l2arc.write_max                                          8388608
        l2arc_feed_again                                               1
        l2arc_feed_min_ms                                            200
        l2arc_feed_secs                                                1
        l2arc_headroom                                                 2
        l2arc_noprefetch                                               1
        l2arc_norw                                                     0
        l2arc_write_boost                                        8388608
        l2arc_write_max                                          8388608
        l2c_only_size                                                  0
        livelist.condense.new_alloc                                    0
        livelist.condense.sync_cancel                                  0
        livelist.condense.sync_pause                                   0
        livelist.condense.zthr_cancel                                  0
        livelist.condense.zthr_pause                                   0
        livelist.max_entries                                      500000
        livelist.min_percent_shared                                   75
        lua.max_instrlimit                                     100000000
        lua.max_memlimit                                       104857600
        max_async_dedup_frees                                     100000
        max_auto_ashift                                               14
        max_dataset_nesting                                           50
        max_log_walking                                                5
        max_logsm_summary_length                                      10
        max_missing_tvds                                               0
        max_missing_tvds_cachefile                                     2
        max_missing_tvds_scan                                          0
        max_nvlist_src_size                                            0
        max_recordsize                                           1048576
        metaslab.aliquot                                         1048576
        metaslab.bias_enabled                                          1
        metaslab.debug_load                                            0
        metaslab.debug_unload                                          0
        metaslab.df_alloc_threshold                               131072
        metaslab.df_free_pct                                           4
        metaslab.df_max_search                                  16777216
        metaslab.df_use_largest_segment                                0
        metaslab.find_max_tries                                      100
        metaslab.force_ganging                                  16777217
        metaslab.fragmentation_factor_enabled                          1
        metaslab.fragmentation_threshold                              70
        metaslab.lba_weighting_enabled                                 1
        metaslab.load_pct                                             50
        metaslab.max_size_cache_sec                                 3600
        metaslab.mem_limit                                            25
        metaslab.preload_enabled                                       1
        metaslab.preload_limit                                        10
        metaslab.segment_weight_enabled                                1
        metaslab.sm_blksz_no_log                                   16384
        metaslab.sm_blksz_with_log                                131072
        metaslab.switch_threshold                                      2
        metaslab.try_hard_before_gang                                  0
        metaslab.unload_delay                                         32
        metaslab.unload_delay_ms                                  600000
        mfu_data_esize                                       19217815040
        mfu_ghost_data_esize                                 11147075072
        mfu_ghost_metadata_esize                              4231379968
        mfu_ghost_size                                       15378455040
        mfu_metadata_esize                                      99684352
        mfu_size                                             24380199936
        mg.fragmentation_threshold                                    95
        mg.noalloc_threshold                                           0
        min_auto_ashift                                                9
        min_metaslabs_to_flush                                         1
        mru_data_esize                                        2558390272
        mru_ghost_data_esize                                  6857117696
        mru_ghost_metadata_esize                               246057984
        mru_ghost_size                                        7103175680
        mru_metadata_esize                                     223361024
        mru_size                                              5258400256
        multihost.fail_intervals                                      10
        multihost.history                                              0
        multihost.import_intervals                                    20
        multihost.interval                                          1000
        multilist_num_sublists                                         0
        no_scrub_io                                                    0
        no_scrub_prefetch                                              0
        nocacheflush                                                   0
        nopwrite_enabled                                               1
        obsolete_min_time_ms                                         500
        pd_bytes_max                                            52428800
        per_txg_dirty_frees_percent                                   30
        prefetch.array_rd_sz                                     1048576
        prefetch.disable                                               0
        prefetch.max_distance                                   67108864
        prefetch.max_idistance                                  67108864
        prefetch.max_sec_reap                                          2
        prefetch.max_streams                                           8
        prefetch.min_distance                                    4194304
        prefetch.min_sec_reap                                          1
        read_history                                                   0
        read_history_hits                                              0
        rebuild_max_segment                                      1048576
        rebuild_scrub_enabled                                          1
        rebuild_vdev_limit                                      33554432
        reconstruct.indirect_combinations_max                       4096
        recover                                                        0
        recv.queue_ff                                                 20
        recv.queue_length                                       16777216
        recv.write_batch_size                                    1048576
        removal_suspend_progress                                       0
        remove_max_segment                                      16777216
        resilver_disable_defer                                         0
        resilver_min_time_ms                                        9000
        scan_blkstats                                                  0
        scan_checkpoint_intval                                      7200
        scan_fill_weight                                               3
        scan_ignore_errors                                             0
        scan_issue_strategy                                            0
        scan_legacy                                                    0
        scan_max_ext_gap                                         2097152
        scan_mem_lim_fact                                             20
        scan_mem_lim_soft_fact                                        20
        scan_strict_mem_lim                                            0
        scan_suspend_progress                                          0
        scan_vdev_limit                                          4194304
        scrub_min_time_ms                                           1000
        send.corrupt_data                                              0
        send.no_prefetch_queue_ff                                     20
        send.no_prefetch_queue_length                            1048576
        send.override_estimate_recordsize                              0
        send.queue_ff                                                 20
        send.queue_length                                       16777216
        send.unmodified_spill_blocks                                   1
        send_holes_without_birth_time                                  1
        slow_io_events_per_second                                     20
        spa.asize_inflation                                           24
        spa.discard_memory_limit                                16777216
        spa.load_print_vdev_tree                                       0
        spa.load_verify_data                                           1
        spa.load_verify_metadata                                       1
        spa.load_verify_shift                                          4
        spa.slop_shift                                                 5
        space_map_ibs                                                 14
        special_class_metadata_reserve_pct                            25
        standard_sm_blksz                                         131072
        super_owner                                                    0
        sync_pass_deferred_free                                        2
        sync_pass_dont_compress                                        8
        sync_pass_rewrite                                              2
        sync_taskq_batch_pct                                          75
        top_maxinflight                                             1000
        traverse_indirect_prefetch_limit                              32
        trim.extent_bytes_max                                  134217728
        trim.extent_bytes_min                                      32768
        trim.metaslab_skip                                             0
        trim.queue_limit                                              10
        trim.txg_batch                                                32
        txg.history                                                  100
        txg.timeout                                                    5
        unflushed_log_block_max                                   131072
        unflushed_log_block_min                                     1000
        unflushed_log_block_pct                                      400
        unflushed_log_txg_max                                       1000
        unflushed_max_mem_amt                                 1073741824
        unflushed_max_mem_ppm                                       1000
        user_indirect_is_special                                       1
        validate_skip                                                  0
        vdev.aggregate_trim                                            0
        vdev.aggregation_limit                                   1048576
        vdev.aggregation_limit_non_rotating                       131072
        vdev.async_read_max_active                                     3
        vdev.async_read_min_active                                     1
        vdev.async_write_active_max_dirty_percent                     60
        vdev.async_write_active_min_dirty_percent                     30
        vdev.async_write_max_active                                    5
        vdev.async_write_min_active                                    1
        vdev.bio_delete_disable                                        0
        vdev.bio_flush_disable                                         0
        vdev.cache_bshift                                             16
        vdev.cache_max                                             16384
        vdev.cache_size                                                0
        vdev.def_queue_depth                                          32
        vdev.default_ms_count                                        200
        vdev.default_ms_shift                                         29
        vdev.file.logical_ashift                                       9
        vdev.file.physical_ashift                                      9
        vdev.initializing_max_active                                   1
        vdev.initializing_min_active                                   1
        vdev.max_active                                             1000
        vdev.max_auto_ashift                                          14
        vdev.min_auto_ashift                                           9
        vdev.min_ms_count                                             16
        vdev.mirror.non_rotating_inc                                   0
        vdev.mirror.non_rotating_seek_inc                              1
        vdev.mirror.rotating_inc                                       0
        vdev.mirror.rotating_seek_inc                                  5
        vdev.mirror.rotating_seek_offset                         1048576
        vdev.ms_count_limit                                       131072
        vdev.nia_credit                                               10
        vdev.nia_delay                                                 2
        vdev.queue_depth_pct                                        1000
        vdev.read_gap_limit                                        32768
        vdev.rebuild_max_active                                        3
        vdev.rebuild_min_active                                        1
        vdev.removal_ignore_errors                                     0
        vdev.removal_max_active                                        2
        vdev.removal_max_span                                      32768
        vdev.removal_min_active                                        1
        vdev.removal_suspend_progress                                  0
        vdev.remove_max_segment                                 16777216
        vdev.scrub_max_active                                          8
        vdev.scrub_min_active                                          1
        vdev.sync_read_max_active                                     10
        vdev.sync_read_min_active                                     10
        vdev.sync_write_max_active                                    10
        vdev.sync_write_min_active                                    10
        vdev.trim_max_active                                           2
        vdev.trim_min_active                                           1
        vdev.validate_skip                                             0
        vdev.write_gap_limit                                        4096
        version.acl                                                    1
        version.ioctl                                                 15
        version.module                         v2023012500-zfs_9ef0b67f8
        version.spa                                                 5000
        version.zpl                                                    5
        vnops.read_chunk_size                                    1048576
        vol.mode                                                       2
        vol.recursive                                                  0
        vol.unmap_enabled                                              1
        wrlog_data_max                                        8589934592
        xattr_compat                                                   1
        zap_iterate_prefetch                                           1
        zevent.len_max                                               512
        zevent.retain_expire_secs                                    900
        zevent.retain_max                                           2000
        zfetch.max_distance                                     67108864
        zfetch.max_idistance                                    67108864
        zil.clean_taskq_maxalloc                                 1048576
        zil.clean_taskq_minalloc                                    1024
        zil.clean_taskq_nthr_pct                                     100
        zil.maxblocksize                                          131072
        zil.nocacheflush                                               0
        zil.replay_disable                                             0
        zil.slog_bulk                                             786432
        zio.deadman_log_all                                            0
        zio.dva_throttle_enabled                                       1
        zio.exclude_metadata                                           0
        zio.requeue_io_start_cut_in_line                               1
        zio.slow_io_ms                                             30000
        zio.taskq_batch_pct                                           80
        zio.taskq_batch_tpq                                            0
        zio.use_uma                                                    1

VDEV cache disabled, skipping section

ZIL committed transactions:                                         4.3M
        Commit requests:                                          493.4k
        Flushes to stable storage:                                474.7k
        Transactions to SLOG storage pool:          128.3 GiB       1.4M
        Transactions to non-SLOG storage pool:       35.4 MiB       5.2k

 

[Juan Manuel Palacios]

Another interesting change was just committed to ZFS:

More adaptive ARC eviction. by amotin · Pull Request #14359 · openzfs/zfs

Traditionally ARC adaptation was limited to MRU/MFU distribution. But for years people with metadata-centric workload demanded mechanisms to also manage data/metadata distribution, that in origina...

github.com

That definitely seems like an interesting change! Wonder how it'll impact ARC usage numbers...

 

[winnielinnie]

What a step backwards.

Not only is this new version of ZFS (and hence, TrueNAS Core 13.0-U4) leaving so much RAM to waste by keeping it "free"...

Not only is this affecting my metadata in the ARC...

But because of the above I'm experience drops in my performance. Rsync crawls and directory traversals are once again slow, because it must be pulled from spinning HDDs, instead of living in physical RAM.



I ask again: How is this an "improvement"? I know this has nothing to do with iXsystems, as this was a decision made upstream. But what was the ZFS developers' justification again?

To keep MRU-MFU "balanced"?

Okay, that's nice. Still not leveraging unused RAM and experiencing performance hits.

 

[Volts]

Are you seeing the same thing as above - the ARC target is what you expect, but the actual ARC size is smaller?

If you set vfs.zfs.arc_min, perhaps to something in between the current ARC size and the target, does that prevent the actual ARC size from collapsing below that value?

ARC shouldn't shrink without a reason. Perhaps it is being more sensitive to other memory pressure.

 

[winnielinnie]

Are you seeing the same thing as above - the ARC target is what you expect, but the actual ARC size is smaller?

With 32 GiB of RAM...

U3.1 and earlier:
My "target" stayed around 24 GiB. My ARC, likewise, remained around this amount.


U4:
My "target" sinks to comically low levels (around 5 GiB.) My ARC, likewise, remains around this level.

If I forcefully read large amounts of data from the pool, my ARC (and target) will grow, as expected. Yet, after some time, they drop back down.

This is also affecting my metadata in ARC (not just userdata).

If you set vfs.zfs.arc_min, perhaps to something in between the current ARC size and the target, does that prevent the actual ARC size from collapsing below that value?

I'm trying to avoid having to use duct-tape solutions where ZFS is supposed to be "smart" in managing memory.

I'll test this out, to see if it brings some relief. However, I'd rather not have to rely on enforcing a hard minimum ARC value, since it might risk potential instabilities if non-ARC RAM needs to temporarily "pressure away" some ARC.

 

[Volts]

I'm not suggesting that as a permanent solution!

arcstat -f time,c,size,l2size,grow,need,free,avail 1

I'm curious especially about grow, and also if avail matches your expectations. Just wondering if ZFS's perspective of the system matches other tools.

 

[morganL]

Another interesting change was just committed to ZFS:

More adaptive ARC eviction. by amotin · Pull Request #14359 · openzfs/zfs

Traditionally ARC adaptation was limited to MRU/MFU distribution. But for years people with metadata-centric workload demanded mechanisms to also manage data/metadata distribution, that in origina...

github.com

That's an iXsystems contribution.

Metadata caching is important on large systems.

 

[olddog9]

Uh, it is simpler to find ZFS version than that:

Code:

root@truenas[~]# zfs --version
zfs-2.1.6-1
zfs-kmod-2.1.6-1

The zpool command also works, but is 3 letters longer;

Code:

root@truenas[~]# zpool --version
zfs-2.1.6-1
zfs-kmod-2.1.6-1

Oh, no! I have outed myself as a Unix SysAdmin! Using a command that is 3 characters shorter to save typing! The sin, no one should know that. Now I must hide myself from the crowds of villagers that will want to burn me at the stake!

Arwen,

You made me laugh ! Too funny. Thanks, that was the first laugh of the morning.

 

[awasb]

Meanwhile at Villarriba ... grows again ... but slowly.

Code:

root@truenas ~ # arc_summary                                                                                                                               [0]

------------------------------------------------------------------------
ZFS Subsystem Report                            Sun Mar 12 13:52:14 2023
FreeBSD 13.1-RELEASE-p7                                    zpl version 5
Machine: truenas.fritz.box (amd64)                      spa version 5000

ARC status:                                                      HEALTHY
        Memory throttle count:                                         0

ARC size (current):                                    29.6 %   37.5 GiB
        Target size (adaptive):                        88.5 %  112.3 GiB
        Min size (hard limit):                          3.1 %    4.0 GiB
        Max size (high water):                           31:1  126.9 GiB
        Most Frequently Used (MFU) cache size:         83.5 %   30.9 GiB
        Most Recently Used (MRU) cache size:           16.5 %    6.1 GiB
        Metadata cache size (hard limit):              75.0 %   95.2 GiB
        Metadata cache size (current):                  3.2 %    3.0 GiB
        Dnode cache size (hard limit):                 10.0 %    9.5 GiB
        Dnode cache size (current):                     2.2 %  217.3 MiB

ARC hash breakdown:
        Elements max:                                               2.1M
        Elements current:                              30.9 %     647.4k
        Collisions:                                               305.9k
        Chain max:                                                     5
        Chains:                                                    12.4k

ARC misc:
        Deleted:                                                   15.1k
        Mutex misses:                                                255
        Eviction skips:                                             3.9M
        Eviction skips due to L2 writes:                               0
        L2 cached evictions:                                     0 Bytes
        L2 eligible evictions:                                   2.4 GiB
        L2 eligible MFU evictions:                     65.5 %    1.6 GiB
        L2 eligible MRU evictions:                     34.5 %  856.5 MiB
        L2 ineligible evictions:                                98.3 GiB

ARC total accesses (hits + misses):                               660.9M
        Cache hit ratio:                               99.8 %     659.5M
        Cache miss ratio:                               0.2 %       1.4M
        Actual hit ratio (MFU + MRU hits):             99.7 %     659.0M
        Data demand efficiency:                        98.3 %      14.6M
        Data prefetch efficiency:                      34.5 %     890.0k

Cache hits by cache type:
        Most frequently used (MFU):                    98.3 %     648.2M
        Most recently used (MRU):                       1.6 %      10.9M
        Most frequently used (MFU) ghost:             < 0.1 %     267.9k
        Most recently used (MRU) ghost:                 0.1 %     536.6k

Cache hits by data type:
        Demand data:                                    2.2 %      14.3M
        Prefetch data:                                < 0.1 %     307.0k
        Demand metadata:                               92.6 %     610.4M
        Prefetch metadata:                              5.2 %      34.5M

Cache misses by data type:
        Demand data:                                   17.9 %     248.3k
        Prefetch data:                                 42.1 %     583.0k
        Demand metadata:                               13.0 %     179.5k
        Prefetch metadata:                             27.1 %     375.1k

DMU prefetch efficiency:                                            9.2M
        Hit ratio:                                     15.5 %       1.4M
        Miss ratio:                                    84.5 %       7.8M

L2ARC not detected, skipping section

Tunables:
        abd_scatter_enabled                                            1
        abd_scatter_min_size                                        4097
        allow_redacted_dataset_mount                                   0
        anon_data_esize                                                0
        anon_metadata_esize                                            0
        anon_size                                                 529920
        arc.average_blocksize                                       8192
        arc.dnode_limit                                                0
        arc.dnode_limit_percent                                       10
        arc.dnode_reduce_percent                                      10
        arc.evict_batch_limit                                         10
        arc.eviction_pct                                             200
        arc.grow_retry                                                 0
        arc.lotsfree_percent                                          10
        arc.max                                                        0
        arc.meta_adjust_restarts                                    4096
        arc.meta_limit                                                 0
        arc.meta_limit_percent                                        75
        arc.meta_min                                                   0
        arc.meta_prune                                             10000
        arc.meta_strategy                                              1
        arc.min                                                        0
        arc.min_prefetch_ms                                            0
        arc.min_prescient_prefetch_ms                                  0
        arc.p_dampener_disable                                         1
        arc.p_min_shift                                                0
        arc.pc_percent                                                 0
        arc.prune_task_threads                                         1
        arc.shrink_shift                                               0
        arc.sys_free                                                   0
        arc_free_target                                           695855
        arc_max                                                        0
        arc_min                                                        0
        arc_no_grow_shift                                              5
        async_block_max_blocks                      18446744073709551615
        autoimport_disable                                             1
        btree_verify_intensity                                         0
        ccw_retry_interval                                           300
        checksum_events_per_second                                    20
        commit_timeout_pct                                             5
        compressed_arc_enabled                                         1
        condense.indirect_commit_entry_delay_ms                        0
        condense.indirect_obsolete_pct                                25
        condense.indirect_vdevs_enable                                 1
        condense.max_obsolete_bytes                           1073741824
        condense.min_mapping_bytes                                131072
        condense_pct                                                 200
        crypt_sessions                                           1928479
        dbgmsg_enable                                                  1
        dbgmsg_maxsize                                           4194304
        dbuf.cache_shift                                               5
        dbuf.metadata_cache_max_bytes               18446744073709551615
        dbuf.metadata_cache_shift                                      6
        dbuf_cache.hiwater_pct                                        10
        dbuf_cache.lowater_pct                                        10
        dbuf_cache.max_bytes                        18446744073709551615
        dbuf_state_index                                               0
        ddt_data_is_special                                            1
        deadman.checktime_ms                                       60000
        deadman.enabled                                                1
        deadman.failmode                                            wait
        deadman.synctime_ms                                       600000
        deadman.ziotime_ms                                        300000
        debug                                                          0
        debugflags                                                     0
        dedup.prefetch                                                 0
        default_bs                                                     9
        default_ibs                                                   15
        delay_min_dirty_percent                                       60
        delay_scale                                               500000
        dirty_data_max                                        4294967296
        dirty_data_max_max                                    4294967296
        dirty_data_max_max_percent                                    25
        dirty_data_max_percent                                        10
        dirty_data_sync_percent                                       20
        disable_ivset_guid_check                                       0
        dmu_object_alloc_chunk_shift                                   7
        dmu_offset_next_sync                                           1
        dmu_prefetch_max                                       134217728
        dtl_sm_blksz                                                4096
        embedded_slog_min_ms                                          64
        flags                                                          0
        fletcher_4_impl [fastest] scalar superscalar superscalar4 sse2 ssse3
        free_bpobj_enabled                                             1
        free_leak_on_eio                                               0
        free_min_time_ms                                            1000
        history_output_max                                       1048576
        immediate_write_sz                                         32768
        initialize_chunk_size                                    1048576
        initialize_value                            16045690984833335022
        keep_log_spacemaps_at_export                                   0
        l2arc.exclude_special                                          0
        l2arc.feed_again                                               1
        l2arc.feed_min_ms                                            200
        l2arc.feed_secs                                                1
        l2arc.headroom                                                 2
        l2arc.headroom_boost                                         200
        l2arc.meta_percent                                            33
        l2arc.mfuonly                                                  0
        l2arc.noprefetch                                               1
        l2arc.norw                                                     0
        l2arc.rebuild_blocks_min_l2size                       1073741824
        l2arc.rebuild_enabled                                          0
        l2arc.trim_ahead                                               0
        l2arc.write_boost                                        8388608
        l2arc.write_max                                          8388608
        l2arc_feed_again                                               1
        l2arc_feed_min_ms                                            200
        l2arc_feed_secs                                                1
        l2arc_headroom                                                 2
        l2arc_noprefetch                                               1
        l2arc_norw                                                     0
        l2arc_write_boost                                        8388608
        l2arc_write_max                                          8388608
        l2c_only_size                                                  0
        livelist.condense.new_alloc                                    0
        livelist.condense.sync_cancel                                  0
        livelist.condense.sync_pause                                   0
        livelist.condense.zthr_cancel                                  0
        livelist.condense.zthr_pause                                   0
        livelist.max_entries                                      500000
        livelist.min_percent_shared                                   75
        lua.max_instrlimit                                     100000000
        lua.max_memlimit                                       104857600
        max_async_dedup_frees                                     100000
        max_auto_ashift                                               14
        max_dataset_nesting                                           50
        max_log_walking                                                5
        max_logsm_summary_length                                      10
        max_missing_tvds                                               0
        max_missing_tvds_cachefile                                     2
        max_missing_tvds_scan                                          0
        max_nvlist_src_size                                            0
        max_recordsize                                           1048576
        metaslab.aliquot                                         1048576
        metaslab.bias_enabled                                          1
        metaslab.debug_load                                            0
        metaslab.debug_unload                                          0
        metaslab.df_alloc_threshold                               131072
        metaslab.df_free_pct                                           4
        metaslab.df_max_search                                  16777216
        metaslab.df_use_largest_segment                                0
        metaslab.find_max_tries                                      100
        metaslab.force_ganging                                  16777217
        metaslab.fragmentation_factor_enabled                          1
        metaslab.fragmentation_threshold                              70
        metaslab.lba_weighting_enabled                                 1
        metaslab.load_pct                                             50
        metaslab.max_size_cache_sec                                 3600
        metaslab.mem_limit                                            25
        metaslab.preload_enabled                                       1
        metaslab.preload_limit                                        10
        metaslab.segment_weight_enabled                                1
        metaslab.sm_blksz_no_log                                   16384
        metaslab.sm_blksz_with_log                                131072
        metaslab.switch_threshold                                      2
        metaslab.try_hard_before_gang                                  0
        metaslab.unload_delay                                         32
        metaslab.unload_delay_ms                                  600000
        mfu_data_esize                                       24935643136
        mfu_ghost_data_esize                                 10278222336
        mfu_ghost_metadata_esize                                14206464
        mfu_ghost_size                                       10292428800
        mfu_metadata_esize                                    1946552832
        mfu_size                                             33154842624
        mg.fragmentation_threshold                                    95
        mg.noalloc_threshold                                           0
        min_auto_ashift                                                9
        min_metaslabs_to_flush                                         1
        mru_data_esize                                        4853362688
        mru_ghost_data_esize                                  6226951168
        mru_ghost_metadata_esize                                70130688
        mru_ghost_size                                        6297081856
        mru_metadata_esize                                      64941056
        mru_size                                              6538654720
        multihost.fail_intervals                                      10
        multihost.history                                              0
        multihost.import_intervals                                    20
        multihost.interval                                          1000
        multilist_num_sublists                                         0
        no_scrub_io                                                    0
        no_scrub_prefetch                                              0
        nocacheflush                                                   0
        nopwrite_enabled                                               1
        obsolete_min_time_ms                                         500
        pd_bytes_max                                            52428800
        per_txg_dirty_frees_percent                                   30
        prefetch.array_rd_sz                                     1048576
        prefetch.disable                                               0
        prefetch.max_distance                                   67108864
        prefetch.max_idistance                                  67108864
        prefetch.max_sec_reap                                          2
        prefetch.max_streams                                           8
        prefetch.min_distance                                    4194304
        prefetch.min_sec_reap                                          1
        read_history                                                   0
        read_history_hits                                              0
        rebuild_max_segment                                      1048576
        rebuild_scrub_enabled                                          1
        rebuild_vdev_limit                                      33554432
        reconstruct.indirect_combinations_max                       4096
        recover                                                        0
        recv.queue_ff                                                 20
        recv.queue_length                                       16777216
        recv.write_batch_size                                    1048576
        removal_suspend_progress                                       0
        remove_max_segment                                      16777216
        resilver_disable_defer                                         0
        resilver_min_time_ms                                        3000
        scan_blkstats                                                  0
        scan_checkpoint_intval                                      7200
        scan_fill_weight                                               3
        scan_ignore_errors                                             0
        scan_issue_strategy                                            0
        scan_legacy                                                    0
        scan_max_ext_gap                                         2097152
        scan_mem_lim_fact                                             20
        scan_mem_lim_soft_fact                                        20
        scan_strict_mem_lim                                            0
        scan_suspend_progress                                          0
        scan_vdev_limit                                          4194304
        scrub_min_time_ms                                           1000
        send.corrupt_data                                              0
        send.no_prefetch_queue_ff                                     20
        send.no_prefetch_queue_length                            1048576
        send.override_estimate_recordsize                              0
        send.queue_ff                                                 20
        send.queue_length                                       16777216
        send.unmodified_spill_blocks                                   1
        send_holes_without_birth_time                                  1
        slow_io_events_per_second                                     20
        spa.asize_inflation                                           24
        spa.discard_memory_limit                                16777216
        spa.load_print_vdev_tree                                       0
        spa.load_verify_data                                           1
        spa.load_verify_metadata                                       1
        spa.load_verify_shift                                          4
        spa.slop_shift                                                 5
        space_map_ibs                                                 14
        special_class_metadata_reserve_pct                            25
        standard_sm_blksz                                         131072
        super_owner                                                    0
        sync_pass_deferred_free                                        2
        sync_pass_dont_compress                                        8
        sync_pass_rewrite                                              2
        sync_taskq_batch_pct                                          75
        top_maxinflight                                             1000
        traverse_indirect_prefetch_limit                              32
        trim.extent_bytes_max                                  134217728
        trim.extent_bytes_min                                      32768
        trim.metaslab_skip                                             0
        trim.queue_limit                                              10
        trim.txg_batch                                                32
        txg.history                                                  100
        txg.timeout                                                    5
        unflushed_log_block_max                                   131072
        unflushed_log_block_min                                     1000
        unflushed_log_block_pct                                      400
        unflushed_log_txg_max                                       1000
        unflushed_max_mem_amt                                 1073741824
        unflushed_max_mem_ppm                                       1000
        user_indirect_is_special                                       1
        validate_skip                                                  0
        vdev.aggregate_trim                                            0
        vdev.aggregation_limit                                   1048576
        vdev.aggregation_limit_non_rotating                       131072
        vdev.async_read_max_active                                     3
        vdev.async_read_min_active                                     1
        vdev.async_write_active_max_dirty_percent                     60
        vdev.async_write_active_min_dirty_percent                     30
        vdev.async_write_max_active                                    5
        vdev.async_write_min_active                                    1
        vdev.bio_delete_disable                                        0
        vdev.bio_flush_disable                                         0
        vdev.cache_bshift                                             16
        vdev.cache_max                                             16384
        vdev.cache_size                                                0
        vdev.def_queue_depth                                          32
        vdev.default_ms_count                                        200
        vdev.default_ms_shift                                         29
        vdev.file.logical_ashift                                       9
        vdev.file.physical_ashift                                      9
        vdev.initializing_max_active                                   1
        vdev.initializing_min_active                                   1
        vdev.max_active                                             1000
        vdev.max_auto_ashift                                          14
        vdev.min_auto_ashift                                           9
        vdev.min_ms_count                                             16
        vdev.mirror.non_rotating_inc                                   0
        vdev.mirror.non_rotating_seek_inc                              1
        vdev.mirror.rotating_inc                                       0
        vdev.mirror.rotating_seek_inc                                  5
        vdev.mirror.rotating_seek_offset                         1048576
        vdev.ms_count_limit                                       131072
        vdev.nia_credit                                                5
        vdev.nia_delay                                                 5
        vdev.queue_depth_pct                                        1000
        vdev.read_gap_limit                                        32768
        vdev.rebuild_max_active                                        3
        vdev.rebuild_min_active                                        1
        vdev.removal_ignore_errors                                     0
        vdev.removal_max_active                                        2
        vdev.removal_max_span                                      32768
        vdev.removal_min_active                                        1
        vdev.removal_suspend_progress                                  0
        vdev.remove_max_segment                                 16777216
        vdev.scrub_max_active                                          3
        vdev.scrub_min_active                                          1
        vdev.sync_read_max_active                                     10
        vdev.sync_read_min_active                                     10
        vdev.sync_write_max_active                                    10
        vdev.sync_write_min_active                                    10
        vdev.trim_max_active                                           2
        vdev.trim_min_active                                           1
        vdev.validate_skip                                             0
        vdev.write_gap_limit                                        4096
        version.acl                                                    1
        version.ioctl                                                 15
        version.module                         v2023012500-zfs_9ef0b67f8
        version.spa                                                 5000
        version.zpl                                                    5
        vnops.read_chunk_size                                    1048576
        vol.mode                                                       2
        vol.recursive                                                  0
        vol.unmap_enabled                                              1
        wrlog_data_max                                        8589934592
        xattr_compat                                                   1
        zap_iterate_prefetch                                           1
        zevent.len_max                                               512
        zevent.retain_expire_secs                                    900
        zevent.retain_max                                           2000
        zfetch.max_distance                                     67108864
        zfetch.max_idistance                                    67108864
        zil.clean_taskq_maxalloc                                 1048576
        zil.clean_taskq_minalloc                                    1024
        zil.clean_taskq_nthr_pct                                     100
        zil.maxblocksize                                          131072
        zil.nocacheflush                                               0
        zil.replay_disable                                             0
        zil.slog_bulk                                             786432
        zio.deadman_log_all                                            0
        zio.dva_throttle_enabled                                       1
        zio.exclude_metadata                                           0
        zio.requeue_io_start_cut_in_line                               1
        zio.slow_io_ms                                             30000
        zio.taskq_batch_pct                                           80
        zio.taskq_batch_tpq                                            0
        zio.use_uma                                                    1

VDEV cache disabled, skipping section

ZIL committed transactions:                                         4.9M
        Commit requests:                                          684.9k
        Flushes to stable storage:                                659.3k
        Transactions to SLOG storage pool:          136.9 GiB       1.6M
        Transactions to non-SLOG storage pool:       42.7 MiB       6.4k

 

[Volts]

That's an iXsystems contribution.

There are so many real world refinements to ZFS and FreeBSD from iXsystems. Hugely appreciated.

 

[dak180]

Metadata caching is important on large systems.

@morganL so, bringing me back to my original question: is this intended behavior or should I make a bug report? If I make a bug report I can attach a netdata (fully instrumented including network traffic levels and smart monitoring of disks) snapshot file covering anytime from several months before the upgrade to present day at whatever resolution you would care for.