cache vs slog

[Bane]

Hi all,

I've been looking at cache and slog. Can someone explain the benefits of each? I'm wondering if I should do one or the other or both on my TrueNAS. Both improve performance, but in different ways. For some background, I have a RAIDZ2 composed of 5, 10 TB HGST drives plus a hot spare. The TrueNAS is on a 10 GB network with two 10Gbe NIC's in a LACP configuration. I use the NAS mainly as a true file store with some streaming with Plex.

Thanks,

 

[sretalla]

I have a RAIDZ2

SLOG is already ruled out... if you're doing block storage (although it seems you're not), you need to read this before considering SLOG: https://www.truenas.com/community/threads/the-path-to-success-for-block-storage.81165/

SLOG will only help to make sync writes not go a lot slower... they will still be slower than async writes.

If by Cache, you mean L2ARC... it depends... do you gave 64GB of RAM or more?

Is arc_summary telling you your hit ratio is too low? (is your working set larger than ARC?)

 

[Bane]

SLOG is already ruled out... if you're doing block storage (although it seems you're not), you need to read this before considering SLOG: https://www.truenas.com/community/threads/the-path-to-success-for-block-storage.81165/

SLOG will only help to make sync writes not go a lot slower... they will still be slower than async writes.

If by Cache, you mean L2ARC... it depends... do you gave 64GB of RAM or more?

Is arc_summary telling you your hit ratio is too low? (is your working set larger than ARC?)

I have 16gb RAM. The option I was looking at is the option for adding a cache drive to the pool. I would use an SSD for the cache. That post was interesting, but didn't really clear up whether or not there would be a benefit to adding a cache SSD to the pool.

 

[sretalla]

didn't really clear up whether or not there would be a benefit to adding a cache SSD to the pool.

If by Cache, you mean L2ARC... it depends... do you gave 64GB of RAM or more?

Is arc_summary telling you your hit ratio is too low? (is your working set larger than ARC?)

L2ARC will only help if you're accessing data that can't be held in RAM (and you have enough RAM to make storing stuff there interesting.

Recommendations from the forum are to focus on adding RAM up to 64GB before messing around with L2ARC (what you're calling cache), since the tables for content in L2ARC are stealing RAM from ARC, which is much (hundreds of times) faster.

As I already recommended, look at arc_summary before you waste money on anything.

You're looking for this section at the top of the output which will tell you if it's worth it or not:

Code:

ARC total accesses (hits + misses):                                67.9G
        Cache hit ratio:                               99.9 %      67.8G
        Cache miss ratio:                               0.1 %      98.3M
        Actual hit ratio (MFU + MRU hits):             99.8 %      67.8G
        Data demand efficiency:                        99.9 %      32.6G
        Data prefetch efficiency:                       9.8 %      31.5M

In my case, that's nearly 68 billion requests where ARC has an opportunity to serve content instead of going back to disk (ARC hit).

in only about 100 million cases, the answer was, "It's not here" (0.1%) (ARC Miss)

Those ARC Miss cases are a chance where L2ARC might have held that content.

L2ARC will potentially slow down your system and waste (burn out early... like in a year or so, depending on how busy your pool is) a cheap SSD as it will write a lot of data as you read or write from the pool and then just chuck it out without even using it a lot of the time (because ARC already holds the most likely data to hit).

 

[Bane]

L2ARC will only help if you're accessing data that can't be held in RAM (and you have enough RAM to make storing stuff there interesting.

Recommendations from the forum are to focus on adding RAM up to 64GB before messing around with L2ARC (what you're calling cache), since the tables for content in L2ARC are stealing RAM from ARC, which is much (hundreds of times) faster.

As I already recommended, look at arc_summary before you waste money on anything.

I found information on arc_summary.py, but it doesn't seem to exist on my TrueNAS. Is there another way to run it un TrueNAS 12?

Never mind, it appears they exist without the .py extension now. My summary is below. What should I be looking at here?

Code:

------------------------------------------------------------------------
ZFS Subsystem Report                            Wed Apr 27 10:33:04 2022
FreeBSD 12.2-RELEASE-p14                                   zpl version 5
Machine: GamakBase.matrixinfosec.com (amd64)            spa version 5000

ARC status:                                                      HEALTHY
        Memory throttle count:                                         0

ARC size (current):                                    77.4 %   11.6 GiB
        Target size (adaptive):                        77.6 %   11.6 GiB
        Min size (hard limit):                          3.3 %  509.6 MiB
        Max size (high water):                           29:1   14.9 GiB
        Most Frequently Used (MFU) cache size:          1.2 %  129.4 MiB
        Most Recently Used (MRU) cache size:           98.8 %   10.5 GiB
        Metadata cache size (hard limit):              75.0 %   11.2 GiB
        Metadata cache size (current):                 14.4 %    1.6 GiB
        Dnode cache size (hard limit):                 10.0 %    1.1 GiB
        Dnode cache size (current):                    39.3 %  450.7 MiB

ARC hash breakdown:
        Elements max:                                             509.0k
        Elements current:                              43.6 %     221.7k
        Collisions:                                                 7.7M
        Chain max:                                                     5
        Chains:                                                    11.0k

ARC misc:
        Deleted:                                                  128.1M
        Mutex misses:                                                342
        Eviction skips:                                           149.2k

ARC total accesses (hits + misses):                               261.0M
        Cache hit ratio:                               99.6 %     259.8M
        Cache miss ratio:                               0.4 %       1.1M
        Actual hit ratio (MFU + MRU hits):             99.6 %     259.8M
        Data demand efficiency:                        89.2 %       1.4M
        Data prefetch efficiency:                       0.1 %     736.6k

Cache hits by cache type:
        Most frequently used (MFU):                    97.8 %     254.2M
        Most recently used (MRU):                       2.2 %       5.6M
        Most frequently used (MFU) ghost:             < 0.1 %      81.4k
        Most recently used (MRU) ghost:               < 0.1 %     118.6k

Cache hits by data type:
        Demand data:                                    0.5 %       1.3M
        Demand prefetch data:                         < 0.1 %        650
        Demand metadata:                               99.5 %     258.5M
        Demand prefetch metadata:                     < 0.1 %      38.4k

Cache misses by data type:
        Demand data:                                   13.9 %     156.9k
        Demand prefetch data:                          65.1 %     736.0k
        Demand metadata:                               18.9 %     213.7k
        Demand prefetch metadata:                       2.2 %      24.9k

DMU prefetch efficiency:                                           18.5M
        Hit ratio:                                      1.7 %     322.6k
        Miss ratio:                                    98.3 %      18.1M

L2ARC not detected, skipping section

Tunables:
        abd_chunk_size                                              4096
        abd_scatter_enabled                                            1
        allow_redacted_dataset_mount                                   0
        anon_data_esize                                                0
        anon_metadata_esize                                            0
        anon_size                                                1644032
        arc.average_blocksize                                       8192
        arc.dnode_limit                                                0
        arc.dnode_limit_percent                                       10
        arc.dnode_reduce_percent                                      10
        arc.evict_batch_limit                                         10
        arc.eviction_pct                                             200
        arc.grow_retry                                                 0
        arc.lotsfree_percent                                          10
        arc.max                                                        0
        arc.meta_adjust_restarts                                    4096
        arc.meta_limit                                                 0
        arc.meta_limit_percent                                        75
        arc.meta_min                                                   0
        arc.meta_prune                                             10000
        arc.meta_strategy                                              1
        arc.min                                                        0
        arc.min_prefetch_ms                                            0
        arc.min_prescient_prefetch_ms                                  0
        arc.p_dampener_disable                                         1
        arc.p_min_shift                                                0
        arc.pc_percent                                                 0
        arc.prune_task_threads                                         1
        arc.shrink_shift                                               0
        arc.sys_free                                                   0
        arc_free_target                                            86655
        arc_max                                                        0
        arc_min                                                        0
        arc_no_grow_shift                                              5
        async_block_max_blocks                      18446744073709551615
        autoimport_disable                                             1
        ccw_retry_interval                                           300
        checksum_events_per_second                                    20
        commit_timeout_pct                                             5
        compressed_arc_enabled                                         1
        condense.indirect_commit_entry_delay_ms                        0
        condense.indirect_obsolete_pct                                25
        condense.indirect_vdevs_enable                                 1
        condense.max_obsolete_bytes                           1073741824
        condense.min_mapping_bytes                                131072
        condense_pct                                                 200
        crypt_sessions                                              2589
        dbgmsg_enable                                                  1
        dbgmsg_maxsize                                           4194304
        dbuf.cache_shift                                               5
        dbuf.metadata_cache_max_bytes               18446744073709551615
        dbuf.metadata_cache_shift                                      6
        dbuf_cache.hiwater_pct                                        10
        dbuf_cache.lowater_pct                                        10
        dbuf_cache.max_bytes                        18446744073709551615
        dbuf_state_index                                               0
        ddt_data_is_special                                            1
        deadman.checktime_ms                                       60000
        deadman.enabled                                                1
        deadman.failmode                                            wait
        deadman.synctime_ms                                       600000
        deadman.ziotime_ms                                        300000
        debug                                                          0
        debugflags                                                     0
        dedup.prefetch                                                 0
        default_bs                                                     9
        default_ibs                                                   15
        delay_min_dirty_percent                                       60
        delay_scale                                               500000
        dirty_data_max                                        1709953024
        dirty_data_max_max                                    4274882560
        dirty_data_max_max_percent                                    25
        dirty_data_max_percent                                        10
        dirty_data_sync_percent                                       20
        disable_ivset_guid_check                                       0
        dmu_object_alloc_chunk_shift                                   7
        dmu_offset_next_sync                                           0
        dmu_prefetch_max                                       134217728
        dtl_sm_blksz                                                4096
        flags                                                          0
        fletcher_4_impl [fastest] scalar superscalar superscalar4 sse2 ssse3
        free_bpobj_enabled                                             1
        free_leak_on_eio                                               0
        free_min_time_ms                                            1000
        history_output_max                                       1048576
        immediate_write_sz                                         32768
        initialize_chunk_size                                    1048576
        initialize_value                            16045690984833335022
        keep_log_spacemaps_at_export                                   0
        l2arc.feed_again                                               1
        l2arc.feed_min_ms                                            200
        l2arc.feed_secs                                                1
        l2arc.headroom                                                 2
        l2arc.headroom_boost                                         200
        l2arc.meta_percent                                            33
        l2arc.mfuonly                                                  0
        l2arc.noprefetch                                               1
        l2arc.norw                                                     0
        l2arc.rebuild_blocks_min_l2size                       1073741824
        l2arc.rebuild_enabled                                          0
        l2arc.trim_ahead                                               0
        l2arc.write_boost                                        8388608
        l2arc.write_max                                          8388608
        l2arc_feed_again                                               1
        l2arc_feed_min_ms                                            200
        l2arc_feed_secs                                                1
        l2arc_headroom                                                 2
        l2arc_noprefetch                                               1
        l2arc_norw                                                     0
        l2arc_write_boost                                        8388608
        l2arc_write_max                                          8388608
        l2c_only_size                                                  0
        livelist.condense.new_alloc                                    0
        livelist.condense.sync_cancel                                  0
        livelist.condense.sync_pause                                   0
        livelist.condense.zthr_cancel                                  0
        livelist.condense.zthr_pause                                   0
        livelist.max_entries                                      500000
        livelist.min_percent_shared                                   75
        lua.max_instrlimit                                     100000000
        lua.max_memlimit                                       104857600
        max_async_dedup_frees                                     100000
        max_auto_ashift                                               16
        max_dataset_nesting                                           50
        max_log_walking                                                5
        max_logsm_summary_length                                      10
        max_missing_tvds                                               0
        max_missing_tvds_cachefile                                     2
        max_missing_tvds_scan                                          0
        max_nvlist_src_size                                            0
        max_recordsize                                           1048576
        metaslab.aliquot                                          524288
        metaslab.bias_enabled                                          1
        metaslab.debug_load                                            0
        metaslab.debug_unload                                          0
        metaslab.df_alloc_threshold                               131072
        metaslab.df_free_pct                                           4
        metaslab.df_max_search                                  16777216
        metaslab.df_use_largest_segment                                0
        metaslab.force_ganging                                  16777217
        metaslab.fragmentation_factor_enabled                          1
        metaslab.fragmentation_threshold                              70
        metaslab.lba_weighting_enabled                                 1
        metaslab.load_pct                                             50
        metaslab.max_size_cache_sec                                 3600
        metaslab.mem_limit                                            75
        metaslab.preload_enabled                                       1
        metaslab.preload_limit                                        10
        metaslab.segment_weight_enabled                                1
        metaslab.sm_blksz_no_log                                   16384
        metaslab.sm_blksz_with_log                                131072
        metaslab.switch_threshold                                      2
        metaslab.unload_delay                                         32
        metaslab.unload_delay_ms                                  600000
        mfu_data_esize                                                 0
        mfu_ghost_data_esize                                    65821184
        mfu_ghost_metadata_esize                                19659776
        mfu_ghost_size                                          85480960
        mfu_metadata_esize                                       1494016
        mfu_size                                               135635968
        mg.fragmentation_threshold                                    95
        mg.noalloc_threshold                                           0
        min_auto_ashift                                                9
        min_metaslabs_to_flush                                         1
        mru_data_esize                                       10095573504
        mru_ghost_data_esize                                    81682432
        mru_ghost_metadata_esize                               927950336
        mru_ghost_size                                        1009632768
        mru_metadata_esize                                      37464576
        mru_size                                             11287740416
        multihost.fail_intervals                                      10
        multihost.history                                              0
        multihost.import_intervals                                    20
        multihost.interval                                          1000
        multilist_num_sublists                                         0
        no_scrub_io                                                    0
        no_scrub_prefetch                                              0
        nocacheflush                                                   0
        nopwrite_enabled                                               1
        obsolete_min_time_ms                                         500
        pd_bytes_max                                            52428800
        per_txg_dirty_frees_percent                                    5
        prefetch.array_rd_sz                                     1048576
        prefetch.disable                                               0
        prefetch.max_distance                                    8388608
        prefetch.max_idistance                                  67108864
        prefetch.max_streams                                           8
        prefetch.min_sec_reap                                          2
        read_history                                                   0
        read_history_hits                                              0
        rebuild_max_segment                                      1048576
        reconstruct.indirect_combinations_max                       4096
        recover                                                        0
        recv.queue_ff                                                 20
        recv.queue_length                                       16777216
        recv.write_batch_size                                    1048576
        reference_tracking_enable                                      0
        removal_suspend_progress                                       0
        remove_max_segment                                      16777216
        resilver_disable_defer                                         0
        resilver_min_time_ms                                        3000
        scan_checkpoint_intval                                      7200
        scan_fill_weight                                               3
        scan_ignore_errors                                             0
        scan_issue_strategy                                            0
        scan_legacy                                                    0
        scan_max_ext_gap                                         2097152
        scan_mem_lim_fact                                             20
        scan_mem_lim_soft_fact                                        20
        scan_strict_mem_lim                                            0
        scan_suspend_progress                                          0
        scan_vdev_limit                                          4194304
        scrub_min_time_ms                                           1000
        send.corrupt_data                                              0
        send.no_prefetch_queue_ff                                     20
        send.no_prefetch_queue_length                            1048576
        send.override_estimate_recordsize                              0
        send.queue_ff                                                 20
        send.queue_length                                       16777216
        send.unmodified_spill_blocks                                   1
        send_holes_without_birth_time                                  1
        slow_io_events_per_second                                     20
        spa.asize_inflation                                           24
        spa.discard_memory_limit                                16777216
        spa.load_print_vdev_tree                                       0
        spa.load_verify_data                                           1
        spa.load_verify_metadata                                       1
        spa.load_verify_shift                                          4
        spa.slop_shift                                                 5
        space_map_ibs                                                 14
        special_class_metadata_reserve_pct                            25
        standard_sm_blksz                                         131072
        super_owner                                                    0
        sync_pass_deferred_free                                        2
        sync_pass_dont_compress                                        8
        sync_pass_rewrite                                              2
        sync_taskq_batch_pct                                          75
        top_maxinflight                                             1000
        traverse_indirect_prefetch_limit                              32
        trim.extent_bytes_max                                  134217728
        trim.extent_bytes_min                                      32768
        trim.metaslab_skip                                             0
        trim.queue_limit                                              10
        trim.txg_batch                                                32
        txg.history                                                  100
        txg.timeout                                                    5
        unflushed_log_block_max                                   262144
        unflushed_log_block_min                                     1000
        unflushed_log_block_pct                                      400
        unflushed_max_mem_amt                                 1073741824
        unflushed_max_mem_ppm                                       1000
        user_indirect_is_special                                       1
        validate_skip                                                  0
        vdev.aggregate_trim                                            0
        vdev.aggregation_limit                                   1048576
        vdev.aggregation_limit_non_rotating                       131072
        vdev.async_read_max_active                                     3
        vdev.async_read_min_active                                     1
        vdev.async_write_active_max_dirty_percent                     60
        vdev.async_write_active_min_dirty_percent                     30
        vdev.async_write_max_active                                    5
        vdev.async_write_min_active                                    1
        vdev.bio_delete_disable                                        0
        vdev.bio_flush_disable                                         0
        vdev.cache_bshift                                             16
        vdev.cache_max                                             16384
        vdev.cache_size                                                0
        vdev.def_queue_depth                                          32
        vdev.default_ms_count                                        200
        vdev.default_ms_shift                                         29
        vdev.file.logical_ashift                                       9
        vdev.file.physical_ashift                                      9
        vdev.initializing_max_active                                   1
        vdev.initializing_min_active                                   1
        vdev.max_active                                             1000
        vdev.max_auto_ashift                                          16
        vdev.min_auto_ashift                                           9
        vdev.min_ms_count                                             16
        vdev.mirror.non_rotating_inc                                   0
        vdev.mirror.non_rotating_seek_inc                              1
        vdev.mirror.rotating_inc                                       0
        vdev.mirror.rotating_seek_inc                                  5
        vdev.mirror.rotating_seek_offset                         1048576
        vdev.ms_count_limit                                       131072
        vdev.nia_credit                                                5
        vdev.nia_delay                                                 5
        vdev.queue_depth_pct                                        1000
        vdev.read_gap_limit                                        32768
        vdev.rebuild_max_active                                        3
        vdev.rebuild_min_active                                        1
        vdev.removal_ignore_errors                                     0
        vdev.removal_max_active                                        2
        vdev.removal_max_span                                      32768
        vdev.removal_min_active                                        1
        vdev.removal_suspend_progress                                  0
        vdev.remove_max_segment                                 16777216
        vdev.scrub_max_active                                          3
        vdev.scrub_min_active                                          1
        vdev.sync_read_max_active                                     10
        vdev.sync_read_min_active                                     10
        vdev.sync_write_max_active                                    10
        vdev.sync_write_min_active                                    10
        vdev.trim_max_active                                           2
        vdev.trim_min_active                                           1
        vdev.validate_skip                                             0
        vdev.write_gap_limit                                        4096
        version.acl                                                    1
        version.ioctl                                                 15
        version.module                         v2022012000-zfs_e870bd8cf
        version.spa                                                 5000
        version.zpl                                                    5
        vnops.read_chunk_size                                    1048576
        vol.mode                                                       2
        vol.recursive                                                  0
        vol.unmap_enabled                                              1
        zap_iterate_prefetch                                           1
        zevent.cols                                                   80
        zevent.console                                                 0
        zevent.len_max                                               512
        zevent.retain_expire_secs                                    900
        zevent.retain_max                                           2000
        zfetch.max_distance                                      8388608
        zfetch.max_idistance                                    67108864
        zil.clean_taskq_maxalloc                                 1048576
        zil.clean_taskq_minalloc                                    1024
        zil.clean_taskq_nthr_pct                                     100
        zil.maxblocksize                                          131072
        zil.nocacheflush                                               0
        zil.replay_disable                                             0
        zil.slog_bulk                                             786432
        zio.deadman_log_all                                            0
        zio.dva_throttle_enabled                                       1
        zio.exclude_metadata                                           0
        zio.requeue_io_start_cut_in_line                               1
        zio.slow_io_ms                                             30000
        zio.taskq_batch_pct                                           80
        zio.taskq_batch_tpq                                            0
        zio.use_uma                                                    1

VDEV cache disabled, skipping section

ZIL committed transactions:                                         4.0k
        Commit requests:                                            1.2k
        Flushes to stable storage:                                  1.2k
        Transactions to SLOG storage pool:            0 Bytes          0
        Transactions to non-SLOG storage pool:        7.7 MiB        941

 

[sretalla]

run arc_summary as I told you...

 

[Bane]

run arc_summary as I told you...

Here's the output, what should I be looking at?

Code:

------------------------------------------------------------------------
ZFS Subsystem Report                            Wed Apr 27 10:33:04 2022
FreeBSD 12.2-RELEASE-p14                                   zpl version 5
Machine: GamakBase.matrixinfosec.com (amd64)            spa version 5000

ARC status:                                                      HEALTHY
        Memory throttle count:                                         0

ARC size (current):                                    77.4 %   11.6 GiB
        Target size (adaptive):                        77.6 %   11.6 GiB
        Min size (hard limit):                          3.3 %  509.6 MiB
        Max size (high water):                           29:1   14.9 GiB
        Most Frequently Used (MFU) cache size:          1.2 %  129.4 MiB
        Most Recently Used (MRU) cache size:           98.8 %   10.5 GiB
        Metadata cache size (hard limit):              75.0 %   11.2 GiB
        Metadata cache size (current):                 14.4 %    1.6 GiB
        Dnode cache size (hard limit):                 10.0 %    1.1 GiB
        Dnode cache size (current):                    39.3 %  450.7 MiB

ARC hash breakdown:
        Elements max:                                             509.0k
        Elements current:                              43.6 %     221.7k
        Collisions:                                                 7.7M
        Chain max:                                                     5
        Chains:                                                    11.0k

ARC misc:
        Deleted:                                                  128.1M
        Mutex misses:                                                342
        Eviction skips:                                           149.2k

ARC total accesses (hits + misses):                               261.0M
        Cache hit ratio:                               99.6 %     259.8M
        Cache miss ratio:                               0.4 %       1.1M
        Actual hit ratio (MFU + MRU hits):             99.6 %     259.8M
        Data demand efficiency:                        89.2 %       1.4M
        Data prefetch efficiency:                       0.1 %     736.6k

Cache hits by cache type:
        Most frequently used (MFU):                    97.8 %     254.2M
        Most recently used (MRU):                       2.2 %       5.6M
        Most frequently used (MFU) ghost:             < 0.1 %      81.4k
        Most recently used (MRU) ghost:               < 0.1 %     118.6k

Cache hits by data type:
        Demand data:                                    0.5 %       1.3M
        Demand prefetch data:                         < 0.1 %        650
        Demand metadata:                               99.5 %     258.5M
        Demand prefetch metadata:                     < 0.1 %      38.4k

Cache misses by data type:
        Demand data:                                   13.9 %     156.9k
        Demand prefetch data:                          65.1 %     736.0k
        Demand metadata:                               18.9 %     213.7k
        Demand prefetch metadata:                       2.2 %      24.9k

DMU prefetch efficiency:                                           18.5M
        Hit ratio:                                      1.7 %     322.6k
        Miss ratio:                                    98.3 %      18.1M

L2ARC not detected, skipping section

Tunables:
        abd_chunk_size                                              4096
        abd_scatter_enabled                                            1
        allow_redacted_dataset_mount                                   0
        anon_data_esize                                                0
        anon_metadata_esize                                            0
        anon_size                                                1644032
        arc.average_blocksize                                       8192
        arc.dnode_limit                                                0
        arc.dnode_limit_percent                                       10
        arc.dnode_reduce_percent                                      10
        arc.evict_batch_limit                                         10
        arc.eviction_pct                                             200
        arc.grow_retry                                                 0
        arc.lotsfree_percent                                          10
        arc.max                                                        0
        arc.meta_adjust_restarts                                    4096
        arc.meta_limit                                                 0
        arc.meta_limit_percent                                        75
        arc.meta_min                                                   0
        arc.meta_prune                                             10000
        arc.meta_strategy                                              1
        arc.min                                                        0
        arc.min_prefetch_ms                                            0
        arc.min_prescient_prefetch_ms                                  0
        arc.p_dampener_disable                                         1
        arc.p_min_shift                                                0
        arc.pc_percent                                                 0
        arc.prune_task_threads                                         1
        arc.shrink_shift                                               0
        arc.sys_free                                                   0
        arc_free_target                                            86655
        arc_max                                                        0
        arc_min                                                        0
        arc_no_grow_shift                                              5
        async_block_max_blocks                      18446744073709551615
        autoimport_disable                                             1
        ccw_retry_interval                                           300
        checksum_events_per_second                                    20
        commit_timeout_pct                                             5
        compressed_arc_enabled                                         1
        condense.indirect_commit_entry_delay_ms                        0
        condense.indirect_obsolete_pct                                25
        condense.indirect_vdevs_enable                                 1
        condense.max_obsolete_bytes                           1073741824
        condense.min_mapping_bytes                                131072
        condense_pct                                                 200
        crypt_sessions                                              2589
        dbgmsg_enable                                                  1
        dbgmsg_maxsize                                           4194304
        dbuf.cache_shift                                               5
        dbuf.metadata_cache_max_bytes               18446744073709551615
        dbuf.metadata_cache_shift                                      6
        dbuf_cache.hiwater_pct                                        10
        dbuf_cache.lowater_pct                                        10
        dbuf_cache.max_bytes                        18446744073709551615
        dbuf_state_index                                               0
        ddt_data_is_special                                            1
        deadman.checktime_ms                                       60000
        deadman.enabled                                                1
        deadman.failmode                                            wait
        deadman.synctime_ms                                       600000
        deadman.ziotime_ms                                        300000
        debug                                                          0
        debugflags                                                     0
        dedup.prefetch                                                 0
        default_bs                                                     9
        default_ibs                                                   15
        delay_min_dirty_percent                                       60
        delay_scale                                               500000
        dirty_data_max                                        1709953024
        dirty_data_max_max                                    4274882560
        dirty_data_max_max_percent                                    25
        dirty_data_max_percent                                        10
        dirty_data_sync_percent                                       20
        disable_ivset_guid_check                                       0
        dmu_object_alloc_chunk_shift                                   7
        dmu_offset_next_sync                                           0
        dmu_prefetch_max                                       134217728
        dtl_sm_blksz                                                4096
        flags                                                          0
        fletcher_4_impl [fastest] scalar superscalar superscalar4 sse2 ssse3
        free_bpobj_enabled                                             1
        free_leak_on_eio                                               0
        free_min_time_ms                                            1000
        history_output_max                                       1048576
        immediate_write_sz                                         32768
        initialize_chunk_size                                    1048576
        initialize_value                            16045690984833335022
        keep_log_spacemaps_at_export                                   0
        l2arc.feed_again                                               1
        l2arc.feed_min_ms                                            200
        l2arc.feed_secs                                                1
        l2arc.headroom                                                 2
        l2arc.headroom_boost                                         200
        l2arc.meta_percent                                            33
        l2arc.mfuonly                                                  0
        l2arc.noprefetch                                               1
        l2arc.norw                                                     0
        l2arc.rebuild_blocks_min_l2size                       1073741824
        l2arc.rebuild_enabled                                          0
        l2arc.trim_ahead                                               0
        l2arc.write_boost                                        8388608
        l2arc.write_max                                          8388608
        l2arc_feed_again                                               1
        l2arc_feed_min_ms                                            200
        l2arc_feed_secs                                                1
        l2arc_headroom                                                 2
        l2arc_noprefetch                                               1
        l2arc_norw                                                     0
        l2arc_write_boost                                        8388608
        l2arc_write_max                                          8388608
        l2c_only_size                                                  0
        livelist.condense.new_alloc                                    0
        livelist.condense.sync_cancel                                  0
        livelist.condense.sync_pause                                   0
        livelist.condense.zthr_cancel                                  0
        livelist.condense.zthr_pause                                   0
        livelist.max_entries                                      500000
        livelist.min_percent_shared                                   75
        lua.max_instrlimit                                     100000000
        lua.max_memlimit                                       104857600
        max_async_dedup_frees                                     100000
        max_auto_ashift                                               16
        max_dataset_nesting                                           50
        max_log_walking                                                5
        max_logsm_summary_length                                      10
        max_missing_tvds                                               0
        max_missing_tvds_cachefile                                     2
        max_missing_tvds_scan                                          0
        max_nvlist_src_size                                            0
        max_recordsize                                           1048576
        metaslab.aliquot                                          524288
        metaslab.bias_enabled                                          1
        metaslab.debug_load                                            0
        metaslab.debug_unload                                          0
        metaslab.df_alloc_threshold                               131072
        metaslab.df_free_pct                                           4
        metaslab.df_max_search                                  16777216
        metaslab.df_use_largest_segment                                0
        metaslab.force_ganging                                  16777217
        metaslab.fragmentation_factor_enabled                          1
        metaslab.fragmentation_threshold                              70
        metaslab.lba_weighting_enabled                                 1
        metaslab.load_pct                                             50
        metaslab.max_size_cache_sec                                 3600
        metaslab.mem_limit                                            75
        metaslab.preload_enabled                                       1
        metaslab.preload_limit                                        10
        metaslab.segment_weight_enabled                                1
        metaslab.sm_blksz_no_log                                   16384
        metaslab.sm_blksz_with_log                                131072
        metaslab.switch_threshold                                      2
        metaslab.unload_delay                                         32
        metaslab.unload_delay_ms                                  600000
        mfu_data_esize                                                 0
        mfu_ghost_data_esize                                    65821184
        mfu_ghost_metadata_esize                                19659776
        mfu_ghost_size                                          85480960
        mfu_metadata_esize                                       1494016
        mfu_size                                               135635968
        mg.fragmentation_threshold                                    95
        mg.noalloc_threshold                                           0
        min_auto_ashift                                                9
        min_metaslabs_to_flush                                         1
        mru_data_esize                                       10095573504
        mru_ghost_data_esize                                    81682432
        mru_ghost_metadata_esize                               927950336
        mru_ghost_size                                        1009632768
        mru_metadata_esize                                      37464576
        mru_size                                             11287740416
        multihost.fail_intervals                                      10
        multihost.history                                              0
        multihost.import_intervals                                    20
        multihost.interval                                          1000
        multilist_num_sublists                                         0
        no_scrub_io                                                    0
        no_scrub_prefetch                                              0
        nocacheflush                                                   0
        nopwrite_enabled                                               1
        obsolete_min_time_ms                                         500
        pd_bytes_max                                            52428800
        per_txg_dirty_frees_percent                                    5
        prefetch.array_rd_sz                                     1048576
        prefetch.disable                                               0
        prefetch.max_distance                                    8388608
        prefetch.max_idistance                                  67108864
        prefetch.max_streams                                           8
        prefetch.min_sec_reap                                          2
        read_history                                                   0
        read_history_hits                                              0
        rebuild_max_segment                                      1048576
        reconstruct.indirect_combinations_max                       4096
        recover                                                        0
        recv.queue_ff                                                 20
        recv.queue_length                                       16777216
        recv.write_batch_size                                    1048576
        reference_tracking_enable                                      0
        removal_suspend_progress                                       0
        remove_max_segment                                      16777216
        resilver_disable_defer                                         0
        resilver_min_time_ms                                        3000
        scan_checkpoint_intval                                      7200
        scan_fill_weight                                               3
        scan_ignore_errors                                             0
        scan_issue_strategy                                            0
        scan_legacy                                                    0
        scan_max_ext_gap                                         2097152
        scan_mem_lim_fact                                             20
        scan_mem_lim_soft_fact                                        20
        scan_strict_mem_lim                                            0
        scan_suspend_progress                                          0
        scan_vdev_limit                                          4194304
        scrub_min_time_ms                                           1000
        send.corrupt_data                                              0
        send.no_prefetch_queue_ff                                     20
        send.no_prefetch_queue_length                            1048576
        send.override_estimate_recordsize                              0
        send.queue_ff                                                 20
        send.queue_length                                       16777216
        send.unmodified_spill_blocks                                   1
        send_holes_without_birth_time                                  1
        slow_io_events_per_second                                     20
        spa.asize_inflation                                           24
        spa.discard_memory_limit                                16777216
        spa.load_print_vdev_tree                                       0
        spa.load_verify_data                                           1
        spa.load_verify_metadata                                       1
        spa.load_verify_shift                                          4
        spa.slop_shift                                                 5
        space_map_ibs                                                 14
        special_class_metadata_reserve_pct                            25
        standard_sm_blksz                                         131072
        super_owner                                                    0
        sync_pass_deferred_free                                        2
        sync_pass_dont_compress                                        8
        sync_pass_rewrite                                              2
        sync_taskq_batch_pct                                          75
        top_maxinflight                                             1000
        traverse_indirect_prefetch_limit                              32
        trim.extent_bytes_max                                  134217728
        trim.extent_bytes_min                                      32768
        trim.metaslab_skip                                             0
        trim.queue_limit                                              10
        trim.txg_batch                                                32
        txg.history                                                  100
        txg.timeout                                                    5
        unflushed_log_block_max                                   262144
        unflushed_log_block_min                                     1000
        unflushed_log_block_pct                                      400
        unflushed_max_mem_amt                                 1073741824
        unflushed_max_mem_ppm                                       1000
        user_indirect_is_special                                       1
        validate_skip                                                  0
        vdev.aggregate_trim                                            0
        vdev.aggregation_limit                                   1048576
        vdev.aggregation_limit_non_rotating                       131072
        vdev.async_read_max_active                                     3
        vdev.async_read_min_active                                     1
        vdev.async_write_active_max_dirty_percent                     60
        vdev.async_write_active_min_dirty_percent                     30
        vdev.async_write_max_active                                    5
        vdev.async_write_min_active                                    1
        vdev.bio_delete_disable                                        0
        vdev.bio_flush_disable                                         0
        vdev.cache_bshift                                             16
        vdev.cache_max                                             16384
        vdev.cache_size                                                0
        vdev.def_queue_depth                                          32
        vdev.default_ms_count                                        200
        vdev.default_ms_shift                                         29
        vdev.file.logical_ashift                                       9
        vdev.file.physical_ashift                                      9
        vdev.initializing_max_active                                   1
        vdev.initializing_min_active                                   1
        vdev.max_active                                             1000
        vdev.max_auto_ashift                                          16
        vdev.min_auto_ashift                                           9
        vdev.min_ms_count                                             16
        vdev.mirror.non_rotating_inc                                   0
        vdev.mirror.non_rotating_seek_inc                              1
        vdev.mirror.rotating_inc                                       0
        vdev.mirror.rotating_seek_inc                                  5
        vdev.mirror.rotating_seek_offset                         1048576
        vdev.ms_count_limit                                       131072
        vdev.nia_credit                                                5
        vdev.nia_delay                                                 5
        vdev.queue_depth_pct                                        1000
        vdev.read_gap_limit                                        32768
        vdev.rebuild_max_active                                        3
        vdev.rebuild_min_active                                        1
        vdev.removal_ignore_errors                                     0
        vdev.removal_max_active                                        2
        vdev.removal_max_span                                      32768
        vdev.removal_min_active                                        1
        vdev.removal_suspend_progress                                  0
        vdev.remove_max_segment                                 16777216
        vdev.scrub_max_active                                          3
        vdev.scrub_min_active                                          1
        vdev.sync_read_max_active                                     10
        vdev.sync_read_min_active                                     10
        vdev.sync_write_max_active                                    10
        vdev.sync_write_min_active                                    10
        vdev.trim_max_active                                           2
        vdev.trim_min_active                                           1
        vdev.validate_skip                                             0
        vdev.write_gap_limit                                        4096
        version.acl                                                    1
        version.ioctl                                                 15
        version.module                         v2022012000-zfs_e870bd8cf
        version.spa                                                 5000
        version.zpl                                                    5
        vnops.read_chunk_size                                    1048576
        vol.mode                                                       2
        vol.recursive                                                  0
        vol.unmap_enabled                                              1
        zap_iterate_prefetch                                           1
        zevent.cols                                                   80
        zevent.console                                                 0
        zevent.len_max                                               512
        zevent.retain_expire_secs                                    900
        zevent.retain_max                                           2000
        zfetch.max_distance                                      8388608
        zfetch.max_idistance                                    67108864
        zil.clean_taskq_maxalloc                                 1048576
        zil.clean_taskq_minalloc                                    1024
        zil.clean_taskq_nthr_pct                                     100
        zil.maxblocksize                                          131072
        zil.nocacheflush                                               0
        zil.replay_disable                                             0
        zil.slog_bulk                                             786432
        zio.deadman_log_all                                            0
        zio.dva_throttle_enabled                                       1
        zio.exclude_metadata                                           0
        zio.requeue_io_start_cut_in_line                               1
        zio.slow_io_ms                                             30000
        zio.taskq_batch_pct                                           80
        zio.taskq_batch_tpq                                            0
        zio.use_uma                                                    1

VDEV cache disabled, skipping section

ZIL committed transactions:                                         4.0k
        Commit requests:                                            1.2k
        Flushes to stable storage:                                  1.2k
        Transactions to SLOG storage pool:            0 Bytes          0
        Transactions to non-SLOG storage pool:        7.7 MiB        941

 

[winnielinnie]

Cache miss ratio: 0.4 % 1.1M

Currently, as it stands, about half-a-percent of requests for data "miss" the ARC.

Doesn't seem worth it to invest in an L2ARC, and as others have alluded, you'd want a minimum of maybe 32GB or 64GB to give enough breathing room for the indexing of what lives within the L2ARC.

But the upgrade in memory will likely do more for you than simply adding an L2ARC.

 

[Bane]

Currently, as it stands, about half-a-percent of requests for data "miss" the ARC.

Doesn't seem worth it to invest in an L2ARC, and as others have alluded, you'd want a minimum of maybe 32GB or 64GB to give enough breathing room for the indexing of what lives within the L2ARC.

But the upgrade in memory will likely do more for you than simply adding an L2ARC.

I have a couple of 16gb SSD's I could use for cache. Would it cause any issue to add one to the pool, I suspect not.

 

[Patrick M. Hausen]

With only 16 GB of memory adding an L2ARC will probably decrease your performance because part of that RAM will be used to manage the L2ARC instead of using it as cache directly. And no SSD you could add is as fast as your RAM. Hence the 64 GB or more recommendation.

 

[Bane]

With only 16 GB of memory adding an L2ARC will probably decrease your performance because part of that RAM will be used to manage the L2ARC instead of using it as cache directly. And no SSD you could add is as fast as your RAM. Hence the 64 GB or more recommendation.

Would the 16gb SSD be useful for SLOG or in my configuration would SLOG not be valuable? Also, if I went with intel Optane for l2arc would then be valuable to do a cache on the pool?

 

[Patrick M. Hausen]

If you don't provide e.g. block storage for VMs via iSCSI or NFS but just file sharing via SMB, an SLOG device does precisely nothing.

An SLOG device can speed up synchronous writes, which in regular file sharing never happen. Second in regular operation an SLOG will only be written and never read. Only after a power failure or other unplanned reboot will SLOG contents be synced to the pool.
The constant writing demands special suitable SSDs like Intel's Optane series. Any run-of-the-mill product used as an SLOG will fail quickly.

You want to speed up your NAS? Add more RAM.

What could be valuable if you like to toy around and this is a private system, is a metadata special device. If you have huge directories (thousands of files) shared via SMB, keeping the directory information on SSDs will significantly speed up the user experience when browsing the shares in e.g. Windows Explorer.

But you will need to add at least 2, better 3 reliable SSDs in a mirror, because if your metadata special device fails, your entire pool is toast. No recovery.

 

[winnielinnie]

What could be valuable if you like to toy around and this is a private system, is a metadata special device. If you have huge directories (thousands of files) shared via SMB, keeping the directory information on SSDs will significantly speed up the user experience when browsing the shares in e.g. Windows Explorer.

Or with enough RAM, try this first, before introducing a special vdev:

ZFS "ARC" doesn't seem that smart...

Without yet resorting to adding an L2ARC, is there a "tuneable" that I can test which instructs the ARC to prioritize metadata? I upgraded from 16GB to 32GB ECC RAM. Yet there is zero change in this behavior. This keeps happening: I run regular rsync tasks from a few local clients, which is...

www.truenas.com

 

[Bane]

If you don't provide e.g. block storage for VMs via iSCSI or NFS but just file sharing via SMB, an SLOG device does precisely nothing.

An SLOG device can speed up synchronous writes, which in regular file sharing never happen. Second in regular operation an SLOG will only be written and never read. Only after a power failure or other unplanned reboot will SLOG contents be synced to the pool.
The constant writing demands special suitable SSDs like Intel's Optane series. Any run-of-the-mill product used as an SLOG will fail quickly.

You want to speed up your NAS? Add more RAM.

What could be valuable if you like to toy around and this is a private system, is a metadata special device. If you have huge directories (thousands of files) shared via SMB, keeping the directory information on SSDs will significantly speed up the user experience when browsing the shares in e.g. Windows Explorer.

But you will need to add at least 2, better 3 reliable SSDs in a mirror, because if your metadata special device fails, your entire pool is toast. No reIt sounds like once again not very valuable and RAM upgrade it is..

I guess I'll be upgrading the RAM. I have a supermicro x9scm-l board and it lists up to 32gb capacity. Do you think going from 16 to 32gb will make a difference?

 

[Patrick M. Hausen]

You already have a 99.6% cache hit rate. How many clients are using your system concurrently? What is the user perceived metric you want to speed up?

 

[Bane]

You already have a 99.6% cache hit rate. How many clients are using your system concurrently? What is the user perceived metric you want to speed up?It's not many lients

It's my home lab, so not many clients. It's mainly my family of 5 and my XCP-NG server using the NAS. I'm not seeing close to 10 gigabit transfers in and out of my TrueNAS, so I was hoping that more ram or cache would help that.

 

[Patrick M. Hausen]

So the first thing to check would be if your TrueNAS and one client system can saturate 10G with e.g. iperf.

 

[Bane]

So the first thing to check would be if your TrueNAS and one client system can saturate 10G with e.g. iperf.

Looks like it is probably my XCP-NG that is not saturating the network. I ran iperf between it and my TrueNAS, I see the below between XCP-NG and TrueNAS.

------------------------------------------------------------
Client connecting to 192.168.2.15, TCP port 5001
TCP window size: 0.42 MByte (default)
------------------------------------------------------------
[ 3] local 192.168.2.32 port 58284 connected with 192.168.2.15 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.0 sec 8212 MBytes 6889 Mbits/sec


I also ran the same test between my primary and backup TrueNAS boxes and I see full speed.

iperf -c 192.168.2.15 -p 5001 -f m
------------------------------------------------------------
Client connecting to 192.168.2.15, TCP port 5001
TCP window size: 0.25 MByte (default)
------------------------------------------------------------
[ 3] local 192.168.2.18 port 27201 connected with 192.168.2.15 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.0 sec 9836 MBytes 8250 Mbits/sec


I also ran DD on my primary TrueNAS with the below results.

dd if=/dev/zero of=/mnt/GamakBase/testfile bs=4M count=10000
10000+0 records in
10000+0 records out
41943040000 bytes transferred in 10.279666 secs (4080194775 bytes/sec)

The same DD on XCP-NP resulted in:

16871473152 bytes (17 GB) copied, 184.689 s, 91.4 MB/s


So it appears the issue lies in the disk subsystem of my XCP-NG box.