I have 64gb, but is 32gb enough?

ThreeDee

Guru
Joined
Jun 13, 2013
Messages
698
I'm trying to downsize a bit and recoup some money from where I can if it makes sense ..

I currently have 4 x 16GB DDR4 ECC UDIMM's in my TrueNAS box .. I will at some point clean up my hard drive configuration to add redundancy and what not instead of the hodge podge array I have now .. 1x 8TB, 1 x 5tb, 3 x 4TB, 1 x 2TB SAS, 256GB NVMe SATA for Jails (going to replace my current 128GB that is at 90% full).

I only run jails for Plex and Unifi controller plus some SMB shares for my 4 windows setups.

Would 32GB suffice for the foreseeable future without any performance deficit or should I stick with 64GB?
 
Last edited:

marcevan

Patron
Joined
Dec 15, 2013
Messages
432
I ran on 16GB RAM from 9.2 to to current and only recently went to 32GB RAM.

Of course I upped the CPU to 6 core so that was probably the best thing.

ZFS cache will grab most of our RAM anyway so I think 32GB is fine.
 

HoneyBadger

actually does care
Administrator
Moderator
iXsystems
Joined
Feb 6, 2014
Messages
5,110
You can make some guesses based on your current ARC hit rates, and MFU:MRU ratio, as well as how often the "ghost" lists get a hit.

Drop the results of arc_summary.pl in CODE tags and let's have a look.
 

ThreeDee

Guru
Joined
Jun 13, 2013
Messages
698
I work nights .. I'll post the results of arc_summary as soon as I get home from work .. it seems that the .pl isn't needed for the command in TrueNAS
 
Last edited:

ThreeDee

Guru
Joined
Jun 13, 2013
Messages
698
Code:
root@truenas[~]# arcstat
    time  read  miss  miss%  dmis  dm%  pmis  pm%  mmis  mm%  size     c  avail
10:31:27     0     0      0     0    0     0    0     0    0   47G   49G   512M
 

ThreeDee

Guru
Joined
Jun 13, 2013
Messages
698
Code:
ZFS Subsystem Report                            Sun May 09 10:26:18 2021
FreeBSD 12.2-RELEASE-p6                                    zpl version 5
Machine: truenas.local (amd64)                          spa version 5000

ARC status:                                                      HEALTHY
        Memory throttle count:                                         0

ARC size (current):                                    78.5 %   49.3 GiB
        Target size (adaptive):                        78.4 %   49.3 GiB
        Min size (hard limit):                          3.2 %    2.0 GiB
        Max size (high water):                           31:1   62.8 GiB
        Most Frequently Used (MFU) cache size:         77.5 %   37.1 GiB
        Most Recently Used (MRU) cache size:           22.5 %   10.7 GiB
        Metadata cache size (hard limit):              75.0 %   47.1 GiB
        Metadata cache size (current):                  6.5 %    3.1 GiB
        Dnode cache size (hard limit):                 10.0 %    4.7 GiB
        Dnode cache size (current):                    15.5 %  749.2 MiB

ARC hash breakdown:
        Elements max:                                               1.1M
        Elements current:                              99.9 %       1.1M
        Collisions:                                                 1.3M
        Chain max:                                                     4
        Chains:                                                    68.2k

ARC misc:
        Deleted:                                                    4.2M
        Mutex misses:                                                 55
        Eviction skips:                                           165.9k

ARC total accesses (hits + misses):                               785.3M
        Cache hit ratio:                               99.3 %     779.9M
        Cache miss ratio:                               0.7 %       5.4M
        Actual hit ratio (MFU + MRU hits):             99.3 %     779.8M
        Data demand efficiency:                        99.8 %     265.0M
        Data prefetch efficiency:                       1.1 %       4.3M

Cache hits by cache type:
        Most frequently used (MFU):                    97.8 %     763.1M
        Most recently used (MRU):                       2.1 %      16.7M
        Most frequently used (MFU) ghost:               0.1 %     607.0k
        Most recently used (MRU) ghost:               < 0.1 %     292.2k

Cache hits by data type:
        Demand data:                                   33.9 %     264.5M
        Demand prefetch data:                         < 0.1 %      49.3k
        Demand metadata:                               66.1 %     515.3M
        Demand prefetch metadata:                     < 0.1 %      65.8k

Cache misses by data type:
        Demand data:                                    7.9 %     428.3k
        Demand prefetch data:                          79.3 %       4.3M
        Demand metadata:                               11.4 %     615.1k
        Demand prefetch metadata:                       1.4 %      76.1k

DMU prefetch efficiency:                                          240.3M
        Hit ratio:                                      1.6 %       3.7M
        Miss ratio:                                    98.4 %     236.5M

L2ARC not detected, skipping section

Tunables:
        abd_chunk_size                                              4096
        abd_scatter_enabled                                            1
        allow_redacted_dataset_mount                                   0
        anon_data_esize                                                0
        anon_metadata_esize                                            0
        anon_size                                                3207168
        arc.average_blocksize                                       8192
        arc.dnode_limit                                                0
        arc.dnode_limit_percent                                       10
        arc.dnode_reduce_percent                                      10
        arc.evict_batch_limit                                         10
        arc.eviction_pct                                             200
        arc.grow_retry                                                 0
        arc.lotsfree_percent                                          10
        arc.max                                                        0
        arc.meta_adjust_restarts                                    4096
        arc.meta_limit                                                 0
        arc.meta_limit_percent                                        75
        arc.meta_min                                                   0
        arc.meta_prune                                             10000
        arc.meta_strategy                                              1
        arc.min                                                        0
        arc.min_prefetch_ms                                            0
        arc.min_prescient_prefetch_ms                                  0
        arc.p_dampener_disable                                         1
        arc.p_min_shift                                                0
        arc.pc_percent                                                 0
        arc.shrink_shift                                               0
        arc.sys_free                                                   0
        arc_free_target                                           347571
        arc_max                                                        0
        arc_min                                                        0
        arc_no_grow_shift                                              5
        async_block_max_blocks                      18446744073709551615
        autoimport_disable                                             1
        ccw_retry_interval                                           300
        checksum_events_per_second                                    20
        commit_timeout_pct                                             5
        compressed_arc_enabled                                         1
        condense.indirect_commit_entry_delay_ms                        0
        condense.indirect_vdevs_enable                                 1
        condense.max_obsolete_bytes                           1073741824
        condense.min_mapping_bytes                                131072
        condense_pct                                                 200
        crypt_sessions                                                 0
        dbgmsg_enable                                                  1
        dbgmsg_maxsize                                           4194304
        dbuf.cache_shift                                               5
        dbuf.metadata_cache_max_bytes               18446744073709551615
        dbuf.metadata_cache_shift                                      6
        dbuf_cache.hiwater_pct                                        10
        dbuf_cache.lowater_pct                                        10
        dbuf_cache.max_bytes                        18446744073709551615
        dbuf_state_index                                               0
        ddt_data_is_special                                            1
        deadman.failmode                                            wait
        deadman.synctime_ms                                       600000
        deadman.ziotime_ms                                        300000
        deadman_checktime_ms                                       60000
        deadman_enabled                                                1
        debug                                                          0
        debugflags                                                     0
        dedup.prefetch                                                 0
        default_bs                                                     9
        default_ibs                                                   15
        delay_min_dirty_percent                                       60
        delay_scale                                               500000
        dirty_data_max                                        4294967296
        dirty_data_max_max                                    4294967296
        dirty_data_max_max_percent                                    25
        dirty_data_max_percent                                        10
        dirty_data_sync_percent                                       20
        disable_ivset_guid_check                                       0
        dmu_object_alloc_chunk_shift                                   7
        dmu_offset_next_sync                                           0
        dmu_prefetch_max                                       134217728
        dtl_sm_blksz                                                4096
        flags                                                          0
        fletcher_4_impl [fastest] scalar superscalar superscalar4 sse2 ssse3 avx2
        free_bpobj_enabled                                             1
        free_leak_on_eio                                               0
        free_min_time_ms                                            1000
        history_output_max                                       1048576
        immediate_write_sz                                         32768
        initialize_chunk_size                                    1048576
        initialize_value                            16045690984833335022
        keep_log_spacemaps_at_export                                   0
        l2arc.feed_again                                               1
        l2arc.feed_min_ms                                            200
        l2arc.feed_secs                                                1
        l2arc.headroom                                                 2
        l2arc.headroom_boost                                         200
        l2arc.meta_percent                                            33
        l2arc.mfuonly                                                  0
        l2arc.noprefetch                                               1
        l2arc.norw                                                     0
        l2arc.rebuild_blocks_min_l2size                       1073741824
        l2arc.rebuild_enabled                                          0
        l2arc.trim_ahead                                               0
        l2arc.write_boost                                        8388608
        l2arc.write_max                                          8388608
        l2arc_feed_again                                               1
        l2arc_feed_min_ms                                            200
        l2arc_feed_secs                                                1
        l2arc_headroom                                                 2
        l2arc_noprefetch                                               1
        l2arc_norw                                                     0
        l2arc_write_boost                                        8388608
        l2arc_write_max                                          8388608
        l2c_only_size                                                  0
        livelist.condense.new_alloc                                    0
        livelist.condense.sync_cancel                                  0
        livelist.condense.sync_pause                                   0
        livelist.condense.zthr_cancel                                  0
        livelist.condense.zthr_pause                                   0
        livelist.max_entries                                      500000
        livelist.min_percent_shared                                   75
        lua.max_instrlimit                                     100000000
        lua.max_memlimit                                       104857600
        max_async_dedup_frees                                     100000
        max_auto_ashift                                               16
        max_dataset_nesting                                           50
        max_log_walking                                                5
        max_logsm_summary_length                                      10
        max_missing_tvds                                               0
        max_missing_tvds_cachefile                                     2
        max_missing_tvds_scan                                          0
        max_nvlist_src_size                                            0
        max_recordsize                                           1048576
        metaslab.aliquot                                          524288
        metaslab.bias_enabled                                          1
        metaslab.debug_load                                            0
        metaslab.debug_unload                                          0
        metaslab.df_alloc_threshold                               131072
        metaslab.df_free_pct                                           4
        metaslab.df_max_search                                  16777216
        metaslab.df_use_largest_segment                                0
        metaslab.force_ganging                                  16777217
        metaslab.fragmentation_factor_enabled                          1
        metaslab.fragmentation_threshold                              70
        metaslab.lba_weighting_enabled                                 1
        metaslab.load_pct                                             50
        metaslab.max_size_cache_sec                                 3600
        metaslab.mem_limit                                            75
        metaslab.preload_enabled                                       1
        metaslab.preload_limit                                        10
        metaslab.segment_weight_enabled                                1
        metaslab.sm_blksz_no_log                                   16384
        metaslab.sm_blksz_with_log                                131072
        metaslab.switch_threshold                                      2
        metaslab.unload_delay                                         32
        metaslab.unload_delay_ms                                  600000
        mfu_data_esize                                       36507808768
        mfu_ghost_data_esize                                 12182749184
        mfu_ghost_metadata_esize                               991018496
        mfu_ghost_size                                       13173767680
        mfu_metadata_esize                                     610089984
        mfu_size                                             39786716672
        mg.fragmentation_threshold                                    95
        mg.noalloc_threshold                                           0
        min_auto_ashift                                                9
        min_metaslabs_to_flush                                         1
        mru_data_esize                                       10394875904
        mru_ghost_data_esize                                 38452731904
        mru_ghost_metadata_esize                              1306700800
        mru_ghost_size                                       39759432704
        mru_metadata_esize                                       7562752
        mru_size                                             11518043136
        multihost.fail_intervals                                      10
        multihost.history                                              0
        multihost.import_intervals                                    20
        multihost.interval                                          1000
        multilist_num_sublists                                         0
        no_scrub_io                                                    0
        no_scrub_prefetch                                              0
        nocacheflush                                                   0
        nopwrite_enabled                                               1
        obsolete_min_time_ms                                         500
        pd_bytes_max                                            52428800
        per_txg_dirty_frees_percent                                    5
        prefetch.array_rd_sz                                     1048576
        prefetch.disable                                               0
        prefetch.max_distance                                    8388608
        prefetch.max_idistance                                  67108864
        prefetch.max_streams                                           8
        prefetch.min_sec_reap                                          2
        read_history                                                   0
        read_history_hits                                              0
        rebuild_max_segment                                      1048576
        reconstruct.indirect_combinations_max                       4096
        recover                                                        0
        recv.queue_ff                                                 20
        recv.queue_length                                       16777216
        recv.write_batch_size                                    1048576
        reference_tracking_enable                                      0
        removal_suspend_progress                                       0
        remove_max_segment                                      16777216
        resilver_disable_defer                                         0
        resilver_min_time_ms                                        3000
        scan_checkpoint_intval                                      7200
        scan_fill_weight                                               3
        scan_ignore_errors                                             0
        scan_issue_strategy                                            0
        scan_legacy                                                    0
        scan_max_ext_gap                                         2097152
        scan_mem_lim_fact                                             20
        scan_mem_lim_soft_fact                                        20
        scan_strict_mem_lim                                            0
        scan_suspend_progress                                          0
        scan_vdev_limit                                          4194304
        scrub_min_time_ms                                           1000
        send.corrupt_data                                              0
        send.no_prefetch_queue_ff                                     20
        send.no_prefetch_queue_length                            1048576
        send.override_estimate_recordsize                              0
        send.queue_ff                                                 20
        send.queue_length                                       16777216
        send.unmodified_spill_blocks                                   1
        send_holes_without_birth_time                                  1
        slow_io_events_per_second                                     20
        spa.asize_inflation                                           24
        spa.discard_memory_limit                                16777216
        spa.load_print_vdev_tree                                       0
        spa.load_verify_data                                           1
        spa.load_verify_metadata                                       1
        spa.load_verify_shift                                          4
        spa.slop_shift                                                 5
        space_map_ibs                                                 14
        special_class_metadata_reserve_pct                            25
        standard_sm_blksz                                         131072
        super_owner                                                    0
        sync_pass_deferred_free                                        2
        sync_pass_dont_compress                                        8
        sync_pass_rewrite                                              2
        sync_taskq_batch_pct                                          75
        top_maxinflight                                             1000
        trim.extent_bytes_max                                  134217728
        trim.extent_bytes_min                                      32768
        trim.metaslab_skip                                             0
        trim.queue_limit                                              10
        trim.txg_batch                                                32
        txg.history                                                  100
        txg.timeout                                                    5
        unflushed_log_block_max                                   262144
        unflushed_log_block_min                                     1000
        unflushed_log_block_pct                                      400
        unflushed_max_mem_amt                                 1073741824
        unflushed_max_mem_ppm                                       1000
        user_indirect_is_special                                       1
        validate_skip                                                  0
        vdev.aggregate_trim                                            0
        vdev.aggregation_limit                                   1048576
        vdev.aggregation_limit_non_rotating                       131072
        vdev.async_read_max_active                                     3
        vdev.async_read_min_active                                     1
        vdev.async_write_active_max_dirty_percent                     60
        vdev.async_write_active_min_dirty_percent                     30
        vdev.async_write_max_active                                    5
        vdev.async_write_min_active                                    1
        vdev.bio_delete_disable                                        0
        vdev.bio_flush_disable                                         0
        vdev.cache_bshift                                             16
        vdev.cache_max                                             16384
        vdev.cache_size                                                0
        vdev.def_queue_depth                                          32
        vdev.default_ms_count                                        200
        vdev.default_ms_shift                                         29
        vdev.file.logical_ashift                                       9
        vdev.file.physical_ashift                                      9
        vdev.initializing_max_active                                   1
        vdev.initializing_min_active                                   1
        vdev.max_active                                             1000
        vdev.max_auto_ashift                                          16
        vdev.min_auto_ashift                                           9
        vdev.min_ms_count                                             16
        vdev.mirror.non_rotating_inc                                   0
        vdev.mirror.non_rotating_seek_inc                              1
        vdev.mirror.rotating_inc                                       0
        vdev.mirror.rotating_seek_inc                                  5
        vdev.mirror.rotating_seek_offset                         1048576
        vdev.ms_count_limit                                       131072
        vdev.nia_credit                                                5
        vdev.nia_delay                                                 5
        vdev.queue_depth_pct                                        1000
        vdev.read_gap_limit                                        32768
        vdev.rebuild_max_active                                        3
        vdev.rebuild_min_active                                        1
        vdev.removal_ignore_errors                                     0
        vdev.removal_max_active                                        2
        vdev.removal_max_span                                      32768
        vdev.removal_min_active                                        1
        vdev.removal_suspend_progress                                  0
        vdev.remove_max_segment                                 16777216
        vdev.scrub_max_active                                          3
        vdev.scrub_min_active                                          1
        vdev.sync_read_max_active                                     10
        vdev.sync_read_min_active                                     10
        vdev.sync_write_max_active                                    10
        vdev.sync_write_min_active                                    10
        vdev.trim_max_active                                           2
        vdev.trim_min_active                                           1
        vdev.validate_skip                                             0
        vdev.write_gap_limit                                        4096
        version.acl                                                    1
        version.ioctl                                                 15
        version.module                         v2021040600-zfs_79bb891fc
        version.spa                                                 5000
        version.zpl                                                    5
        vnops.read_chunk_size                                    1048576
        vol.mode                                                       2
        vol.recursive                                                  0
        vol.unmap_enabled                                              1
        zap_iterate_prefetch                                           1
        zevent.cols                                                   80
        zevent.console                                                 0
        zevent.len_max                                               256
        zevent.retain_expire_secs                                    900
        zevent.retain_max                                           2000
        zfetch.max_distance                                      8388608
        zfetch.max_idistance                                    67108864
        zil.clean_taskq_maxalloc                                 1048576
        zil.clean_taskq_minalloc                                    1024
        zil.clean_taskq_nthr_pct                                     100
        zil.maxblocksize                                          131072
        zil.nocacheflush                                               0
        zil.replay_disable                                             0
        zil.slog_bulk                                             786432
        zio.deadman_log_all                                            0
        zio.dva_throttle_enabled                                       1
        zio.exclude_metadata                                           0
        zio.requeue_io_start_cut_in_line                               1
        zio.slow_io_ms                                             30000
        zio.taskq_batch_pct                                           75
        zio.use_uma                                                    1

VDEV cache disabled, skipping section

ZIL committed transactions:                                         3.5M
        Commit requests:                                           41.5k
        Flushes to stable storage:                                 41.2k
        Transactions to SLOG storage pool:          256.0 KiB          2
        Transactions to non-SLOG storage pool:        4.4 GiB      68.3k

root@truenas[~]# arcstat
    time  read  miss  miss%  dmis  dm%  pmis  pm%  mmis  mm%  size     c  avail
10:31:27     0     0      0     0    0     0    0     0    0   47G   49G   512M
 

marcevan

Patron
Joined
Dec 15, 2013
Messages
432
At that ratio (MRU MFU) you're completely find for 32GB
 

pschatz100

Guru
Joined
Mar 30, 2014
Messages
1,184
I don't think you need to guess at all. Remove some memory and run for a while with 32GB. The system won't crash or do anything dramatic.

Keep an eye on the system swap. If it doesn't use any swap, then you have enough memory.
 

ThreeDee

Guru
Joined
Jun 13, 2013
Messages
698
I know 16GB would probably be "enough" .. let alone 32GB .. I'm just wondering if anybody had insight as to whether or not I'd lose any performance going from 64gb to 32gb.

I probably wouldn't notice either way .. still pondering :rolleyes:
 

rvassar

Guru
Joined
May 2, 2018
Messages
971
I run on 16GB, with Plex & Zoneminder jails, three pools, and some NFS traffic. I used to run Gitea/Gogs and a general purpose MySQL server, but it displaced enough RAM that I noticed a performance hit. I'd like to have 32GB, and my little X9SCL supports it, but I refuse to pay the "Mac Pro Premium" for the 8GB ECC UDIMM's. My only performance complaint at the moment is the Zoneminder zone alarms lag a second or two behind the events. I'm thinking an Xeon E3v2 might cure this, but I'd rather save the money for a 32GB+ DDR4 replacement build.

Code:
------------------------------------------------------------------------
ZFS Subsystem Report                            Wed May 12 08:26:11 2021
FreeBSD 12.2-RELEASE-p6                                    zpl version 5
Machine: (redacted) (amd64)                 spa version 5000

ARC status:                                                      HEALTHY
        Memory throttle count:                                         0

ARC size (current):                                    72.2 %    9.1 GiB
        Target size (adaptive):                        72.2 %    9.1 GiB
        Min size (hard limit):                          3.9 %  509.6 MiB
        Max size (high water):                           25:1   12.7 GiB
        Most Frequently Used (MFU) cache size:         20.4 %    1.7 GiB
        Most Recently Used (MRU) cache size:           79.6 %    6.7 GiB
        Metadata cache size (hard limit):              75.0 %    9.5 GiB
        Metadata cache size (current):                 12.0 %    1.1 GiB
        Dnode cache size (hard limit):                 10.0 %  972.7 MiB
        Dnode cache size (current):                    28.0 %  272.3 MiB

ARC hash breakdown:
        Elements max:                                               1.7M
        Elements current:                              27.3 %     469.8k
        Collisions:                                                32.6M
        Chain max:                                                     9
        Chains:                                                    45.0k

ARC misc:
        Deleted:                                                   43.1M
        Mutex misses:                                               1.6k
        Eviction skips:                                           644.6k

ARC total accesses (hits + misses):                                 3.3G
        Cache hit ratio:                               99.3 %       3.3G
        Cache miss ratio:                               0.7 %      21.6M
        Actual hit ratio (MFU + MRU hits):             99.3 %       3.3G
        Data demand efficiency:                        97.4 %     328.7M
        Data prefetch efficiency:                      21.2 %       2.2M

Cache hits by cache type:
        Most frequently used (MFU):                    95.0 %       3.1G
        Most recently used (MRU):                       5.0 %     164.2M
        Most frequently used (MFU) ghost:             < 0.1 %     543.6k
        Most recently used (MRU) ghost:               < 0.1 %     650.7k

Cache hits by data type:
        Demand data:                                    9.8 %     320.2M
        Demand prefetch data:                         < 0.1 %     466.2k
        Demand metadata:                               73.0 %       2.4G
        Demand prefetch metadata:                      17.2 %     564.2M

Cache misses by data type:
        Demand data:                                   39.0 %       8.4M
        Demand prefetch data:                           8.0 %       1.7M
        Demand metadata:                               40.7 %       8.8M
        Demand prefetch metadata:                      12.3 %       2.7M

DMU prefetch efficiency:                                          323.2M
        Hit ratio:                                      1.8 %       5.7M
        Miss ratio:                                    98.2 %     317.5M

L2ARC not detected, skipping section

Tunables:
        abd_chunk_size                                              4096
        abd_scatter_enabled                                            1
        allow_redacted_dataset_mount                                   0
        anon_data_esize                                                0
        anon_metadata_esize                                            0
        anon_size                                              101937152
        arc.average_blocksize                                       8192
        arc.dnode_limit                                                0
        arc.dnode_limit_percent                                       10
        arc.dnode_reduce_percent                                      10
        arc.evict_batch_limit                                         10
        arc.eviction_pct                                             200
        arc.grow_retry                                                 0
        arc.lotsfree_percent                                          10
        arc.max                                              13598774528
        arc.meta_adjust_restarts                                    4096
        arc.meta_limit                                                 0
        arc.meta_limit_percent                                        75
        arc.meta_min                                                   0
        arc.meta_prune                                             10000
        arc.meta_strategy                                              1
        arc.min                                                        0
        arc.min_prefetch_ms                                            0
        arc.min_prescient_prefetch_ms                                  0
        arc.p_dampener_disable                                         1
        arc.p_min_shift                                                0
        arc.pc_percent                                                 0
        arc.shrink_shift                                               0
        arc.sys_free                                                   0
        arc_free_target                                            86655
        arc_max                                              13598774528
        arc_min                                                        0
        arc_no_grow_shift                                              5
        async_block_max_blocks                      18446744073709551615
        autoimport_disable                                             1
        ccw_retry_interval                                           300
        checksum_events_per_second                                    20
        commit_timeout_pct                                             5
        compressed_arc_enabled                                         1
        condense.indirect_commit_entry_delay_ms                        0
        condense.indirect_vdevs_enable                                 1
        condense.max_obsolete_bytes                           1073741824
        condense.min_mapping_bytes                                131072
        condense_pct                                                 200
        crypt_sessions                                                 0
        dbgmsg_enable                                                  1
        dbgmsg_maxsize                                           4194304
        dbuf.cache_shift                                               5
        dbuf.metadata_cache_max_bytes               18446744073709551615
        dbuf.metadata_cache_shift                                      6
        dbuf_cache.hiwater_pct                                        10
        dbuf_cache.lowater_pct                                        10
        dbuf_cache.max_bytes                        18446744073709551615
        dbuf_state_index                                               0
        ddt_data_is_special                                            1
        deadman.failmode                                            wait
        deadman.synctime_ms                                       600000
        deadman.ziotime_ms                                        300000
        deadman_checktime_ms                                       60000
        deadman_enabled                                                1
        debug                                                          0
        debugflags                                                     0
        dedup.prefetch                                                 0
        default_bs                                                     9
        default_ibs                                                   15
        delay_min_dirty_percent                                       60
        delay_scale                                               500000
        dirty_data_max                                        1709965721
        dirty_data_max_max                                    4274914304
        dirty_data_max_max_percent                                    25
        dirty_data_max_percent                                        10
        dirty_data_sync_percent                                       20
        disable_ivset_guid_check                                       0
        dmu_object_alloc_chunk_shift                                   7
        dmu_offset_next_sync                                           0
        dmu_prefetch_max                                       134217728
        dtl_sm_blksz                                                4096
        flags                                                          0
        fletcher_4_impl [fastest] scalar superscalar superscalar4 sse2 ssse3
        free_bpobj_enabled                                             1
        free_leak_on_eio                                               0
        free_min_time_ms                                            1000
        history_output_max                                       1048576
        immediate_write_sz                                         32768
        initialize_chunk_size                                    1048576
        initialize_value                            16045690984833335022
        keep_log_spacemaps_at_export                                   0
        l2arc.feed_again                                               1
        l2arc.feed_min_ms                                            200
        l2arc.feed_secs                                                1
        l2arc.headroom                                                 2
        l2arc.headroom_boost                                         200
        l2arc.meta_percent                                            33
        l2arc.mfuonly                                                  0
        l2arc.noprefetch                                               0
        l2arc.norw                                                     0
        l2arc.rebuild_blocks_min_l2size                       1073741824
        l2arc.rebuild_enabled                                          0
        l2arc.trim_ahead                                               0
        l2arc.write_boost                                       40000000
        l2arc.write_max                                         10000000
        l2arc_feed_again                                               1
        l2arc_feed_min_ms                                            200
        l2arc_feed_secs                                                1
        l2arc_headroom                                                 2
        l2arc_noprefetch                                               0
        l2arc_norw                                                     0
        l2arc_write_boost                                       40000000
        l2arc_write_max                                         10000000
        l2c_only_size                                                  0
        livelist.condense.new_alloc                                    0
        livelist.condense.sync_cancel                                  0
        livelist.condense.sync_pause                                   0
        livelist.condense.zthr_cancel                                  0
        livelist.condense.zthr_pause                                   0
        livelist.max_entries                                      500000
        livelist.min_percent_shared                                   75
        lua.max_instrlimit                                     100000000
        lua.max_memlimit                                       104857600
        max_async_dedup_frees                                     100000
        max_auto_ashift                                               16
        max_dataset_nesting                                           50
        max_log_walking                                                5
        max_logsm_summary_length                                      10
        max_missing_tvds                                               0
        max_missing_tvds_cachefile                                     2
        max_missing_tvds_scan                                          0
        max_nvlist_src_size                                            0
        max_recordsize                                           1048576
        metaslab.aliquot                                          524288
        metaslab.bias_enabled                                          1
        metaslab.debug_load                                            0
        metaslab.debug_unload                                          0
        metaslab.df_alloc_threshold                               131072
        metaslab.df_free_pct                                           4
        metaslab.df_max_search                                  16777216
        metaslab.df_use_largest_segment                                0
        metaslab.force_ganging                                  16777217
        metaslab.fragmentation_factor_enabled                          1
        metaslab.fragmentation_threshold                              70
        metaslab.lba_weighting_enabled                                 1
        metaslab.load_pct                                             50
        metaslab.max_size_cache_sec                                 3600
        metaslab.mem_limit                                            75
        metaslab.preload_enabled                                       1
        metaslab.preload_limit                                        10
        metaslab.segment_weight_enabled                                1
        metaslab.sm_blksz_no_log                                   16384
        metaslab.sm_blksz_with_log                                131072
        metaslab.switch_threshold                                      2
        metaslab.unload_delay                                         32
        metaslab.unload_delay_ms                                  600000
        mfu_data_esize                                        1588703232
        mfu_ghost_data_esize                                  2427846656
        mfu_ghost_metadata_esize                              4621679104
        mfu_ghost_size                                        7049525760
        mfu_metadata_esize                                      78632448
        mfu_size                                              1854327808
        mg.fragmentation_threshold                                    95
        mg.noalloc_threshold                                           0
        min_auto_ashift                                                9
        min_metaslabs_to_flush                                         1
        mru_data_esize                                        6559852032
        mru_ghost_data_esize                                   141819904
        mru_ghost_metadata_esize                              2417380352
        mru_ghost_size                                        2559200256
        mru_metadata_esize                                     123365376
        mru_size                                              7246237184
        multihost.fail_intervals                                      10
        multihost.history                                              0
        multihost.import_intervals                                    20
        multihost.interval                                          1000
        multilist_num_sublists                                         0
        no_scrub_io                                                    0
        no_scrub_prefetch                                              0
        nocacheflush                                                   0
        nopwrite_enabled                                               1
        obsolete_min_time_ms                                         500
        pd_bytes_max                                            52428800
        per_txg_dirty_frees_percent                                    5
        prefetch.array_rd_sz                                     1048576
        prefetch.disable                                               0
        prefetch.max_distance                                   33554432
        prefetch.max_idistance                                  67108864
        prefetch.max_streams                                           8
        prefetch.min_sec_reap                                          2
        read_history                                                   0
        read_history_hits                                              0
        rebuild_max_segment                                      1048576
        reconstruct.indirect_combinations_max                       4096
        recover                                                        0
        recv.queue_ff                                                 20
        recv.queue_length                                       16777216
        recv.write_batch_size                                    1048576
        reference_tracking_enable                                      0
        removal_suspend_progress                                       0
        remove_max_segment                                      16777216
        resilver_disable_defer                                         0
        resilver_min_time_ms                                        3000
        scan_checkpoint_intval                                      7200
        scan_fill_weight                                               3
        scan_ignore_errors                                             0
        scan_issue_strategy                                            0
        scan_legacy                                                    0
        scan_max_ext_gap                                         2097152
        scan_mem_lim_fact                                             20
        scan_mem_lim_soft_fact                                        20
        scan_strict_mem_lim                                            0
        scan_suspend_progress                                          0
        scan_vdev_limit                                          4194304
        scrub_min_time_ms                                           1000
        send.corrupt_data                                              0
        send.no_prefetch_queue_ff                                     20
        send.no_prefetch_queue_length                            1048576
        send.override_estimate_recordsize                              0
        send.queue_ff                                                 20
        send.queue_length                                       16777216
        send.unmodified_spill_blocks                                   1
        send_holes_without_birth_time                                  1
        slow_io_events_per_second                                     20
        spa.asize_inflation                                           24
        spa.discard_memory_limit                                16777216
        spa.load_print_vdev_tree                                       0
        spa.load_verify_data                                           1
        spa.load_verify_metadata                                       1
        spa.load_verify_shift                                          4
        spa.slop_shift                                                 5
        space_map_ibs                                                 14
        special_class_metadata_reserve_pct                            25
        standard_sm_blksz                                         131072
        super_owner                                                    0
        sync_pass_deferred_free                                        2
        sync_pass_dont_compress                                        8
        sync_pass_rewrite                                              2
        sync_taskq_batch_pct                                          75
        top_maxinflight                                             1000
        trim.extent_bytes_max                                  134217728
        trim.extent_bytes_min                                      32768
        trim.metaslab_skip                                             0
        trim.queue_limit                                              10
        trim.txg_batch                                                32
        txg.history                                                  100
        txg.timeout                                                    5
        unflushed_log_block_max                                   262144
        unflushed_log_block_min                                     1000
        unflushed_log_block_pct                                      400
        unflushed_max_mem_amt                                 1073741824
        unflushed_max_mem_ppm                                       1000
        user_indirect_is_special                                       1
        validate_skip                                                  0
        vdev.aggregate_trim                                            0
        vdev.aggregation_limit                                   1048576
        vdev.aggregation_limit_non_rotating                       131072
        vdev.async_read_max_active                                     3
        vdev.async_read_min_active                                     1
        vdev.async_write_active_max_dirty_percent                     60
        vdev.async_write_active_min_dirty_percent                     30
        vdev.async_write_max_active                                    5
        vdev.async_write_min_active                                    1
        vdev.bio_delete_disable                                        0
        vdev.bio_flush_disable                                         0
        vdev.cache_bshift                                             16
        vdev.cache_max                                             16384
        vdev.cache_size                                                0
        vdev.def_queue_depth                                          32
        vdev.default_ms_count                                        200
        vdev.default_ms_shift                                         29
        vdev.file.logical_ashift                                       9
        vdev.file.physical_ashift                                      9
        vdev.initializing_max_active                                   1
        vdev.initializing_min_active                                   1
        vdev.max_active                                             1000
        vdev.max_auto_ashift                                          16
        vdev.min_auto_ashift                                           9
        vdev.min_ms_count                                             16
        vdev.mirror.non_rotating_inc                                   0
        vdev.mirror.non_rotating_seek_inc                              1
        vdev.mirror.rotating_inc                                       0
        vdev.mirror.rotating_seek_inc                                  5
        vdev.mirror.rotating_seek_offset                         1048576
        vdev.ms_count_limit                                       131072
        vdev.nia_credit                                                5
        vdev.nia_delay                                                 5
        vdev.queue_depth_pct                                        1000
        vdev.read_gap_limit                                        32768
        vdev.rebuild_max_active                                        3
        vdev.rebuild_min_active                                        1
        vdev.removal_ignore_errors                                     0
        vdev.removal_max_active                                        2
        vdev.removal_max_span                                      32768
        vdev.removal_min_active                                        1
        vdev.removal_suspend_progress                                  0
        vdev.remove_max_segment                                 16777216
        vdev.scrub_max_active                                          3
        vdev.scrub_min_active                                          1
        vdev.sync_read_max_active                                     10
        vdev.sync_read_min_active                                     10
        vdev.sync_write_max_active                                    10
        vdev.sync_write_min_active                                    10
        vdev.trim_max_active                                           2
        vdev.trim_min_active                                           1
        vdev.validate_skip                                             0
        vdev.write_gap_limit                                        4096
        version.acl                                                    1
        version.ioctl                                                 15
        version.module                         v2021040600-zfs_79bb891fc
        version.spa                                                 5000
        version.zpl                                                    5
        vnops.read_chunk_size                                    1048576
        vol.mode                                                       2
        vol.recursive                                                  0
        vol.unmap_enabled                                              1
        zap_iterate_prefetch                                           1
        zevent.cols                                                   80
        zevent.console                                                 0
        zevent.len_max                                                64
        zevent.retain_expire_secs                                    900
        zevent.retain_max                                           2000
        zfetch.max_distance                                     33554432
        zfetch.max_idistance                                    67108864
        zil.clean_taskq_maxalloc                                 1048576
        zil.clean_taskq_minalloc                                    1024
        zil.clean_taskq_nthr_pct                                     100
        zil.maxblocksize                                          131072
        zil.nocacheflush                                               0
        zil.replay_disable                                             0
        zil.slog_bulk                                             786432
        zio.deadman_log_all                                            0
        zio.dva_throttle_enabled                                       1
        zio.exclude_metadata                                           0
        zio.requeue_io_start_cut_in_line                               1
        zio.slow_io_ms                                             30000
        zio.taskq_batch_pct                                           75
        zio.use_uma                                                    1

VDEV cache disabled, skipping section

ZIL committed transactions:                                        42.5M
        Commit requests:                                           14.9M
        Flushes to stable storage:                                 14.9M
        Transactions to SLOG storage pool:          372.2 GiB      17.0M
        Transactions to non-SLOG storage pool:      385.7 MiB      17.7k


 

Arwen

MVP
Joined
May 17, 2014
Messages
3,600
A while back I moved my media server from a tiny box with only 1 SO-DIMM to an exact same CPU & general hardware, but with 2 SO-DIMMs. Those SO-DIMMs were on the same memory channel, but hey, it allowed me to double the amount of memory it had. Since they were identical CPUs and support hardware, I thought I would test the old with 1 SO-DIMM, and the new with 2 identical SO-DIMMs. The new dual SO-DIMM computer was noticeably faster in a memory test, more than double. Don't know why there was such a difference.

So, for my new AMD Epyc NAS, I wanted to start with 2 separate memory channels and 2 DIMMs. Plus some expansion memory channels, (not 2nd slot on the same channels). (The lower end embedded Epycs only have 2 memory channels.)

All that said, the average person probably won't notice much of a change going from 2 DIMMs to 1 DIMM. Or 4 to 2. Just try and use as many memory channels as you can.
 

HoneyBadger

actually does care
Administrator
Moderator
iXsystems
Joined
Feb 6, 2014
Messages
5,110
I work nights .. I'll post the results of arc_summary as soon as I get home from work .. it seems that the .pl isn't needed for the command in TrueNAS
Sorry about the delay here. As others have mentioned you should be fine going to 32GB.

You're getting a 99% hit rate, with about 2/3rd of your hits serving metadata. The remaining 1/3rd is almost exclusively (97%) MFU hits.

Basically, what's frequently hit (the SMB shares, Plex/Unifi jail, and metadata) is being easily served out of the MFU piece. The MRU chunk is just basically prefetch and speculation about what might get hit, but it's not doing a particularly good job and won't really be harmed by its absence. Your jails are taking a fair bit off the top so to speak though (I'd guess between 8-10GB depending on how much is allocated for your "Services" under TrueNAS) so I wouldn't suggest going anywhere below 32GB.

@Arwen While more channels is always better, I would always go for capacity first with "number of channels" being secondary. Eg: if I have an X11SCM, I'm going to do my best to afford a 1x32GB DIMM instead of 2x16GB (or 1x16GB instead of 2x8GB) just to have the upgrade path clear for as much capacity as possible. For secondhand gear it's still awfully hard to beat the price-to-ARC ratio of the X9 SuperMicro boards stuffed to the gills with cheap DDR3 RDIMMs.
 

Arwen

MVP
Joined
May 17, 2014
Messages
3,600
@HoneyBadger Absolutely. Leaving an upgrade path is always good. So 1 x 32GB DIMM when you have 4 slots, (2 channels, 2 slots per channel), is a perfectly acceptable compromise over 2 x 16GB DIMMs.

In my case, AMD Epycs support 8 memory channels and the smaller board I bought only has 6 DIMM slots. So, 1 DIMM per channel... and the price was acceptable for 2 x 32GB DIMMs even though as a backup server, this NAS really only needs 16GBs. Leaves me the option to add another 4 x 32GB DIMMs. (Their were few qualified DIMMs... so 6 x 8GB or 4 x 16GB DIMMs was not really an option :-(.
 
Top