SOLVED Invalid Argument on tunable vfs.zfs.arc_max

Snow

Patron
Joined
Aug 1, 2014
Messages
309
So for awhile now I have been fighting with this tunable. On boot the tunable gives Invalid Argument when trying to set any value.
This then defaults to 16GB. I have tried adjusting the vm.kmem and max to 64GB this does not fix the issue. It still acts the same.
I have been trying to set arc_max between 32GB and 48GB. Due to arc taking up all of my 128GB system memory. I understand that
arc is behaving correctly with default memory usage. I would like to make sure its limited so I can utilize the memory for VM'S & other use's.

Any and all help would be great thanks.

Code:
root@freenas:~ # zfs-stats -M -s -A
------------------------------------------------------------------------
sysctl: unknown oid 'kstat.zfs.misc.arcstats.l2_writes_hdr_miss'
sysctl: unknown oid 'kstat.zfs.misc.arcstats.recycle_miss'
sysctl: unknown oid 'kstat.zfs.misc.zfetchstats.bogus_streams'
sysctl: unknown oid 'kstat.zfs.misc.zfetchstats.colinear_hits'
sysctl: unknown oid 'kstat.zfs.misc.zfetchstats.colinear_misses'
sysctl: unknown oid 'kstat.zfs.misc.zfetchstats.reclaim_failures'
sysctl: unknown oid 'kstat.zfs.misc.zfetchstats.reclaim_successes'
sysctl: unknown oid 'kstat.zfs.misc.zfetchstats.streams_noresets'
sysctl: unknown oid 'kstat.zfs.misc.zfetchstats.streams_resets'
sysctl: unknown oid 'kstat.zfs.misc.zfetchstats.stride_hits'
sysctl: unknown oid 'kstat.zfs.misc.zfetchstats.stride_misses'
ZFS Subsystem Report                            Mon Feb  4 22:03:53 2019
------------------------------------------------------------------------
System Memory Statistics:
        Physical Memory:                        130998.93M
        Kernel Memory:                          1479.76M
        DATA:                           97.14%  1437.48M
        TEXT:                           2.85%   42.27M
------------------------------------------------------------------------
ZFS Tunable (sysctl):
        kern.maxusers=8523
        vfs.zfs.vol.immediate_write_sz=32768
        vfs.zfs.vol.unmap_sync_enabled=0
        vfs.zfs.vol.unmap_enabled=1
        vfs.zfs.vol.recursive=0
        vfs.zfs.vol.mode=2
        vfs.zfs.sync_pass_rewrite=2
        vfs.zfs.sync_pass_dont_compress=5
        vfs.zfs.sync_pass_deferred_free=2
        vfs.zfs.zio.dva_throttle_enabled=1
        vfs.zfs.zio.exclude_metadata=0
        vfs.zfs.zio.use_uma=1
        vfs.zfs.zil_slog_bulk=786432
        vfs.zfs.cache_flush_disable=0
        vfs.zfs.zil_replay_disable=0
        vfs.zfs.version.zpl=5
        vfs.zfs.version.spa=5000
        vfs.zfs.version.acl=1
        vfs.zfs.version.ioctl=7
        vfs.zfs.debug=0
        vfs.zfs.super_owner=0
        vfs.zfs.immediate_write_sz=32768
        vfs.zfs.standard_sm_blksz=131072
        vfs.zfs.dtl_sm_blksz=4096
        vfs.zfs.min_auto_ashift=12
        vfs.zfs.max_auto_ashift=13
        vfs.zfs.vdev.queue_depth_pct=1000
        vfs.zfs.vdev.write_gap_limit=4096
        vfs.zfs.vdev.read_gap_limit=32768
        vfs.zfs.vdev.aggregation_limit=1048576
        vfs.zfs.vdev.trim_max_active=64
        vfs.zfs.vdev.trim_min_active=1
        vfs.zfs.vdev.scrub_max_active=2
        vfs.zfs.vdev.scrub_min_active=1
        vfs.zfs.vdev.async_write_max_active=10
        vfs.zfs.vdev.async_write_min_active=1
        vfs.zfs.vdev.async_read_max_active=3
        vfs.zfs.vdev.async_read_min_active=1
        vfs.zfs.vdev.sync_write_max_active=10
        vfs.zfs.vdev.sync_write_min_active=10
        vfs.zfs.vdev.sync_read_max_active=10
        vfs.zfs.vdev.sync_read_min_active=10
        vfs.zfs.vdev.max_active=1000
        vfs.zfs.vdev.async_write_active_max_dirty_percent=60
        vfs.zfs.vdev.async_write_active_min_dirty_percent=30
        vfs.zfs.vdev.mirror.non_rotating_seek_inc=1
        vfs.zfs.vdev.mirror.non_rotating_inc=0
        vfs.zfs.vdev.mirror.rotating_seek_offset=1048576
        vfs.zfs.vdev.mirror.rotating_seek_inc=5
        vfs.zfs.vdev.mirror.rotating_inc=0
        vfs.zfs.vdev.trim_on_init=1
        vfs.zfs.vdev.bio_delete_disable=0
        vfs.zfs.vdev.bio_flush_disable=0
        vfs.zfs.vdev.cache.bshift=16
        vfs.zfs.vdev.cache.size=0
        vfs.zfs.vdev.cache.max=16384
        vfs.zfs.vdev.default_ms_shift=29
        vfs.zfs.vdev.min_ms_count=16
        vfs.zfs.vdev.max_ms_count=200
        vfs.zfs.vdev.trim_max_pending=10000
        vfs.zfs.txg.timeout=5
        vfs.zfs.trim.enabled=1
        vfs.zfs.trim.max_interval=1
        vfs.zfs.trim.timeout=30
        vfs.zfs.trim.txg_delay=32
        vfs.zfs.spa_min_slop=134217728
        vfs.zfs.spa_slop_shift=5
        vfs.zfs.spa_asize_inflation=24
        vfs.zfs.deadman_enabled=1
        vfs.zfs.deadman_checktime_ms=5000
        vfs.zfs.deadman_synctime_ms=1000000
        vfs.zfs.debug_flags=0
        vfs.zfs.debugflags=0
        vfs.zfs.recover=0
        vfs.zfs.spa_load_verify_data=1
        vfs.zfs.spa_load_verify_metadata=1
        vfs.zfs.spa_load_verify_maxinflight=10000
        vfs.zfs.max_missing_tvds_scan=0
        vfs.zfs.max_missing_tvds_cachefile=2
        vfs.zfs.max_missing_tvds=0
        vfs.zfs.spa_load_print_vdev_tree=0
        vfs.zfs.ccw_retry_interval=300
        vfs.zfs.check_hostid=1
        vfs.zfs.mg_fragmentation_threshold=85
        vfs.zfs.mg_noalloc_threshold=0
        vfs.zfs.condense_pct=200
        vfs.zfs.metaslab_sm_blksz=4096
        vfs.zfs.metaslab.bias_enabled=1
        vfs.zfs.metaslab.lba_weighting_enabled=1
        vfs.zfs.metaslab.fragmentation_factor_enabled=1
        vfs.zfs.metaslab.preload_enabled=1
        vfs.zfs.metaslab.preload_limit=3
        vfs.zfs.metaslab.unload_delay=8
        vfs.zfs.metaslab.load_pct=50
        vfs.zfs.metaslab.min_alloc_size=33554432
        vfs.zfs.metaslab.df_free_pct=4
        vfs.zfs.metaslab.df_alloc_threshold=131072
        vfs.zfs.metaslab.debug_unload=0
        vfs.zfs.metaslab.debug_load=0
        vfs.zfs.metaslab.fragmentation_threshold=70
        vfs.zfs.metaslab.force_ganging=16777217
        vfs.zfs.free_bpobj_enabled=1
        vfs.zfs.free_max_blocks=18446744073709551615
        vfs.zfs.zfs_scan_checkpoint_interval=7200
        vfs.zfs.zfs_scan_legacy=0
        vfs.zfs.no_scrub_prefetch=0
        vfs.zfs.no_scrub_io=0
        vfs.zfs.resilver_min_time_ms=3000
        vfs.zfs.free_min_time_ms=1000
        vfs.zfs.scan_min_time_ms=1000
        vfs.zfs.scan_idle=50
        vfs.zfs.scrub_delay=4
        vfs.zfs.resilver_delay=2
        vfs.zfs.top_maxinflight=32
        vfs.zfs.delay_scale=500000
        vfs.zfs.delay_min_dirty_percent=60
        vfs.zfs.dirty_data_sync=67108864
        vfs.zfs.dirty_data_max_percent=10
        vfs.zfs.dirty_data_max_max=4294967296
        vfs.zfs.dirty_data_max=4294967296
        vfs.zfs.max_recordsize=1048576
        vfs.zfs.default_ibs=15
        vfs.zfs.default_bs=9
        vfs.zfs.zfetch.array_rd_sz=1048576
        vfs.zfs.zfetch.max_idistance=67108864
        vfs.zfs.zfetch.max_distance=8388608
        vfs.zfs.zfetch.min_sec_reap=2
        vfs.zfs.zfetch.max_streams=8
        vfs.zfs.prefetch_disable=0
        vfs.zfs.send_holes_without_birth_time=1
        vfs.zfs.mdcomp_disable=0
        vfs.zfs.per_txg_dirty_frees_percent=30
        vfs.zfs.nopwrite_enabled=1
        vfs.zfs.dedup.prefetch=1
        vfs.zfs.dbuf_cache_lowater_pct=10
        vfs.zfs.dbuf_cache_hiwater_pct=10
        vfs.zfs.dbuf_cache_shift=5
        vfs.zfs.dbuf_cache_max_bytes=4150591872
        vfs.zfs.arc_min_prescient_prefetch_ms=6
        vfs.zfs.arc_min_prefetch_ms=1
        vfs.zfs.l2c_only_size=0
        vfs.zfs.mfu_ghost_data_esize=0
        vfs.zfs.mfu_ghost_metadata_esize=0
        vfs.zfs.mfu_ghost_size=0
        vfs.zfs.mfu_data_esize=88935424
        vfs.zfs.mfu_metadata_esize=2140160
        vfs.zfs.mfu_size=3585968128
        vfs.zfs.mru_ghost_data_esize=0
        vfs.zfs.mru_ghost_metadata_esize=0
        vfs.zfs.mru_ghost_size=0
        vfs.zfs.mru_data_esize=10654292992
        vfs.zfs.mru_metadata_esize=14049280
        vfs.zfs.mru_size=14751316480
        vfs.zfs.anon_data_esize=0
        vfs.zfs.anon_metadata_esize=0
        vfs.zfs.anon_size=52619776
        vfs.zfs.l2arc_norw=1
        vfs.zfs.l2arc_feed_again=1
        vfs.zfs.l2arc_noprefetch=0
        vfs.zfs.l2arc_feed_min_ms=200
        vfs.zfs.l2arc_feed_secs=1
        vfs.zfs.l2arc_headroom=2
        vfs.zfs.l2arc_write_boost=8388608
        vfs.zfs.l2arc_write_max=8388608
        vfs.zfs.arc_meta_limit=33204734976
        vfs.zfs.arc_free_target=696397
        vfs.zfs.arc_kmem_cache_reap_retry_ms=0
        vfs.zfs.compressed_arc_enabled=1
        vfs.zfs.arc_grow_retry=60
        vfs.zfs.arc_shrink_shift=7
        vfs.zfs.arc_average_blocksize=8192
        vfs.zfs.arc_no_grow_shift=5
        vfs.zfs.arc_min=16606053120
        vfs.zfs.arc_max=132818939904
        vfs.zfs.abd_chunk_size=4096
        vm.kmem_size=267785363456
        vm.kmem_size_scale=1
        vm.kmem_size_min=0
        vm.kmem_size_max=549755813888


root@freenas:~ # sysctl -a |grep vm.kmem
vm.kmem_map_free: 260812484608
vm.kmem_map_size: 6972878848
vm.kmem_size_scale: 1
vm.kmem_size_max: 549755813888
vm.kmem_size_min: 0
vm.kmem_zmax: 65536
vm.kmem_size: 267785363456
 
Last edited:

DrKK

FreeNAS Generalissimo
Joined
Oct 15, 2013
Messages
3,630
Can you show us exactly what command you are using to set this tunable?
 

Snow

Patron
Joined
Aug 1, 2014
Messages
309
vfs.zfs.arc_max
 

DrKK

FreeNAS Generalissimo
Joined
Oct 15, 2013
Messages
3,630
That's not a command. I understand you are setting this tunable. Please show me what command you are using to set it.
 

Snow

Patron
Joined
Aug 1, 2014
Messages
309
I am sorry not using a command just adding it to tunable as a sysctl, Is this not how you are suppose to set up arc_max as a tunable? If I do a reboot to see if it took thats where I get the Invalid Argument from.
 
Last edited:

Jessep

Patron
Joined
Aug 19, 2018
Messages
379
This is what I have set:

System>Tunables>Add Tunable

freenas system tunable.PNG
 

Snow

Patron
Joined
Aug 1, 2014
Messages
309
yes I do it the same way and it will not set! no matter what I set arc to it uses like 110Gb.
vfs.zfs.arc_max 274,877,906,944 sysctl Enabled yes I had it set to bits and not bytes sorry Some one was reading me back the numbers last night and I was like why is that number so big. I am such a noob
 
Last edited:

DrKK

FreeNAS Generalissimo
Joined
Oct 15, 2013
Messages
3,630
ok, so if you set to a number that's impossible, which is what would happen with bits, and not bytes, since it is too high, then you're going to get defaults.

Is it working now?
 

Snow

Patron
Joined
Aug 1, 2014
Messages
309
Yes Sir it is I feel Like a dork I know better to Lol :).
 

DrKK

FreeNAS Generalissimo
Joined
Oct 15, 2013
Messages
3,630
Very good sir, I am glad you were able to solve it.
 
Top