SOLVED Invalid Argument on tunable vfs.zfs.arc_max

[Snow]

So for awhile now I have been fighting with this tunable. On boot the tunable gives Invalid Argument when trying to set any value.
This then defaults to 16GB. I have tried adjusting the vm.kmem and max to 64GB this does not fix the issue. It still acts the same.
I have been trying to set arc_max between 32GB and 48GB. Due to arc taking up all of my 128GB system memory. I understand that
arc is behaving correctly with default memory usage. I would like to make sure its limited so I can utilize the memory for VM'S & other use's.

Any and all help would be great thanks.

Code:

root@freenas:~ # zfs-stats -M -s -A
------------------------------------------------------------------------
sysctl: unknown oid 'kstat.zfs.misc.arcstats.l2_writes_hdr_miss'
sysctl: unknown oid 'kstat.zfs.misc.arcstats.recycle_miss'
sysctl: unknown oid 'kstat.zfs.misc.zfetchstats.bogus_streams'
sysctl: unknown oid 'kstat.zfs.misc.zfetchstats.colinear_hits'
sysctl: unknown oid 'kstat.zfs.misc.zfetchstats.colinear_misses'
sysctl: unknown oid 'kstat.zfs.misc.zfetchstats.reclaim_failures'
sysctl: unknown oid 'kstat.zfs.misc.zfetchstats.reclaim_successes'
sysctl: unknown oid 'kstat.zfs.misc.zfetchstats.streams_noresets'
sysctl: unknown oid 'kstat.zfs.misc.zfetchstats.streams_resets'
sysctl: unknown oid 'kstat.zfs.misc.zfetchstats.stride_hits'
sysctl: unknown oid 'kstat.zfs.misc.zfetchstats.stride_misses'
ZFS Subsystem Report                            Mon Feb  4 22:03:53 2019
------------------------------------------------------------------------
System Memory Statistics:
        Physical Memory:                        130998.93M
        Kernel Memory:                          1479.76M
        DATA:                           97.14%  1437.48M
        TEXT:                           2.85%   42.27M
------------------------------------------------------------------------
ZFS Tunable (sysctl):
        kern.maxusers=8523
        vfs.zfs.vol.immediate_write_sz=32768
        vfs.zfs.vol.unmap_sync_enabled=0
        vfs.zfs.vol.unmap_enabled=1
        vfs.zfs.vol.recursive=0
        vfs.zfs.vol.mode=2
        vfs.zfs.sync_pass_rewrite=2
        vfs.zfs.sync_pass_dont_compress=5
        vfs.zfs.sync_pass_deferred_free=2
        vfs.zfs.zio.dva_throttle_enabled=1
        vfs.zfs.zio.exclude_metadata=0
        vfs.zfs.zio.use_uma=1
        vfs.zfs.zil_slog_bulk=786432
        vfs.zfs.cache_flush_disable=0
        vfs.zfs.zil_replay_disable=0
        vfs.zfs.version.zpl=5
        vfs.zfs.version.spa=5000
        vfs.zfs.version.acl=1
        vfs.zfs.version.ioctl=7
        vfs.zfs.debug=0
        vfs.zfs.super_owner=0
        vfs.zfs.immediate_write_sz=32768
        vfs.zfs.standard_sm_blksz=131072
        vfs.zfs.dtl_sm_blksz=4096
        vfs.zfs.min_auto_ashift=12
        vfs.zfs.max_auto_ashift=13
        vfs.zfs.vdev.queue_depth_pct=1000
        vfs.zfs.vdev.write_gap_limit=4096
        vfs.zfs.vdev.read_gap_limit=32768
        vfs.zfs.vdev.aggregation_limit=1048576
        vfs.zfs.vdev.trim_max_active=64
        vfs.zfs.vdev.trim_min_active=1
        vfs.zfs.vdev.scrub_max_active=2
        vfs.zfs.vdev.scrub_min_active=1
        vfs.zfs.vdev.async_write_max_active=10
        vfs.zfs.vdev.async_write_min_active=1
        vfs.zfs.vdev.async_read_max_active=3
        vfs.zfs.vdev.async_read_min_active=1
        vfs.zfs.vdev.sync_write_max_active=10
        vfs.zfs.vdev.sync_write_min_active=10
        vfs.zfs.vdev.sync_read_max_active=10
        vfs.zfs.vdev.sync_read_min_active=10
        vfs.zfs.vdev.max_active=1000
        vfs.zfs.vdev.async_write_active_max_dirty_percent=60
        vfs.zfs.vdev.async_write_active_min_dirty_percent=30
        vfs.zfs.vdev.mirror.non_rotating_seek_inc=1
        vfs.zfs.vdev.mirror.non_rotating_inc=0
        vfs.zfs.vdev.mirror.rotating_seek_offset=1048576
        vfs.zfs.vdev.mirror.rotating_seek_inc=5
        vfs.zfs.vdev.mirror.rotating_inc=0
        vfs.zfs.vdev.trim_on_init=1
        vfs.zfs.vdev.bio_delete_disable=0
        vfs.zfs.vdev.bio_flush_disable=0
        vfs.zfs.vdev.cache.bshift=16
        vfs.zfs.vdev.cache.size=0
        vfs.zfs.vdev.cache.max=16384
        vfs.zfs.vdev.default_ms_shift=29
        vfs.zfs.vdev.min_ms_count=16
        vfs.zfs.vdev.max_ms_count=200
        vfs.zfs.vdev.trim_max_pending=10000
        vfs.zfs.txg.timeout=5
        vfs.zfs.trim.enabled=1
        vfs.zfs.trim.max_interval=1
        vfs.zfs.trim.timeout=30
        vfs.zfs.trim.txg_delay=32
        vfs.zfs.spa_min_slop=134217728
        vfs.zfs.spa_slop_shift=5
        vfs.zfs.spa_asize_inflation=24
        vfs.zfs.deadman_enabled=1
        vfs.zfs.deadman_checktime_ms=5000
        vfs.zfs.deadman_synctime_ms=1000000
        vfs.zfs.debug_flags=0
        vfs.zfs.debugflags=0
        vfs.zfs.recover=0
        vfs.zfs.spa_load_verify_data=1
        vfs.zfs.spa_load_verify_metadata=1
        vfs.zfs.spa_load_verify_maxinflight=10000
        vfs.zfs.max_missing_tvds_scan=0
        vfs.zfs.max_missing_tvds_cachefile=2
        vfs.zfs.max_missing_tvds=0
        vfs.zfs.spa_load_print_vdev_tree=0
        vfs.zfs.ccw_retry_interval=300
        vfs.zfs.check_hostid=1
        vfs.zfs.mg_fragmentation_threshold=85
        vfs.zfs.mg_noalloc_threshold=0
        vfs.zfs.condense_pct=200
        vfs.zfs.metaslab_sm_blksz=4096
        vfs.zfs.metaslab.bias_enabled=1
        vfs.zfs.metaslab.lba_weighting_enabled=1
        vfs.zfs.metaslab.fragmentation_factor_enabled=1
        vfs.zfs.metaslab.preload_enabled=1
        vfs.zfs.metaslab.preload_limit=3
        vfs.zfs.metaslab.unload_delay=8
        vfs.zfs.metaslab.load_pct=50
        vfs.zfs.metaslab.min_alloc_size=33554432
        vfs.zfs.metaslab.df_free_pct=4
        vfs.zfs.metaslab.df_alloc_threshold=131072
        vfs.zfs.metaslab.debug_unload=0
        vfs.zfs.metaslab.debug_load=0
        vfs.zfs.metaslab.fragmentation_threshold=70
        vfs.zfs.metaslab.force_ganging=16777217
        vfs.zfs.free_bpobj_enabled=1
        vfs.zfs.free_max_blocks=18446744073709551615
        vfs.zfs.zfs_scan_checkpoint_interval=7200
        vfs.zfs.zfs_scan_legacy=0
        vfs.zfs.no_scrub_prefetch=0
        vfs.zfs.no_scrub_io=0
        vfs.zfs.resilver_min_time_ms=3000
        vfs.zfs.free_min_time_ms=1000
        vfs.zfs.scan_min_time_ms=1000
        vfs.zfs.scan_idle=50
        vfs.zfs.scrub_delay=4
        vfs.zfs.resilver_delay=2
        vfs.zfs.top_maxinflight=32
        vfs.zfs.delay_scale=500000
        vfs.zfs.delay_min_dirty_percent=60
        vfs.zfs.dirty_data_sync=67108864
        vfs.zfs.dirty_data_max_percent=10
        vfs.zfs.dirty_data_max_max=4294967296
        vfs.zfs.dirty_data_max=4294967296
        vfs.zfs.max_recordsize=1048576
        vfs.zfs.default_ibs=15
        vfs.zfs.default_bs=9
        vfs.zfs.zfetch.array_rd_sz=1048576
        vfs.zfs.zfetch.max_idistance=67108864
        vfs.zfs.zfetch.max_distance=8388608
        vfs.zfs.zfetch.min_sec_reap=2
        vfs.zfs.zfetch.max_streams=8
        vfs.zfs.prefetch_disable=0
        vfs.zfs.send_holes_without_birth_time=1
        vfs.zfs.mdcomp_disable=0
        vfs.zfs.per_txg_dirty_frees_percent=30
        vfs.zfs.nopwrite_enabled=1
        vfs.zfs.dedup.prefetch=1
        vfs.zfs.dbuf_cache_lowater_pct=10
        vfs.zfs.dbuf_cache_hiwater_pct=10
        vfs.zfs.dbuf_cache_shift=5
        vfs.zfs.dbuf_cache_max_bytes=4150591872
        vfs.zfs.arc_min_prescient_prefetch_ms=6
        vfs.zfs.arc_min_prefetch_ms=1
        vfs.zfs.l2c_only_size=0
        vfs.zfs.mfu_ghost_data_esize=0
        vfs.zfs.mfu_ghost_metadata_esize=0
        vfs.zfs.mfu_ghost_size=0
        vfs.zfs.mfu_data_esize=88935424
        vfs.zfs.mfu_metadata_esize=2140160
        vfs.zfs.mfu_size=3585968128
        vfs.zfs.mru_ghost_data_esize=0
        vfs.zfs.mru_ghost_metadata_esize=0
        vfs.zfs.mru_ghost_size=0
        vfs.zfs.mru_data_esize=10654292992
        vfs.zfs.mru_metadata_esize=14049280
        vfs.zfs.mru_size=14751316480
        vfs.zfs.anon_data_esize=0
        vfs.zfs.anon_metadata_esize=0
        vfs.zfs.anon_size=52619776
        vfs.zfs.l2arc_norw=1
        vfs.zfs.l2arc_feed_again=1
        vfs.zfs.l2arc_noprefetch=0
        vfs.zfs.l2arc_feed_min_ms=200
        vfs.zfs.l2arc_feed_secs=1
        vfs.zfs.l2arc_headroom=2
        vfs.zfs.l2arc_write_boost=8388608
        vfs.zfs.l2arc_write_max=8388608
        vfs.zfs.arc_meta_limit=33204734976
        vfs.zfs.arc_free_target=696397
        vfs.zfs.arc_kmem_cache_reap_retry_ms=0
        vfs.zfs.compressed_arc_enabled=1
        vfs.zfs.arc_grow_retry=60
        vfs.zfs.arc_shrink_shift=7
        vfs.zfs.arc_average_blocksize=8192
        vfs.zfs.arc_no_grow_shift=5
        vfs.zfs.arc_min=16606053120
        vfs.zfs.arc_max=132818939904
        vfs.zfs.abd_chunk_size=4096
        vm.kmem_size=267785363456
        vm.kmem_size_scale=1
        vm.kmem_size_min=0
        vm.kmem_size_max=549755813888

root@freenas:~ # sysctl -a |grep vm.kmem
vm.kmem_map_free: 260812484608
vm.kmem_map_size: 6972878848
vm.kmem_size_scale: 1
vm.kmem_size_max: 549755813888
vm.kmem_size_min: 0
vm.kmem_zmax: 65536
vm.kmem_size: 267785363456

 

[DrKK]

Can you show us exactly what command you are using to set this tunable?

 

[Snow]

vfs.zfs.arc_max

 

[DrKK]

That's not a command. I understand you are setting this tunable. Please show me what command you are using to set it.

 

[Snow]

I am sorry not using a command just adding it to tunable as a sysctl, Is this not how you are suppose to set up arc_max as a tunable? If I do a reboot to see if it took thats where I get the Invalid Argument from.

 

[Jessep]

This is what I have set:

System>Tunables>Add Tunable

 

[Snow]

yes I do it the same way and it will not set! no matter what I set arc to it uses like 110Gb.
vfs.zfs.arc_max 274,877,906,944 sysctl Enabled yes I had it set to bits and not bytes sorry Some one was reading me back the numbers last night and I was like why is that number so big. I am such a noob

 

[DrKK]

ok, so if you set to a number that's impossible, which is what would happen with bits, and not bytes, since it is too high, then you're going to get defaults.

Is it working now?

 

[Snow]

Yes Sir it is I feel Like a dork I know better to Lol :).

 

[DrKK]

Very good sir, I am glad you were able to solve it.