To set vfs.zfs.arc_max or not?

[Elliot Dierksen]

I did some forum searching to see if setting vfs.zfs.arc_max is required/recommended. I know it would be if you are running VM's and you want to make sure some memory is reserved for the VM's, but that is not my use case. I am running my units bare metal storage only. Both units are Cisco C240 M4's with 256G RAM, E5-2637 v3 CPU's, LSI 3108 based HBA, Chelsio T580 40G NIC's on the storage side. My gut is that I shouldn't need vfs.zfs.arc_max, but I don't want to shoot myself in the foot. This was all brought to mind by a recent post from @Spearfoot regarding 10G network tuning. Both units are tuned the same as follows:

Code:

Variable            Value        Type    Description        Enabled
dev.t5nex.0.toe.ddp        1        sysctl    T5 manual tune        no
dev.t5nex.0.toe.tx_zcopy    1        sysctl    T5 manual tune        no
hw.pci.honor_msi_blacklist    0        loader    Spearfoot        yes
kern.ipc.maxsockbuf        16777216    sysctl    Soearfoot        yes
kern.ipc.nmbclusters        16352042    sysctl    Generated by autotune    yes
kern.ipc.soacceptqueue        1024        sysctl    Spearfoot        yes
net.inet.ip.intr_queue_maxlen    2048        sysctl    Spearfoot        yes
net.inet.tcp.blackhole        2        sysctl    Disable firewall    no
net.inet.tcp.delayed_ack    0        sysctl    Generated by autotune    yes
net.inet.tcp.mssdflt        1448        sysctl    Generated by autotune    yes
net.inet.tcp.recvbuf_inc    4194304        sysctl    Spearfoot        yes
net.inet.tcp.recvbuf_max    16777216    sysctl    Spearfoot        yes
net.inet.tcp.recvspace        4194304        sysctl    Spearfoot        yes
net.inet.tcp.sendbuf_inc    4194304        sysctl    Spearfoot        yes
net.inet.tcp.sendbuf_max    16777216    sysctl    Spearfoot        yes
net.inet.tcp.sendspace        4194304        sysctl    Spearfoot        yes
net.inet.udp.blackhole        1        sysctl    Disable firewall    no
net.route.netisr_maxqlen    2048        sysctl    Spearfoot        yes
vfs.usermount            1        sysctl    Replication        no
vfs.zfs.arc_max            247311000000    sysctl    Generated by autotune    yes
vfs.zfs.dirty_data_max        ‭34359738368‬    sysctl    Manual tune to 32G    yes
vfs.zfs.l2arc_headroom        2        sysctl    Generated by autotune    yes
vfs.zfs.l2arc_noprefetch    0        sysctl    Generated by autotune    yes
vfs.zfs.l2arc_norw        0        sysctl    Generated by autotune    yes
vfs.zfs.l2arc_write_boost    40000000    sysctl    Generated by autotune    yes
vfs.zfs.l2arc_write_max        10000000    sysctl    Generated by autotune    yes
vfs.zfs.metaslab.lba_weighting_enabled    1    sysctl    Generated by autotune    yes
vfs.zfs.zfetch.max_distance    33554432    sysctl    Generated by autotune    yes

I did have auto-tune enabled at one point, but it isn't enabled now. I used that as a starting point. My desire is to wring as much performance out of my 40G connections as possible. Both units have an Optane 900P SLOG on each pool since I use NFS from ESXi 6.7 for sharing storage to the ESXi hosts.

 

[Elliot Dierksen]

Perhaps I rambled too much. Is it advisable/required to set vfs.arc_max in a bare metal storage only use case?

 

[hescominsoon]

If i read this correct the nas is being used as a data store only and not running the actual vm's themselves? If this is the case I would think not.