Increasing SCALE ARC size

[Kris Moore]

We are pleased to see that upstream OpenZFS has merged changes which will increase the default ARC sizes on Linux to match what we had on FreeBSD. This change is expected to land in the next major release of TrueNAS SCALE 24.04 (Dragonfish).

In the meantime, it is possible for you to manually set this tunable to help us test and validate that this is working as expected. If you want to change the default tunings of ZFS ARC, you can do so by using the following commands:

To set a new larger ARC Size:

# echo <SIZE IN BYTES> >> /sys/module/zfs/parameters/zfs_arc_max

For example:

# echo 47191459840 >> /sys/module/zfs/parameters/zfs_arc_max


To make the setting persist you will need to setup a post-init script to run it at bootup. You can reset it to '0' to return to default behavior.

EDIT:

The original instructions didn't apply correctly. Adjusted to correct procedure.

 

[HoneyBadger]

Further to Kris' note above, ensure that when tuning this setting, you account for additional workloads on your system such as Apps, VMs, and the TrueNAS middleware itself. While ARC can be released as needed when other system services demand memory, maintaining a relatively consistent ARC target value will let the ZFS algorithms do their best work balancing your most frequently vs. recently used files, prefetching, and other optimizations.

 

[sfatula]

Is this for Cobia only (the ability to set the kernal option)?

 

[Kris Moore]

Cobia and Bluefin should both be fine using this.

 

[PhilD13]

I am using the command:
echo 88046829568 >> /sys/module/zfs/parameters/zfs_arc_max
in a PostInit and that has been working fine for me.

Would there be any difference to doing it this way (from cli):
system advanced update kernel_extra_options="zfs_arc_max=88046829568"

 

[HoneyBadger]

I am using the command:
echo 88046829568 >> /sys/module/zfs/parameters/zfs_arc_max
in a PostInit and that has been working fine for me.

Would there be any difference to doing it this way (from cli):
system advanced update kernel_extra_options="zfs_arc_max=88046829568"

Shouldn't be a functional difference, zfs_arc_max can be adjusted dynamically. Note that reducing it below the current ARC size doesn't cause it to shrink immediately; something else still has to demand the memory be released.

 

[probain]

Will it be persistent?
I've experienced that zfs_arc_max can get reset. Which is why I've got a Cron job, instead of post-init

 

[sfatula]

I have read here before that if someone restart VMs or maybe even middlewared, that Scale sets the zfs_arc_max back to default, is this not the case?

If it is reset, would that apply to the kernel option, or is it just resetting it to the value of the kernel option in which case the kernel option is superior?

 

[PhilD13]

Shouldn't be a functional difference, zfs_arc_max can be adjusted dynamically. Note that reducing it below the current ARC size doesn't cause it to shrink immediately; something else still has to demand the memory be released.

Thanks, I am aware of the ability to shrink it also by providing a smaller value and that it won't shrink until something asks for a release. Setting a smaller value would also be handy for a system with limited memory where you could set the value to less than 50% of the memory value.

I set mine to use more than the 50% default value but is still conservative and it works fine for my setup.

 

[probain]

I have read here before that if someone restart VMs or maybe even middlewared, that Scale sets the zfs_arc_max back to default, is this not the case?

If it is reset, would that apply to the kernel option, or is it just resetting it to the value of the kernel option in which case the kernel option is superior?

Shutting down a VM, seems to swiftly eject all of the ARC above the default 50% threshhold.
It does not reset the kernel_extra_options="zfs_arc_max=<bytes>"-value. ARC will slowly fill up again (I believe).

Update: Added caveat that I believe the ARC will re-fill itself.
Update 2: I no longer believe that the ARC fills up again automatically. Monitoring this actively now.

 

[probain]

Update 2: I no longer believe that the ARC fills up again automatically. Monitoring this actively now.

So we are getting conflicting things happening.

Code:

username@truenas:~$ cat /sys/module/zfs/parameters/zfs_arc_max
67460859904 (This was previously manually reset to 0)
username@truenas:~$ sudo cli
[sudo] password for username
[truenas]> system advanced conf
Namespace conf not found
[truenas]> system advanced config
+----------------------------------+-------------------------+
|                               id | 1                       |
|                      consolemenu | true                    |
|                    serialconsole | false                   |
|                       serialport | ttyS0                   |
|                      serialspeed | 9600                    |
|                      powerdaemon | false                   |
|                      swapondrive | 2                       |
|                    overprovision | <null>                  |
|                        traceback | true                    |
|                     advancedmode | false                   |
|                         autotune | false                   |
|                      debugkernel | false                   |
|                      uploadcrash | true                    |
|                        anonstats | true                    |
|                  anonstats_token |                         |
|                             motd | Welcome to TrueNAS      |
|                       boot_scrub | 2                       |
|                      fqdn_syslog | false                   |
|                         sed_user | USER                    |
|                      sysloglevel | F_INFO                  |
|                     syslogserver |                         |
|                 syslog_transport | UDP                     |
|                    kdump_enabled | false                   |
|             isolated_gpu_pci_ids | 0000:c6:00.0            |
|             kernel_extra_options | zfs_arc_max=96636764160 |
|           syslog_tls_certificate | <null>                  |
| syslog_tls_certificate_authority | <null>                  |
|                       consolemsg | false                   |
+----------------------------------+-------------------------+
[truenas]>

 

[Sawtaytoes]

I have 256GB of RAM in my system. What should I be setting this to?

Code:

# system advanced update kernel_extra_options="zfs_arc_max=47191459840"

I think 32GB of RAM is plenty for services:

 

[probain]

I have 256GB of RAM in my system. What should I be setting this to?

Code:

# system advanced update kernel_extra_options="zfs_arc_max=47191459840"

That/his example would be ~43.9GB

Use this bytes converter, to get a reasonable value corresponding to how many GB's you want to allocate for ARC. Just give yourself some breathing room, if something would need to claim extra RAM somehow.

But be aware, that this tunable doesn't seem to work quite as intended. Especially if you use VM's in Scale

Update: I re-read the question :)

 

[probain]

I believe that this whole thing is not working as intended. Having trouble reproducing the behaviour reliably. Which I'm trying to nail down before filing another JIRA.

 

[Sawtaytoes]

Oh, I see. That's why that command doesn't work?

 

[probain]

Oh, I see. That's why that command doesn't work?

View attachment 71755

you have to go into the "TrueNAS CLI". You can invoke it by using

Code:

cli

Or by choosing it on the console menu on your NAS physical screen, if you have one attached. Or via IPMI

 

[sfatula]

Oh, I see. That's why that command doesn't work?

View attachment 71755

You have to first type cli and hit return

 

[Sawtaytoes]

I see now. It was confusing because CLI and Console are also words used for terminals.

I was able to run it, but nothing's changed yet.

 

[probain]

JIRA: NAS-124864
I've now written down my train of steps, showing how I believe that this kernel_extra_options isn't working as intended/expected. And also how the VM's and middleware are affecting this behaviour. I do not know if it's all related, or if these are two separate issues. But that will be up to the great devs to decide :)

 

[PhilD13]

Your ticket shows using the postinit method. You have to reboot after applying that method and you also must select postinit as when to run so it runs after boot but before the services start.
I asked the question because Tom Lawrence posted a video recently about doing the change using the method I posted and wanted to ask Kris Moore if the cli method and the postinit method would work the same as I suspected it would.
Have you tried the cli method Kris posted?
# cli
# system advanced update kernel_extra_options="zfs_arc_max=<SIZE IN BYTES>"

For example:

# system advanced update kernel_extra_options="zfs_arc_max=47191459840"

To clear and go back to the defaults:

# system advanced update kernel_extra_options=""

With both methods you need to reboot after changing the arc_max value for the system to read the new value. The setting is applied at a certain point in the boot process, changing it after that point will not have an effect until next reboot.

When I apply the setting I use for my system, I go from about 62GB (50%) for cache before applying the new value, to 82GB cache which is what I set my value to, after reboot.

I also noticed you are setting to 90GB which is less than 50% of your 256GB You can do that but still need to reboot.