SOLVED ZFS Cache Zeroed Out after ARC Tuning

[visamp]

Hello,

I am running TrueNAS-SCALE-22.12.3.2 in a Dell 7920 Workstation with Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz and 250 GB of Samsung ECC memory (2666 hz).

I followed the Lawrence Systems tutorial to "How To Get the Most From Your TrueNAS Scale: Arc Memory Cache Tuning". When I did this, and set my memory to 185 for max cache (which left plenty of memory for the rest of the system apps, eg. Nextcloud, Collabora, and Cloudflare Tunnel) using the shell in TrueNAS, the dashboard showed ZFS cache dropped down to nearly zero.

So, I undid the setting in /sys/module/zfs/parameters/zfs_arc_max and set it back to 0, turned off the init script, rebooted twice, but the ZFS cache still remains at nearly zero. I have opened up several video files to see if I could get it to rise, but nothing. Here is what you can see when I cat the zfs_arc_min:

You can see here on the 20th when I followed the tutorial to tune the ARC caching, the in use memory flatlined.

I'm not sure how to troubleshoot this further. I do not notice slowness in the system, but I am using an NVME pool for my primary pool. Any ideas if this is a bug or something that I can troubleshoot further and fix myself?

 

[sfatula]

You are saying two different things here, you said you set arc_max, but your shell command which shows the current setting shows arc_min. So, not really sure what you did.

 

[visamp]

Sorry, I mispoke. Here is what I have set now for both min and max:

I have this as the current ZFS cache with the above screenshot:

Originally, I changed the zfs_arc_max. I did nothing with the zfs_arc_min.

 

[sfatula]

I notice mine, which was never set/overriden by me, is set to 0's in both cases also. I know there is a Scale bug dealing with the arc max, maybe somehow it is hindering your arc size, not sure. But that was never a problem on ubuntu/debian, 0 meant 50% of ram.

I saw this message from IX Systems:
With any VMs configured, manual tuning can be overridden because middleware automatically adjusts the ARC max as needed.

What does this command show:

arc_summary -s arc

 

[visamp]

The only thing that I can think is that when I made that change to the arc_max, it flushed the ARC completely and now there is nothing saved in it. Is that possible? Meaning that there isn't a mechanism in place within TrueNAS/ZFS to restore ARC, ie, there isn't a baseline of data stored in it and it can be flushed and need to be rebuilt by using files, etc.?

 

[sfatula]

The cache can be flushed, but changing the max shouldn't do that. See if the target size increases over time. It actually looks correct to me. It should go up with usage. You said you rebooted, that would have cleared the arc.

 

[NickF]

From the manpage about ARC generally:

Adaptive Replacement Cache​

For decades, operating systems have used RAM as a cache to avoid the necessity of waiting on disk IO, which is extremely slow. This concept is called page replacement. Until ZFS, virtually all filesystems used the Least Recently Used (LRU) page replacement algorithm in which the least recently used pages are the first to be replaced. Unfortunately, the LRU algorithm is vulnerable to cache flushes, where a brief change in workload that occurs occasionally removes all frequently used data from cache. The Adaptive Replacement Cache (ARC) algorithm was implemented in ZFS to replace LRU. It solves this problem by maintaining four lists:

1. A list for recently cached entries.
2. A list for recently cached entries that have been accessed more than once.
3. A list for entries evicted from #1.
4. A list of entries evicited from #2.

Data is evicted from the first list while an effort is made to keep data in the second list. In this way, ARC is able to outperform LRU by providing a superior hit rate.

In addition, a dedicated cache device (typically a SSD) can be added to the pool, with zpool add POOLNAME cache DEVICENAME. The cache device is managed by the L2ARC, which scans entries that are next to be evicted and writes them to the cache device. The data stored in ARC and L2ARC can be controlled via the primarycache and secondarycache zfs properties respectively, which can be set on both zvols and datasets. Possible settings are all, none and metadata. It is possible to improve performance when a zvol or dataset hosts an application that does its own caching by caching only metadata. One example would be a virtual machine using ZFS. Another would be a database system which manages its own cache (Oracle for instance). PostgreSQL, by contrast, depends on the OS-level file cache for the majority of cache.

There is specific logic that went into creating SANE defaults for ZFS on Linux. This is not a SCALE problem, its a Linux one. While I like Tom Lawrence's videos, I disagree with the approach to this video. You should not tune your ARC blindly based on the information in that video, it lacks all context to make a quick 5 minute video for "noobs". IMO These values should not be tuned by "noobs".

For your specific issue:

Changing ARC settings will have an impact on ARC…obviously. I think the behavior you saw if because you mistakenly changed both min and max to the same value which flushed everything in ARC. Additionally, you rebooted the system several times. While L2ARC can be set to persist between reboots, ARC will not because it's in volatile system memory (RAM).

l2arc_rebuild_enabled=1|0 (int)Rebuild the L2ARC when importing a pool (persistent L2ARC). This can be disabled if there are problems importing a pool or attaching an L2ARC device (e.g. the L2ARC device is slow in reading stored log metadata, or the metadata has become somehow fragmented/unusable).

Setting them to 0 is not the correct first step. .It should have resulted in returning them to default behavior.

zfs_arc_max​

Maximum size of ARC in bytes.

If set to 0 then the maximum size of ARC is determined by the amount of system memory installed:

• Linux: 1/2 of system memory
• FreeBSD: the larger of all_system_memory - 1GB and 5/8 × all_system_memory

zfs_arc_max can be changed dynamically with some caveats. It cannot be set back to 0 while running and reducing it below the current ARC size will not cause the ARC to shrink without memory pressure to induce shrinking.

Your current problem is most likely that the ARC cache takes time to populate and you need to be patient. Additionally, based on the man page, you now need to provide it with a value other than 0. Once you set it statically, you can never go back.

If you insist on making it larger than the 1/2 default, be aware that you can potentially cause your system to crash when it runs out of memory. There's a big difference between memory management in Linux and memory management in FreeBSD. The developers of ZFS didn't arbitrarily decide to make ARC caches smaller on Linux, there's a reason for it.

That being said, I personally have my max set to about 2/3 of my system RAM and have seen no ill effects. I have toyed around with ARC min as well, setting it very high, but I don't have any good information on what you should set it to as a sane default.

 

[sfatula]

There is indeed some logic to it. I feel it gets out of hand though the larger system memory someone has. But I've seen zfs go oom as it did not clear the memory pressure in time. This is another reason some swap space is quite useful on Linux. It can compensate and provide the time for zfs to clear said memory and release it back due to pressure.

I was not aware he set min and max to the same value?

For max, one can monitor their free memory over time as an approximation of the most arc memory they could add. I would subtract from that number though.

The doc actually says what is below here. It does not say you can never set it back to 0. I don't think this is correct, that "Once you set it statically, you can never go back", never seen that behavior.

"It cannot be set back to 0 while running". To me, that means you set it to 0, but it doesn't go back to default until you reboot.

 

[NickF]

I agree, both that I may have misread the bit about going back to 0 and that the default can get out of hand as memory density increases.

I also think I opened up a TrueNAS ticket recently. specifically requesting a larger SWAP size, specifically on smaller systems, on pool creation. There’s another thread somewhere about that too.

My assumption has always been that the devs set it to 1/2 out of an abundance of caution. I also have assumed that they have spent some effort in trying to refine this default. I just think they haven’t found a non-arbitrary ratio…that’s specifically backed by test data to be stable and considers a non linear growth as memory increases

 

[sfatula]

Oh I agree as well, they had their reasons. And yes, larger swap on smaller makes sense for the same reason that the 50% gets out of hand when going larger.

 

[visamp]

"It cannot be set back to 0 while running". To me, that means you set it to 0, but it doesn't go back to default until you reboot.

I will let you know what happens as (hopefully) the ARC rebuilds over time. I hope that what I did is undone for now as I have rebooted the system (yesterday) once again to see if this restores the settings to the default as I think this has value for the devs, the forums, and for others like myself who may do this.

There is specific logic that went into creating SANE defaults for ZFS on Linux. This is not a SCALE problem, its a Linux one. While I like Tom Lawrence's videos, I disagree with the approach to this video. You should not tune your ARC blindly based on the information in that video, it lacks all context to make a quick 5 minute video for "noobs". IMO These values should not be tuned by "noobs".

I agree. I have built what I have so far by breaking it as I learn, having a good cry, learning, then trying again.

I was not aware he set min and max to the same value?

I didn't. I only adjusted the max value per the instructional video. I mispoke at one point in my initial post.

Thank you both for taking the time to think through this with me. It has been weighing on me very much. Appreciate it!

 

[NickF]

Yeah man for sure. I learned most of what I know by breaking things :)

Let us know how it goes.

 

[Davvo]

@LawrenceSystems

Also, iX System is working on moving that 1/2 ARC restriction to a more reasonable value (iirc 2/3).

 

[sfatula]

@LawrenceSystems

Also, iX System is working on moving that 1/2 ARC restriction to a more reasonable value (iirc 2/3).

Hopefully based on memory size.

 

[NickF]

That is all

 

[visamp]

ARC size is growing back again. I think I just jumped the gun because it zeroed out in the first place and I thought I corrupted my system. Glad I learned this. Thank you all again.