L2ARC

ZFS has several features to help improve performance for frequent access data read operations. One is Adaptive Replacement Cache (ARC), which uses the server memory (RAM). The other is second level adaptive replacement cache (L2ARC), which uses cache drives added to ZFS storage pools. These cache drives are multi-level cell (MLC) SSD drives and, while slower than system memory, are still much faster than standard hard drives. ZFS (including TrueNAS) uses all of the RAM installed in a system to make the ARC as large as possible, but this can be very expensive. Cache drives provide a cheaper alternative to RAM for frequently accessed data.

How Does L2ARC Work?

When a system gets read requests, ZFS uses ARC (RAM) to serve those requests. When the ARC is full and there are L2ARC drives allocated to a ZFS pool, ZFS uses the L2ARC to serve the read requests that overflowed from the ARC. This reduces the use of slower hard drives and therefore increases system performance.

Implementation in TrueNAS

TrueNAS integrates L2ARC management in the web interface Storage section. Specifically, adding a Cache vdev to a new or existing pool and allocating drives to that pool enables L2ARC for that specific storage pool.

Cached drives are always striped, not mirrored. To increase an existing L2ARC size, stripe another cache device with it. You cannot share dedicated L2ARC devices between ZFS pools.

A cache device failure does not affect the integrity of the pool, but it might impact read performance depending on the workload and the dataset size to cache size ratio.

Persistent L2ARC in TrueNAS

By default, the L2ARC cache empties when the system reboots. When Persistent L2ARC is enabled, a sysctl repopulates the cache device mapping during the reboot process. Persistent L2ARC preserves L2ARC performance even after a system reboot.

However, persistent L2ARC for large data pools can drastically slow the reboot process, degrading middleware and web interface performance. Because of this, we have disabled persistent L2ARC by default in TrueNAS, but you can manually activate it.

Activating Persistent L2ARC

Go to System > Tunables and click ADD. For the Variable, enter vfs.zfs.l2arc.rebuild_enabled. Set the Value to 1 and the Type to sysctl. We recommend noting in the Description that this is the persistent L2ARC activation. Make sure Enabled is selected and click SUBMIT.

PersistentL2ARCTunable

CLI Instructions
TrueNAS does not write settings changed through the CLI to the configuration database. TrueNAS resets them on reboot.

In a command line, enter sysctl vfs.zfs.l2arc.rebuild_enabled=1. When successful, the output reads: vfs.zfs.l2arc.rebuild_enabled: 0 -> 1

TrueNAS enables persistent L2ARC by default. We do not recommend users disable it.

Device Recommendations

Like all complicated features, deciding whether L2ARC is effective or not requires a strong understanding of your storage environment, performance goals, and the software you are using.

However, we have a few recommendations for L2ARC devices:

  • Using multiple L2ARC devices helps reduce latency and improve performance.

  • Using large capacity L2ARC SSDs can benefit random Read Heavy workloads. L2ARC SSDs are faster than the existing data storage drives.

  • Using an L2ARC device that is much faster than the data storage devices makes better use of its larger capacity. Sequential or streaming workloads need very fast, low-latency L2ARC devices. We recommend Enterprise-grade NVMe devices. L2ARC device capacity depends on how much faster it is than the data storage devices.

Resources