Register for the iXsystems Community to get an ad-free experience and exclusive discounts in our eBay Store.
Resource icon

ZFS Feature Flags in TrueNAS

Western Digital Drives - The Preferred Drives of FreeNAS and TrueNAS CORE
OpenZFS' distributed development led to the introduction of Feature Flags. Instead of incrementing version numbers, support for OpenZFS features is indicated by Feature Flags.

Feature Flag states

Feature flags exist in one of three states:

The Feature Flag is not used by the pool. The pool can be imported on systems that do not support this feature flag.

The feature has been enabled for use in this pool, but no changes are in effect. The pool can be imported on systems that do not support this feature flag.

The on-disk format of the pool includes the changes needed for this feature. Some features may allow for the pool to be imported read-only, while others make the pool completely incompatible with systems that do not support the Feature Flag in question.

Note that many ZFS features, such as compressed ARC or sequential scrub/resilver, do not require on-disk format changes. They do not introduce feature flags and pools used with these features are compatible with systems lacking them.

Feature Flags support in TrueNAS and FreeNAS

The following Feature Flags are supported in FreeNAS 9.10.2:
  • Asynchronous Destroy
  • Empty Block Pointer Objects
  • LZ4 compression
  • Multi-vdev crash dumps
  • Spacemap histogram
  • Enabled TXG (not user-facing)
  • Hole Birth
  • Extensible dataset (not user-facing)
  • Embedded data in Block Pointers
  • Bookmarks
  • Filesystem Limits
  • Large Blocks (>128KB)
For details on these features, consult the manual page ( man zpool-features).

The following Feature Flags are introduced in FreeNAS 11.0:
  • skein
  • sha512
Both of these features are optional checksums, of interest to those running deduplication or ZFS native encryption, as they are faster than the default cryptographically-secure sha256. In both cases, the feature is active if any datasets use it. It will return to the enabled state after the dataset is destroyed (data can be preserved by using zfs send | zfs recv to replicate to a dataset which does not use these checksums).
As such, pools can be upgraded while preserving backward compatibility.

The following Feature Flags are introduced in FreeNAS 11.2:
  • Device removal
  • Obsolete counts
  • Zpool checkpoint
Device removal allows for vdevs to be removed from a pool, subject to limitations. This feature flag becomes active when a vdev is removed using the zpool remove subcommand and will never return to enabled. Obsolete counts is an optimization of Device removal and behaves identically.

Zpool checkpoint provides the zpool checkpoint subcommand, which allows for a checkpoint - a sort of super-snapshot of the whole pool capable of reverting even dataset creation and deletion - to be made, subject to significant limitations. The Feature Flag is active while a checkpoint exists and returns to the enabled state after the checkpoint is deleted or the pool is rewound to the checkpoint.

The following Feature Flags are introduced in FreeNAS 11.3:
  • Spacemap v2
Spacemap v2 allows for more efficient and flexible encoding of spacemaps. However, this is immediately backwards-incompatible - the feature will become active as soon as it is enabled and will never return to the enabled state. Again, upgrading a pool with feature@spacemap_v2 breaks backwards compatibility. Note that the pool can still be imported read-only for disaster recovery, though.

TrueNAS 12.0 supports the following new Feature Flags (Bold feature flags are read-only backwards compatible, italicized feature flags are very easy to return to the enabled state):
  • Allocation Classes
  • Bookmarks v2
  • Bookmark written
  • Sequential Rebuilds [device_rebuild]
  • Encryption
  • Large dnodes
  • Livelist
  • Log Spacemap
  • Project Quota
  • Redacted datasets
  • Redaction bookmarks
  • Resilver defer
  • Userobj accounting
  • zstd compression
Allocation classes allow for vdevs dedicated to specific types of data, namely dedup tables or special (generic metadata). This feature becomes active if a dedup or special vdev is created and reverts to enabled once all such datasets have been destroyed.

V2 Bookmarks are necessary to support a number of other features, such as encryption and redacted send. The Bookmark written Feature Flag adds accounting for space written since a bookmark. Both features can revert to enabled if all v2 Bookmarks are deleted.

Sequential Rebuilds (also known as device_rebuild) allow for disks to be reconstructed sequentially by copying from the source disk to the new disk. This process allows for faster recoveries, if the old disk is on its way out. Since this approach precludes verifying checksums, a traditional scrub will be issued after the rebuild is completed to check the pool's integrity. The whole process is longer than a traditional resilver, but does allow for faster recovery.

Encryption adds support for ZFS native encryption, the details of which are outside the scope of this document. Encryption becomes active once an encrypted dataset is created and can be reverted to enabled by destroying all encrypted datasets

Large dnodes are a performance/storage optimization. However, the large_dnode feature flag breaks zfs send to systems that do not support this flag.

is a performance optimization for clone deletion.
The feature is activated when a clone is created and remains active until all clones have been destroyed.

Log Spacemap is a performance optimization for heavily-fragmented pools. It becomes active as soon as it is enabled and will never return to being enabled.

Project Quota adds support for quotas against project identifiers. It becomes active as soon as it is enabled and will never return to being enabled.

Redaction bookmarks and Redacted datasets allow sending and receiving datasets with redacted data.

Resilver defer allows a resilver to be deferred until after a currently-running resilver has completed, avoiding unwanted restarts of the operation. The feature becomes active once a resilver has been deferred and returns to being enabled when the deferred resilver begins.

Userobj accounting adds support for accounting object user information by user/group. It becomes active as soon as it is enabled and will never return to being enabled.

zstd compression provides very flexible compression. It is typically slower than lz4, but much faster than gzip, and can provide compression ratios better than gzip while still being faster. The features becomes active as soon as a dataset is set to use zstd compression. It will return to the enabled state if all datasets that have used zstd at any point in time are destroyed.

Overview of commands

To see the Feature Flags supported by the version of ZFS you're running, use man zpool-features.
To view the status of Feature Flags on a pool, use zpool get all poolname | grep feature.
To view available Feature Flags, use zpool upgrade. Feature Flags can be enabled using zpool upgrade poolname.
Feature flags can be selectively enabled at import time with zpool import -o feature@feature_name=enabled poolname. To enable multiple features at once, specify -o feature@feature1=enabled -o feature@feature2=enabled ... for each feature.
First release
Last update
5.00 star(s) 10 ratings

More resources from Ericloewe

Latest updates

  1. TrueNAS 12.0 Beta 2.1 Update

    Updated the TrueNAS 12 section to account for device_rebuild (sequential reconstruction in LBA...
  2. Preliminary TrueNAS 12.0 feature list

    I've added in the preliminary list of TrueNAS 12.0 features. There is still information missing...
  3. spacemap_v2

    Updated the resource for FreeNAS 11.3. The only addition is feature@spacemap_v2, which I...
  4. More selective feature enabling fun

    It turns out that selectively enabling features on import is even less fun than it sounds when...
  5. Selectively enabling features

    Going through the man page for the zpool command, I noticed that there is an option to...

Latest reviews

Thanks for the helpful explanation of feature flags by FreeNAS/TrueNAS version.
Thanks a lot! Definitely needed this information for my new build.
This is a great resource to help users determine if they can roll back to the previous version of FreeNAS
I am one of those people that don't update unless an update offers a solution for a problem or offers a new/altered feature that I wan't. Don't fix it if it ain't broken. The ZFS Feature flags where a bit of a problem for me as I was not enable to judge if I should update or not. Now I can act informed and that is great.
Excellent - Concise/Understandable and very comprehensive for such a short article! I posted a question regarding interoperability of ZFS Linux/FreeNAS, and was having trouble understanding the answers - this made it all come together!
Excellent/current information without excessive words.
Great reference for which version supports which feature flags.
I was looking for this information. This told me exactly what I wanted to know in simple and concise format!
It explains the feature flag status, and what the flags are. What more could you want from a feature flags resource?
Apart from an earlier from posting by Ericloewe we didn't have an explanation of the disabled/enabled/active states of ZFS feature flags here in this forum, at least nothing that I'm aware of. Extra Bonus: Feature Flags introduced in 11.2.
Running the nightlies, I noticed the new feature flags had arrived. That was what pushed me to write this stuff down and keep track of new feature flags from now on.