Resource icon

ZFS Feature Flags in TrueNAS

Ericloewe

Server Wrangler
Moderator
Joined
Feb 15, 2014
Messages
20,175
Ericloewe submitted a new resource:

ZFS Feature Flags in FreeNAS - Overview of ZFS Feature Flags and their support in FreeNAS

OpenZFS' distributed development led to the introduction of Feature Flags. Instead of incrementing version numbers, support for OpenZFS features is indicated by Feature Flags.

Feature Flag states

Feature flags exist in one of three states:

disabled
The Feature Flag is not used by the pool. The pool can be imported on systems that do not support this feature flag.

enabled
The feature has been enabled for use in this pool, but no changes are in effect....

Read more about this resource...
 

danb35

Hall of Famer
Joined
Aug 16, 2011
Messages
15,456
Device removal allows for vdevs to be removed from a pool, subject to limitations.
This is coming in 11.2? Wow. I knew it was on the way, but didn't realize it was that close. Bit of a bummer that you lose the ability to use that pool with an older version though.
 

Ericloewe

Server Wrangler
Moderator
Joined
Feb 15, 2014
Messages
20,175
This is coming in 11.2? Wow. I knew it was on the way, but didn't realize it was that close. Bit of a bummer that you lose the ability to use that pool with an older version though.
Yes, AFAIK it only works for single-disk vdevs for now. So mirrors need to be demoted first and RAIDZ is out of luck.

It's been in production at Delphix for a while now. Apparently they had a bunch of machines where they went "oops, we gave them a bunch of space that they'll never use, we should remove them to save some cash", which is an unusual scenario, to say the least.
 

Stux

MVP
Joined
Jun 2, 2016
Messages
4,358
This is coming in 11.2? Wow. I knew it was on the way, but didn't realize it was that close. Bit of a bummer that you lose the ability to use that pool with an older version though.

It’s still something you want to avoid though, as you end up with permanent block redirects in your pool.

Idea I guess is for the “oh whoops” case that shouldn’t matter, since the vdev wouldn’t have seen much usage before it’s removal

@Ericloewe thanks for putting this resource together
 

Ericloewe

Server Wrangler
Moderator
Joined
Feb 15, 2014
Messages
20,175
It’s still something you want to avoid though, as you end up with permanent block redirects in your pool.
That's what obsolete_counts is for. It removes obsolete entries so that you're not attached to the table forever.
 

Stux

MVP
Joined
Jun 2, 2016
Messages
4,358
It's been in production at Delphix for a while now. Apparently they had a bunch of machines where they went "oops, we gave them a bunch of space that they'll never use, we should remove them to save some cash", which is an unusual scenario, to say the least.

Illustrates nicely the driving force behind ZFS feature development ;)
 

Ericloewe

Server Wrangler
Moderator
Joined
Feb 15, 2014
Messages
20,175
It's kind of an ironic outcome. Oracle wanted to suppress ZFS outside its grasp, but they ended up creating the perfect environment for ZFS development to spread. A special method of version tracking needed to be created to accommodate the spread-out development.
 

Ericloewe

Server Wrangler
Moderator
Joined
Feb 15, 2014
Messages
20,175

Ericloewe

Server Wrangler
Moderator
Joined
Feb 15, 2014
Messages
20,175

Yorick

Wizard
Joined
Nov 4, 2018
Messages
1,912
@Ericloewe , can you add the spacemap_v2 feature flag for 11.3?

I believe TrueNAS 12 is ZoL master, which would be "a whole bunch", but I'm not solid on that.
 

Ericloewe

Server Wrangler
Moderator
Joined
Feb 15, 2014
Messages
20,175
Yeah, I'll do that immediately. My home NAS units are on 11.2, but the work NAS is on 11.3, so I should've gotten to this earlier.

I believe TrueNAS 12 is ZoL master, which would be "a whole bunch", but I'm not solid on that.
Let's let that stabilize a bit to figure out what Linux feature flags will show up.
 

Ericloewe

Server Wrangler
Moderator
Joined
Feb 15, 2014
Messages
20,175
Ericloewe updated ZFS Feature Flags in FreeNAS with a new update entry:

spacemap_v2

Updated the resource for FreeNAS 11.3. The only addition is feature@spacemap_v2, which I consider the first fundamentally incompatible new feature since the 9.10 days. Device removal may have a shot at that title, but it's a niche thing and pools aren't suddenly incompatible if you accidentally upgrade them - it's only after removing a device that compatibility is broken.

Read the rest of this update entry...
 

Ericloewe

Server Wrangler
Moderator
Joined
Feb 15, 2014
Messages
20,175
Always good to emphasize:

Spacemap_v2 breaks compatibility. Immediately. Do not upgrade pools unless you are certain that you won't need/want to go back.

Note that it is read-only compatible, though.

FreeNAS 12 may bring in a ton of new Feature Flags, if every single Linux Feature Flag in master becomes available on FreeBSD. That's going to be:
  • bookmark_v2
  • encryption
  • resilver_defer
  • bookmark_written
  • livelist
  • log_spacemap
  • redacted_datasets
  • redacted_bookmarks
  • edonr
  • allocation_classes
  • large_dnode
  • project_quota
  • userobj_accounting

That's 13 new feature flags compared to 18 at present. A few are completely incompatible. Notably, encryption cannot be deactivated back into the enabled state, so the send | recv to a new dataset trick won't work, if I understood things correctly. I don't think any of these will immediately break downgrades if not used, but I'll review the matter closer to 12's release.
 

Yorick

Wizard
Joined
Nov 4, 2018
Messages
1,912
Notably, encryption cannot be deactivated back into the enabled state

Reading the man page, it can, by destroying all encrypted datasets.

However, log_spacemap becomes active and will never go back to being enabled. Same for project_quota and userobj_accounting.

It looks like we have a number of features that aren't read-only compatible but can be brought back into "enabled" state by stopping their use; and three features that will remain active forever, making the pool read-only in earlier versions.

That's actually not bad. "As long as you don't actually use the new features, it'll be read-only in earlier versions once you upgrade the pool" - that's the same result as a move to 11.3.
 

Ericloewe

Server Wrangler
Moderator
Joined
Feb 15, 2014
Messages
20,175
Reading the man page, it can, by destroying all encrypted datasets.
Interesting, I seem to recall bringing this up once after someone said that encryption was one-way and being told that a keystore or something was added and kind of threw a wrench into the destroying encrypted datasets to go back to older versions plan.

I'll have to look into this
 

Yorick

Wizard
Joined
Nov 4, 2018
Messages
1,912
Beta 1 is out, I think the plan now is to stabilize and go towards RC and then release, no more pool features to be added: Time to update the resource?
 

Ericloewe

Server Wrangler
Moderator
Joined
Feb 15, 2014
Messages
20,175
I'll spin up a VM and see if I can get a preliminary list going.
 

Ericloewe

Server Wrangler
Moderator
Joined
Feb 15, 2014
Messages
20,175

Ericloewe

Server Wrangler
Moderator
Joined
Feb 15, 2014
Messages
20,175
That was somewhat painful.

I still have a few things that need to be cleared up (hint hint):
  • v2 bookmarks - are all new bookmarks v2 once this is enabled? If not, what are the exact rules?
  • I thought I understood redacted send/recv, but the man page left more confused than I was going in. Redacted datasets aren't safe to mount? But then why send/recv them? It would make sense to me if you still had a viable dataset, just missing some data that you really don't want to leak. Also, how do they behave? I'm assuming it's standard "active until all such datasets/bookmarks are destroyed" behavior.
  • I was previously told that encryption would break backwards compatibility, but the man page disagrees.
  • How much extra warning does large_dnode need? It's a nasty one, since it breaks replication and I've been bitten by it trying to send from Linux to a FreeNAS server.
  • Are projects a real thing or just a Linux invention nobody supports?
  • Are "special" vdevs really generic metadata vdevs?
 

Ericloewe

Server Wrangler
Moderator
Joined
Feb 15, 2014
Messages
20,175
Also note: edonr is still not supported, not that it'd be useful, since it apparently isn't even that cryptographically secure, at which point fletcher makes more sense, if sha256/512 or skein are not needed.
 
Top