RAIDZ expansion, it's happening ... someday!

[Ericloewe]

I am terrified about the post expansion era… the flood of lost pools that will kill off consumer confidence in ZFS

Not likely. One of the reasons it takes so long is that this isn't btrfs where movement is key and if it happens to be into the abyss, who cares, it's movement.

That would be a nice feature for truenas.

I tend to disagree for most situations. If you're re-writing everything from scratch.... why aren't you doing that in a new pool? Obviously, the answer is "I don't have disks for a new pool", but the intersection of the set of people expanding their pools with the set of people who care about this is, I suspect, fairly small.

 

[Whattteva]

I guess something like the rebalance script is something that may become popular after running the expansions... https://github.com/markusressel/zfs-inplace-rebalancing

Would be nice if they just made a command for that like they did zpool-scrub or maybe even include it as an option of zpool-scrub.

 

[Davvo]

Would be nice if they just made a command for that like they did zpool-scrub or maybe even include it as an option of zfs-scrub.

Theoretically it should be possibile to create and alias.

 

[Whattteva]

I tend to disagree for most situations. If you're re-writing everything from scratch.... why aren't you doing that in a new pool? Obviously, the answer is "I don't have disks for a new pool", but the intersection of the set of people expanding their pools with the set of people who care about this is, I suspect, fairly small.

I'm actually one of those people that don't care about this feature as I run striped mirrors, but after being on the forums for the last few years, I think the amount of people that want this seems to be significant. I've seen a lot of posts from people that tend to make their RAIDZ arrays without knowing that the vdev size is fixed. They tend to come from people with gamer gear and low post count and I guess only barely read guides/watch YouTube videos barely enough just to get things up and running without fully knowing what the implications of their choices are. Fast forward a few years down the line, they come back to the forums to either ask about resizing the vdev or panicking about their degraded or umountable pools.

 

[Davvo]

I would see the use of this feature only if it would get possible to increase the parity as well. It makes sense to want go from 6 wide RAIDZ2 to a 9 or 12 wide RAIDZ3 just by adding drives.

 

[Ericloewe]

That's actually sort-of-possible, as future work.

 

[sretalla]

The point of it all (if there is one) is to give "people" back the functionality they expect or feel that they lost when transitioning from hardware RAID (or other SW RAID) that allows starting an array with 3 disks and adding as space requires.

As already mentioned, I don't like the idea in ZFS and appreciate the obligation to design well up-front that is currently there, but some people don't like giving up lazy features that they used to use (and assumed they would just be able to get with any other system they liked the idea of.

Don't even get me started on what will come when performance gets questioned in an expanded VDEV/pool.

 

[Etorix]

Fast forward a few years down the line, they come back to the forums to either ask about resizing the vdev or panicking about their degraded or umountable pools.

There's nothing fundamentally wrong with being user-friendly or allowing some flexibility… but I suppose that the major error in pool design by newcomers is going for raidz1 with modern HDDs and vdev expansion is unfortunately NOT going to be the way into raidz2 safety.

I would see the use of this feature only if it would get possible to increase the parity as well. It makes sense to want go from 6 wide RAIDZ2 to a 9 or 12 wide RAIDZ3 just by adding drives.

If you think that going from 6-wide Z2 to 9-wide Z3 is a good idea, I think you need to read about more raidz expansion, its caveats, and spend some time thinking about this implies for cascading expansions.
Of course, running the rewrite/rebalance script inbetween expansions would obviate any issue… but at this stage, and for the time that the process is going to take, the "old way", backup-destroy-restore, becomes very attractive again.

 

[Davvo]

If you think that going from 6-wide Z2 to 9-wide Z3 is a good idea, I think you need to read about more raidz expansion, its caveats, and spend some time thinking about this implies for cascading expansions.

How do I understand the caveats of something that's not here yet? Don't we have mostly speculations?

Also my post was based on it being actually feasible from more than a technical point of view.

 

[Yorick]

There's certainly a subset of users clamoring for raidz expansion who, upon further query, actually don't want to run ZFS any more. Thinking of someone specific who did a 4-wide raidz1 with 18TB drives and heavy dedup in a QNAP, without architecting for dedup - and is now running out of space. They're considering moving it all to Windows Server and tbh, it may be a far better fit for them.

 

[garm]

There's certainly a subset of users clamoring for raidz expansion who, upon further query, actually don't want to run ZFS any more. Thinking of someone specific who did a 4-wide raidz1 with 18TB drives and heavy dedup in a QNAP, without architecting for dedup - and is now running out of space. They're considering moving it all to Windows Server and tbh, it may be a far better fit for them.

This… expansion is a unicorn… there is no real need…

 

[Ericloewe]

How do I understand the caveats of something that's not here yet? Don't we have mostly speculations?

The design is actually pretty locked-down. If there are no show-stopper design flaws, it's just about getting all the edge cases right.

 

[Davvo]

The design is actually pretty locked-down. If there are no show-stopper design flaws, it's just about getting all the edge cases right.

I see. Can someone give me a brief recap so that I don't need to search 13 pages of discussion?

 

[Whattteva]

How do I understand the caveats of something that's not here yet? Don't we have mostly speculations?

Also my post was based on it being actually feasible from more than a technical point of view.

I mean..... tell that to BTRFS. All these years and still a mess.... even though it's certainly technically feasible.

I like my file system to be reliable and very boring, not exciting and disastrous. I suspect most people also would share that sentiment, so I think the skepticism is warranted.

 

[Whattteva]

The design is actually pretty locked-down. If there are no show-stopper design flaws, it's just about getting all the edge cases right.

The edge cases.... is what software is all about though lol. It's usually where all sorts of security exploits and "seemingly intermittent" bugs come from. And any programmer worth their salt will agree with me on this. And this is especially even more important on a mission-critical software like a file system.

 

[Etorix]

I see. Can someone give me a brief recap so that I don't need to search 13 pages of discussion?

See post #27 and the linked video to a presentation by Matt Ahrens. I generally just skip anything which says "video" and go for written documentation instead, but this one is a primary source of information.
And what's worth considering for successive expansions is the fourth bullet point.

 

[Tigersharke]

Surely said or conveyed or thought already:

• An appliance market (aka TrueNAS, etc) is unlikely to add the expansion feature until it is most certainly bulletproofed in every direction imaginable.
• FreeBSD itself or Linux or where ever could roll it out in some way for it to be tested and tried well before then.

 

[jgreco]

So, still a year or three off.

 

[Philip Robar]

"The Moore" just dropped this comment in the PR: "FWIW, we on the iX side are taking a serious look at this right now to determine when to incorporate it into TrueNAS SCALE. Once we have something ready for testing we'll be sure to ping back in case folks want to help us beat it up and get it into shape for inclusion into upstream."

So is this the first sign of what I predicted when Scale was announced—BSD based TrueNAS gradually becoming an afterthought that will eventually be discontinued?

 

[morganL]

So is this the first sign of what I predicted when Scale was announced—BSD based TrueNAS gradually becoming an afterthought that will eventually be discontinued?

History: iX funded the original RAID-Z expansion work for openZFS - we sponsored via the FreeBSD foundation.
iX provided a lot of the work to ensure OpenZFS works well on Linux and BSD equally
So, RAID-Z expansion will be available on both freeBSD and Linux at about the same time. via OpenZFS

We've been very clear... SCALE is being used for development of new features and UIs
Making new features, reliable, and supportable is a lot of work (both us and community).
Developing new features in parallel is too expensive (time and money)

CORE/Enterprise are focussed on stability, performance, quality for existing storage users
Sidegrading (apart from apps is relatively easy) - so getting to the new features is possible
Backporting features is possible if stable and there is demand.

In the case of RAID-Z expansion, we'd expect most existing users don't need it, they've worked around the restrictions.

The expansion is better for small users that can't afford to buy full sets of drives, but need efficiency. It grows the potential market for ZFS.

While we are doing the work, until it's tested and verified as reliable, it's still at risk of schedule slip.