Any ETA on pool expansion?

[koberulz]

It turns out I rather underestimated my storage requirements, and my 40TB pool is now 71% full just three days after I started adding to it.

I know pool expansion isn't currently a thing (unless it slipped in very recently), but my thought was it would be in place by the time I needed it, a couple of years from now. Obviously that hasn't worked out very well.

So, does anyone have any sort of an idea of how much longer it's going to be?

 

[HoneyBadger]

I assume you mean "RAIDZ vdev expansion" as "pool expansion" has always been possible (by adding more vdevs) - the status on that remains as "when it's done" but that date may be closer than we think.

A pull request was opened in June of 2021, and the testing thus far has been quite successful by my understanding, with multiple users reporting success at expanding Z1/Z2 vdevs.

RAIDZ Expansion feature by ahrens · Pull Request #12225 · openzfs/zfs

Motivation and Context This feature allows disks to be added one at a time to a RAID-Z group, expanding its capacity incrementally. This feature is especially useful for small pools (typically wit...

github.com

Note that this can't change the parity level (an expanded Z1 is still a Z1) but you will be able to add capacity.

 

[koberulz]

It can't be closer than I think, because I have absolutely no clue what the timeframe is. Thus the question.

 

[techfan]

It would be helpful if someone from TrueNAS could comment here. Or, it would probably be even more helpful if TrueNAS could volunteer some development resources to help speed this up.

Like @koberulz, I am also interested in hearing some sort of ETA. This pull request has been languishing for almost two years now. Based on developer conversations, and an apparent lack of urgency, it seems likely to languish for at least a couple more years. Many of the participants in that thread, and contributors, seem to think there are currently workarounds and that we all have unlimited budgets.

I am in the process of considering a DIY NAS build. I am on a tight budget so the initial build is likely to have limited storage. I need to grow it slowly, one disk at a time, to stay on budget. Without this capability, I am forced to consider alternatives other than TrueNAS.

I suspect there are many enthusiasts, on a tight budget like myself, looking to grow their NAS in a similar fashion. To those, who seem to exist solely to suggest costly alternatives to this mechanism, please don't bother. Here are some of the suggestions I've previously seen, and why I'm not considering them...

Add a new vdev.
I need to spend as little as possible on each increment of growth. I do however value some redundancy. To add a "single" disk vdev, using RAIDZ1 would double the cost and RAIDZ2 would triple the cost. Yes, growing in multi-disk increments would lessen the overhead expense, but it wouldn't significantly lower the overall expense. I simply don't have that kind of money.

Copy the data elsewhere and rebuild the vdev.
You're kidding, right? If I could afford that kind of storage, I would have just added all the needed disks in the original build.

Mirroring provides better redundancy and is easier to expand.
It sure does. I truly wish I could afford to spend twice as much and travel more too. Regrettably, I live in the real world where my budget is limited.

 

[jgreco]

It would be helpful if someone from TrueNAS could comment here. Or, it would probably be even more helpful if TrueNAS could volunteer some development resources to help speed this up.

Like @koberulz, I am also interested in hearing some sort of ETA. This pull request has been languishing for almost two years now. Based on developer conversations, and an apparent lack of urgency, it seems likely to languish for at least a couple more years. Many of the participants in that thread, and contributors, seem to think there are currently workarounds and that we all have unlimited budgets.

You're not likely to get any sort of encouraging official commentary. It is very likely to languish for some additional time, just as it took about four or five years for ZFS to even be adopted as a stable filesystem by the FreeBSD community to begin with. The UNIX and Linux communities both have plenty of examples of filesystems going "out the door" without sufficient testing; some of the EXT variants and BTRFS are both fine examples of this. ZFS is a filesystem that is relied on to store massive amounts of data and it doesn't have an fsck to fix artificial errors, so developers do more than just running it on a single PC in a test environment in order to trust it.

Like I say with the LSI HBA's, we trust them because they've got billions of aggregate problem-free run-hours under their belt. It is quite likely that RAIDZ vdev expansion will not be adopted in TrueNAS until there's a large installed base of systems where it has been used and it is demonstrated to be not problematic. This may indeed be some years out.

Many of the participants in that thread, and contributors, seem to think there are currently workarounds and that we all have unlimited budgets.

There are workarounds, such as using mirrors, or periodically rebuilding your pool and reflecting it back and forth between your primary and backup servers. And if you came to a product that was originally sold by Sun Microsystems for massive storage systems, and expected that you could get by on a shoestring budget, well, hrm.

I suspect there are many enthusiasts, on a tight budget like myself, looking to grow their NAS in a similar fashion. To those, who seem to exist solely to suggest costly alternatives to this mechanism, please don't bother. Here are some of the suggestions I've previously seen, and why I'm not considering them...

Sure. And no one here will think less of you if you go and adopt UnRAID or one of the other various alternatives that are targeted at such enthusiast populations. There are lots of wheeled vehicles out there in the world. For some people, a dirt bike is fine. Some like sedans, some like SUV's, some like pickup trucks. Some people need a semi. TrueNAS is more of a locomotive and freight train solution; it isn't good at going off-roading like a dirt bike is, but if you need to handle a petabyte of data, your options do not include four-wheeled vehicles.

Please bear in mind that TrueNAS is not here to fulfill your enthusiast dreams of the perfect cheap storage solution. TrueNAS CORE is the testing and debugging platform for TrueNAS Enterprise, which is the platform that pays the salaries of the iX developers. iX is selling large storage systems, and you and I are helping them test the software out. This is one of the reasons that we here in the forums try to steer people towards solutions that resemble the platforms that iX is selling. The closer your platform resembles what they're selling, the better your outcomes will be.

 

[Patrick M. Hausen]

The UNIX and Linux communities both have plenty of examples of filesystems going "out the door" without sufficient testing; some of the EXT variants and BTRFS are both fine examples of this.

Don't forget the infamous ReißwolfFS - Reißwolf is German for shredder. SuSE shipped it rather prematurely to have some USP or so.

@techfan Seriously though if RAIDZn expansion ever becomes available in a non-development release of OpenZFS, this will make the headlines on Phoronix, The Register, Heise (German), ... about any IT news platform you can think of. It's going to be huge. You won't miss it.

And once it's in an OpenZFS release it will come to TrueNAS, I expect in a rather timely fashion. But definitely not earlier.

 

[techfan]

There are workarounds, such as using mirrors, or periodically rebuilding your pool and reflecting it back and forth between your primary and backup servers. And if you came to a product that was originally sold by Sun Microsystems for massive storage systems, and expected that you could get by on a shoestring budget, well, hrm.

As mentioned, I can't afford to mirror my disks, so I seem to have misplaced my "backup server" somewhere. That's some funny stuff...thanks for the laughs :)

If TrueNAS is hoping for wider adoption, a larger market share, and maybe more money? it should try to help this effort along.

It's not only enthusiasts like myself that have limited budgets. Most small businesses have fairly limited budgets as well.

And, having worked for a number of Fortune 25 companies over the decades, mostly in the telecom industry. I can also assure you that it's not uncommon for departmental budgets within even the largest corporations to get stretched pretty thin.

Or, they can ignore the issue for a few more years. Arrogance works great for tech companies. They never, ever fail because of it :)

 

[Patrick M. Hausen]

Now who's arrogant?

You get the definitely most scalable and reliable filesystem in existence for free, because it is open source. Use it to your advantage or leave it.
As long as OpenZFS does not ship RAIDZ expansion how do you expect a company that is essentially an integrator to provide that feature?

It's open source. It's open for your contributions.

iX is not Microsoft. They are a consumer/user of ZFS, not a major contributor or architect. Yet they build a damn well engineered product on top of available open source technology. And I am not implying that they do not give back/upstream improvements. Their entire system is BSD licensed. I am arguing the main "drive" in OpenZFS is happening elsewhere and iX have to wait for that feature just like everybody else.

'nuff said.

 

[Davvo]

I want it so they should do it and here are my made-up reasons because they should do so, otherwise they are arrogant.

Show : edit

Anyway it's not "someone from TrueNAS" (product) but "someone from iX Systems" (company).
And they likely know their customers better than you.

 

[jgreco]

As mentioned, I can't afford to mirror my disks, so I seem to have misplaced my "backup server" somewhere. That's some funny stuff...thanks for the laughs :)

Sorry, it is what it is. You ever seen one of those YouTube videos with someone who's piled a bunch of wood on top of their car? You can feel sorry for the fellow who was desperate while simultaneously cringing. None of it makes it a better idea to try that trick though.

If TrueNAS is hoping for wider adoption, a larger market share, and maybe more money? it should try to help this effort along.

iXsystems has been contributing to ZFS development for years; if you look at the contributors list, you'll note that the FreeBSD support has been very heavy, and a good bit of that has happened via iXsystems.

However, ZFS also cannot afford negative publicity of the sort garnered by catastrophic filesystems such as BTRFS. So there is little appetite for charging blindly forward with cutting edge, poorly tested features, especially stuff that substantially reworks ZFS fundamentals.

As it is, iXsystems was ranked in 2021 as one of the fastest growing storage companies.

Blog - iXsystems: Fastest Growing Storage Systems Company in 2021 with TrueNAS

iXsystems reported 70% annual growth for its TrueNAS storage systems in 2021, which was the highest reported growth in the industry for storage systems. According to the analysis of 2021 data by Storage Newsletter, iXsystems was reported to stand alone in the systems category among the...

www.truenas.com

If Enlyft is to be believed --

Data Storage Hardware

• FreeNAS (0.67%)
• NetApp Storage Systems (15.31%)
• HP Storage Systems (6.46%)
• EMC SAN (Storage Area Network) (3.89%)

And their charge continues into 2022 with the opening of their new Maryville, TN Technology Campus. They are doing just fine for a small privately held technology company that has grown from three employees 20 years ago to more than 200 today.

It's not only enthusiasts like myself that have limited budgets. Most small businesses have fairly limited budgets as well.

And, having worked for a number of Fortune 25 companies over the decades, mostly in the telecom industry. I can also assure you that it's not uncommon for departmental budgets within even the largest corporations to get stretched pretty thin.

Yes, so iXsystems will even generously allow you to run TrueNAS on your own hardware, for free. The last time I ran the numbers against a quote from iXsystems, it was about 35-40% the cost to build your own TrueNAS system. Of course you do not get the white glove service or tech support from iXsystems if you do that. Here in these forums, I've specialized in advising people on how to build their own high quality TrueNAS systems, ranging from the original Hardware Suggestions thread to very specific engineering advice about PSU's and disk attachment. So basically that's a meaningless argument. There will always be someone who wants a smaller, cheaper system, and at a certain point there's no way to compete with a repurposed old desktop running Openfiler. TrueNAS isn't trying to serve that end of the market and iXsystems has made it clear that they only service the "small" systems because doing so helps with product resilience.

Or, they can ignore the issue for a few more years. Arrogance works great for tech companies. They never, ever fail because of it :)

But usually people wait until they start showing signs of failure before whining about such "arrogance". Even the forum post volume here shows the robustness of the growth of the product in its current format.

 

[techfan]

But usually people wait until they start showing signs of failure before whining about such "arrogance". Even the forum post volume here shows the robustness of the growth of the product in its current format.

If you go back to the original post, I only asked for two things from TrueNAS, an ETA and a request that they volunteer some extra time to help speed up the effort. The latter was based on comments in the thread indicating that things like code reviews and testing were holding up the release.

In the hope of avoiding wasting time, I expressed frustration at the number of posts (on the PR) offering advice that is frankly rather obvious if money were not an issue. I requested that people not bother repeating that advice. To that end, I am unclear regarding the purpose of your replies. They mostly repeat information from the PR, though with less brevity.

As someone who has been an active and fairly successful software engineer since the late 70s, continuing until today, I am not unfamiliar with the software engineering process or the challenges of adequate testing.

 

[jgreco]

If you go back to the original post, I only asked for two things from TrueNAS, an ETA and a request that they volunteer some extra time to help speed up the effort. The latter was based on comments in the thread indicating that things like code reviews and testing were holding up the release.

Testing is something that will hold this up for a very long time, because it requires that people actually perform pool transformations and then identify any issues that result. This is going to be a very slow process, just as the original integration and validation of ZFS into FreeBSD was, with the various adjustments being made to major kernel subsystems.

In the hope of avoiding wasting time, I expressed frustration at the number of posts (on the PR) offering advice that is frankly rather obvious if money were not an issue. I requested that people not bother repeating that advice.

That's nice, but I don't necessarily read PR's and have no idea what PR you're even referring to.

To that end, I am unclear regarding the purpose of your replies. They mostly repeat information from the PR, though with less brevity.

The purpose of my replies is to answer issues you've posted here on the forum. If they are repeating information that you already obtained from some PR, then you should get the strong impression that this is well known information. Additionally, you could potentially have avoided asking questions to which you already had been given answers, though generally we are forgiving about that on the forums.

As someone who has been an active and fairly successful software engineer since the late 70s, continuing until today, I am not unfamiliar with the software engineering process or the challenges of adequate testing.

Then my suggestion, respectfully: act like it.

For my part, I've lost interest in this thread, and won't be monitoring it further, but please feel free to keep "educating" me if you feel the need.

That is inappropriate and rude. Please review the Forum Rules, conveniently linked at the top of every page in red, so that you better understand the expectations we have of people being friendly and helpful.

 

[danb35]

For my part, I've lost interest in this thread

I'm suspecting you didn't have a great deal of interest in the subject in the first place. Otherwise, you would surely have found the existing discussion thread discussing this feature:

RAIDZ expansion, it's happening ... someday!

If you've not already seen it, RAIDZ expansion is happening. The OpenZFS project tweeted this earlier today. RAIDZ expansion (most requested ZFS feature ever?) is coming, courtesy of @freebsdfndation. Sneak preview at OpenZFS DevSummit! https://twitter.com/OpenZFS/status/921042446275944448 So...

www.truenas.com

...where, had you reviewed it, you would have seen that iX have contributed a great deal toward this very feature, and there are comments there from both the iX CTO and the SCALE product manager, the latter within the last couple of weeks. But (noting that the thread is 5+ years old) it appears getting it right, and reliable, has proven to be more difficult than expected.

 

[jgreco]

That's probably a good note on which to lock this thread.