RAIDZ expansion, it's happening ... someday!

[DayBlur]

Okay I've updated the sheet. It now shows two calculations. Better language to make clear what those are would be welcome.

Calc 1: Expand vdev when it's full to threshold; expand to final width then rewrite data: How much space did we gain.

Calc 2 : Expand vdev when it's full to threshold; expand by just 1 drive; expand again when vdev is full to threshold; keep doing this until final width, then rewrite data: How much space did we gain.

The space gain with calc 2 is larger because there is more data to be rewritten that was not written with the final parity to disk ratio.

This now uses a "spaceLoop" function. I don't know whether this comes across when making a copy. In case it does not:

Code:

function spaceLoop(initial, final, size, parity, threshold) {
  var spacesaved = initial * size * threshold * ((parity / initial) - (parity / final));
  for (var drives = initial + 1; drives < final; drives++) {
     spacesaved += size * threshold * ((parity / drives) - (parity / final));
  }
 
  return spacesaved;
}

Based on the equation in my last post (still waiting for moderator approval at this time), you should be able to use the following formulas instead. The first bit uses the equation I posted directly, which is a little simpler but algebraically equivalent to what you had. The other avoids your custom function, although summing the reciprocals is still annoyingly awkward (I'm not a spreadsheet engineer, is there a better way?).

Code:

[Cell B14]: =$B$4*B15
[Cell B15]: =$B$6*$B$5*($B$8-$B$7)/$B$8

[Cell B19]: =$B$4*B20
[Cell B20]: =$B$6*$B$5*SUM(ARRAYFORMULA(DIVIDE(1,SEQUENCE(1,$B$8-$B$7,$B$7+1,1))))

 

[Yorick]

Let me know if there's any problem with that.

None at all. You can consider it public domain or MIT licensed, whichever you prefer.

 

[Louwrentius]

Here's a simple example you can use to compare your understanding and calculations for these types of scenarios:
...
Hopefully the above clarifies and formalizes my understanding of what's going on and will help with any spreadsheets/calculators. Let me know if you find I've missed something or made an error.

Thank you Yorick and DayBlur, I think I now do understand how it works. I think this is rather interesting because this overhead doesn't seem that big of a deal when you look at the examples provided by Matthew Ahrends. But if you work through this home user scenario, where you start small and grow to - for example - 10 drives, the losses become quite significant. I think it would be helpful if the the larger home user storage enthusiast is aware of this.

However, what matters most to me is the fact that you can rewrite the data and the overhead will be *poof* gone. That is awesome. I can see how just maybe moving files over to a new file system is enough to trigger this rewrite. Just let it run and things will be OK.

Again thanks for the help, I'm not sure if I would have figured this all out just by myself.

 

[Louwrentius]

So I have created a blog post about this topic, I hope I haven't made any mistakes, feedback is always welcome.

Edit: by request I also created a resource on the forums here, it is a PDF-version of my article.

 

[Yorick]

Based on the equation in my last post (still waiting for moderator approval at this time), you should be able to use the following formulas instead.

Clever. Thanks! I've made that change, no more reliance on custom functions.

 

[Arwen]

So I have created a blog post about this topic, I hope I haven't made any mistakes, feedback is always welcome.

...

One note about the beginning of the article, it more or less assumes that the only way to expand a pool is to add another vDev;

For example, if you run a ZFS pool based on a single 3-disk RAIDZ vdev (RAID5 equivalent1), the only way to expand a pool is to add another 3-disk RAIDZ vdev.

This is not true. You can replace each drive in a vDev with a larger drive, one at a time. Whence all drives are replaced, the pool can be expanded. (Automatically if "autoexpand" is set to "on".)

I am not suggesting that this is the "best", "only" or "ideal" way to expand a pool. It's just another option that exists now.

 

[Louwrentius]

One note about the beginning of the article, it more or less assumes that the only way to expand a pool is to add another vDev;

This is not true. You can replace each drive in a vDev with a larger drive, one at a time. Whence all drives are replaced, the pool can be expanded. (Automatically if "autoexpand" is set to "on".)

I am not suggesting that this is the "best", "only" or "ideal" way to expand a pool. It's just another option that exists now.

That's a fair point. I was going to explain why I didn't add this information but just adding a footnote is just as much work, so the article is updated. Thank you.

 

[HarambeLives]

RAIDZ expansion capabilities will be a huge blow to Synology. The main reason to go with a Synology is their flexible SHR implementation

 

[Constantin]

Not sure I agree. Synology focuses on a different end of the market, offering attractive enclosures with adequate performance for many folk. There is overlap between TrueNAS and Synology users but TrueNAS is far more difficult to set up and administer than Synology.

On the other hand, TrueNAS has features that Synology users can only dream of at the moment and moreover TrueNAS users aren’t locked into specific software and hardware gardens like Synology users are.

To each his/her/their own. I’ve voted with my feet on account of data integrity alone.

 

[HarambeLives]

The TrueNAS Mini boxes have really started pulling users over recently, and everyone I know including myself who loves Synology, has started to look into TrueNAS, better expansion is the icing on the cake

And as crazy as it sounds, the TrueNAS name vs FreeNAS. Too many times in the past people I spoke to assumed FreeNAS was some kind of budget alternative to a real NAS.

 

[Constantin]

I hope you’re right. I’ve used both and TrueNAS is a far superior software platform in many respects … but I found it’s learning curve also steeper.

The splashy UI of Synology combined with limited configuration options make it a much more mass-market SOHO storage appliance. The TrueNAS community in general also plans a lot more ahead, optimizing their hardware platforms / software configurations for specific use cases.

Then there is the issue of open-source support. As I understand it, iXSystems contributed materially to open-source projects like Netatalk while the other three simply absorbed improvements. Similarly, all three alternatives may make claims about data integrity but the many posts of customers with data loss issues attest to BTRFS continuing to be a beta-quality file system.

I’ve owned / used Synology, QNAP, and ReadyNAS and found all three inferior to FreeNAS and now TrueNAS. No NAS is perfect, backups (ideally offsite) are 100% necessary but once you work your way through all the idiosyncrasies of the TrueNAS UI, it’s easy enough to use.

Reminds me somewhat of learning EAGLE or AutoCAD.

 

[IOSonic]

The TrueNAS Mini boxes have really started pulling users over recently, and everyone I know including myself who loves Synology, has started to look into TrueNAS, better expansion is the icing on the cake

Err, I have 7 folks on my team who are IT pros (we support 40,000+ users), and I'd estimate we average out at ~15 years of experience. Except for one person who uses unraid and me, Syno is the way--and they've all tried FreeNAS or TrueNAS at some point.

The reason is that TrueNAS is too difficult to use in comparison to Synology for their purposes. For cloud replication, surveillance, easy-button-containers, simple SMB & AFP shares and iSCSI exports, Syno just makes these a lot easier. FOSS support can also be unpleasant and depends on forum users/developers deem worthy to help with, so it's easy to hit dead ends.

I suffered through the complexity and am happy with my decision, but it took a lot to get it all working. I invested it because TrueNAS had some cool features, but also because I had the time and inclination to learn something new--as well as some background to troubleshoot when things didn't work. It depends on your knowledge level, requirements and patience. Synology democratizes tech a lot more, so I don't ever see TrueNAS completely eating its lunch until it does the same. That may not be an iXsystems objective, and that's fine too. However, I don't think TrueNAS will supplant other proprietary systems unless it becomes more accessible.

 

[jgreco]

Err, I have 7 folks on my team who are IT pros (we support 40,000+ users), and I'd estimate we average out at ~15 years of experience. Except for one person who uses unraid and me, Syno is the way--and they've all tried FreeNAS or TrueNAS at some point.

This is the way of tech. It has gotten complicated enough that people no longer really want to own the problem, as they have enough problems that they're forced to own without a choice. This has led to the popularity of outsourced stuff en masse, appliance devices, and other easy-making things like cloud.

The reason is that TrueNAS is too difficult to use in comparison to Synology for their purposes. For cloud replication, surveillance, easy-button-containers, simple SMB & AFP shares and iSCSI exports, Syno just makes these a lot easier. FOSS support can also be unpleasant and depends on forum users/developers deem worthy to help with, so it's easy to hit dead ends.

Yes, because the Synology solves a lot of the hardware design questions, and doesn't let an end user make bad choices like trying to undersize RAM or repurpose a gaming PC for use as a NAS. TrueNAS has addressed that particular part of the problem with the TrueNAS Mini, as @HarambeLives notes, which wipes out an entire class of complexity. ZFS still adds a lot of complexity and TrueNAS is still aimed at a higher end enterprise market that is able to put forth the hardware resources, which has always meant that the cheap little $300 NAS devices have a huge market share simply because they are cheap to deploy. It also means that there isn't an entire team of developers trying to churn out value-adds for TrueNAS.

TrueNAS sites basically break down into three general categories:

1) Enterprise deployments looking for massive scale storage, willing to pay for it, and aren't really looking at value-adds like surveillance camera stuff. These are the sites iXsystems is focused on, because these people pay the bills.

2) End users who just want something cheap and easy to deploy, who usually have TrueNAS experiences that end badly when they try to build the cheapest possible thing out of a 2009-era PC with 4GB of RAM and some scrounged disks. Here in the forums, we try to salvage these people into better setups where possible.

3) End users who want the power of ZFS and are willing to put forth the cash and effort to get the substantially-better-than-Synology result, but this still requires more effort than just unboxing a Syno and running Synology Assistant.

If you've ever had a Synology tragedy, you might discover their support to be more along the lines of "disinterested" or "nonexistent". iXsystems offers support with their TrueNAS server products, and is well known for it. The free software product they have released has no parallel at Synology, and the user community here that supports the free product has an amazing number of highly skilled and knowledgeable contributors, all of whom share their knowledge with this community for free. If you find that unpleasant, that's unfortunate, but there isn't even such a thing at Synology since they don't have a free product at all. They do have a sort of mediocre community support forum for their paid stuff.

 

[winnielinnie]

TrueNAS sites basically break down into three general categories:

1) Enterprise deployments looking for massive scale storage, willing to pay for it, and aren't really looking at value-adds like surveillance camera stuff. These are the sites iXsystems is focused on, because these people pay the bills.

2) End users who just want something cheap and easy to deploy, who usually have TrueNAS experiences that end badly when they try to build the cheapest possible thing out of a 2009-era PC with 4GB of RAM and some scrounged disks. Here in the forums, we try to salvage these people into better setups where possible.

3) End users who want the power of ZFS and are willing to put forth the cash and effort to get the substantially-better-than-Synology result, but this still requires more effort than just unboxing a Syno and running Synology Assistant.

That should be a sticky.

 

[IOSonic]

If you find that unpleasant, that's unfortunate, but there isn't even such a thing at Synology since they don't have a free product at all. They do have a sort of mediocre community support forum for their paid stuff.

This response seemed a little indignantly defensive, and it mischaracterized some of what I said. I agree with almost everything in your reply, but you appear to mistakenly believe I am criticizing this project when I am making objective observations about who will use it and why.

I never said I find the user community here is unpleasant. I can think of a few FOSS project communities that routinely get pretty uncivil, but this is not one of them. For users having problems, the necessity of relying on a community forum can be difficult, simply because it may be the only avenue to solving a problem if one lacks the information to fix it themselves. Experience-shy users often require a lot of catch-up, too (this has been me many times ). I sometimes see those kinds of posts go unanswered, likely because there's a lot of legwork involved to bring the user up to speed. Personally, I try to help someone whenever I log in, but I only have so many hours in a day to spend on forums.

I think your breakdown of users who may choose TrueNAS over Syno, or vice versa, is spot on. TrueNAS is the more powerful, performant and secure product. Still, some prefer not to go through the SLOG of getting some things to work (do you see what I did there ?). Investing the extra effort to learn TrueNAS got me the best solution in the end, personally. For others, they may want an easy-button, and TrueNAS will not be for them. Either decision is ok; it just depends on what is most appropriate for that person personally.

 

[Constantin]

To centralize or decentralize in IT is like breathing for lungs. First we had standalone computers, then we built centralized minicomputers that allowed multiple users, then we went back to stand-alone PCs, now we’re going back into the cloud. Work may fit your computer with an SSD but all that speed may be for naught if every file you save has to be uploaded to the cloud first.

I get why organizations want to cast off their IT businesses as unnecessary cost centers that get in the way of innovation, and so on. It’s expensive to plan for & stand up performant IT infrastructure, maintain it well, and so on. Being able to spool up a big platform and then collapse it as a function of actual, not projected, demand significantly lowers business risk. Anything with spiky or seasonal loads is a natural candidate for the cloud.

But by outsourcing it, you also hitch your collective IT wagon to go to wherever the Amazon AWS or whatever alternative cloud computing juggernaut is headed. Given the oligopolistic market structure now in place, that may get expensive - not only in the day to day but also in terms of switching costs if you ever want to unhitch your wagon.

Further, I don’t see why a competently run IT department for any organization at scale should be more expensive to run than using the cloud for baseline operations. Amazon is reaping really attractive profit margins from their AWS platform ATM. Scale has to do with but scale on its own doesn’t explain it.

 

[danb35]

I don’t see why a competently run IT department for any organization at scale should be more expensive to run than using the cloud

This seems obvious--if Amazon (or Google, or Microsoft, or IBM) can do the work and make a profit from it, you should be able to do the work yourself and keep the profit. But there are a couple of sticking points buried in this statement.

The first, I think, is obvious: "at scale." The large majority of organizations aren't large enough to economically support a full-time IT staff, so they either pay a full-time IT person to sit on his hands most of the time, use a local support organization, or use the cloud--and often, for such a smaller organization, the cloud really is the more economical way to go.

The second may be less obvious: "competently run". It's been my impression that assembling a truly competently-run IT department is actually very hard, particularly when that isn't the organization's main focus. Witness the many large organizations in the last year or two that got bit by ransomware, proof positive that their IT departments were not competently run (first because it happened in the first place, second because they were not able to quickly restore from backup and move on).

 

[Elliot Dierksen]

To centralize or decentralize in IT is like breathing for lungs.

I have been in IT 35+ years, and I refer to it as a tidal cycle. Start where you will, but I have seen it go like this many times.
Everything takes too long and we can't get anything done. Push it all out to distributed systems!
Everything is out of control and we can't integrate anything. Pull it all back in to centralized systems!

There is no one size fits all/magic bullet. The right thing depends on what you are trying to accomplish.

 

[Constantin]

Part of being a competent IT department is knowing what to outsource and what to keep in house and why. There are good arguments for the cloud, just as there are for decentralized operations. Usually, a mix of the two makes the most sense for any organization.

However, I’d put an emphasis on doing and saving as much locally as possible and then uploading to the cloud as necessary vs. the reverse (which adds a lot of latency for the user). Just creating a folder on your abstracted “desktop” can take a minute, if said creation has to be coordinated with some faraway cloud center first. Saving large files “locally” becomes a coffee break.

I put IT into the same bucket as HR, as in its a service that is usually not the focus of an organization, it’s hard to get right, etc. But, once it’s broken, it can ruin an organization. Ironically and tragically one of my former employers went bankrupt because of a bungled IT project started by HR. As a employee-owned company, that cost 10k+ people their retirement.

 

[jgreco]

This response seemed a little indignantly defensive, and it mischaracterized some of what I said.

Perhaps, perhaps not. I've been here doing this for more than a decade, so I have quite a few years of additional context with which to interpret some of what you said, some of where it might have gone next, and some of where it has actually gone to from there in past discussions. In other words, it wasn't quite a direct response specifically to you, but rather to the general class of users who say something like it.

Experience-shy users often require a lot of catch-up, too (this has been me many times ). I sometimes see those kinds of posts go unanswered, likely because there's a lot of legwork involved to bring the user up to speed. Personally, I try to help someone whenever I log in, but I only have so many hours in a day to spend on forums.

Me too; at a point early on, I switched to a model of trying to create sticky posts which I could then point people at, because the task of writing bespoke responses to individual inquiries is not generally tenable in this sort of support forum. It turns out to be easier to be able to point people at the things they need to know, rather than to inadequately answer their questions directly... and it is therefore important to have high quality informational sources available to point people to.

I think your breakdown of users who may choose TrueNAS over Syno, or vice versa, is spot on. TrueNAS is the more powerful, performant and secure product. Still, some prefer not to go through the SLOG of getting some things to work (do you see what I did there ?). Investing the extra effort to learn TrueNAS got me the best solution in the end, personally. For others, they may want an easy-button, and TrueNAS will not be for them. Either decision is ok; it just depends on what is most appropriate for that person personally.

This may underplay TrueNAS somewhat, as it has come leaps and bounds over the years, and is much better at new user interactions now than it used to be. Still, there is a significant problem for users approaching TrueNAS from the "assemble random pile of junk, expect it to work" aspect, and as previously discussed, that is going to continue to be a challenge because Synology, QNAP, etc., just don't have that issue, because you can't run their stuff on anything but a blessed platform. They also have a substantially easier model to work with, because their platforms are generally just lightly modified Linux EXT or BTRFS platforms, and by being designed for smaller systems with fewer options, there are definitely less choices that can be made which would be true configuration mistakes. TrueNAS is also very heavyweight. A small NAS platform like a Synology DS416slim (4x 2.5" bays, 1GHz CPU, 512MB RAM, dual ethernet, less than $300) has no competition within the TrueNAS world, and is highly useful in many contexts, but it is not a performance superstar. We still use the Syno's here for base tier shared VM storage, because the cost of a pair of decent TrueNAS appliances is still hideously greater than a quartet of crappy DS416slim's.