New ZFS version or feature flags -> Update-Time for 17TB?

[longtom]

Hello together!
Is the zfs update taking minutes/hours/days for about 17TB?
Thanks!

 

[Ericloewe]

Upgrading the pool is instant, though some feature flags may introduce background operations to optimize something.

That said, you most probably do not want to jump into that pool without careful consideration of the pros and cons:

ZFS Feature Flags in TrueNAS

OpenZFS' distributed development led to the introduction of Feature Flags. Instead of incrementing version numbers, support for OpenZFS features is indicated by Feature Flags. Feature Flag states Feature flags exist in one of three states...

www.truenas.com

 

[longtom]

Ok, do with 13.0 only draid flag is introduced.

I found following from here

Distributed RAID vdevs are mostly intended for large storage servers—OpenZFS draid design and testing revolved largely around 90-disk systems. At smaller scale, traditional vdevs and spares remain as useful as they ever were.

 

[Ericloewe]

This is one of those cases where you can just blindly enable the draid feature flag, because it won’t actually be active until you have a DRAID vdev, which you probably won’t.

 

[TrumanHW]

This is one of those cases where you can just blindly enable the draid feature flag, because it won’t actually be active until you have a DRAID vdev, which you probably won’t.

I think (speaking of my own interest) that dRAID is a feature that sounds like it should

- increase performance per HD
- improve re-silver time (substantially)
- improves the ratio of HW iops to iops available for storage
- costing the space ... but retaining the performance ... you specify to "reserve" (same as RAIDz).

Knowing any 'cons' would make it easier to get over my fascination.

Are IOPs slower?
Are they less reliable in some way?
Incompatible with Special vDevs..?

Bc, with known pros ... and no stated 'cons' ... it's not:
Why like it, but rather, why wouldn't everyone like & use it?
(To the extinction of RAIDz?)

(And Eric, I asking you you're a smart and knowledgeable guy. I'm sure you have a reason driving your decision, bc I've been reading your answers for years, & you have reliably good logic. You're just not always in the mood to reveal your logic with an explanation .... and instead report your conclusion. But devs worked on this feature for years, only for it to be relevant to a miniscule % of users who have 90 (or 70, or 50) ... HDs ..? Really?)

 

[danb35]

But devs worked on this feature for years, only for it to be relevant to a miniscule % of users who have 90 (or 70, or 50) ... HDs ..? Really?)

Given that ZFS is designed for the enterprise market, why does this surprise you? But I'm wondering what "decision" of Eric's you're referring to; the post you quote is simply stating that it's unlikely @longtom will have a dRAID vdev.

If you're reading in Eric's post a recommendation against using dRAID, I don't see it, but I'd think a perfectly valid reason for such a recommendation would be "TrueNAS doesn't yet support it."

 

[Ericloewe]

I don't recommend dRAID, but I also don't recommend against it. To be honest, I'm not 100% on some of the details I'd need to understand in order to recommend it.

And Eric, I asking you you're a smart and knowledgeable guy. I'm sure you have a reason driving your decision, bc I've been reading your answers for years, & you have reliably good logic. You're just not always in the mood to reveal your logic with an explanation .... and instead report your conclusion.

If you're reading in Eric's post a recommendation against using dRAID

This is one of those cases where you can just blindly enable the draid feature flag, because it won’t actually be active until you have a DRAID vdev, which you probably won’t.

Allow me to clarify my original meaning:
dRAID is not the sort of thing you'd just throw in with older vdevs. It just doesn't make sense, other than for testing edge cases. You'd use it as a replacement for (multiple) RAIDZ vdevs.

Knowing any 'cons' would make it easier to get over my fascination.

Are IOPs slower?
Are they less reliable in some way?
Incompatible with Special vDevs..?

IOPS should be comparable to similar RAIDZ setups, taking into account that a single dRAID vdev can take the place of multiple RAIDZ vdevs. Conceptually anyway. Reliability should also be on par with RAIDZ, apart from bugs and the like. It should also have zero impact on special vdevs.

The primary compromise in dRAID's design versus RAIDZ is that it deals poorly - worse than RAIDZ - with small blocks. The realistic expectations are something that will have to be built up as it gets more use.

 

[Etorix]

Why like it, but rather, why wouldn't everyone like & use it?
(To the extinction of RAIDz?)

Because dRAID doesn't make sense for a single vdev. dRAID is useful for many vdevs—typically enterprise setups with 50+ drives on a big SAS backplane. Home users are unlikely to have enough drives for dRAID.
Which is fine. Most home users do not need a L2ARC or a SLOG either. And ZFS gets a lot easier once one understands that its many features operate on a "need-to-know-basis".

 

[ChrisRJ]

But devs worked on this feature for years, only for it to be relevant to a miniscule % of users who have 90 (or 70, or 50) ... HDs ..? Really?)

How do you come to the conclusion that only a fraction of ZFS users has that number of HDDs?

In addition, for the context of this topic, we need to distinguish paying enterprise customers and hobbyists, who are basically free riders here.

All in all, 90 disks is not really large for an enterprise deployment. Of course, no playground anymore, but nothing exceptional either. You can easily find cases that put this amount of disks into 4U. And with cases of 60 disks/4U you end up with 600 HDDs in a full-size rack.

 

[TrumanHW]

How do you come to the conclusion that only a fraction of ZFS users has that number of HDDs?

In addition, for the context of this topic, we need to distinguish paying enterprise customers and hobbyists, who are basically free riders here.

All in all, 90 disks is not really large for an enterprise deployment. Of course, no playground anymore, but nothing exceptional either. You can easily find cases that put this amount of disks into 4U. And with cases of 60 disks/4U you end up with 600 HDDs in a full-size rack.

Statistics.

There are many passenger rental car companies. There are many Fortune corporations which have car fleets. The number of people / corporations that own a vehicle is LESS THAN 1 vehicle based on the total registered vehicles in the U.S. (including corporations).

There were just 263.6-million registered vehicles in the US in 2015
Only most of which were passenger cars, meaning, including trucks/commercial.

Yet, the number is still less than 1 per US Citizen. if I understood wikipedia accurately.

So yes, I highly doubt there are anywhere near as many 90-drive arrays as there are home-labs with the minimum ... many of whom use RAID-5 bc they can't afford RAID-6 despite knowing we've long since scoffed at that decision ...

In fact, I bet ~30% of those who "buy a RAID" use RAID-0 bc it's the default setting ...
Knowingly lying to themselves about the risk of getting some fault tolerance ...
Because they can't afford the performance and fault tolerance.

What % of free vs Enterprise TrueNAS deployments do you think there are?


We're we talking about the number of drives in a RAID / SAN / etc including consumers, whatever it is, it'll be described by some math model that asymptotically approaches ±1 HD over [min HD +1] per RAID type used, eg.,

• RAIDz1 will avg 4 HD ±1 HD
• RAIDz2 will avg 5 HD ±1 HD

(Due to the psych aversion to no-more security than RAID 10 which outperforms it - if checksummed like ZFS)

But that curve is gonna be smooth! The number of people & corps (including SOHOs) which own arrays comprised of more drives ...?

Decreases as the HD number increases, asymptoting to 0 at some obscenely high number, with gaps at the extreme end of the tail of maximum HD in use.

What may be an impressive & surprising number is perhaps however many HDs are used in that tiny number of ginormous arrays by the monstrosities (as is inferable) by the annual Backblaze reliability-report & other monstrosities like governments (some on AWS) Google, Apple, DropBox, TeleComs, etc. even ignoring businesses with sub-500 HD arrays.

But to think there might ... just be as many 90-HD arrays as those of 8 or less??

Not to be rude, but I'd bet on it ... (and we probably agree: money is truth serum).

 

[Ericloewe]

I mean, people paid developers to implement dRAID. If that's not a statement to the effect of "we really want to use this", I don't know what is.

 

[danb35]

Would you have guessed that US citizens owned an average number of cars below 1..?

I probably would have, particularly if you consider that many citizens are under driving age, and in larger cities, it's fairly common for a family to not have a car at all. But WTF does that have to do with anything?

Your incredulity aside, the fact remains that ZFS was developed for enterprise use, so it shouldn't be surprising that at least some of the features being developed (and recently released) are going to be applicable to applications with large drive counts. But if you want to be the guinea pig with your data, and you're fine managing it at the CLI, there's nothing stopping you.

 

[TrumanHW]

Because dRAID doesn't make sense for a single vdev. dRAID is useful for many vdevs—typically enterprise setups with 50+ drives on a big SAS backplane. Home users are unlikely to have enough drives for dRAID.
Which is fine. Most home users do not need a L2ARC or a SLOG either. And ZFS gets a lot easier once one understands that its many features operate on a "need-to-know-basis".

Politely, this is the same issue which prompted my question.
This says "what" but doesn't describe "why." I've yet to see a reason.
Do you believe things in the absence of reasons?

Can you even support your own statement with evidence or public statements..?
(I.e., from iXsystems?)

What feature(s) or flaw(s) makes it better for large arrays & worse for small arrays..?

Politely & out of curiosity – have you used dRAID?
Can we assume it retains all features of RAIDz ... like Special vDevs ..?

 

[TrumanHW]

I mean, people paid developers to implement dRAID. If that's not a statement to the effect of "we really want to use this", I don't know what is.

Now THIS makes sense to me.

All the other speculations about dRAID?
Why it's irrelevant
It's [only] good for large arrays but it's bad for small arrays. (very weird statement)

And noone citing a single reason; just baseless assertions
(argument from authority / ad verecundiam).

My optimism may be unfounded ... but if it's SAFE for large arrays?
Doesn't sound bad to experiment with at least ...

It's seeming more and more like a rorschach test than candid array analysis.
It's how we look at features / OS / apps before they're released.
Do we look at them with optimism or pessimism ..?
And, that, all optimists confuse realists for pessimists.

I'm optimistic because I have confidence in iXsystems and their dev-team.
And I've listened to lectures by Matt Ahrens, etc., or maybe it's confirmation bias.

 

[TrumanHW]

I probably would have, particularly if you consider that many citizens are under driving age, and in larger cities, it's fairly common for a family to not have a car at all. But WTF does that have to do with anything?

Your incredulity aside, the fact remains that ZFS was developed for enterprise use, so it shouldn't be surprising that at least some of the features being developed (and recently released) are going to be applicable to applications with large drive counts. But if you want to be the guinea pig with your data, and you're fine managing it at the CLI, there's nothing stopping you.

There's 265-million americans over 16 yo ... but ignoring the parallels etc ...
You're making some sort of an argument that the feature is "good for enterprise" in a way that is bad for smaller arrays. Are there any other array configurations that make sense for enterprise that don't make sense for people with 6-16 drives ..?

I've already addressed that the feature's relegation to CLI are the devs way of saying "not ready for primetime yet." I GET THAT. But YOU are claiming to know something about it's fitment for a particular use. I'm asking you for YOUR reasons by which you come to this conclusion. We may as well be discussing the existence of a god. At least deists / theists cite (bad) reasons. You cite none. Aka, ad verecundiam. And I don't mean that you're being arrogant, but saying nothing would've been equally convincing to the arguments (reasons) you've provided. Which is all I've been asking for.

Do YOU accept explanations telling you what to do, instead of telling you the behavior, cost, pros and cons of tech to enable you to make decisions predicated on your own logic?

Do you know enough about it to do that?

And if you don't (neither theoretical nor from experience, which none of us have yet) ... then why are you so confidently telling me what to do? This is not only a question for me, but for you to consider. I get dudes telling girls "how to play pool" when they absolutely suck ... they have an ulterior motive; fair enough. But you have no ulterior motive ... so you either should have reasons (which I'd like you to articulate) ... or you're literally making an argumentum ad ignorantiam, and you might want to audit yourself for this tendency as engaging in one of the most well-known fallacies seems a poor policy ... and probably contributes to more decisions than you're aware of, not just this.

 

[TrumanHW]

I don't recommend dRAID, but I also don't recommend against it. To be honest, I'm not 100% on some of the details I'd need to understand in order to recommend it.



Allow me to clarify my original meaning:
dRAID is not the sort of thing you'd just throw in with older vdevs. It just doesn't make sense, other than for testing edge cases. You'd use it as a replacement for (multiple) RAIDZ vdevs.

IOPS should be comparable to similar RAIDZ setups, taking into account that a single dRAID vdev can take the place of multiple RAIDZ vdevs. Conceptually anyway. Reliability should also be on par with RAIDZ, apart from bugs and the like. It should also have zero impact on special vdevs.

The primary compromise in dRAID's design versus RAIDZ is that it deals poorly - worse than RAIDZ - with small blocks. The realistic expectations are something that will have to be built up as it gets more use.

Spectacular answer. Finally ... actual reasons, clearly expressed. Thank you!

And I actually have a pair of P5800 Optanes that I've yet to use (bc I got them at a decent price) which I could use as a Special vDev for small blocks ... and make a pair of partitions that are ~5s x the line-speed (I have an SFP28 switch but using it will depend on whether I can find SFP28 for reasonable prices which I can connect to my MacOS machines) ... so, either 10Gb or 28Gb ... in a mirrored pair, and the balance for the small blocks.

And, of course, [IF] I wind up creating a dRAID, it'll be sync'd to a RAIDz pool minimizing 5 years of bug reports and updates ... just as is happening (on an accelerated pace) with TrueNAS Scale (debian kernel).

Greatly appreciated.

 

[TrumanHW]

Given that ZFS is designed for the enterprise market, why does this surprise you? But I'm wondering what "decision" of Eric's you're referring to; the post you quote is simply stating that it's unlikely @longtom will have a dRAID vdev.

If you're reading in Eric's post a recommendation against using dRAID, I don't see it, but I'd think a perfectly valid reason for such a recommendation would be "TrueNAS doesn't yet support it."

Since several people mistook my statement, I must've been ambiguous & apologize.
I just meant Eric's general acumen and was nothing he'd specifically said about dRAID.

 

[Etorix]

Politely, this is the same issue which prompted my question.
This says "what" but doesn't describe "why." I've yet to see a reason.
Do you believe things in the absence of reasons?

Can you even support your own statement with evidence or public statements..?
(I.e., from iXsystems?)

Since iXSystem does not even describe dRAID in its documentation at this stage, I'll make do with

dRAID, Finally! - Klara Systems

Learn about dRAID in OpenZFS 2.1. Explore its data protection capabilities, distributed hot spares, faster resilvering, and enhanced performance in this article.

klarasystems.com

Conclusion:

The dRAID offers a solution for large arrays, vdevs with fewer than 20 spindles will have limited benefits from the new option. The performance and resilver result will be similar to RAIDZ for small numbers of spindles.

Or just from reading the description of dRAID.
A single dRAID vdev holds the equivalent of multiple raidz1-3 vdevs with spares, which are "pre-filled" with data to speed up subsequent resilvers. Multiple vdevs = lots of drives. Default stripe width is 8; multiples of that is going to be a lot of drives—at least from the point of view of a home user.
With a single raidz vdev, having hot spares is of no use: Just increase the raidz level. If you're at raidz3 and still trying to add hot spares to sustain the loss of more than three drives in relatively short sucession and without human intervention to replace failed drives, I have to ask: What is exactly your use case?

dRAID is intended to speed up resilver of parity arrays to guard against the risk of further drive failures during resilver. Although resilver is in itself a cause of strain which may cause further failures, hard drives are reasonably reliable; a single failure is a rare event and multiple failures have to be a even more rare event. Again, it takes a large number of drives in a single array to reach the point where the risk of multiple failures is not acceptable.

I do not use dRAID, and do not plan to, because I do not think I have enough drives to warrant it for my home use.