BUILD Updated build thoughts?

[Nick2253]

Okay now i'm getting it, the abstraction clicked. vdevs for each drive, zpools to organize the drives. So long as the vdevs are maintained, drive failures aren't a (likely) concern.

As long as each drive is a vdev, I can change the zpool as I see fit (within some limitations). IE adding vdevs to a RAIDZ2/3 zpool or taking an existing (striped/stand alone) drive/vdev that is its own zpool and adding a second vdev to the pool for mirroring without losing existing data in the pool.

Assuming a RAIDz2 pool with 4 drives (or 2 mirrored drives) . If a drive fails, the vdev isn't necessarily compromised and neither is the zpool (my data). If the vdev fails, the drive isn't necessarily compromised but the zpool (my data) is. Correct?

No, this is wrong. You don't have a RAIDZ2 pool; you have a RAIDZ2 vdev. All a pool does is pool the vdevs together. It provides no redundancy: that's what the vdevs are for.

The confusion comes from the fact that most people create a zpool with a single vdev, so attribute pool properties to vdevs and vice versa. However, they are distinct concepts.

 

[ChriZ]

Absolutely correct and to clarify my previous post:
A raidz2 pool of 4 drives is a raidz2 vdev which is the only vdev in the pool.
A two mirrors pool is a pool consisting of two vdevs; each vdev contains two mirrored drives.
Bottom line:
A drive is ... well, a drive
A vdev is a virtual drive consisting of one or more drives.
A pool is a "container" ( sorry could not find a better word) consisting of one or more vdevs.

 

[Something]

Any recommendations as far as the hardware goes?

No, this is wrong. You don't have a RAIDZ2 pool; you have a RAIDZ2 vdev. All a pool does is pool the vdevs together. It provides no redundancy: that's what the vdevs are for.

The confusion comes from the fact that most people create a zpool with a single vdev, so attribute pool properties to vdevs and vice versa. However, they are distinct concepts.

Okay, thank you.

In a 4drive raidz2 pool you can lose any 2 drives and data stays intact.
In a two mirrored pairs zpool (two raid1s which create a raid10, in traditional raid naming), you can afford to lose 2 drives, but not from the same mirror.
In this aspect a raidz2 using 4 drives is safer. But due to huge waste of space, the best raidz2 to begin with is a 6drive one.

Alright, now I see why it's preferential. I think for now i'll hold off on buying 6 drives, that's a bit beyond me in pricing.

 

[Robert Trevellyan]

raidz2 using 4 drives is safer. But due to huge waste of space

I think the phrase "waste of space" is a bit loaded here. The space you devote to redundancy in any vdev isn't wasted, it's allocated to improving data availability. Some might consider a 4 drive RAIDZ2 vdev extravagant, but if the alternative is buying more expensive system that supports more drives, when your storage needs are modest but you value your data, it's a great solution. It's also very easy to upgrade.

 

[Something]

I think the phrase "waste of space" is a bit loaded here. The space you devote to redundancy in any vdev isn't wasted, it's allocated to improving data availability. Some might consider a 4 drive RAIDZ2 vdev extravagant, but if the alternative is buying more expensive system that supports more drives, when your storage needs are modest but you value your data, it's a great solution. It's also very easy to upgrade.

It is possible to go from mirroring vdev to RAIDz2 vdev without losing data in the vdev, correct?

 

[danb35]

No, that is not possible.

 

[Something]

I didn't think so though I wished it... I take it RAIDZ2 vdevs can't be changed to RAIDZ3 vdevs (assuming a sufficient number of drives).

Actually, I think i've been convinced on the usefulness of RAIDZ2. It'd be perfect for expansion. I doubt i'll go for more than 8-10 drives in my NAS and 10x4TB RAIDZ2 is still safe, no?

E: scratch that, even at 3% annual failure rate i'm looking at a 3.2% shot of losing my array, 5% or 7% results in catastrophic likelihood of things going very, very bad. I probably will stop at 6 and if need be fracture things off to a second zpool and mirror.

 

[Darren Myers]

x10 4's is acceptable, if it was me I'd do x11 4's in a RAIDz3 but you loose some performance but with my x7 4Tb Reds in RAIDz3 I can scrub at close to 600; typically chills around 530-550. But that's an extra drive of protection but at a cost


Sent from my iPhone using Tapatalk

 

[Something]

Any other thoughts? If not, i'll start placing orders for parts. Ideally 1 day shipping on that UPS...

Big round of thanks everyone, this is really helping me to plan out where i'll go for the future in ways I hadn't previously considered. I think RAIDZ2 is the most sensible and if need be, in the future, move to a bigger system. Oh who am I kidding, in the future i'm going to end up with a much more crazy system. 10GbE, RAM a plenty, and so, so very much 10e-15 URE storage space. 3-5 years is plenty of time for all kinds of advancements to bring that stuff to more affordability.

For now, i'll work with 4x4TB vdevs in RAIDZ2 (6.1TB of space is fine for now) and expand up to 6x4TB (do I want to even consider what'll necessitate 12.4TBs of storage? Yes, yes I do, but my wallet doesn't). From there, start a new RAIDZ2. I'm not ready to play too risky with that data. Some of it, if I lose, it's gone for good. When it hits that point of uncomfortability, i'll consider an external drive or 2 as cold storage redundancy if not something smarter. The chances with 6x4TB vdevs with a 10e-14 URE are still pretty good so perhaps not, but I think i've decently planned for the next years and years. A 4c8T E3 is plenty of performance, 32GB DDR3 ECC is a bit on the small side but if my RAM needs increase a year from now I can upgrade rather than feature creep now, and RAIDZ2 with 4TB drives gives me a nice balance between cost efficiency, redundancy, safety and performance.

x10 4's is acceptable, if it was me I'd do x11 4's in a RAIDz3 but you loose some performance but with my x7 4Tb Reds in RAIDz3 I can scrub at close to 600; typically chills around 530-550. But that's an extra drive of protection but at a cost


Sent from my iPhone using Tapatalk

RAIDZ2 seems to be all around a winner, I can see why it's got a good amount of love here. So long as the drive size, number of drives, and number of redundancies are planned for decently, I can enjoy faster speeds with decent safety through redundancy and versatility in expansion.

Have any calculations into 5TB, 6TB and 8TB drives been done?

 

[Bidule0hm]

I'm not ready to play too risky with that data. Some of it, if I lose, it's gone for good.

Just remember that RAID doesn't replace backups. If you have important data you still need to do backups of it.

 

[Darren Myers]

Just remember that RAID doesn't replace backups. If you have important data you still need to do backups of it.

This , this all day


Sent from my iPhone using Tapatalk

 

[Nick2253]

For now, i'll work with 4x4TB vdevs in RAIDZ2 (6.1TB of space is fine for now) and expand up to 6x4TB (do I want to even consider what'll necessitate 12.4TBs of storage? Yes, yes I do, but my wallet doesn't). From there, start a new RAIDZ2.

I think you're still misunderstanding the expandability issue.

First off, let's be precise with your language: you cannot combine vdevs in to RAIDZ2. RAIDZ2 (or RAIDZ1 or mirror) is a property of a single vdev. You can combine 4, 4TB drives in to a single RAIDZ2 vdev. You could also create 4x vdevs (each drive as a single vdev), but you cannot combine those in any way that provides redundancy.

Second, you cannot trivially expand a RAIDZx vdev. The only vdev you can expand (i.e. add drives to) is a mirrored vdev. However, all that does is add an additional mirror (i.e., the data is now mirrored across three drives instead of two).

If you want to add space, you have two options: replace all the drives in a single vdev with larger drives, or add an additional vdev. You cannot convert a 4 drive RAIDZ2 vdev into a 6 drive RAIDZ2 vdev without destroying the vdev.

 

[Something]

Just remember that RAID doesn't replace backups. If you have important data you still need to do backups of it.

Well the data doesn't exist elsewhere, so it fundamentally can't be backed up. Always nice to hear though.

I think you're still misunderstanding the expandability issue.

Yes, yes I have. It's okay though, I didn't want to have money anyways.

First off, let's be precise with your language: you cannot combine vdevs in to RAIDZ2. RAIDZ2 (or RAIDZ1 or mirror) is a property of a single vdev. You can combine 4, 4TB drives in to a single RAIDZ2 vdev. You could also create 4x vdevs (each drive as a single vdev), but you cannot combine those in any way that provides redundancy.

Second, you cannot trivially expand a RAIDZx vdev. The only vdev you can expand (i.e. add drives to) is a mirrored vdev. However, all that does is add an additional mirror (i.e., the data is now mirrored across three drives instead of two).

So buy for a zdev i'm living with for the next number of years.

4TB drives are $150 each. Assuming 5TB drives at $200, 6TB drives at $250 for upgrading, 77.5% usable drive capacity...
$300 now, 6.2TB storage. 4x4TB -> 4x5TB = $800, 7.75TB. 4x4TB -> 4x6TB = $1000, 9.3TB.
$450 now, 9.3TB storage. 5x4TB -> 5x5TB = $1000, 11.625TB. 5x4TB -> 5x6TB = $1250, 13.95TB.
$600 now, 12.4TB storage. 6x4TB -> 6x5TB = $1200, 15.5TB. 6x4TB -> 6x6TB = $1500, 18.6TB.

So I should invest in more drives now and make the vdev large enough to accommodate future growth, rather than attempt to expand the vdev down the line. I think based on my storage needs that 3 to 4 drives with 2 redundant would be ideal. So $450 or $600 now.

If you want to add space, you have two options: replace all the drives in a single vdev with larger drives, or add an additional vdev. You cannot convert a 4 drive RAIDZ2 vdev into a 6 drive RAIDZ2 vdev without destroying the vdev.

I'm sure i'm missing something and shouldn't ask why it isn't possible right now, but is there something fundamental that makes such a thing impossible or is it just a choice of implementation because of things like unreliability and there being little guarantee of success?

 

[Bidule0hm]

Well the data doesn't exist elsewhere, so it fundamentally can't be backed up.

I don't understand. If the data exist at least at one place then you can (and should) backup it.

So I should invest in more drives now and make the vdev large enough to accommodate future growth, rather than attempt to expand the vdev down the line.

It's a complex question. First the drives with the most TB/$ are the 4 TB ones so I'd chose those ones. Then it depends on your current needs and your future needs.

but is there something fundamental that makes such a thing impossible or is it just a choice of implementation because of things like unreliability and there being little guarantee of success?

ZFS is a very complex file system. It's not impossible (everything is possible... well, more or less...) but it's probably very complex to implement this so the ZFS devs haven't implemented it because the complexity outweighs the utility.

 

[Robert Trevellyan]

is there something fundamental that makes such a thing impossible or is it just a choice of implementation

The current implementation of ZFS does not allow modification of any type of vdev other than a mirror. My comment about a 4-drive vdev being easy to upgrade was based on the idea that replacing 4 drives is easier and cheaper than replacing more than 4 drives.

For a while there has been talk of the underlying ZFS implementation being enhanced to enable vdev modification, but you shouldn't plan your storage based on that expectation. If you want to be able to expand your storage ad-hoc, ZFS is not for you. The closest you can get with ZFS is to build your storage entirely with mirrors.

 

[Robert Trevellyan]

even at 3% annual failure rate i'm looking at a 3.2% shot of losing my array, 5% or 7% results in catastrophic likelihood of things going very, very bad

I think your math may be a bit off here. It looks like you're assuming a failed drive doesn't get replaced.

 

[Something]

I don't understand. If the data exist at least at one place then you can (and should) backup it.

Oh definitely. The language I used is unclear. "The data can't be backed up" can mean the data itself can't be backed up or the data doesn't have the property of being backed up. I meant the latter, since the data doesn't exist on another drive (not redundant, ideally not in the same machine), it isn't backed up.

RAID not being a backup is always one of those things that makes me double take on some level though I know the reasons why. At surface level it makes sense to consider it a backup. Though really its only a way to minimize downtimes, not backup data.

It's a complex question. First the drives with the most TB/$ are the 4 TB ones so I'd chose those ones. Then it depends on your current needs and your future needs.

4TB drives are what I'm rocking, I made at least one good decision as far as my NAS goes. Question is quantity as expansion is frightfully costly to convert from 4TB to 5, 6 or 8. A new vdev will likely be the next course of action, though I'll have to wait on that front as it necessitates a new machine or at the very least a larger case. Future problems.

ZFS is a very complex file system. It's not impossible (everything is possible... well, more or less...) but it's probably very complex to implement this so the ZFS devs haven't implemented it because the complexity outweighs the utility.

Hmm...

Contraction would be incredibly difficult (it makes a number of assumptions) but I wonder about expansion. At a (very) high level it seems easily possible. But, like many things in computer science, that doesn't translate well to reality. I'd need a low level understanding of how ZFS operates including what is stored on each drive, how the drives are managed, and how differing drive counts affect RAIDZ# to even hazard a high level guess to how one would implement such a feature.

It would be a really interesting problem to try and tackle though I agree, it probably isn't an efficient problem to solve given the dev cost and added maintenance complexity along with it being a feature most (larger) companies wouldn't likely care for. Opportunity cost says doing what FreeNAS already does better and with less bugs is more important given what FreeNAS is for.

The current implementation of ZFS does not allow modification of any type of vdev other than a mirror. My comment about a 4-drive vdev being easy to upgrade was based on the idea that replacing 4 drives is easier and cheaper than replacing more than 4 drives.

Thank you for clarifying. I think even 4 drives is way too pricey to upgrade.

For a while there has been talk of the underlying ZFS implementation being enhanced to enable vdev modification, but you shouldn't plan your storage based on that expectation. If you want to be able to expand your storage ad-hoc, ZFS is not for you. The closest you can get with ZFS is to build your storage entirely with mirrors.

Mirrors was how I originally thought I would do things, but I think investing now into RAUDZ2 is a more efficient solution and can see why its preferred. It seems a direct upgrade from mirroring with one small cost.

I'm not going to bet on ZFS having such an enhancement, cool at is it would be. With not going ECC and taking the advice of the professionals (i'm not normally one for such arrogance) I have learned that ZFS and FreeNAS are what they are. Really effective storage and backup solutions albeit VERY much not for Joe Average. It's professional stuff for professional usage and I have to bend to it, not vice versa. Planning, learning, and a large learning curve are the price paid for what ZFS excels at.

At the very least, this gives me something to say I learned as an engineer by messing up. I'm just thankful it hasn't been an irrecoverable mess up costing me data and instead a financial one.

 

[Bidule0hm]

Question is quantity as expansion is frightfully costly to convert from 4TB to 5, 6 or 8. A new vdev will likely be the next course of action, though I'll have to wait on that front as it necessitates a new machine or at the very least a larger case. Future problems.

Also, don't forget the prices decrease with time and you'll need to replace failed drives sooner or later. So, if you replace the failed drives with bigger drives then you'll expand the pool when all drives will be replaced. It's what I chose to do. You need to be sure that the current pool will cover your space needs for about 5 years.

I'd need a low level understanding of how ZFS operates including what is stored on each drive, how the drives are managed, and how differing drive counts affect RAIDZ# to even hazard a high level guess to how one would implement such a feature.

OpenZFS code is open-source, FreeNAS code too, feel free to read them :)

 

[danb35]

Contraction would be incredibly difficult (it makes a number of assumptions) but I wonder about expansion. At a (very) high level it seems easily possible. But, like many things in computer science, that doesn't translate well to reality. I'd need a low level understanding of how ZFS operates including what is stored on each drive, how the drives are managed, and how differing drive counts affect RAIDZ# to even hazard a high level guess to how one would implement such a feature.

It's something that's been discussed since the early days of ZFS, and the consensus appears to be that it just isn't going to happen. If it did, it would make a lot of things better. Add drives to a vdev? Done. Convert RAIDZ1 to RAIDZ2? Done. Enable (or disable) compression or deduplication? Not a problem. But getting it there seems to be a task of epic proportions. If you want to read up on the issue, the term you'll want to look for is "block pointer rewrite".

One semi-related area in which some progress is being made is the ability to remove a vdev from a pool. That's still far from production-ready, but it's a feature that seems to be moving a bit.

One filesystem that's semi-comparable to ZFS is btrfs. It's my understanding that it does support at least adding drives to existing RAID sets; not sure about changing RAID levels or other things.

 

[Something]

I think your math may be a bit off here. It looks like you're assuming a failed drive doesn't get replaced.

I used these calculations from Nick. https://forums.freenas.org/index.php?threads/the-math-on-hard-drive-failure.21110/

Also, don't forget the prices decrease with time and you'll need to replace failed drives sooner or later. So, if you replace the failed drives with bigger drives then you'll expand the pool when all drives will be replaced. It's what I chose to do. You need to be sure that the current pool will cover your space needs for about 5 years.

I anticipate being able to replace the entire machine if not add a much larger machine within the next 2-3 years. I shouldn't need more than 9.3TB (5 drives total RAIDZ2) at any point soon and if I did, I'd likely have the funding to afford for a new machine. Off the top of my head the only thing to necessitate such space would be uncompressed recording and video storage. Replacing drives with larger ones is an interesting take.

Regarding HD prices I'm a bit fed up with that racket. Prices still operate at their 2010(?) Panic high. A 4TB external was $140 3 years ago, and I'm paying $150 for 4TB internals today. I can't wait till Seagate and other oust their CEOs and management when profits crumble as SSDs cut into their markets. Enterprise SSDs of an affordable nature are moving fast and the HD manufacturers aren't keeping up.

Aren't failed drives covered by warranty?

OpenZFS code is open-source, FreeNAS code too, feel free to read them :)

Heh, what can I say, it was a tacit admission I'm too lazy too (for now).

It's something that's been discussed since the early days of ZFS, and the consensus appears to be that it just isn't going to happen. If it did, it would make a lot of things better. Add drives to a vdev? Done. Convert RAIDZ1 to RAIDZ2? Done. Enable (or disable) compression or deduplication? Not a problem. But getting it there seems to be a task of epic proportions. If you want to read up on the issue, the term you'll want to look for is "block pointer rewrite".

Oh boy. From low level to VERY low level. I'll check it out cheers.

One semi-related area in which some progress is being made is the ability to remove a vdev from a pool. That's still far from production-ready, but it's a feature that seems to be moving a bit.

Huh, seems like it should be the opposite. How does one go about replacing a drive in a vdev that fails?

One filesystem that's semi-comparable to ZFS is btrfs. It's my understanding that it does support at least adding drives to existing RAID sets; not sure about changing RAID levels or other things.

Hmmm. I'll see what the hisory on brtfs was.