Planning a build, need some clarification

[SweetAndLow]

Creating multiple vdevs allows you to have much larger pools with smaller rebuild times, you only have to rebuild a single vdev not the entire pool. The danger isn't in loosing a drive, that is expected, the danger is during the rebuild. The faster you can get through the rebuild process the faster you can be back at full protection.

 

[Wallybanger]

Creating multiple vdevs allows you to have much larger pools with smaller rebuild times, you only have to rebuild a single vdev not the entire pool. The danger isn't in loosing a drive, that is expected, the danger is during the rebuild. The faster you can get through the rebuild process the faster you can be back at full protection.

OK, that makes sense. so that brings me back to the question of whether you can have more than one zpool....?

Furthermore, at what point is it necessary to have more than one VDev? It sounds like, in order to make the most of your drives, you need at least 5 or 6 drives for a RAIDZ2 array, am I right? Can you even do a RAIDZ2 array with 3 drives? At 4 drives your storage is cut by 50% (so you might as well just mirror the drives) and that percentage of cut storage decreases with every device added. So it sounds like you would need at least 10 drives to make a 2 VDev zpool reasonable.

With my plan being to use 8x 4TB drives, is there any point in running more than one VDev?

 

[Bidule0hm]

It sounds like, in order to make the most of your drives, you need at least 5 or 6 drives for a RAIDZ2 array, am I right? Can you even do a RAIDZ2 array with 3 drives?

No, you need at least 4 drives for a RAID-Z2 vdev.

With my plan being to use 8x 4TB drives, is there any point in running more than one VDev?

Nop.

 

[danb35]

whether you can have more than one zpool....?

Of course you can.

at what point is it necessary to have more than one VDev?

It's recommended to not have more than 10-11 drives in one vdev.

At 4 drives your storage is cut by 50% (so you might as well just mirror the drives)

You use half your space for redundancy, just like with mirrors, but the redundancy is better. With a four-disk RAIDZ2 vdev, you can lose any two drives. With striped mirrors, if you lose both disks in one mirror, that vdev is done, so your pool is lost.

 

[danb35]

having 1 VDev kill the entire pool still sounds like a design flaw.

Again, data is striped across all vdevs. Think of it like RAID0, but with vdevs rather than individual drives. In RAID0, if you lose one drive, all the data in that array is gone. What else would you do with multiple vdevs?

 

[Bidule0hm]

What else would you do with multiple vdevs?

I'd love ZFS to be recursive, for example a stripe of vdevs made of vdevs that are mirrors of 2 vdevs each made of 6 drives RAID-Z2 vdevs :D

 

[Wallybanger]

No, you need at least 4 drives for a RAID-Z2 vdev.



Nop.

Of course you can.

It's recommended to not have more than 10-11 drives in one vdev.

You use half your space for redundancy, just like with mirrors, but the redundancy is better. With a four-disk RAIDZ2 vdev, you can lose any two drives. With striped mirrors, if you lose both disks in one mirror, that vdev is done, so your pool is lost.

Boom. One VDev in my Zpool. Well that problem is solved :p And it's good to know it's possible to have more than one zpool.

Again, data is striped across all vdevs. Think of it like RAID0, but with vdevs rather than individual drives. In RAID0, if you lose one drive, all the data in that array is gone. What else would you do with multiple vdevs?

No idea. That's the point. So, aside from drive arrays of more than 10-11 disks that require a large number of simultaneous IO, it sounds like multiple VDevs in one zpool is borderline irresponsible.... especially since you can have more than one Zpool. If performance isn't a concern it sounds like the better thing to do would be to have 2 zpools, each with one VDev.... and even then, wouldn't you still have the advantage of more IO since the arrays are separate from one another?

 

[Ericloewe]

it sounds like multiple VDevs in one zpool is borderline irresponsible.

No, it isn't. The sum of two very small risks is still a very small risk.

wouldn't you still have the advantage of more IO since the arrays are separate from one another?

No, unless you have a fantasy workload that is magically spread across different shares.

 

[danb35]

multiple VDevs in one zpool is borderline irresponsible.

There are many scenarios where it's at least convenient, often useful, and sometimes even necessary, to put all your storage into one single pool. I can remember having free space split across several drives, and not having enough on any one drive to do what I wanted to; this is one of the problems ZFS was built to solve. If that's not of any value to you, very well, but it is to a lot of people.

Let me give a real-world example, my own. I had six disks in RAIDZ2 configuration in a 12-bay chassis, and my pool was getting full. It was trivial, after properly testing them, to install six more disks and add them to my pool--took about 15 seconds in the web GUI*. Nothing else had to be reconfigured. All the shares were in the same place, no scripts were moved, I didn't have to move anything to the new disks, nothing. Pool running out of space? Add storage to the pool. That's the way ZFS is designed to work. And I'll never run into a situation where I have 100 GB free here, and another 100 GB free there, and I need 150 GB to do whatever I want to do.

* Actually, my situation was a bit more complicated, because my pool was two three-disk RAIDZ1 vdevs--poor planning when I first built it. I had to build a new pool with the new disks (single RAIDZ2 vdev), replicate my data to the new pool, destroy the old pool, and rename the new pool to the old pool's name--at that point, everything was back up and running. I could then add the old disks to the new pool as a second RAIDZ2 vdev. I need more space again, and RAIDZ2 continues to be adequate, so I'll do what I described above.

 

[David Dyer-Bennet]

Damage to a single vdev is damage to the pool, so for example three mirror vdevs in one zpool is single redundancy -- two drives dieing can kill that pool (it won't if the drives are part of different mirrors, but it will if both drives are in the same mirror).

Now, that's the way I ran my home server since 2006 -- started with one mirror, added a second mirror, then a third, then upgraded the disks in the first, etc. The biggest of those disks was 2TB, I was willing to go with single redundancy for that (I also had backups, including offsite).

For a bigger chunk of storage, of course RAIDZ is more efficient. A 4+2 pool (RAIDZ2 on 6 disks) has double redundancy, it can withstand the failure of any two drives in the array, so that's safer than the same 6 disks arranged in three mirrors. However, the three mirrors setup is more flexible and easier to upgrade -- the three mirrors can be different sizes, and a mirror can be upgraded without ever lowering the redundancy, whereas a RAIDZ cannot. So...tradeoffs.

As the number of disks starts hitting the number of physical or controller slots in your host, the efficiency of RAIDZ becomes more attractive . I'm now in the process of re-working the physical chassis of my original server to also contain a big array for off-site backup for my partner in the video business; the backup array will be 8+3 of 6TB drives. I've managed to make room both physically and on the disk controllers for 14 drives total, which leaves me replacing my 3 two-way mirrors with just three drives; I'm going to do that as a three-way mirror of 6TB drives, because with drives that big single redundancy just doesn't cut it (resilver time is so long, and of course that increases stress on the rest of the array, that something else might go!).

There are modest space gains from making the base size of a RAIDZ array (before adding parity disks) a power of 2; hence I'm going with 2, 4, and 8-disk base arrays. Not actually using any 2-disk arrays; I was going to do a 2+2 for my production array, but with disk sizes so big today, a three-way mirror gets me sufficient redundancy and sufficient size in just three drives, allowing me to add a third parity disk to the 8-disk array, which really needed it. Anyway -- a 6+2 array won't give you quite the space you expected, is the point here.

 

[Robert Trevellyan]

a mirror can be upgraded without ever lowering the redundancy, whereas a RAIDZ cannot

This is not correct. See http://doc.freenas.org/9.3/freenas_storage.html#replacing-drives-to-grow-a-zfs-pool

 

[Bidule0hm]

There are modest space gains from making the base size of a RAIDZ array (before adding parity disks) a power of 2; hence I'm going with 2, 4, and 8-disk base arrays. Not actually using any 2-disk arrays; I was going to do a 2+2 for my production array, but with disk sizes so big today, a three-way mirror gets me sufficient redundancy and sufficient size in just three drives, allowing me to add a third parity disk to the 8-disk array, which really needed it. Anyway -- a 6+2 array won't give you quite the space you expected, is the point here.

This is not correct too. See https://forums.freenas.org/index.php?threads/misaligned-pools-and-lost-space.40288/

 

[David Dyer-Bennet]

Apologies for the misinformation! BiduleOhm's article looks to be well-supported, and it certainly explains the situation fairly clearly. And I hadn't found it yet; glad I did now! It's not too late, I don't have the next "final" (several years, anyway) form of any of the new pools yet, so there's time for me to benefit from this.

I'm going to have to study my distribution of file sizes. In addition to the small files, there's also the fact that each *big* file wastes on average half a block (if the file sizes are distributed independently of the block size, which I believe they are), so increasing the block size increases that waste. But this gives me the tools to evaluate what makes sense.

 

[David Dyer-Bennet]

And again; didn't realize the preemptive "replace" worked that way. Learn something new every day (I've been running ZFS since 2006).

 

[Bidule0hm]

And I hadn't found it yet

I didn't put it in my signature yet because I wanted to wait the add of the real case example (I have the data from my own server, just need the time to make it pretty and understandable).

 

[David Dyer-Bennet]

I didn't put it in my signature yet because I wanted to wait the add of the real case example (I have the data from my own server, just need the time to make it pretty and understandable).

Yeah, writing things up clearly with suitable documentation is work!

 

[Wallybanger]

There are many scenarios where it's at least convenient, often useful, and sometimes even necessary, to put all your storage into one single pool. I can remember having free space split across several drives, and not having enough on any one drive to do what I wanted to; this is one of the problems ZFS was built to solve. If that's not of any value to you, very well, but it is to a lot of people.

Let me give a real-world example, my own. I had six disks in RAIDZ2 configuration in a 12-bay chassis, and my pool was getting full. It was trivial, after properly testing them, to install six more disks and add them to my pool--took about 15 seconds in the web GUI*. Nothing else had to be reconfigured. All the shares were in the same place, no scripts were moved, I didn't have to move anything to the new disks, nothing. Pool running out of space? Add storage to the pool. That's the way ZFS is designed to work. And I'll never run into a situation where I have 100 GB free here, and another 100 GB free there, and I need 150 GB to do whatever I want to do.

* Actually, my situation was a bit more complicated, because my pool was two three-disk RAIDZ1 vdevs--poor planning when I first built it. I had to build a new pool with the new disks (single RAIDZ2 vdev), replicate my data to the new pool, destroy the old pool, and rename the new pool to the old pool's name--at that point, everything was back up and running. I could then add the old disks to the new pool as a second RAIDZ2 vdev. I need more space again, and RAIDZ2 continues to be adequate, so I'll do what I described above.

Well that scenario sounds fine and doesn't really fall into the category of "irresponsible" since you still have 2 VDevs with 2 drive redundancy. I guess the odds of 3 drives failing in either VDev before rebuild are fairly low. lol I don't know what you do but I don't think I will EVER be in a situation where I have 2 drives with 100gb free and I need 150. I've been running on 5tb (mirrored) for years now and I'm only just outgrowing it now. The 24(ish) terabytes I'm considering now are going to be vastly overkill for my needs and I'm actually kind of second guessing spending $2500-$3000 on a NAS. I've been looking at 12TB and 16TB WD Duo mirror drives which run about $96-$104 per TB and the NAS is going to cost about $104/TB. I'm sure the NAS will offer better data security but that's a hefty price tag which demands serious consideration. I'm not a tycoon :p Thanks for your thoughts on the subject :)

Damage to a single vdev is damage to the pool, so for example three mirror vdevs in one zpool is single redundancy -- two drives dieing can kill that pool (it won't if the drives are part of different mirrors, but it will if both drives are in the same mirror).

Now, that's the way I ran my home server since 2006 -- started with one mirror, added a second mirror, then a third, then upgraded the disks in the first, etc. The biggest of those disks was 2TB, I was willing to go with single redundancy for that (I also had backups, including offsite).

For a bigger chunk of storage, of course RAIDZ is more efficient. A 4+2 pool (RAIDZ2 on 6 disks) has double redundancy, it can withstand the failure of any two drives in the array, so that's safer than the same 6 disks arranged in three mirrors. However, the three mirrors setup is more flexible and easier to upgrade -- the three mirrors can be different sizes, and a mirror can be upgraded without ever lowering the redundancy, whereas a RAIDZ cannot. So...tradeoffs.

As the number of disks starts hitting the number of physical or controller slots in your host, the efficiency of RAIDZ becomes more attractive :). I'm now in the process of re-working the physical chassis of my original server to also contain a big array for off-site backup for my partner in the video business; the backup array will be 8+3 of 6TB drives. I've managed to make room both physically and on the disk controllers for 14 drives total, which leaves me replacing my 3 two-way mirrors with just three drives; I'm going to do that as a three-way mirror of 6TB drives, because with drives that big single redundancy just doesn't cut it (resilver time is so long, and of course that increases stress on the rest of the array, that something else might go!).

There are modest space gains from making the base size of a RAIDZ array (before adding parity disks) a power of 2; hence I'm going with 2, 4, and 8-disk base arrays. Not actually using any 2-disk arrays; I was going to do a 2+2 for my production array, but with disk sizes so big today, a three-way mirror gets me sufficient redundancy and sufficient size in just three drives, allowing me to add a third parity disk to the 8-disk array, which really needed it. Anyway -- a 6+2 array won't give you quite the space you expected, is the point here.

According to [URL='https://forums.freenas.org/index.php?members/bidule0hm.31801/']Bidule0hm's ZFS RAID size and reliability calculator[/url], RAIDZ2 arrays of 6, 8 & 10 drives all yeild about 78.7% usable space so it doesn't sound like there is any cost advantage/disadvantage to any of those scenarios.

Biduleohm, is any of the stuff from your "Misaligned Pools and Lost Space" thread incorporated into your drive size calculator? With 8*4tb drives would I be looking at lost space or inefficiency? "The app assumes ashift=12, recordsize=128k and no compression." but there is no change is usable space between 6, 8 or 10 drives.

P.S. the minimum RAM calculation would be massively helpful ;) Thanks for the calculator, it's pretty awesome.

 

[Wallybanger]

Nevermind, looks like none of the percentage features work.

Usable space and price per TB are as follows (assuming $200 per 4tb drive)
6*4tb = 52.5% --> $95.24 per TB
8*4tb = 59.0% --> $84.70 per TB
10*4tb = 63.0% --> $79.40 per TB

So 8 drives vs 6 drives you're saving more than $10 per TB but that drops to $5 per TB savings between 8 and 10 drives. You would need to be loosing a lot of space and performance to make up for the price gap between a 6 and 8 drive RAIDZ2 array.

Biduleohm, can those numbers be improved by messing with the recordsize value? Why do you say "if we ignore VMs and DBs, and we can as we are talking about RAID-Zx vdevs only"? Are VMs and DBs majorly affected by changing the recordsize value? I probably won't be running a DB (at least not one big enough to suffer from performance issues) but I may want to run a VM at some point for security cam software or something.

 

[Robert Trevellyan]

I've been running on 5tb (mirrored) for years now and I'm only just outgrowing it now.

I'm actually kind of second guessing spending $2500-$3000 on a NAS

So don't. Put four 6TB or 8TB WD Reds in a PowerEdge T20 with 32GB of RAM. You'll have more than double your current storage, for about half the price.

 

[danb35]

Well that scenario sounds fine and doesn't really fall into the category of "irresponsible"

I'm glad you think so.

I don't think I will EVER be in a situation where I have 2 drives with 100gb free and I need 150.

If you've never experienced free space fragmentation, good for you. If you really don't see the value in combining all of your storage into a single pool, I don't know what to tell you. The ZFS designers saw quite a lot of value in that, enough so that that they created a filesystem to enable it.

So don't. Put four 6TB or 8TB WD Reds in a PowerEdge T20 with 32GB of RAM. You'll have more than double your current storage, for about half the price.

...and to actually do the math on that:

• Dell T20: $180
• 32 GB of ECC RAM: 4 x $65 = $260
• 6 x WD Red 6 TB drives: 4 x $239 = $956
• Total: $1,396

You could probably even reduce the RAM to 16 GB and save $130. That will give you 12 TB, or 10.8 TiB, of accessible space, about 9 TiB usable if you follow the 80% rule. The numbers are the same whether you put the drives in RAIDZ2 or striped mirrors.