RaidZ2 Question

[silvesj]

Hello All,

I've read over documentation and other tutorials. I'm familiar with FreeNAS and comfortable and I've used RAID6 in other enterprise systems such as EMC storage, Dell Equallogic, etc, but those are bit more streamlined.

The question I have is, when I go to configure the disks via volume manager it seems weird. I have 8 3TB drives, and when I add them and select RaidZ2 it appears to be mirroring each disk 1-to-1. And I'm left with about 10TB of Space. See attached img file.

If I drag the sliders up and then down so all 8 disks are in the 1st row, and select RaidZ2, am I actually configure RaidZ2?

2nd question. If I need to accomplish RaidZ2, should I be configuring 6 disks in a RaidZ2, and then 2 disks in a separate pool called spares correct?

Ultimately, waiting on them to come in, but I will have a total of 12 disks when the rest arrive and would like to accomplish RaidZ2. Please let me know if I'm missing anything.

 

[nojohnny101]

What your attached screenshot is showing is correct. That is how FreeNAS represents raudz2 graphically.

I think most on here would say that 12 discs is getting a bit high for raidz2. why not either go raidz3 or just do 2 raidz2? Still gives you 2 disk fails in each raidz2 but you can keep a bit more space.

I'm sure others will chime in. Everyone approaches it different and it is all about risk management and what you're comfortable with/how important your data is to you.

 

[gpsguy]

Yes.

If I drag the sliders up and then down so all 8 disks are in the 1st row, and select RaidZ2, am I actually configure RaidZ2?

How were you planning to use the 12 disks? We don't recommend a single 12 disk RAIDz2 vdev. 10 disks would probably be the widest. Often users will go with 2 x 6 disks RAIDz2 vdev's, striped together. Since you can't add additional disks into a RAIDzX vdev once it's been created, I'd hold off, until the other disks arrive.

Ultimately, waiting on them to come in, but I will have a total of 12 disks when the rest arrive and would like to accomplish RaidZ2. Please let me know if I'm missing anything.

 

[gpsguy]

I believe the OP's screenshot represents 2 x 4 disk RAIDz2 vdev's, striped together.

What your attached screenshot is showing is correct. That is how FreeNAS represents raudz2 graphically.

 

[Spearfoot]

I believe the OP's screenshot represents 2 x 4 disk RAIDz2 vdev's, striped together.

I agree. I fired up my VirtualBox FreeNAS test VM and created a volume matching his layout and ended up with 2 RAIDZ2 vdevs in the pool, which makes sense in light of the 4x2x3.0TB description the FreeNAS Volume Manager dialog displays.

When I re-created the pool with the drives selected as shown below I ended up with a single 8-drive RAIDZ2 array (8x1x8.6GB)

I agree with @gpsguy that 12 drives is probably too 'wide', but that really depends on how @silvesj plans to use the server. It would be okay if pokey, slow storage is the goal. Otherwise, either six mirrors or 2 mirrored RAIDZ2 vdevs would both provide more IOPS.

So the original pool layout posted by @silvesj may end up being a better design, only with 6 drives in each vdev instead of the 4 he has now.

 

[gpsguy]

You actually ended up with a RAIDz3 array. ;-)

I also did the same experiment earlier. When I selected the 8 disks, volume manager guessed that I wanted RAIDz3. I changed it to RAIDz2 and created the volume. Once it was done, I confirmed it with "zpool status".

When I re-created the pool with the drives selected as shown below I ended up with a single 8-drive RAIDZ2 array (8x1x8.6GB)

 

[Spearfoot]

You actually ended up with a RAIDz3 array. ;-)

I also did the same experiment earlier. When I selected the 8 disks, volume manager guessed that I wanted RAIDz3. I changed it to RAIDz2 and created the volume. Once it was done, I confirmed it with "zpool status".

Heh, you're right!
Now, where are my glasses...

 

[silvesj]

Right, I know you can't expand the pool once it's created. I was purely experimenting while waiting for the disks since I've never setup RaidZ2 in FreeNAS.

So everyone is saying it's too wide for Z2, and what are the downsides of that? Just slowness? Again based on my familiarity with other products, understanding that FreeNAS is different with the ZFS, other NAS/SAN offer high input/output using even more drives (16, 24, etc). What creates this limitation in FreeNAS?

And in the end the server will really be used for hosting files, system images, and media.

 

[Bidule0hm]

Right, I know you can't expand the pool once it's created.

You can expand a pool (by adding one or more vdev(s) to it), but not a vdev.

So everyone is saying it's too wide for Z2, and what are the downsides of that? Just slowness?

Yep, and a higher risk to lose the pool, but it's still safer than a 3 drives RAID-Z1 so... Well, I'd recommend a RAID-Z3 here ;)

What creates this limitation in FreeNAS?

It's not a FreeNAS limitation but a ZFS limitation. ZFS can get very quickly very complicated so if you want more info I recommend to search directly with ZFS in your keywords.

 

[Mirfster]

So everyone is saying it's too wide for Z2, and what are the downsides of that? Just slowness?

In a nutshell "Yes".

Per "ZFS Best Practices Guide":

RAIDZ Configuration Requirements and Recommendations
A RAIDZ configuration with N disks of size X with P parity disks can hold approximately (N-P)*X bytes and can withstand P device(s) failing before data integrity is compromised.

• Start a single-parity RAIDZ (raidz) configuration at 3 disks (2+1)
• Start a double-parity RAIDZ (raidz2) configuration at 6 disks (4+2)
• Start a triple-parity RAIDZ (raidz3) configuration at 9 disks (6+3)
• (N+P) with P = 1 (raidz), 2 (raidz2), or 3 (raidz3) and N equals 2, 4, or 6
• The recommended number of disks per group is between 3 and 9. If you have more disks, use multiple groups.

Per "A Closer Look at ZFS, Vdevs and Performance":

When writing to RAID-Z vdevs, each filesystem block is split up into its own stripe across (potentially) all devices of the RAID-Z vdev. This means that each write I/O will have to wait until all disks in the RAID-Z vdev are finished writing. Therefore, from the point of view of a single application waiting for its IO to complete, you'll get the IOPS write performance of the slowest disk in the RAID-Z vdev.

So performance wise, it would be better to use 2 x 6 Drive RaidZ2 vDevs (more IOPS).

If you are "raring to go" and have already properly burned in your Hardware, RAM and Drives (you did do this right?); then I would say go ahead and make your pool with one 6 Drive RaidZ2 vDev. When you have the other six (6) drives in then make a second 6 Drive RaidZ2 vDev and add it to the existing pool.

 

[Spearfoot]

@silvesj : How much memory do you have in your system? If you have 64GB or greater, you should get a performance boost by adding an L2ARC device (an SSD) of roughly 256GB.

RAIDZ2 vdevs should be fine for your use-case, as pointed out by @Mirfster

 

[jgreco]

@silvesj : How much memory do you have in your system? If you have 64GB or greater, you should get a performance boost by adding an L2ARC device (an SSD) of roughly 256GB.[/USER]

Potentially, but don't race out and add one unless your pool is fairly busy and the stats suggest you're short on ARC. Adding an L2ARC without needing one actually robs the system of some resources.

 

[silvesj]

So I know we talked about how many disks in a Raidz2, but I guess I should clarify it would really only be 10 disks, with 2 of those being setup as spares. Does that change the opinion of this setup?

 

[depasseg]

So I know we talked about how many disks in a Raidz2, but I guess I should clarify it would really only be 10 disks, with 2 of those being setup as spares. Does that change the opinion of this setup?

Spares are not part of the RAIDZ, is that really what you mean? Spares really only make sense if you have multiple vdevs and don't want to tie a disk to a specific one.

And in the GUI, each row is a vdev (and the vdev is configured as what you select in the drop down).

A 10 disk RAID Z2 has 8 disks with of data and 2 disks worth of parity. Rather than adding 2 disks as spares, consider using an 10 or 11 disk RAIDZ3 and a spare.

And to clarify, the risk of pool loss is in the case of a drive failure, during the resilvering process all the other drives are pushed hard for a very long long time, so the risk of a second disk (and third) failing is greater. So as you make the vdev wider, the likelihood of additional disk failures during resilvering is greater.

 

[jgreco]

Spares also make sense if you're already out at RAIDZ3 and you want more resiliency. In particular, because 12/24/36 are popular tiers for number-of-drive-bays, an 11 disk RAIDZ3 plus a spare is a reasonable way to get massive space with great redundancy and resiliency characteristics.

 

[Mirfster]

True, but if you wanted to upgrade that single vDev you are looking at replacing all 11 drives before it will auto-expand. Each layout has its pros and cons; which makes it all the more vital to take the time to consider ones design. All part of the fun... ;)

 

[silvesj]

Well what I thought would be an easier convo now appears to have taken a turn haha. If I were to go with RaidZ3 then would 12 disks be within recommendation? And I can't imagine even at RaidZ3 that I will be needing to upgrade anytime soon.

 

[jgreco]

Well, RAIDZ3 at 11 disks wide gives you 8 "data disks" worth, which is on the high end of things that are expected to work fine. The archival pool here is built on 11 disk RAIDZ3, and it isn't exactly fast, but it's absolutely suitable for storing long term data, ISO images, archived VM's, and anything else that is basically large data files that are simply being stored.

 

[silvesj]

The thing that sucks about the 2 RaidZ2 is I lose 3TB of storage space. :\

When I originally planned, I had planned on doing 12 disk z2, but I shall follow the recommendations and do two z2s I suppose.

 

[silvesj]

So I had to re-read all the posts to make sure I had all my info clear. And sorry as I missed it before, yes I did do all the burn in testing. I did have 1 drive fail during the process and immediately exchanged it, and re-ran it.

The original plan for me was to do a 12 Disk RaidZ2 or z3. While I don't know that I will plan on doing anything that requires fast IOPS but MAY down the line place a vmware datastore on here. So I think in light of all the info I'll sacrafic a bit of storage and do 2 RaidZ2s.