Switching from RAIDZ1 to Mirrored vDevs - Sanity Check

[Zaimor Kai]

Hi! New to the forums but been using FreeNAS virtualized via ESXi for about a year. I'm looking for someone with more knowledge than I to sanity check what I'm about to do. Current build:

Code:

- Gigabyte GA-990FXA-UD3 R5 board
- AMD FX-8370
- 2x Crucial 8GB ECC DDR3-1600 RAM (16GB)
- Samsung 850 EVO 120GB drive hosting ESXi 6.0 and a few VMs
- 4x Seagate Desktop (I know, I messed up there) 4TB HDs passed through to the FreeNAS VM, one vDev, one zPool, RaidZ1 (I have about 10TB of data on that pool)

What I'd like to do is purchase 4 new 4TB drives (WD Red NAS). I'm planning on setting up two of the drives in one mirrored vDev in a new zPool, then copying ~4TB of data to the new pool. I'd then add two more drives in a new mirrored vDev, add it to the pool, and fill it up with another ~4TB chunk of data.

Next, I'd make a couple external backups of the remaining existing data, destroy the old pool, create 2 more 4TBx2 mirrored vDevs, add them to the new pool, and copy the remaining data.

I'm hoping this will give me a few more TBs of space while converting to a mirrored vDev setup, a little at a time, allowing me better redundancy and the ability to expand/add drives later.

Does this all sound like a good idea, or am I missing anything important?

Thanks in advance. Also, a big part of choosing to go this route was @cyberjock 's presentation, which was extremely helpful in adding to my FreeNAS knowledge!

 

[Stux]

Why do you not create a 2x2 pool to start?

 

[Zaimor Kai]

Why do you not create a 2x2 pool to start?

Giving myself practice on adding a vDev to an existing zPool and seeing how it all works :)

 

[Stux]

Okay.

There is another option if you want to go to 8-way Raidz2 instead for about 24TB of storage, instead of 16.

With no more extra disks and at least one disk worth of redundancy the entire time.

 

[Arwen]

Giving myself practice on adding a vDev to an existing zPool and seeing how it all works :)

It's not ideal for data on pools. To sum it up, if you start with 2 x vDevs, all data written will stripe across the 2 x vDevs.

But, if you start with one mirrored vDev, all the original data will be written only to this vDev. When you add a second
vDev, you end up with ZFS deciding to choose the new, (and mostly empty), vDev for all new writes. If the files are smallish,
then it's not much of an issue. However, for larger files, having them striped across 2 vDevs, (or potentially 3 or 4), gets a
bit better read performance. Especially for VMs.

That said, if the files are dynamic / changing, ZFS will eventually copy on write newly written data across all vDevs.

If you don't have the slots to add all 4 new disks in, the neat thing about mirrors, is that you can create a striped pool with
half of the mirrors. Copy the data over, then remove the old disks, (your RAID-Z1 pool), and add the new disks in as a set
of mirrors.

 

[danb35]

and add the new disks in as a set of mirrors.

...which still can't be done from the GUI.

 

[Arwen]

...which still can't be done from the GUI.

Yes, I was afraid of that.

 

[Ericloewe]

...which still can't be done from the GUI.

I filed a feature request, which is currently targeted for 11.1. Is this the one? I guess we'll find out in August or so.

 

[Zaimor Kai]

It's not ideal for data on pools. To sum it up, if you start with 2 x vDevs, all data written will stripe across the 2 x vDevs.

But, if you start with one mirrored vDev, all the original data will be written only to this vDev. When you add a second vDev, you end up with ZFS deciding to choose the new, (and mostly empty), vDev for all new writes. If the files are smallish, then it's not much of an issue. However, for larger files, having them striped across 2 vDevs, (or potentially 3 or 4), gets a bit better read performance. Especially for VMs.

I thought of that but didn't know how it would affect performance. 90% of the files are media files 200MB-2GB in size. The array is mostly media storage.

If you don't have the slots to add all 4 new disks in, the neat thing about mirrors, is that you can create a striped pool with half of the mirrors. Copy the data over, then remove the old disks, (your RAID-Z1 pool), and add the new disks in as a set of mirrors.

I have lots of slots, that's not an issue. Is your suggestion to create a striped vDev consisting of the 4 new drives, copy the data, then add a vDev to the pool and make that a mirror? If it is, that would limit my ability to replace/expand the drive in a modular fashion, wouldn't it?

 

[danb35]

Is your suggestion to create a striped vDev consisting of the 4 new drives, copy the data, then add a vDev to the pool and make that a mirror?

Not exactly. When you create a four-disk striped pool, it consists of four single-disk vdevs striped together (this is something I misunderstood myself until recently). You can add mirrors to any or all of these at any time. If you add mirrors to each of the disks, you'd end up with four striped sets of two-way mirror vdevs.

 

[Zaimor Kai]

Not exactly. When you create a four-disk striped pool, it consists of four single-disk vdevs striped together (this is something I misunderstood myself until recently). You can add mirrors to any or all of these at any time. If you add mirrors to each of the disks, you'd end up with four striped sets of two-way mirror vdevs.

So four single-disk vDevs striped in a new pool, copy the data, then destroy the old pool, add a disk to each one, turning it into four two-disk mirror vDevs as in the original plan? And based on above comments, I'd have to do this via the CLI?

 

[danb35]

Pretty much. There are threads on here on how to properly add a disk to create a mirror, and have it be consistent with the "FreeNAS way", but it boils down to (1) create the swap partition at the beginning of the disk, and (2) add the device to the pool by gptid, not by adan designation.

 

[Zaimor Kai]

That sounds like the best plan, thanks for the ideas!

What's the best (read: safest) way to copy the data between the pools while ensuring integrity?

 

[Zaimor Kai]

Okay.

There is another option if you want to go to 8-way Raidz2 instead for about 24TB of storage, instead of 16.

With no more extra disks and at least one disk worth of redundancy the entire time.

That was my original plan, but I read some articles on how much faster silvering is on mirrored vs raid, and considered how much easier it is to grow the pool using smaller vDevs. This way I can add a few terabytes at a time when I need, don't have to worry about rebuilding the pool, and have more redundancy since my dummy self started with desktop drives lol.

Also, using @danb35 's suggestion, I end up with one new and one old drive per vDev, which will be perfect if my desktop drive error starts catching up with me.

 

[danb35]

• zfs snapshot -r oldpool@migrate
• zfs send -R oldpool@migrate | zfs recv -F newpool

You'd want to check the case on those flags; I'm going from memory here.

 

[Arwen]

...
Also, using @danb35 's suggestion, I end up with one new and one old drive per vDev, which will be perfect if my desktop drive error starts catching up with me.

<indignation>Heh, that was MY suggestion!</indignation>

But yes, one option is to use your older, desktop drives as the mirror for your vDevs.

Please note, my indignation was intended to be humorous :) I don't mind others be credited for my careful and well thought out work.

 

[Zaimor Kai]

<indignation>Heh, that was MY suggestion!</indignation>

But yes, one option is to use your older, desktop drives as the mirror for your vDevs.

Please note, my indignation was intended to be humorous :) I don't mind others be credited for my careful and well thought out work.

Haha, sorry, I misunderstood the end state of your suggestion. Thank you all for the help! :)

 

[Stux]

...which still can't be done from the GUI.

I think the GUI supports "extending" a "volume" with a mirror. That's the only functionality you need

Edit: thought Arwen suggested striped mirrors. Not mirroring the stripes ;)