Create zfs mirror by adding a drive?

cyberjock

Inactive Account
Joined
Mar 25, 2012
Messages
19,526
Yes, and I also posted more than 2 years ago that these steps aren't correct for the latest version of FreeNAS, which is now an ancient 2014 version. The reality is that nobody should be using this guide anymore. :P
 

Philip Robar

Contributor
Joined
Jun 10, 2014
Messages
116
Yes, and I also posted more than 2 years ago that these steps aren't correct for the latest version of FreeNAS, which is now an ancient 2014 version. The reality is that nobody should be using this guide anymore. :p

It's not clear what you mean by "latest version of FreeNAS, which is now an ancient 2014 version".

Given that the GUI in the release version of FreeNAS, 9.10.2, doesn't have a way to turn a single drive VDEV into a mirror how do you suggest that one go about it?
 

Stux

MVP
Joined
Jun 2, 2016
Messages
4,367
Thanks, this helped me. I followed the steps given and now I can see my newly added drive is presently reslivering. Perfect. For completeness, I'm quoting the steps, with 1.5 inserted (as #2 below), as I could have ended up missing 1.5, and others may end up missing it.. Thanks a lot Dusan!

----------------------------------------------------------------------
Dusan & James' rad guide to
ZFS: Adding a drive to create a mirror
of another in an existing ZFS pool
(circa FreeNAS 9.1.1)

Here we're adding a drive labelled /dev/ada1 - Your exact drive name obviously matters vastly. Make sure you know the correct name for the drive you mean to add to your pool!




Upon completion of these steps, running zpool status should return a screen similar to mine, note the "resilver" comments in the output:



Note in this case, I have a drive for a zfs slog - I expect most won't have the logs section I do.. ^_^

I've been looking for this guide for hours... finally found it... and I believe its still correct (even though @cyberjock says its not). Can anyone (@Ericloewe) resource-ify it?
 

MrToddsFriends

Documentation Browser
Joined
Jan 12, 2015
Messages
1,338
I've been looking for this guide for hours... finally found it... and I believe its still correct (even though @cyberjock says its not). Can anyone (@Ericloewe) resource-ify it?

I successfully used Dusan & James' rad guide to create a mirror last year. AFAICT the underlying FreeNAS version was either FreeNAS-9.10-STABLE-201605021851 (35c85f7) or FreeNAS-9.10-STABLE-201605240427 (64fcd8e), in any case a much more recent version than anything cyberjock's comment from Aug 9, 2014 could refer to.

There seems to be a little glitch in that guide: In step 5 replace ada1p1 by ada1p2.
 

James Snell

Explorer
Joined
Jul 25, 2013
Messages
50
I was summoned back to this thread via email notifications. I was amused to re-read this thread and have zero recollection of this. I tend to forget a lot of stuff, which is why I like to write things down. I was about to wander away when I saw the skeptical comments about it no longer working. That invoked emotions (I get like that when I feel some advice is mistaken and unreasonably authoritive about it). But, I didn't know if it would still work. So I tried it.

I downloaded the latest version of FreeNAS as of this evening (FreeNAS 11). I created a VM and installed it. 1 OS drive, 2 data drives. When the OS was up, I used the FreeNAS GUI to create a single drive ZFS volume. Then I opened my steps documented in this thread and re-traced them. It worked completely fine. But since some folks need more convincing than others, I did a screencast of the whole damn thing, here:
Then I went on to throw down a post on my personal blog about this, with some tiny updates to the steps, here: https://dawning.ca/2017/freenas-11-add-a-drive-to-create-a-mirrored-zfs-volume/

I doubt these steps will go completely out-of-style any time soon, since I think doing so would mean deep changes to ZFS, which seems unnecessary. Anyway, it still works. Have fun. (and proceed at your own risk)
 

Ericloewe

Server Wrangler
Moderator
Joined
Feb 15, 2014
Messages
20,176
Can anyone (@Ericloewe) resource-ify it?
I never followed up on this one, sorry...

I don't have much time to spare these days, and I'd want to verify it before submitting as a resource.

I do welcome any community members who would be so kind as to do the following:
  1. Test the process in a VM
  2. Verify that the resulting disk is indistinguishable from one partitioned by FreeNAS
  3. Write this up as a resource
Then I went on to throw down a post on my personal blog about this, with some tiny updates to the steps, here: https://dawning.ca/2017/freenas-11-add-a-drive-to-create-a-mirrored-zfs-volume/
Would you like to submit it as a resource? Even if you prefer to link to your blog, a resource would provide easier discoverability. It's easy for you, since the hard part of documenting it has been done. ;)
 

Nigel Dunning

Dabbler
Joined
May 1, 2015
Messages
12
edit: FYI posted issue about error trying to make a stripe into mirror, tried with another disk and it worked now.
 
Last edited:

Nigel Dunning

Dabbler
Joined
May 1, 2015
Messages
12
You tried with another destination disk and it worked? That's interesting...

I'm not overly surprised, this system and the disks are all about 5 years old, and it's had a tough life, was previously used as primary storage for doing fluid/particle simulations on (thousands of machines in a farm all simultaneously writing data 24/7, so lots of stress on the disks), and when it was not able to keep up with that it was just turned into a parking space for data before being archived.

So we've seen a few strange problems on this server crop up on this server, disks dying without warning etc.
 

ezra

Contributor
Joined
Jan 15, 2015
Messages
124
Confirmed to work in 11.1. Shows up in the GUI as mirror now, resilvered right away. Thank you!
 

James Snell

Explorer
Joined
Jul 25, 2013
Messages
50
Confirmed to work in 11.1. Shows up in the GUI as mirror now, resilvered right away. Thank you!

Good stuff. Yeah, I actually was just dealing with a mirror drive replacement last night. I no longer had the failed drive and when I tried to use the FreeNAS 11.0-U4 GUI to 'replace' the failed/removed disk, with another of the precise same make and model, the gui returned an extensive middleware fail report which included a comment about the disk being too small. I then managed to get the whole disk added to the ZFS volume as a mirror member from reading some generic FreeNAS ZFS guide, but it entered the pool as "ada4" rather than by a UUID. I found I could detach the drive from the pool while resilvering was happening and it seemed fine. Then I detached the failed drive from the pool (it was physically removed, but still a member in the pool). This actually changed the mirror in to a plain unmirrored pool. Then I actually just followed my own steps here and successfully added the new/replacement drive to the pool. Resilvering is nearly done now. Yay.

Would you like to submit it as a resource?
Yes, I should make this a resource, it would have actually helped me a little faster with my own latest FreeNAS episode
 

Ericloewe

Server Wrangler
Moderator
Joined
Feb 15, 2014
Messages
20,176
Yes, I should make this a resource, it would have actually helped me a little faster with my own latest FreeNAS episode
Okay, great! Any questions, just PM me.
 

tortue

Dabbler
Joined
Jan 13, 2018
Messages
11

pjc

Contributor
Joined
Aug 26, 2014
Messages
187
Bump. I'm setting up some new drives and am wondering whether I should use the '-a 4k' flag. Seems like it couldn't hurt?
 

PhilipS

Contributor
Joined
May 10, 2016
Messages
179
Bump. I'm setting up some new drives and am wondering whether I should use the '-a 4k' flag. Seems like it couldn't hurt?

You probably are done with this now, but yes, -a 4k is the way the middleware does it now. I sleuthed through the python code to figure this out.

Here are the commands I ran to convert a single disk to mirror without detaching from the GUI in 11.1u4 which totally voids the warranty:

Ran gpart list to get the device name of the new disk {newdevname} (ada0 in my case)
Code:
gpart create -s gpt /dev/{newdevname}
gpart add -a 4k -b 128 -t freebsd-swap -s 4194304 {newdevname}
gpart add -a 4k -t freebsd-zfs {newdevname}
gpart bootcode -b /boot/pmbr-datadisk /dev/{newdevname}


Ran zpool status to get gptid of existing disk in the pool {gptid_existing}
Ran glabel status to get gptid of the new zfs partition created with gpart {gptid_new}. The order for the following command is important.
zpool attach {poolname} {gptid_existing} {gptid_new}

The pool now begins to resilver. I checked the partitions of both drives and they are identical.

Instead of rebooting to get mirrored swap, I ran the following commands:

Ran swapinfo to get the existing swap device name (ada2p1 in my case)
Code:
swapoff /dev/{swapname}.eli
geli detach {swapname}.eli
gmirror create -b prefer swap0 {swapname} {newdevname}p1
geli onetime mirror/swap0
swapon /dev/mirror/swap0.eli
 

mgd

Dabbler
Joined
Jan 8, 2017
Messages
46
Then I went on to throw down a post on my personal blog about this, with some tiny updates to the steps, here: https://dawning.ca/2017/freenas-11-add-a-drive-to-create-a-mirrored-zfs-volume/

Great concise post!

My challenge is that the existing drive in the stripe is GELI encrypted (the volume was created in FreeNAS from one drive). If using your guide as is I guess I will get a mirror where one drive is encrypted and the added one is unencrypted which is not what one would want.

I would like the encryption to be set up by (and known to) FreeNAS, so I thought of performing these steps to get the drive partitioned (2GB swap + rest for data) and encrypted the way FreeNAS normally does:
  • Add the new drive to server.
  • From the FreeNAS GUI: create a new encrypted volume from this single drive.
  • From the FreeNAS GUI: detach the new volume (without wiping it)
Now, we have a correctly partitioned and encrypted disk, so we can proceed with your steps:
  • zpool status – to find the gptid of the existing encrypted device in the stripe to be converted to a mirror
  • geli list – to find the gtpid of the new drive's encrypted device
  • zpool attach MyPoolName /dev/gptid/[gptid_of_the_existing_geli_device] /dev/gptid/[gptid_of_the_new_geli_device]
Finally, follow the steps from the guide about replacing an encrypted drive:
  • Rekey the volume, add a passphrase, download the key and add a recovery key.
Any thoughts about whether this will work?
(I will test in a VM before doing anything on my real FreeNAS system.)
 

mgd

Dabbler
Joined
Jan 8, 2017
Messages
46
Ok, I tested the procedure out in a VM running FreeNAS.

Assuming we have an existing one-drive stripe volume called tank and an extra drive of the same size, the steps are outlined here.
GUI = The web interface.
CLI = Command-line interface / shell

1. GUI: Create a new encrypted volume fake
2. CLI: zpool status tank fake
This lists the GEOM IDs of the two encrypted GELI devices used by the two volumes fake and tank

3. CLI: zpool export fake
Detaches the volume fake so we can attach the encrypted disk to volume tank

4. CLI: zpool attach -f tank <GEOM-ID-tank-dev> <GEOM-ID-fake-dev>
This converts the volume tank from a one-drive stripe to a two-drive mirror

5. GUI: Detach volume fake
The volume still existed in the GUI but could not find its disk so we had to detach it manually

6. GUI: Reboot
After the reboot the encrypted device for the second disk is gone so the volume tank is degraded.

7. GUI: In “Volume Status” for volume tank we replace the missing disk with the second disk
FreeNAS will now create a new encrypted device from the second disk, resilver the mirror and all is good.

If you had a passphrase and a recovery key for volume tank please remember to set the passphrase and add a recovery key as this was invalidated by FreeNAS when replacing the drive in step 7.


To explain the procedure a bit, steps 1. to 4. simply converts the volume tank from stripe to mirror.

Was it not for FreeNAS we would have been done by now. However, FreeNAS still knows the volume fake in its configuration even though it no longer exists and reports an error for the volume fake. To get rid of that error we must detach the volume fake in the GUI. Unfortunately, this has the side effect of also removing the encrypted device for the second disk. This is why, after a reboot, the volume tank is degraded, missing one of its encrypted disks. However, it is still a mirror and 6. and 7. brings us back into a state with two healthy devices in our mirror.

---

For those that want all the gory details, they are here:

Initially, we have a volume, tank, which is a stripe set with one encrypted disk. In the CLI we can see:
Code:
root@freenas:~ # zpool status tank
  pool: tank
 state: ONLINE
  scan: none requested
config:

  NAME											  STATE	 READ WRITE CKSUM
  tank											  ONLINE	   0	 0	 0
	gptid/66136357-4b22-11e8-a57c-001c424e6846.eli  ONLINE	   0	 0	 0
errors: No known data errors

From the GUI we create a second volume, fake, which ensures that FreeNAS partitions the disk and setup the encrypted GELI device. In the CLI we now have:
Code:
root@freenas:~ # zpool status fake
  pool: fake
 state: ONLINE
  scan: none requested
config:

  NAME											  STATE	 READ WRITE CKSUM
  fake											  ONLINE	   0	 0	 0
	gptid/15b03546-4b24-11e8-a57c-001c424e6846.eli  ONLINE	   0	 0	 0

errors: No known data errors

From the CLI we detach (export) the volume (pool) fake. (If we do it from the GUI, the GELI device will disappear.)
Code:
root@freenas:~ # zpool export fake
root@freenas:~ # zpool status fake
cannot open 'fake': no such pool

Now, using the two GEOM ids we attach the second encrypted disk to the original volume (pool) tank thus transforming it to a mirror.
Code:
root@freenas:~ # zpool attach -f tank gptid/66136357-4b22-11e8-a57c-001c424e6846.eli gptid/15b03546-4b24-11e8-a57c-001c424e6846.eli

Option '-f' is necessary to override the warning we would get without '-f':
Code:
/dev/gptid/15b03546-4b24-11e8-a57c-001c424e6846.eli is part of exported pool 'fake'

We now have a mirrored volume:
Code:
root@freenas:~ # zpool status tank
  pool: tank
 state: ONLINE
  scan: resilvered 1.27M in 0 days 00:00:00 with 0 errors on Sat Apr 28 13:39:57 2018
config:

  NAME												STATE	 READ WRITE CKSUM
  tank												ONLINE	   0	 0	 0
	mirror-0										  ONLINE	   0	 0	 0
	  gptid/66136357-4b22-11e8-a57c-001c424e6846.eli  ONLINE	   0	 0	 0
	  gptid/15b03546-4b24-11e8-a57c-001c424e6846.eli  ONLINE	   0	 0	 0

errors: No known data errors

If we go to the GUI we will see, that volume tank is HEALTHY and looking at the volume status we also see that it is now a mirror.

However, the fake volume still exists there but has status LOCKED. We simply detach it in the GUI.

Now, for a reboot.

After the reboot, in the GUI we will find the volume tank in a DEGRADED state, the new disk being UNAVAIL. From the CLI it looks like this:
Code:
# zpool status tank
  pool: tank
 state: DEGRADED
status: One or more devices could not be opened.  Sufficient replicas exist for
the pool to continue functioning in a degraded state.
action: Attach the missing device and online it using 'zpool online'.
   see: http://illumos.org/msg/ZFS-8000-2Q
  scan: resilvered 1.13M in 0 days 00:00:00 with 0 errors on Sat Apr 28 14:47:44 2018
config:

  NAME												STATE	 READ WRITE CKSUM
  tank												DEGRADED	 0	 0	 0
	mirror-0										  DEGRADED	 0	 0	 0
	  gptid/2e5e391d-4b2d-11e8-b078-001c424e6846.eli  ONLINE	   0	 0	 0
	  3280147069974124757							 UNAVAIL	  0	 0	 0  was /dev/gptid/3a75cb49-4b2d-11e8-b078-001c424e6846.eli

errors: No known data errors

The problem is that the GELI device from the second disk no longer exists.

From the GUI we select “Volume Status” for volume tank, click the UNAVAIL device and click “Replace” to replace it with our second disk.

Now, the volume is HEALTHY and remains so after a reboot.

Again, remember to create a new recovery key (if you had one) as the existing has been invalidated by replacing the drive.
 
Last edited:

Stux

MVP
Joined
Jun 2, 2016
Messages
4,367
And if FreeNAS supported adding a drive to stripe/mirror vdev in the GUI none of this would be needed.

It’s really a fairly trivial thing. Surprising it’s not there.

Especially since the functionality is there for the boot pool!
 

MrToddsFriends

Documentation Browser
Joined
Jan 12, 2015
Messages
1,338
Top