Correct way to replace/update storage on an encrypted pool/vdev?

[SebbaG]

Hi folks,

I'm about to upgrade my 6x 4TB WD-Green raidZ1 vdev setup and enhance storage capabilities. Therefore I want to replace them with 6x 8TB Seagate IronWolf NAS drives, which arrived today. My Z1-pool is encrypted though and I'm still running the older freenas 9.3 version (FreeNAS-9.3-STABLE-201412090314). So far I've read the user-guide of FreeNas 9.3 in chapter "8.1.11. Replacing Drives to Grow a ZFS Pool" which suggest the following steps to grow an existing vdev:

The safest way to perform this is to use a spare drive port or an eSATA port and a hard drive dock. In this case, you can perform the following steps:

1. Shut down the system.
2. Install one new disk.
3. Start up the system.
4. Go to Storage ‣ Volumes, select the pool to expand and click the “Volume Status” button. Select a disk and click the “Replace” button. Choose the new disk as the replacement.
5. You can view the status of the resilver process by running zpool status. When the new disk has resilvered, the old one will be automatically offlined. You can then shut down the system and physically remove the replaced disk. One advantage of this approach is that there is no loss of redundancy during the resilver.

Now I wonder if this is also applicable to encrypted drives, since a few steps above the guide suggest the following for replacing encrypted drives:

8.1.10.1. Replacing an Encrypted Drive
If the ZFS pool is encrypted, additional steps are needed when replacing a failed drive.
First, make sure that a passphrase has been set using the instructions in Encryption before attempting to replace the failed drive. Then, follow the steps 1 and 2 as described above. During step 3, you will be prompted to input and confirm the passphrase for the pool. Enter this information then click the “Replace Disk” button. Wait until the resilvering is complete.
Next, restore the encryption keys to the pool. If the following additional steps are not performed before the next reboot, you may lose access to the pool permanently.

1. Highlight the pool that contains the disk you just replaced and click the “Encryption Re-key” button in the GUI. You will need to enter the root password.
2. Highlight the pool that contains the disk you just replaced and click the “Create Passphrase” button and enter the new passphrase. You can reuse the old passphrase if desired.
3. Highlight the pool that contains the disk you just replaced and click the “Download Key” button in order to save the new encryption key. Since the old key will no longer function, any old keys can be safely discarded.
4. Highlight the pool that contains the disk you just replaced and click the “Add Recovery Key” button in order to save the new recovery key. The old recovery key will no longer function, so it can be safely discarded.

I do have an additional spare drive port available for the approach documented in 8.1.11 of the user guide. Before I mess things up, I wanted to clarify in the beginning how the proper way of replacing should be done. Has anyone ever done this before? Sorry if this question has already been answered, but I could not find anything to this specific topic with my search skills.

Thx in advance...
Sebastian

 

[SebbaG]

*push* has nobody ever done this?

 

[nojohnny101]

What are you unclear about? The step are outlined right the manual. Could you be more specific? Are you looking for confirmation from someone who has gone through it that what is outlined in the manual is indeed the steps?

Are you still considering running a 6 x 8TB wide vdev on raidz1? This sounds like a disaster waiting to happen and wanted to know if you have carefully considered and understand the risks or if you have a particular reason for choosing raidz1.

 

[depasseg]

I've avoided encryption, so I can't help there. You have a backup right? Because you are going to end up resilvering your pool 6 times to replace all your drives which puts a lot of stress on the drives. This an even more important consideration as you are moving to 8TB drives.

 

[SebbaG]

hi again,

first of all thx for the answers. I was actually wondering if "8.1.11. Replacing Drives to Grow a ZFS Pool" also applies, if the pool is encrypted. By that I mean if I can connect my 8TB drive to an additional sata slot and just simply follow the "Replacing Drives to Grow a ZFS Pool" guide. Or if I need to follow the "Replacing an encrypted drives" guide.

Anyway your comment let me to reconsider my decision! :) Maybe after two years of happily running an RaidZ1 without any issues, I should not further challenge the universe and its destiny for another 2-3 years :)
So maybe I'm going to follow depasseg's how to migrate data guide https://forums.FreeNAS.org/index.ph...te-data-from-one-pool-to-a-bigger-pool.40519/
an create a new encrypted bigger RaidZ2 pool to be on a more safe side.

But then I'll have to buy another 8TB drive, because I will need that amount of space for the upcoming period (7x8TB RaidZ2)...

 

[nojohnny101]

first of all thx for the answers. I was actually wondering if "8.1.11. Replacing Drives to Grow a ZFS Pool" also applies, if the pool is encrypted. By that I mean if I can connect my 8TB drive to an additional sata slot and just simply follow the "Replacing Drives to Grow a ZFS Pool" guide. Or if I need to follow the "Replacing an encrypted drives" guide.

I understand what you are saying now, and out of an abidance of caution, I will refrain from answering because I also do not run an encrypted pool.

Curiosity speaking here, why are you running an encrypted pool? Most people on here would advice that unless you are mandated by law in accordance with any business that your freenas may be involved in, it is most safer to run an unencrypted pool and then encrypt only certain folders or files with the appropriate software (separate from freenas). Too many people have lost everything simply because they made a very simple mistake somewhere along the line and encryption locked them out.

Anyway your comment let me to reconsider my decision! :) Maybe after two years of happily running an RaidZ1 without any issues, I should not further challenge the universe and its destiny for another 2-3 years :)
So maybe I'm going to follow depasseg's how to migrate data guide https://forums.FreeNAS.org/index.ph...te-data-from-one-pool-to-a-bigger-pool.40519/
an create a new encrypted bigger RaidZ2 pool to be on a more safe side.

Sounds like a plan. Especially with 8tb drives, you're just asking for a problem with raidz1.

As @depasseg said, make sure you have a backup before you do anything. Raidz and redundancy is not a substitute for a backup!

 

[SebbaG]

hi,

just finished my upgrade from 6x 4TB WD Greens Z1-pool to 7x 8TB Seagate IronWolf Z2-pool. Everything went well!

Curiosity speaking here, why are you running an encrypted pool? Most people on here would advice that unless you are mandated by law in accordance with any business that your freenas may be involved in, it is most safer to run an unencrypted pool and then encrypt only certain folders or files with the appropriate software (separate from freenas). Too many people have lost everything simply because they made a very simple mistake somewhere along the line and encryption locked them out.

Well because I like security and data-privacy! All my hdd and ssd which I have are encrypted, thats why I wanted to keep my policy :)
Anyway was running an encrypted z1-pool for almost two years now, without any problem! So I went along with an encrypted z2-pool! Hope it will work for the next 3 years as well ;)

thx for all your advices and help
SebbaG

 

[stemplar]

I was actually wondering if "8.1.11. Replacing Drives to Grow a ZFS Pool" also applies, if the pool is encrypted. By that I mean if I can connect my 8TB drive to an additional sata slot and just simply follow the "Replacing Drives to Grow a ZFS Pool" guide. Or if I need to follow the "Replacing an encrypted drives" guide.

I know this is an old thread, so apologies if it's bad form to reply to it, but I find myself in the same boat as the OP. Specifically, I'd like to grow an encrypted vdev by replacing the drives one by one. I've been using freenas for several years (and zfs since it was introduced in solaris) and I've been through failed drive replacement in encrypted vdevs a few times, and the documentation in sections 8.1.10 and 8.1.10.1 was fine. (8.1.10 is for "Replacing a Failed Drive" and 8.1.10.1 is the subsection specifically for replacing a failed encrypted drive). Since my vdevs are encrypted I followed the documentation in 8.1.10.1 whenever a drive failed.

I find the documentation as it exists to be ambiguous though for replacing encrypted drives that have not failed with the intent of growing the capacity of a vdev. 8.1.11 is for replacing a drive to grow a zfs pool, but can I safely follow 8.1.11 with an encrypted pool, or do I need to follow 8.1.10.1 which would have me offline the drive that I want to replace first? I'd rather not offline a drive first for the reasons outlined in 8.1.11. My new drives are already attached, tested, and visible, so if the proper way to grow the encrypted pool is to use the replace function as outlined in 8.1.11 then I'm ready for step #4.

Perhaps a section 8.1.11.1 could be added for Replacing Encrypted Drives to Grow a ZFS Pool ? Even if the right answer is to point us to 8.1.10.1 to follow those steps...

 

[Ericloewe]

If you just normally replace the encrypted drives as if one was failing, everything should just automagically fall into place.

 

[stemplar]

If you just normally replace the encrypted drives as if one was failing, everything should just automagically fall into place.

Thanks for taking the time to respond Eric; I have no doubt you're correct. If I follow the steps in 8.1.10.1 I'd be fine, as I'd done before with failed drives before.

My specific question is, is section 8.1.11 relevant to encrypted drives?

I have hardware to set up a test system but it'll be some time before I can prioritize the work to do so. I'm really curious what would happen if a user with encrypted drives follows section 8.1.11 to upgrade a drive.

 

[Ericloewe]

Yes, same thing. If it doesn't work, that's a bug and needs to be reported.

 

[stemplar]

Just a quick follow-up to say that I did use the 'Replace' function as outlined in 8.1.11 on my encrypted vdev and it worked just fine. Doing so invalidates the recovery key, but that's spelled out in one of the dialog boxes.