Why is a re-key required when replacing a failed encrypted disk?

[RobL]

Hi,
It may seem obvious, or I haven't had enough coffee yet today, but why is a re-key required when replacing an encrypted volume?

Typing in the password to add the new drive seems to bring it on line. What does a re-key do after that, that is needed? Does the geli key for the volume need to be generated? The docs don't describe why, just that you have to re-key when doing this.

Many thanks,
--RobL.

 

[Kerry]

I am also curious about this, especially since the system seems to have everything it needs.

I appreciate any insight anyone is able to bring to this.

 

[Lai Zit Seng]

Hi, has anyone found an answer to this? I am too also keen to find out why the re-key is necessary. The way the docs describe the steps, it seems like we have a window of vulnerability, i.e. the system crashes and reboots (for any reason) after disk replacement but before resilvering is complete. It makes me a little worried. Shouldn't one be able to continue using the old passphrase, geli key or recovery key?

 

[Ericloewe]

Any new GELI devices have their individual, random Master Key, which is stored encrypted at the end of the GELI partition. This Master Key is decrypted using User Keys, which are also random. When replacing drives, the Master Keys for all drives need to be reencrypted with the new User Keys.

That said, it's probably not a strict requirement that new user keys be used, but it makes some sense to avoid leaking information to attackers. FreeNAS makes it a requirement.

Hi, has anyone found an answer to this? I am too also keen to find out why the re-key is necessary. The way the docs describe the steps, it seems like we have a window of vulnerability, i.e. the system crashes and reboots (for any reason) after disk replacement but before resilvering is complete. It makes me a little worried. Shouldn't one be able to continue using the old passphrase, geli key or recovery key?

It's a tiny window. Most of the data is not re-encrypted, it's always encrypted with the Master Key, which never changes. It's this Master Key that is re-encrypted when new keys are generated.

 

[Lai Zit Seng]

Any new GELI devices have their individual, random Master Key, which is stored encrypted at the end of the GELI partition. This Master Key is decrypted using User Keys, which are also random. When replacing drives, the Master Keys for all drives need to be reencrypted with the new User Keys.

That said, it's probably not a strict requirement that new user keys be used, but it makes some sense to avoid leaking information to attackers. FreeNAS makes it a requirement.


It's a tiny window. Most of the data is not re-encrypted, it's always encrypted with the Master Key, which never changes. It's this Master Key that is re-encrypted when new keys are generated.

Ah, thanks for the explanation. So for clarity, even when you say:

"When replacing drives, the Master Keys for all drives need to be reencrypted with the new User Keys"

It's not truly needed, but rather just a, shall we say, strong recommendation?

If you don't do anything, your old passphrase and geli.key would have still worked anyway, yes?

 

[Ericloewe]

Ah, thanks for the explanation. So for clarity, even when you say:

"When replacing drives, the Master Keys for all drives need to be reencrypted with the new User Keys"

It's not truly needed, but rather just a, shall we say, strong recommendation?

If you don't do anything, your old passphrase and geli.key would have still worked anyway, yes?

If you were managing things by hand, yes.

With FreeNAS behind the wheel, no.

 

[gary_1]

It's a tiny window. Most of the data is not re-encrypted, it's always encrypted with the Master Key, which never changes. It's this Master Key that is re-encrypted when new keys are generated.

At which point is freenas generating the new random user keys and re-encrypting the master key?

Is it before the resilver starts, after the resilver completes, or only when the resilver has completed and the user goes through the ui steps to re-key?

 

[Ericloewe]

When manually triggered in the UI.

 

[pro lamer]

Ah, thanks for the explanation. So for clarity, even when you say:

"When replacing drives, the Master Keys for all drives need to be reencrypted with the new User Keys"

It's not truly needed, but rather just a, shall we say, strong recommendation?

If you don't do anything, your old passphrase and geli.key would have still worked anyway, yes?

If you were managing things by hand, yes.

With FreeNAS behind the wheel, no.

Does that mean that one can do it using command line interface while still having FreeNAS... "operational" (running or paused)?

 

[Ericloewe]

Yes, but it is definitely not recommended.

 

[mgd]

Any new GELI devices have their individual, random Master Key, which is stored encrypted at the end of the GELI partition. This Master Key is decrypted using User Keys, which are also random. When replacing drives, the Master Keys for all drives need to be reencrypted with the new User Keys.

That said, it's probably not a strict requirement that new user keys be used, but it makes some sense to avoid leaking information to attackers. FreeNAS makes it a requirement.

I do not understand what information could be leaked by reusing the existing user key for encrypting the master key of the replaced drive. Could you please elaborate?

It's a tiny window. Most of the data is not re-encrypted, it's always encrypted with the Master Key, which never changes. It's this Master Key that is re-encrypted when new keys are generated.

Just to make sure I understand the window of vulnerability. When one press Re-Key in the GUI a new user key is generated and the master key of all GELI encrypted drives in the volume is re-encrypted with the new user key. This means that if the system reboots and the boot volume crashes before the user has downloaded the new user key, then the entire volume is completely lost. Is that correctly understood?

Although this is definitely a tiny window, it is also a huge risk within that tiny window and – as far as I understand – an unnecessary risk. So if there is no compelling reason to force the use of a new user key then maybe FreeNAS should stop doing that?

 

[Ericloewe]

I do not understand what information could be leaked by reusing the existing user key for encrypting the master key of the replaced drive. Could you please elaborate?

You're leaking an entire disk's worth of data encrypted with a key that sits in a known location, encrypted. It's not likely at all that it's going to be decrypted in useful time, but if it is, you can use it on the other old disks.

Just to make sure I understand the window of vulnerability. When one press Re-Key in the GUI a new user key is generated and the master key of all GELI encrypted drives in the volume is re-encrypted with the new user key. This means that if the system reboots and the boot volume crashes before the user has downloaded the new user key, then the entire volume is completely lost. Is that correctly understood?

Mostly. Don't forget you have two independent keys. Now, it's been a while since I've looked into this, so I don't know what the GUI is doing, exactly (the abstracted terms used don't help), but the sane option is to do it for one slot, then the other once the first one has been confirmed to have finished.

 

[mgd]

You're leaking an entire disk's worth of data encrypted with a key that sits in a known location, encrypted. It's not likely at all that it's going to be decrypted in useful time, but if it is, you can use it on the other old disks.

I am discarding a disk that is encrypted with military grade AES256 encryption using a unique, random master key. Yes, the master key sits on the disk but is encrypted with the user key (and possibly a passphrase) in slot 0 and with a recovery key in slot 1.

If someone can derive the master key from that information within reasonable time and decrypt the entire drive, there is a serious problem with GELI and/or AES or with the way FreeNAS generates the user key. Even if they can, they cannot use that information to decrypt any other drive in the pool because each drive has its own unique, random master key – so no, they cannot use it on the other old disks.

In order to decrypt the rest of the drives in the pool they would either have to find their master keys or derive the user key from the master key of the discarded drive.

I am not saying this is not possible – just that it is very hypothetical.

Mostly. Don't forget you have two independent keys. Now, it's been a while since I've looked into this, so I don't know what the GUI is doing, exactly (the abstracted terms used don't help), but the sane option is to do it for one slot, then the other once the first one has been confirmed to have finished.

The problem is (verified on FreeNAS 11.1 U4):

1. After replacing a drive in a pool, the existing recovery key (slot 1) is invalid for the new drive – because it is not known to the system, there is no way to encrypt the new drive's master key with the existing recovery key.
2. Re-keying the pool zaps the passphrase (protecting the user key in slot 0).
3. Creating a new passphrase invalidates the existing recovery key (slot 1) for the remaining drives in the pool – so even if you started out by setting a new recovery key, it would invalidated by now.

So after re-keying and adding a new passphrase, one is in the situation that a crash of the boot device which holds the new user key (generated by re-keying) will have the dire consequence that all your data is lost. And the only reason we are taking that risk is because FreeNAS forces you to re-key after replacing a failed drive – to mitigate a very theoretical risk that someone derives your user key and/or passphrase from the discarded drive.

Even if FreeNAS did not require a re-key after drive replacement, then if you at some point in time wanted to re-key your pool because you feared your user key, recovery key or passphrase had been compromised, you would have the same problem.

A simple solution would be that setting a new passphrase would not invalidate the recovery key – and from my (limited?) knowledge of GELI I do not understand why it is necessary. With that change, one could start out by setting and downloading a new recovery key for the entire pool. With that safely stored, one could then re-key the pool getting a new user key generated and then set a new passphrase. Doing this, there would be no point in time where you did not have information moved off the system necessary for unlocking the pool and the risk would be eliminated.

For details about how GELI is used by FreeNAS, please see this excellent description (which is also linked from section 8.1.1.1 in the official documentation):

• https://forums.freenas.org/index.php?threads/recover-encryption-key.16593/#post-85497

I apologise if there is something I have misunderstood and I welcome everybody to correct me if I am wrong.

 

[Ericloewe]

I agree that the encryption management needs a serious look and potentially significant changes, to reduce needless risk without compromising security.

That said, the risk here is minimal, even if it is completely avoidable. Encryption in general adds a non-trivial risk of data loss via philosophical discussion, which must be weighed against the problems that it solves. Hell, there's a non-zero risk that the PSU will go haywire and fry every hard drive in the system beyond repair, or catch fire and destroy them outright. That's why backups are important.

 

[mgd]

I guess we are very close to being in complete agreement.

Yes, re-keying leaves us with (tiny) window where we risk loosing everything, and there is no need to take that risk.
And yes, backups are important.

That being said, it seems that something has changed in FreeNAS. The documentation in section 8.1.10.1. Replacing an Encrypted Drive is apparently not entirely correct:

1. Set a passphrase before attempting to replace the failed drive: That is not necessary. If no passphrase is set (which is the case if you only want to protect your disks in case of RMA but want auto-mount on reboot) then FreeNAS will not prompt for a passphrase during drive replacement and everything works fine.
2. You must re-key the volume after drive replacement: That is not necessary. The existing user key works fine. The only thing that needs to be replaced is the recovery key – and that is important because (as previously said) the old recovery key will not work for the new drive and if you ever try to unlock the volume using the old recovery key, the replaced drive will not unlock and leave the volume in a state where you will never get that drive online again – not even with the userkey + passphrase. The drive will be forever lost.

So bottom line: The only thing strictly necessary after replacing a drive in an encrypted volume is to generate a new recovery key. There is no need for re-keying (and it is not enforced by FreeNAS) which means that the risk of loosing the entire array we have been discussing in this thread is eliminated.

P.S. I have tested all this thoroughly in a VM using FreeNAS 11.1 U4 in multiple configurations using a 2 drive mirror. What I did was:

• Offline one drive: Ok
• Replace the drive with another: Ok, but you need to supply the existing passphrase
• Reboot to see if volume unlocks successfully with both drives online: Ok (did 2 reboots)
• Lock and unlock with passphrase: Ok
• Lock and unlock with old recovery key: new drive does not unlock as expected, volume is degraded
• Lock and unlock with passphrase: new drive does not unlock, volume is still degraded!

I also tried with a 2 drive mirror volume without a passphrase on the encryption key:

• Offline one drive: Ok
• Replace the drive with another: Ok (no prompt for passphrase)
• Reboot to see if volume unlocks successfully with both drives online: Ok (did 2 reboots)

 

[pro lamer]

I've just come across some older thread:

When looking at the procedures for how to replace an encrypted HD, I was greatly concerned that I would have to re-key after the resilver or I could lose the pool. A resilver could take a long time, which means during that time if anything happened that caused a reboot I wouldn't be able to re-key first. But, like you, I could not figure out why a re-key is necessary so I took a look at the source code. It turns out you and I are correct.

The code asks you for your password, then verifies you entered it correctly by decrypting the existing drives. If that fails, it assumes you entered the wrong password. If the password is correct, it then uses that password plus the existing key in /data/geli to set the master key on the replacement drive. It then decrypts the replacement drive making it ready for use and also verifying the password and key worked correctly. Therefore, a reboot after the resilver would not render your pool useless as long as you know the password.

You would, of course, need to set a new recovery key as the existing recovery key has not been set on the replacement drive. I think the docs need an update and should say that after replacing an encrypted drive, you need to set a new recovery key, either before or after the reboot. All of the other stuff can be removed as nothing else needs to be done as far as keys are concerned.

Is it still valid? It's a bit dated thread...

 

[pro lamer]

Just an update to this old thread (anyway it appeared as a first one when I used the forums search tool)

At which point is freenas generating the new random user keys and re-encrypting the master key?

Is it before the resilver starts, after the resilver completes, or only when the resilver has completed and the user goes through the ui steps to re-key?

When manually triggered in the UI.

Updated: I guess it refers to the disks already in the pool but the newly put disk gets a new key right away
It must have changed since then, have a look into this thread:
https://www.ixsystems.com/community/threads/replace-encrypted-drive.72573/page-2#post-524780

Does that mean that the new disk is being given some other keys into first or second slot (other comparing to the disks already in pool) ?

Yes, which is why it's necessary to re-key the disks.

If you skip it, you need to manually keep track of the keys for each disk.

Now it looks even more I can understand better than before how important to re-key is but at least we don't need to wait for the resilver to complete...

Sent from my phone