TrueNAS 12.0 GELI to Full Drive Encryption Migration

Chris Tobey

Contributor
Joined
Feb 11, 2014
Messages
114
Hi everyone,

I have some questions about migrating my 11.x GELI encrypted pool to a new 12.x encrypted pool. I played with this a bit, and I know some of the features are not there yet, but want to get some advice and clarify things a bit. For reference I am mostly following this: https://www.truenas.com/docs/hub/initial-setup/storage/encryption/

Setup:
zpool-A is a GELI encrypted pool with two datasets {mydata, iocage}.
zpool-B is a new pool that I can do whatever required with.
I want both mydata and iocage (or at least the jails) to be encrypted, so that after a power cycle they are unusable until the key is given.

I have tried creating an unencrypted zpool-B and then copying mydata over with dataset encryption, but the replication commands do not seem to like recursive operations or intermediate snapshots, and I am not sure that copying the jail would work well this way.
# zfs send -v zpool-A/mydata@migration | zfs recv -o encryption=on -o keyformat=passphrase -o keylocation=file:///tmp/pass zpool-B/mydata

So I tried again, destroyed zpool-B, re-created it with full pool encryption, then replicated the datasets without encryption, and this was much nicer.
# zfs send -Rv zpool-A/mydata@migration | zfs recv zpool-B/mydata
BUT - is my data actually encrypted? If I reboot, zpool-B is locked and not mounted, and I cannot see the datasets on it, once unlocked I can see them. Are the data blocks stored on disk actually encrypted if zpool-B has encryption but the child dataset does not?

Is there something better I can be doing? Is there something I should not be doing?
 

Chris Tobey

Contributor
Joined
Feb 11, 2014
Messages
114
So, I did some testing, and I think this picture will help:
1607487177470.png
 

Chris Tobey

Contributor
Joined
Feb 11, 2014
Messages
114
From this, I can see that my commands above lead to the anti-lock symbol, meaning the datasets are not encrypted. I believe, and please correct me if I am wrong, that I can have them inherit the encryption and behave as desire with this command instead:
# zfs send -Rv zpool-A/mydata@migration | zfs recv -x encryption zpool-B/mydata
 
Joined
Oct 22, 2019
Messages
3,589
I'll chime in and try to help, since I had to do the exact same thing. I'm not an "expert", but I've learned a lot about native encryption in a relatively short amount of time to the point where I successfully did a full migration and use a setup that is "friendly" towards encrypted data.

Read this thread in the meantime, and I'll try to reply later; perhaps someone else might chime in too.

 
Joined
Oct 22, 2019
Messages
3,589
If you absolutely want "iocage" (Jails) and ".system" (System Dataset) to be encrypted on the new pool, then you must enable encryption when creating the new pool. The reason for this is because it will force the top-level root dataset to be encrypted, and ".system" as well as "iocage" by default "inherit" the encryption. This is how TrueNAS-12.0 works by default, and unless you get your hands dirty with the command prompt, it's best to accept this.

  • new_pool <--- top-level root dataset, encrypted, keyfile (NOT passphrase)
    • .system <--- inherited encryption scheme, inherits the same keyfile as "new_pool"
    • iocage <--- inherited encryption scheme, inherits the same keyfile as "new_pool"
    • dataroot <--- non-inherited encryption, unlocked by passphrase (can be "locked")
      • multimedia <--- inherited encryption scheme, inherits the same passphrase as "dataroot"
      • downloads <--- inherited encryption scheme, inherits the same passphrase as "dataroot"
      • backups <--- inherited encryption scheme, inherits the same passphrase as "dataroot"
      • legal <--- inherited encryption scheme, inherits the same passphrase as "dataroot"
In the above template, the following datasets highlighted in blue are available and immediately unlocked upon booting the system, because the keyfile (.json) lives on the boot device (until you decide to "export" the pool, in which you MUST have a copy of the .json keyfile saved elsewhere or you will FOREVER LOSE ACCESS):
  • new_pool
    • .system
    • iocage
    • dataroot
      • multimedia
      • downloads
      • backups
      • legal

The following datasets highlighted in blue will not be accessible until you manually unlock them with your passphrase. If you ever forget your passphrase, you will LOSE EVERYTHING:
  • new_pool
    • .system
    • iocage
    • dataroot
      • multimedia
      • downloads
      • backups
      • legal
Encryption happens on a "per dataset" application. The "tree" view is deceiving, as it implies there is a dependancy of children datasets to their encrypted parents. This is not true. The ability of unlocking all datasets at once (or as a nested group) is only for the sake of convenience. You can break encryption inheritance and re-enable inheritance as many times as you want, as long as the dataset was initially created with encryption enabled.
 
Last edited:
Joined
Oct 22, 2019
Messages
3,589
Here is a template I used to migrate over to a new pool with native encryption on TrueNAS-12.0-RELEASE:

First, I created the new pool and enabled encryption. The root dataset is newpool and it is encrypted, which means .system and iocage will inherit its encryption. It uses a keyfile (.json) to automatically unlock upon boot. I saved this keyfile to my computer and to a USB stick.
  • newpool <--- (encrypted, .json keyfile)

Next, I changed the location of the System Dataset (.system) to reside on mainpool, which automatically creates a hidden dataset named .system under newpool, it inherits the encryption properties of newpool, and moves the contents from the previous old .system dataset (FreeNAS 11.x) to the new one. From the main menu, System > System Dataset.
  • newpool <--- (encrypted, .json keyfile)
    • .system <--- (inherited from newpool)

Next, I stopped my jails and used send/recv to migrate my old iocage to the new pool. This forces the new iocage dataset to inherit the newpool encryption properties.
Code:
zfs snap -r oldpool/iocage@migrate-20201209

zfs send -v -R oldpool/iocage@migrate-20201209 | zfs recv -v -d -F -x encryption newpool


After this, I went to Jails > Options and change the location of my jails to newpool.

At this point, the new pool looks like this:
  • newpool <--- (encrypted, .json keyfile)
    • .system <--- (inherited from newpool)
    • iocage <--- (inherited from newpool)

Now I need to finally migrate all my data to the new pool, and make sure it is encrypted upon being received. First I had to create a new pseudo-root dataset under newpool, make sure it is encrypted. (Later I will break its inheritiance and apply a lockable passphrase on it.) I named it zdataroot.
  • newpool <--- (encrypted, .json keyfile)
    • .system <--- (inherited from newpool)
    • iocage <--- (inherited from newpool)
    • zdataroot <--- (inherited from newpool)

With it ready, I have to send and recv, one by one, my child datasets:
Code:
zfs snap -r oldpool@migrate-20201209

zfs send -v -R oldpool/archive@migrate-20201209 | zfs recv -v -d -F -x encryption newpool/zdataroot

zfs send -v -R oldpool/downloads@migrate-20201209 | zfs recv -v -d -F -x encryption newpool/zdataroot

zfs send -v -R oldpool/isos@migrate-20201209 | zfs recv -v -d -F -x encryption newpool/zdataroot

zfs send -v -R oldpool/legal@migrate-20201209 | zfs recv -v -d -F -x encryption newpool/zdataroot

...and on and on and on, yes, it's tedious and time consuming, but was the safest method I could find.



At this point, the new pool looks like this:
  • newpool <--- (encrypted, .json keyfile)
    • .system <--- (inherited from newpool)
    • iocage <--- (inherited from newpool)
    • zdataroot <--- (inherited from newpool)
      • archive <--- (inherited from newpool)
      • downloads <--- (inherited from newpool)
      • isos <--- (inherited from newpool)
      • legal <--- (inherited from newpool)

Finally, I want to make zdataroot and all the child datasets underneath lockable by a passphrase, which means they will be locked upon reboot, and require a passphrase to access their contents. So I click on the encryption properties for zdataroot, uncheck "inherit", and give it a passphrase. Now everything looks like this:
  • newpool <--- (encrypted, .json keyfile)
    • .system <--- (inherited from newpool)
    • iocage <--- (inherited from newpool)
    • zdataroot <--- (inheritance break, passphrase instead)
      • archive <--- (inherited from zdataroot)
      • downloads <--- (inherited from zdataroot)
      • isos <--- (inherited from zdataroot)
      • legal <--- (inherited from zdataroot)


FOR THE LOVE OF EVERYTHING DO NOT DESTROY YOUR OLD POOL UNTIL YOU VERIFY, MORE THAN ONCE, THAT YOU CAN REBOOT AND ACCESS YOUR DATA ON THE NEW POOL USING THE .JSON KEYFILE AND YOUR PASSPHRASE.
 
Last edited:

Patrick M. Hausen

Hall of Famer
Joined
Nov 25, 2013
Messages
7,740
until you decide to "export" the pool, in which you MUST have a copy of the .json keyfile saved elsewhere or you will FOREVER LOSE ACCESS
Oops! Danger, Will Robinson!

Thanks for researching and posting about all of this!
 

Chris Tobey

Contributor
Joined
Feb 11, 2014
Messages
114
@winnielinnie Thank you for the detailed response.
I think I am in the process of something similar, though I changed my pool from key to passphase so that it remains locked on boot (and cannot automatically unlock), and do not plan to move the .system over, I think.
The iocage dataset will also be locked, as intended, and your steps are helpful for that migration.
I currently have about 16h more of copying my datasets and then plan to test a few reboots, update the shares and play with it for a little while before destroying my old pool, which is good advice for anyone!
 

gary_1

Explorer
Joined
Sep 26, 2017
Messages
78
(until you decide to "export" the pool, in which you MUST have a copy of the .json keyfile saved elsewhere or you will FOREVER LOSE ACCESS):
[snip]
The following datasets highlighted in blue will not be accessible until you manually unlock them with your passphrase. If you ever forget your passphrase, you will LOSE EVERYTHING:

You might be able to clear up a couple of questions I've had regarding the new encryption as I've not upgraded yet nor played with 12.x

Firstly, with the key version, it sounds like the key gets stored on the boot drive and there's no way to protect that key (unlike geli where you could have key itself encrypted with password). So that really is only a good choice for data you want to protect when a disk is removed/recycled/RMA'd etc

Moving on to the password protected dataset, I assume that's based on key derivation, so there is no stored key as such. If you use password encrypted dataset, unlock it, then reboot the server without locking it. Will the dataset be in the locked state post reboot and require manual unlocking with the password? (I'm hoping so). I had this question in another thread where it was suggested passworded datasets can be auto-unlocked during a reboot if they were unlocked at time of reboot and that raises some red flags to me and never really got a conclusive answer.

PS I do think key based encryption should work differently. When you first create an encrypted dataset it should force you to download the key. It should then create AND lock the dataset and then require you to unlock the dataset using the key you downloaded, this will prove to the user + freenas that you have a copy of the valid/non corrupt/working key and can unlock the dataset before any data is put at risk. At this point you could be offered the option to store a copy of your key on the boot volume to allow auto-unlocking on reboot + a link to more information on the pros/cons of this. Such as if you don't allow auto-unlock you cannot store .system in an encrypted dataset.

Same should be done for password unlocks too, only here the auto-unlock could offer to save the derived key for auto booting. Any time the generated key or derived key are not available on the boot volume, user is expected to manually unlock.

I assume there's a reason behind why it doesn't work this way, but it seems in terms of helping users avoid shooting themselves in the foot, verifying they can unlock the dataset before allowing data to be placed in it would be a good thing. If users lose the key they downloaded or forget the pass subsequently, well, that's on them as long as docs/ui made it clear you'd need it.
 
Last edited:
Joined
Oct 22, 2019
Messages
3,589
@winnielinnie Thank you for the detailed response.
I think I am in the process of something similar, though I changed my pool from key to passphase so that it remains locked on boot (and cannot automatically unlock), and do not plan to move the .system over, I think.
The iocage dataset will also be locked, as intended, and your steps are helpful for that migration.
I currently have about 16h more of copying my datasets and then plan to test a few reboots, update the shares and play with it for a little while before destroying my old pool, which is good advice for anyone!

Best of luck, and remember, you're not a large corporation or bound by laws and regulations: you can take your time and be patient, there is no rush. Play it safe. I found it helps to create a basic virtual machine to use as a sandbox, with virtual disks, which you can play around and be risky without affecting anything of importance. VirtualBox is free, easy to use, and I made a test environtment specifically for this.

Reading guides, FAQs, Oracle docs, iXsystems docs, forums, Reddit threads, etc, was only half of my learning experience. The other half was getting my hands dirty and playing around in a virtual machine as a hands-on experience.
 
Joined
Oct 22, 2019
Messages
3,589
Firstly, with the key version, it sounds like the key gets stored on the boot drive and there's no way to protect that key (unlike geli where you could have key itself encrypted with password). So that really is only a good choice for data you want to protect when a disk is removed/recycled/RMA'd etc

Until you export the pool. Which means exporting the pool essentially "locks" the root dataset. This is why you MUST have the .json keyfile saved to your laptop, desktop, USB stick, etc. You can use a passphrase locked rootdataset; however, TrueNAS will refuse to allow the .system dataset to reside on that pool. That's why I, personally, have the root dataset unlocked at boot, using a keyfile (default) rather than passphrase.


Moving on to the password protected dataset, I assume that's based on key derivation, so there is no stored key as such. If you use password encrypted dataset, unlock it, then reboot the server without locking it. Will the dataset be in the locked state post reboot and require manual unlocking with the password?
Yes, correct.


I had this question in another thread where it was suggested passworded datasets can be auto-unlocked during a reboot if they were unlocked at time of reboot and that raises some red flags to me.
I tested this myself, and regardless of its state upon when I rebooted, my passphrase-protected datasets were "locked" upon boot. Perhaps those users incorporated a tweak or custom script? I am unfamiliar. The only datasets that automatically unlock in my case are those that use a keyfile. However, if I ever "export" the pool, and then re-import it, I must upload the .json keyfile to unlock those datasets, including the root dataset. Otherwise, even they remain locked. This is why the .system dataset will automatically transfer back over the the boot pool (or any such currently existing pool that does not have a passphrase-protected root dataset) when I try to export my mainpool.


PS I do think key based encryption should work differently. When you first create an encrypted dataset it should force you to download the key.
It doesn't "force" you to, but it STRONGLY RECOMMENDS that you do. :wink:


It should then create AND lock the dataset and then require you to unlock the dataset using the key you downloaded, this will prove to the user + freenas that you have a copy of the working key and can unlock the dataset before any data is put at risk. At this point you could be offered the option to store a copy of your key on the boot volume to allow auto-unlocking on reboot + a link to more information on the pros/cons of this. Such as if you don't allow auto-unlock you cannot store .system in an encrypted dataset.

Same should be done for password unlocks too, only here the auto-unlock could offer to save the derived key for auto booting. Any time the generated key or derived key are not available on the boot volume, user is expected to manually unlock. Some might prefer to use a password derived key even if I was going to allow auto-unlock too. Where the level of protection offered really depends on your password strength, but for some even a low security password would offer sufficient level of protection and be less likely to be lost/forgot than a key.

Clever and interesting! I suppose it's up to the developers to incorporate those safety measures. A Feature Suggestion, maybe? Though sometimes it may come across as overly-complex where only a niche base would find it useful?
 

gary_1

Explorer
Joined
Sep 26, 2017
Messages
78
That's why I, personally, have the root dataset unlocked at boot, using a keyfile (default) rather than passphrase.

That's how I intend to setup too. Auto-unlock for data that requires minimal protection and is more a RMA/recycling protection and then password datasets for the at-rest protection.

Had the option existed to auto-unlock using a password derived key that's stored on the boot volume. I'd have probably gone with that, although maybe that wasn't offered as it would have needed to get across to people that you'd then need a different password for the datasets you wanted at-rest protection for vs the auto-unlock. Maybe that's what drove the key vs password difference for auto-unlock?
 
Joined
Oct 22, 2019
Messages
3,589
(snip)maybe that wasn't offered as it would have needed to get across to people that you'd then need a different password for the datasets you wanted at-rest protection for vs the auto-unlock. Maybe that's what drove the key vs password difference for auto-unlock?
That's possibly one of the reasons why.

I hate to sound like a pessimist, but compared to LUKS on Linux, native ZFS encryption feels like it's still yet to catch up to the practicality of its counterpart.

:cool: LUKS allows up to eight keyslots.
:mad: ZFS only allows one? Is this how it is implemented in TrueNAS, or is this currently a limitation of ZoL / OpenZFS?

:cool: LUKS allows a combination of passhprases and keyfiles.
:mad: ZFS only allows one or the other, not both.

On these two fronts, LUKS is hands down superior. Having these features does not take away from the user's implementation: just because LUKS allows it does not mean the user is forced to use it. However, they still have that option if they so desire. With ZFS? You don't even have those options to use if you so desired.


Here are two feasible and realistic examples with LUKS.

Example A:
LUKS header contains the following keyslots
  1. passphrase: myFav1234password!
  2. unused
  3. unused
  4. unused
  5. unused
  6. unused
  7. unused
  8. unused
In such a simple setup, you decrypt the master key with the passphrase myFav1234password!


Example B:
LUKS header contains the following keyslots
  1. passphrase: myFav1234password!
  2. passphrase: Rabbits'n'Dragons
  3. passphrase: DEE-LICIOUS-encryption2020
  4. keyfile: sister-baby-shower.jpg (saved on your USB flash drive)
  5. keyfile: blob.dat (saved on someone else's computer)
  6. unused
  7. unused
  8. unused
In this setup, you might allow different people to access the master key, and thus, the data within. Perhaps you give a keyfile to a trustworthy friend? You share one of the passphrases with a family member? Maybe you use multiple keyslots to mitigate against the risk of "forgetting a passphrase" or "losing access to one of the keyfiles." Keep in mind, the LUKS header does not store any identifiable information about the location or name of the keyfiles used. (I simply wrote them out for illustrative purposes.)
 

q/pa

Explorer
Joined
Mar 16, 2015
Messages
64
I am thinking about

1. temporarily backing up my three datasets (shares, iocage, maria db) to a ZFS on Linux machine with native encryption enabled

2. create a new pool with native encryption on TrueNAS

3. restore each dataset (not a raw send!) thereby inheriting encryption on the receiving side, i.e. TrueNAS

4. later destroying the temporary backup and regularly backing up the raw (encrypted) data

My biggest concern here is: will 3. work? Can I restore the unencrypted datasets with
Code:
zfs send -Rvb remotebackup/dataset@snap | zfs receive -euv -x encryption newpool
 
Joined
Oct 22, 2019
Messages
3,589
My biggest concern here is: will 3. work? Can I restore the unencrypted datasets with
Code:
zfs send -Rvb remotebackup/dataset@snap | zfs receive -euv -x encryption newpool
Theoretically it should. Never tried using ZFS on Linux yet (plan to someday), but I believe step 3 will require you to decrypt the stream with its key before it can be re-encrypted on the new target.
 
Joined
Oct 22, 2019
Messages
3,589
Is there any way to do this through the UI yet?
TrueNAS does support running a Replication Task manually, which theoretically can do the same thing. But it's not meant solely for "migrating from legacy encryption to native encryption". It would only be a substitute for using the command line to do the same thing i.e, send/recv from old pool to new pool, enabling native encryption on the destination side.
 

storage-junkie

Dabbler
Joined
Jan 17, 2018
Messages
44
I haven't found a way to make the replication task inherit encryption on the remote side, the way you can with the command line.
 

gary_1

Explorer
Joined
Sep 26, 2017
Messages
78
Replication tasks can be set to "Include dataset properties" or "Full Filesystem Replication", both of which should result in a native ZFS encrypted dataset replicating to a destination and retaining its encryption.

The issue you likely face, is that a GELI pool is treated as an unencrypted data set, so neither of those options will result in encryption at the destination.

I could be wrong (winnielinnie has looked into this more than I have, so will hopefully correct me if I am), but I believe the only way to handle this currently is to create a ZFS encrypted destination pool, then use the cmd line to zfs send with the "-R" raw flag and zfs recv with the "-x encryption" flag. That should result in encrypted datasets on the destination that have inherited from your destination parent root (due to -x).

Now at this point, assuming you export/import the destination pool and can unlock/access everything correctly. You can destroy your original pool, create a new pool with native ZFS encryption and then either zfs send back via cmdline with "-R -w" or make a replication task you run manually that has "Include dataset properties" checked and replicate to your new main pool.

I'd certainly suggest making extra backups of the pool before attempting any of this though, just in case a mistake is made.
 

storage-junkie

Dabbler
Joined
Jan 17, 2018
Messages
44
The user guide mentions GELI to ZFS migrations briefly, and says that it will soon be possible with an Advanced Replication Task.
https://www.truenas.com/docs/core/storage/pools/storageencryption/#geli-pool-migrations

But that feature appears to have been implemented as of 12.0-U1. The documentation hasn't been updated though. Which leaves people like me resorting to forum posts like this, or internet searches to see how other people have managed to do it. This should be a straightforward, documented process. Maybe even a wizard for non-power users (this is an appliance after all, right?)

My first time attempting this, like others here, I did a replication task to a new encrypted pool with the expectation that my new datasets would inherit the encryption properties of the parent pool, but that was not the case. Instead, I got the anti-padlock icon (which itself is confusing - would a tooltip be too much to ask?).

For now, I've gotten around it by setting up a replication task for each dataset, checking the 'Encryption' box, setting a passphrase, and after the first time the task completes, I go into the new dataset and change its encryption properties to 'Inherit'. After that, the subsequent replications maintain the inherited encryption of the target dataset. But figuring this out took a decent amount of trial and error.

My goal for this migration was to stand up a new encrypted pool on a new set of disks, replicate all data to it for a period of time until I feel comfortable the new pool is operating correctly, and then at some point do a cutover to the new pool (re-point shares, etc), and eventually retire the old pool. But this seemingly simple task has proved to be unnecessarily confusing.

I should mention that I also have an offsite server that I'm replicating to, but I haven't figured out how I'm going to get that switched over (without having to re-send everything).
 
Top