TrueNAS 12.0 GELI to Full Drive Encryption Migration

[winnielinnie]

Replication tasks can be set to "Include dataset properties" or "Full Filesystem Replication", both of which should result in a native ZFS encrypted dataset replicating to a destination and retaining its encryption.

The issue you likely face, is that a GELI pool is treated as an unencrypted data set, so neither of those options will result in encryption at the destination.

I haven't found a way to make the replication task inherit encryption on the remote side, the way you can with the command line.

Looks like the GUI wasn't designed for this in mind, after all. Going from a non-encrypted source to a destination you wish to "encrypt and inherit all in one go" is either not possible via the GUI, or hidden behind some counter-intuitive combination of settings.

I got some pretty whacky results testing this with the GUI, and so I have to fallback to "it works via the command-line", just as @gary_1 explained above.

Even with the "Recursive" option, and choosing the source dataset (and its children) with "Encryption" checked for the destination option, only seems to encrypt the parent dataset, while leaving the children unencrypted on the destination. It's also not clear how selecting certain options and specific datasets translates into the command line (behind the scenes.) I cannot figure out which option (if any!) does the equivalent of "-x encryption". But then again, maybe it's not a priority from the design standpoint, since it might only be used for a "one-time migration to the new native ZFS encryption".

For example, when selecting a single dataset from the source, and selecting only the poolname for the destination, it complains about "cannot overwrite an encryption root". But then I tried it again by selecting multiple children nested under a parent on the source side, and it dumps them one level below the selected destination when the task completes. So choosing multiple source datasets implies "nest these underneath the selected destination", yet choosing a single source dataset implies "try to overwrite the destination!"

I never found the Replication Tasks GUI to be clear, let alone intuitive nor self-explanatory. It's a shame, because I believe it's ideal to avoid using the command-line as much as possible on a graphical appliance.

 

[winnielinnie]

For now, I've gotten around it by setting up a replication task for each dataset, checking the 'Encryption' box, setting a passphrase, and after the first time the task completes, I go into the new dataset and change its encryption properties to 'Inherit'. After that, the subsequent replications maintain the inherited encryption of the target dataset. But figuring this out took a decent amount of trial and error.

To me that defies expectation! It takes some real digging and fiddling around to figure that out.

The user guide mentions GELI to ZFS migrations briefly, and says that it will soon be possible with an Advanced Replication Task.
https://www.truenas.com/docs/core/storage/pools/storageencryption/#geli-pool-migrations

But that feature appears to have been implemented as of 12.0-U1. The documentation hasn't been updated though. Which leaves people like me resorting to forum posts like this, or internet searches to see how other people have managed to do it. This should be a straightforward, documented process. Maybe even a wizard for non-power users (this is an appliance after all, right?)

My feelings are reflected by your experiences. I just learned to "work around it" in the meantime and call it a day. There seems to be a mismatch between the documentation and the final product. I don't even know where to start, since the GUI and tooltips don't really explain the settings clearly. I haven't the energy to make bug report after bug report, since I don't think in such a compartmentalized manner. To me this is an "over-arching" issue, all inter-related.

 

[winnielinnie]

After that, the subsequent replications maintain the inherited encryption of the target dataset. But figuring this out took a decent amount of trial and error.

My concern with that, upon second review, is that what if an update later down the road affects it in such a way that it breaks the inheritance on the destination side, and you forgot the temporary (and supposedly) one-time password you made a while back?

 

[storage-junkie]

My concern with that, upon second review, is that what if an update later down the road affects it in such a way that it breaks the inheritance on the destination side, and you forgot the temporary (and supposedly) one-time password you made a while back?

Well, my solution to that is my passphrase for all these matches my root dataset encryption key :)

But I am admittedly confused by how you can simply uncheck a box and 'inherit' the encryption properties of a parent dataset, after it's already been encrypted with a (potentially different) passphrase. What is going on there? The new encryption confuses the hell out of me.

Why did they have to get rid of whole-partition encryption again?? I've come very close to just doing a 'replace' of each of the disks in my old pool (since those disks were getting old) and leaving it as legacy. The new stuff makes my head hurt, and worried that I've potentially left something 'in the clear' by accident.

 

[winnielinnie]

But I am admittedly confused by how you can simply uncheck a box and 'inherit' the encryption properties of a parent dataset, after it's already been encrypted with a (potentially different) passphrase. What is going on there?

The Master Key, the Cipher, the Key Width (128, 192, or 256) and the data itself are left untouched when you do this; even if you go back and forth a million times, over and over.

All it is doing is re-encrypting the (unchangeable) Master Key with your new passphrase / HEX string (which requires the dataset to be currently unlocked, since it needs the Master Key itself to do this), and this new "obfuscated" Master Key replaces the old one in the header. Now when you try your old passphrase or HEX string, it will no longer work on the newly saved obfuscated Master Key: only the new passphrase / HEX string will work. (This is why you can "Inherit" from a parent that uses a different Master Key, Cipher, and Key Width, since those things aren't affected.)

"Yeah but what if I manually use the same passphrase for all my datasets, and intentionally uncheck Inherit? How come I can't unlock them all at once? Why do I have to explicitly unlock them individually with the same passphrase? Don't they all use the same passphrase anyways?"

You can settle for that, but you lose out on another feature of "Inherit encryption" that happens quietly behind-the-scenes: the property encryptionroot

When datasets share the same encryptionroot, it is only the encryptionroot that needs to be unlocked: the rest will likewise simultaneously unlock in a seamless fashion. Locking the encryptionroot will also lock all datasets in this group simultaneously.

You can deduce which datasets are part of the same encryptionroot: they all are missing a padlock icon, but if you follow upwards with your eyes, you will bump into a higher dataset with a padlock icon. This is the encryptionroot of all the lower datasets.

To confirm this, you can also list out what encryptionroot your datasets belong to:
zfs get -r -t filesystem encryptionroot poolname
Look under the VALUE column.

 

[storage-junkie]

How things are looking currently:

Code:

NAME                            PROPERTY        VALUE    SOURCE
tank2                           encryptionroot  tank2    -
tank2/dataroot                  encryptionroot  tank2    -
tank2/dataroot/tmp              encryptionroot  tank2    -
tank2/dataroot/tmp/backup       encryptionroot  tank2    -
tank2/dataroot/tmp/scratch      encryptionroot  tank2    -
tank2/dataroot/tmp/software     encryptionroot  tank2    -
tank2/dataroot/tmp/timemachine  encryptionroot  tank2    -

They all appear to be inheriting from the top-level dataset. And there is only one padlock at the top for tank2. I'll be keeping an eye out to see if this changes after a few days of replication.

At some point, I'll probably break the inheritance at the dataroot level (based on your suggestion), so that I can protect everything beneath it with a passphrase instead of a key. But that wouldn't happen until I have fully switched over to using this new pool as the primary.

 

[storage-junkie]

As a followup to this thread - I found a way through the UI to accomplish this. The trick is to UN-CHECK the 'include dataset properties' on the advanced replication task. This box is checked by default. Doing this allows you to replicate a dataset (and all its children, if 'recursive' is checked), from a GELI encrypted pool to a new zfs encrypted pool OR vice-versa. You don't have to set any of the encryption options on the replication task. Simply un-checking the 'include dataset properties' checkbox causes the datasets to inherit the properties of the target datasets once they land at their destination. I confirmed this works by migrating my old pool to a new set of disks, and then by replicating the new pool to my offsite server (which is still using legacy GELI encryption).

I set up a new periodic snapshot task on the new pool, using the same naming convention as the snapshots that had been migrated. Doing that allowed me to continue replicating to my offsite server without having to re-send everything.

It's been running this way for about a week now, and I've re-pointed my shares to the datasets on the new pool and added new data to it and confirmed that it's made it to the offsite just fine. I have a top-level share on that server that lets me browse the filesystem, and my new data shows up and I can open it, as well as older data. My snapshots are aging out according to the policy I set, on both my primary and offsite. Everything appears to be working.

I'm keeping the old pool around for a while, as an additional backup just in case. But I'll eventually export that pool and decommission the disks. Hopefully this thread helps someone else accomplish this much more easily. Myself and many others have spent a lot of hours with trial and error trying to figure this out. It's a shame the documentation isn't more clear about how to do this.

 

[winnielinnie]

The trick is to UN-CHECK the 'include dataset properties' on the advanced replication task. This box is checked by default. Doing this allows you to replicate a dataset (and all its children, if 'recursive' is checked), from a GELI encrypted pool to a new zfs encrypted pool OR vice-versa. You don't have to set any of the encryption options on the replication task. Simply un-checking the 'include dataset properties' checkbox causes the datasets to inherit the properties of the target datasets once they land at their destination. I confirmed this works by migrating my old pool to a new set of disks, and then by replicating the new pool to my offsite server (which is still using legacy GELI encryption).

Myself and many others have spent a lot of hours with trial and error trying to figure this out. It's a shame the documentation isn't more clear about how to do this.

This, among other reasons, is why I went back to using custom scripts invoked by Cron Tasks, rather than the official Replication Task GUI. I really, really want to only use what's available in the GUI, considering that this is an appliance and not a tinkerer's OS. However, it's for reasons you mentioned, as well as a glaring issue in this bug report which will not be fixed in TrueNAS Core, are why I had to ditch the GUI, once again, and use my own custom (and very ugly) replication scripts.



* If the bug report does not load for you, you can get the gist of the issue in this thread I created:

Replication Task "Full Filesystem Replication" does not send intermediate snapshots?

 

[jasonks]

I just went through this process to move from a GELI-encrypted pool on older SSDs to a native-encrypted pool on NVMe drives. Thank you for the very detailed and VERY helpful how-to. The only hitch I ran into was that the GUI wouldn't unlock the pool when I uploaded the JSON key file on pool import, so I had to paste in the keys from within the file instead. Once I did that, everything worked exactly as expected and it persists across reboots.

 

[winnielinnie]

he only hitch I ran into was that the GUI wouldn't unlock the pool when I uploaded the JSON key file on pool import, so I had to paste in the keys from within the file instead. Once I did that, everything worked exactly as expected and it persists across reboots.

If it's working fine and all the datasets are currently unlocked, you can export your pool's dataset key(s) as a new .json file that should have the correct format.

Glad to hear you're up and running!