Questions about how not to make mistakes with encryption

[John Childermass]

Yes, I’m afraid this is another question about encryption! I’ve read a lot of the previous threads, but would like to check I’ve understood. I’m aware that encryption can lead to data loss if I do it wrong, so I’m trying to be careful.

I’m considering using the standard native ZFS encryption in TrueNAS CORE 12 when setting up my pools, primarily so that I don’t have to worry when disposing of drives. I therefore think I’d be happy using keys instead of passphrases. I know this wouldn't help me if someone steals the whole system.

1. As long as I download the json file from “Export Dataset Keys" and keep this file somewhere secure such as my password manager, does this protect me from the main risk of losing my encrypted data, i.e. losing the keys? Is this file all I need to regain access to those datasets (e.g. if I have to put the drives in a new system)?
2. I noticed that not all encrypted datasets are mentioned in the json file. Is that because if a dataset simply inherits its encryption from the parent, no extra key is needed?
3. I’ve read that the keys are stored in the system dataset. I assume this means that the system dataset itself cannot be encrypted. Does that mean that I should move the system dataset to the boot drive and not keep it on a storage pool, because otherwise if I dispose of those storage disks the encryption keys would potentially be accessible on the disks alongside the datasets that they decrypt?
4. Are there any other classic "beginner assumes encryption is automatically a good thing and ends up losing data" pitfalls that I should be aware of?

Thank you in advance!

 

[Arwen]

I don't have many answers, but some comments:

• Have a backup of your data, (or 2 backups!)
• Your expectations seem sound:
  • Done wrong, ZFS encryption can lead to data loss
  • Using ZFS encryption for drive disposal, (at rest encryption does not help against network exploits)
  • ZFS encryption using keys won't help if someone steals the hardware

It does sound like you want to encrypt the whole pool and any datasets below it. Though it is possible to limit encryption to specific ZFS datasets that use private information.

So test after enabling ZFS encryption. Try a reboot or 2. Try replacing the OS boot drive, (to simulate OS boot drive failure).

 

[winnielinnie]

1. As long as I download the json file from “Export Dataset Keys" and keep this file somewhere secure such as my password manager, does this protect me from the main risk of losing my encrypted data, i.e. losing the keys? Is this file all I need to regain access to those datasets (e.g. if I have to put the drives in a new system)?

In a sense, "yes". But it still depends on you not losing the .json file. If you lose it, or the keyfile becomes corrupt, you're out of luck. They keyfile is really just a basic text file that contains a 64-character HEX string for each encryptionroot. You can literally copy + paste this long string to unlock the dataset. (Two methods are acceptable to unlock: (1) copy + paste the string or (2) upload the .json file

If you need the dataset(s) to be available at boot up (automatically unlocked) and/or your System Dataset lives on this pool, your top-level root dataset cannot use a passphrase. This is not a ZFS limitation: it's by design for TrueNAS specifically, since it stores the keyfile on the boot drive, which it automatically imports at boot time to automatically unlock and make accessible the System Dataset. Technically, we should be able to manually lock/unlock over and over again datasets protected with a keystring (as we can with a passphrase), but for some reason the GUI doesn't allow this. And the predicament of the System Dataset (which must be immediately available) adds to the complexity.

2. I noticed that not all encrypted datasets are mentioned in the json file. Is that because if a dataset simply inherits its encryption from the parent, no extra key is needed?

It's due to encryptionroots. With ZFS, you only ever "unlock" encryptionroots. Any datasets part of the same enryptionroot family are simultaneously unlocked/locked. You can view your encryptionroots like this:
zfs list -r -t filesystem -o name,encryptionroot poolName

A "common" scenario is the top-level root dataset is encrypted, and is your only encryptionroot, with every nested child inheriting this.

3. I’ve read that the keys are stored in the system dataset. I assume this means that the system dataset itself cannot be encrypted. Does that mean that I should move the system dataset to the boot drive and not keep it on a storage pool, because otherwise if I dispose of those storage disks the encryption keys would potentially be accessible on the disks alongside the datasets that they decrypt?

No and no. They're stored on the boot-pool (boot drive). The System Dataset can be an encrypted dataset. However, it can only use a keystring/keyfile (not a passphrase). And TrueNAS doesn't let you control this with the GUI. In fact, it always inherits from your top-level root dataset; which means your top-level root dataset cannot be protected with a passphrase, if it will house the System Dataset. (This is to allow it to be immediately unlocked and available automatically upon bootup.)

If you want this to be a non-issue, just put the System Dataset on your boot-pool. This simplifies everything else. (As long as your boot-pool is not a USB stick.) Now you can freely use a passphrase instead of a keyfile, without worrying about the System Dataset, nor having a keyfile on the plain (non-encrypted) boot drive.

4. Are there any other classic "beginner assumes encryption is automatically a good thing and ends up losing data" pitfalls that I should be aware of?

Favor passphrase over keyfile, whenever applicable. You can always "remember" a passphrase, but you will be hard-pressed to "remember" a 64-character HEX string that you probably never even read once. While you can "lose" or forget to copy a keyfile/keystring, and while it must exist somewhere (on a USB stick, laptop drive, cloud storage, etc), a passphrase lives in your mind, and can't be accidentally deleted.

If you do use a keyfile/keystring, make multiple, safe copies of it, and even open it up in a text editor and copy + paste the 64-character HEX string(s) into a secure password manager. If you lose the file or forget where you saved it, you can still use the string(s).

Inspecting the .json file in a text editor, you will see it only lists the encryptionroot datasets, and the HEX string for each one.

Also, don't forget to re-export your keyfile and re-paste your HEX strings any time you make changes to the encryption properties and/or break "inheritance" with child datasets.

 

[John Childermass]

Thank you very much @Arwen and @winnielinnie ! This is all very helpful and I think I know how to proceed. I'll definitely test replacing the OS boot drive, too.

I’ve read that the keys are stored in the system dataset. I assume this means that the system dataset itself cannot be encrypted. Does that mean that I should move the system dataset to the boot drive and not keep it on a storage pool, because otherwise if I dispose of those storage disks the encryption keys would potentially be accessible on the disks alongside the datasets that they decrypt?

No and no. They're stored on the boot-pool (boot drive). The System Dataset can be an encrypted dataset. However, it can only use a keystring/keyfile (not a passphrase). And TrueNAS doesn't let you control this with the GUI. In fact, it always inherits from your top-level root dataset; which means your top-level root dataset cannot be protected with a passphrase, if it will house the System Dataset. (This is to allow it to be immediately unlocked and available automatically upon bootup.)

This part was still a bit surprising/confusing to me. In the docs for the System Dataset it says "The system dataset stores debugging core files, encryption keys for encrypted pools, and Samba4 metadata such as the user and group cache and share level permissions" (emphasis mine). Are the docs wrong, or am I misunderstanding something here?

Later on that page it says "Moving the system dataset to an encrypted pool disables that volume’s passphrase capability" which, as you say, implies that it can still be encrypted with a key. I was already confused by this part because I didn't understand how the System Dataset could be encrypted (even with a key) if those encryption keys are stored on the System Dataset!

 

[winnielinnie]

I was already confused by this part because I didn't understand how the System Dataset could be encrypted (even with a key) if those encryption keys are stored on the System Dataset!

Exactly. If it was stored on the System Dataset, but the System Dataset is locked, how would TrueNAS be able to automatically unlock the System Dataset if the keys are stored in this still locked dataset? (Chicken and egg.)

The keys are embedded inside the SQLite file /data/freenas-v1.db, which lives in plain non-encrypted space, on the boot drive.

Are the docs wrong, or am I misunderstanding something here?

Wouldn't be the first time...

 

[John Childermass]

Exactly. If it was stored on the System Dataset, but the System Dataset is locked, how would TrueNAS be able to automatically unlock the System Dataset if the keys are stored in this still locked dataset? (Chicken and egg.)

The keys are embedded inside the SQLite file /data/freenas-v1.db, which lives in plain non-encrypted space, on the boot drive.

Thanks! So is that part of the official documentation just wrong, then?

 

[winnielinnie]

Thanks! So is that part of the official documentation just wrong, then?

I've filed bug reports against incorrect documentation and tooltip descriptions (of the GUI itself.) There's been a handful lately.

• Ambiguous description of SSH cipher options
• Incorrect description of encryption during new pool creation
• Incorrect description of checksum property (and it's relation to Dedup)
• ...and now this: incorrect documentation about where the keys for encrypted datasets are stored

 

[John Childermass]

Thank you very much again! I'll start with key-based encryption for now, but if I understand correctly I can switch between keys and passphrases later — as long as I start with encryption (of either kind) when creating the datasets.

I have some datasets and zvols on my old pool, which were created as unencrypted. When I transfer them over to the new pool, are the following steps sufficient to get them into the new pool, correctly encrypted?

Let's say I have "original-pool" which is all unencrypted. I would create an encrypted pool on the new disks, as "new-pool":

• original-pool (FILESYSTEM, unencrypted) (this is the automatically created root dataset of the pool)
  • media (FILESYSTEM, unencrypted)
  • vms (FILESYSTEM, unencrypted)
    • hostname1-abcdef (VOLUME, unencrypted)
    • hostname2-ghijkl (VOLUME, unencrypted)
• new-pool (FILESYSTEM, created as encrypted) (this is the automatically created root dataset of the pool)
  • nothing in here yet

I think the steps are:

1. In the GUI, create a snapshot of the "original-pool" dataset (let's name it "for-replication"), Recursive must be checked.
2. In the shell (I don't think this is possible from the GUI):
   1. zfs send -v -R original-pool/media@for-replication | zfs recv -x encryption new-pool/media
   2. zfs send -v -R original-pool/vms@for-replication | zfs recv -x encryption new-pool/vms

The -R is to replicate everything below the specified dataset, e.g. include the zvols under "vms". The -x is to make the replicated datasets and zvols inherit the encryption settings at their destination (otherwise they are unencrypted at the destination).

I've tested this and it seems to work, but am I missing something that will come back and bite me later? The destination datasets/zvols show in the GUI that they are inheriting encryption settings from their parent. When I export the encryption keys json there's only one key in the file, for "new-pool", which is what I expect if the inheritance is working.

By the way, in the GUI the only way I found to tell if a dataset or zvol is encrypted is to click on the three vertical dots and see if "Encryption Actions"/"Encryption Options" appears in the menu or not (if it appears, then it is encrypted). Is there a better way?

 

[winnielinnie]

By the way, in the GUI the only way I found to tell if a dataset or zvol is encrypted is to click on the three vertical dots and see if "Encryption Actions"/"Encryption Options" appears in the menu or not (if it appears, then it is encrypted).

The way they designed the TrueNAS GUI (i.e, the "Pools" page), is to only denote encrypted or plain (with an icon) whenever there is a "break" in inheritance or encryption property.

When the top-level root dataset is encrypted, look downwards until you see another padlock icon. This is either where a new encryptionroot begins (with another padlock icon) or when a non-encrypted dataset begins (with a "anti" padlock icon). Otherwise, if the lower child inherits from above (regardless of encryption), there will be no icon.

In my opinion, I think there should have been three different icons, to make things more clear:

• A gold and/or large padlock icon, to denote an encryptionroot
• A normal and/or small padlock icon, to denote an encrypted dataset (but one that inherits from the encryptionroot)
• A blank and/or "anti" padlock icon, to denote a plain (non-encrypted) dataset

Then every dataset can clearly have an icon, and you wouldn't have to remember or "infer" or guess.

Is there a better way?

As I alluded to it in my previous post. You can list all the encryptionroots in a pool.
zfs list -r -t filesystem -o name,encryptionroot,encryption mypool

I've tested this and it seems to work, but am I missing something that will come back and bite me later?

Looks like it's working.

Only thing I would add is to use "-s" on the receiving side, in order to generate a "resume token". The dataset will update its resume token every 10 seconds or so, which means you can resume where you left off if there was an interruption or crash.

 

[John Childermass]

Thanks, @winnielinnie. Yes, I agree that the padlock icons in the GUI are really confusing. Something like your suggestion would be helpful. Luckily the zfs list command you suggested is a good way to see what's going on more clearly.

I did the replications as described above (including the "-s", but the transfers weren't interrupted), and everything went smoothly!