Can't ZFS Send between encrypted pools

[Sprint]

So this is off the back of another thread which I've marked as solved, but here's the brief history.

I decided I wanted to encrypt my pools, incase I ever need to send a drive off for repair or RMA.

I started with two Pools
Primary_Array = no Encryption
Secondary_Array = Legacy Encryption

I started by moving everything off of the Secondary_Array onto the Primary_Array, so I could re-encrypt the Secondary using the new Encryption system in TrueNAS. This was done using:

This takes the entire dataset and its snapshots with it.

Once I'd moved all the data off the Secondary_Array, I tore down the pool, and recreated it using the new encryption options.

When I tried to copy everything back to the newly encrypted Secondary, I discovered that I needed to use the "-x encryption" tag to my zfs send commands in order for it to work.

This again works fine, and takes all the snapshots with it.

Once the Primary_Array was clear, I did a full DD wipe on all 8 drives, and re-creating the pool with the same new encryption method. I'm now looking at move my data back into their final homes, which means moving around 20Tb's of Datasets back to the Primary_Array, and this is where it goes wrong.

When I attempt to zfs send datasets from the Secondary_Array back to the Primary_Array (both encrypted using the same method now) I'm getting errors, and I'll be honest, I'm a little out of my depth, and a little nervous as my backup server took a nose dive this morning :(

This is what I see....
root@Nitrogen[~]# zfs send -Rv Secondary_Array/Public@copy2 | zfs receive -F -x encryption Primary_Array/Public
cannot send Secondary_Array/Public@copy2: encrypted dataset Secondary_Array/Public may not be sent with properties without the raw flag
warning: cannot send 'Secondary_Array/Public@copy2': backup failed
cannot receive: failed to read from stream

I've tried it both with and without the -x flag, but the error is the same.

It DOES work if I remove the -R tag at the start, but then it ONLY takes the snapshot named in the command (@copy2), and not the earlier snapshots.

What am I doing wrong?

(as a side question for bonus points, when I send some datasets over, I also want to change the compression, is there a flag todo that?)

Thanks all in advanced :)

Sprint

 

[winnielinnie]

You can't use -R without -w for a native-encrypted dataset recursive send (all snapshots). -R includes all properties of the dataset, and this is only possible to keep in tact with a raw stream, -w. Using -w means you're sending the data "as is", block for block, all properties "as is", including encryption and compression. Every single block is perfectly replicated. You can later change the newly copied dataset's compression option after-the-fact, but it will only apply to newly saved / modified files; not existing ones.

Otherwise, you would have to drop the -R and -w flags, decrypt the streams on-the-fly (using the key), and re-encrypt it on the destination, as well as apply a different compression option.

I have not done nor tested this myself, so someone who already has can hopefully drop in what command and options should be used.

In the meantime, you might find this helpful, since 12.x I've started to use "pseudo-roots" on all of my pools: Pseudo-roots method, for TrueNAS 12+

 

[winnielinnie]

If you prefer, you can also try this:
zfs send -Rwv Secondary_Array/Public@copy2 | zfs recv -F -d Primary_Array

This will send the dataset Public (from Secondary_Array) "as is", with everything, including all snapshots; and once complete it will live nested under the top-level root dataset Primary_Array. Afterwards, you can tell it to inherit its encryption options, which will have it adopt the keystring/passphrase from Primary_Array. (The dataset Public cannot exist on the destination prior to this send/recv operation, or it will refuse to continue.)

Keep in mind, you can change certain encryption properties as many times as you want without needing to destroy any data or re-encrypt anything. See this post: https://www.truenas.com/community/threads/help-moving-pool-to-new-disks.89555/post-620034

However, if you leave the default options when creating new datasets, it will by default "inherit" its parent's encryption options: when the new dataset is created it will generate a new Master Key for itself, inherit the Cipher of the parent (e.g, AES GCM), inherit the Key Width of the parent (e.g, 256-bit), inherit the Keyfile or Passphrase of the parent, and inherit the Iterations of the parent.

The following CANNOT be changed later, at any time:

• Master Key
• Cipher
• Key Width

The following CAN be changed after the dataset's initial creation, without affecting the saved data, files, and folders, simply by unchecking the "Inherit" box for the dataset's encryption properties:

• Keyfile
• Passphrase
• Iterations

 

[Sprint]

Hi Winnie

Thanks for coming back so soon. I tried the -w flag, and it does indeed copy the entire dataset (snapshots included), however the dataset on the Primary_Array appears locked? I tried unlocking with both the key for the Primary and Secondary array, but without success.

This is the command I used, does this look correct?

zfs send -Rvw Secondary_Array/Public@copy2 | zfs receive -F Primary_Array/Public

(Haven't had a chance to read your Pseudo-roots method article yet but will certainly do so, think I'll leave compression for now, as I really doubt I'll gain much from it.)

 

[winnielinnie]

I tried unlocking with both the key for the Primary and Secondary array, but without success.

Since it is under a different pool, the .json file (which contains the keystrings for each encryptionroot dataset) will not match the new names. You can manually modify the .json file in a text editor, or just copy + paste the 64-character HEX string to unlock Public. After doing so, change Public encryption options to "inherit", so that it will use the same keystring/passphrase as Primary_Array, and it will be part of the same "encryptionroot". (This can only be done when it is currently unlocked.)

Every time, after you change an encryption property, enable/disable "inherit", change a keystring/passphrase, or any such similar action, make sure to backup the pool's encryption key file (.json) and save it somewhere safe and accessible.

Locate Pool > Cogwheel > Export Dataset Keys

For reference, check out this post why using the same .json file will not work for a new pool (different names, different nested layouts, different "encryptionroots"): https://www.truenas.com/community/threads/help-moving-pool-to-new-disks.89555/post-620656

 

[Sprint]

Since it is under a different pool, the .json file (which contains the keystrings for each encryptionroot dataset) will not match the new names. You can manually modify the .json file in a text editor, or just copy + paste the 64-character HEX string to unlock Public. After doing so, change Public encryption options to "inherit", so that it will use the same keystring/passphrase as Primary_Array, and it will be part of the same "encryptionroot". (This can only be done when it is currently unlocked.)

Every time, after you change an encryption property, enable/disable "inherit", change a keystring/passphrase, or any such similar action, make sure to backup the pool's encryption key file (.json) and save it somewhere safe and accessible.

Locate Pool > Cogwheel > Export Dataset Keys

For reference, check out this post why using the same .json file will not work for a new pool (different names, different nested layouts, different "encryptionroots"): https://www.truenas.com/community/threads/help-moving-pool-to-new-disks.89555/post-620656

So I've tried this on a smaller dataset, and Bingo, so far its work, bigger dataset now moving over, will repeat and retest, but confident it'll work :) Am I right in my understanding that once I unlock it, and change it to inherit the key from the parent, I should never need to use the other pools key again for that dataset? It should fall under the master key of the parent? (Don't want to have to unlock half a dozen datasets after a reboot or something). When I add a dataset to the parents encryption, the parents key shouldn't change should it?

Thanks again for helping me get this sorted :) Greatly appreciated.

 

[winnielinnie]

Am I right in my understanding that once I unlock it, and change it to inherit the key from the parent, I should never need to use the other pools key again for that dataset?

That's exactly correct! Make sure to Export Dataset Keys from Primary_Array once you do all of that. (It's relevant to encrypted datasets that use a 64-character HEX keystring; yet irrelevant to those that use a passphrase since you are responsible to "remember" the passphrase).

There is one (possible) caveat to your question, however. Doing another replication, from Secondary_Array, even incremental, will reset the dataset's properties on the target (Primary_Array). But if the Secondary_Array pool will never again be used as a "source" for a send, then you don't need to worry about it. Sounds like Secondary_Array is only being used for this migration transition, after which Primary_Array will be the main pool, and thus the "source" for future replications / copies to backup pools?

It should fall under the master key of the parent? (Don't want to have to unlock half a dozen datasets after a reboot or something).

I think I know what you are asking, but even as far back as early FreeNAS versions, terms used for encryption have been confused and mixed up. * I'll explain further. But yes, it will use the inhertible encryption properties of the parent: HEX keystring (or passphrase) and iterations, as well as "relate" to its parent, sharing the same "encryptionroot". (Refer to my post above with the quoted red and green text.)

When I add a dataset to the parents encryption, the parents key shouldn't change should it?

Correct. Unless you manually change it at a later time, in which the same change will happen (under-the-hood) to the children that have the option "Inherit" enabled for their encryption properties.


* I don't always use the official terminlogy, and when it comes to native ZFS encryption I try my best to make it clearly distinct "what is what". I prefer terms that reflect what something actually "does", such as a "User Key" being the user's responsibility.

• Master Key
  • This is the key used in the symmetric block encryption algorithm which actually encrypts/decrypts your data.
  • Can be 128-bit, 192-bit, or 256-bit (default)
  • Randomly generated every time you create a new encrypted dataset, even when creating a new child that inherits encryption properties
  • Stored in the dataset's metadata in an encrypted format (useless for attackers)
  • Decrypted using another seperate key (User Key) provided by the user (HEX keystring or passphrase)
  • Cannot be changed at any later time after dataset creation (nor would you ever want to, nor is it even possible)
  • Plays no relevant role in how securely it protects your data from unwanted access (the User Keys do, however)
• User Key
  • This is the key used to decrypt the actual Master Key, which is stored in an encrypted format in the dataset's metadata
  • Can be in the form of a keystring (64-character HEX, exportable) or passphrase (memorized by the user)
  • The stronger and more complex, the better it protects the Master Key from being decrypted (and hence, exposing your data using the now decrypted Master Key)
  • The User Key is fed into a separate algorithm (*Passphrase-Based Key Derivation Function) to decrypt the Master Key
  • Can be changed as many times as you want, as long as the dataset is currently unlocked (this action requires access to the decrypted Master Key in order to cryptographically relate them together)
• Encryptionroot
  • This is the encrypted dataset that serves as the encryption parent to other related encrypted datasets further down
  • When a child dataset "inherits" its encryption properties, it essentially sets its "encryptionroot" property to reflect the parent dataset
  • If a child dataset "breaks" encryption inheritance, it essentially becomes its own Encryptionroot, and thus children further down the nest that "inherit" encryption will treat this as their "encryptionroot"
  • In the TrueNAS GUI, an Encryptionroot can be identified by the padlock icon (however, not to be confused with the crossed-out padlock icon, which symbolizes "not encryped", such as Pluto_Replication in your screenshot)
  • An Encryptionroot can be viewed as an "encryption group": all datasets that have the same "encryptionroot" are part of the same "encryption group" (or "family", if you prefer)
  • Only the Encryptionroot is specifically "unlocked", whether with a HEX keystring or passphrase: the other datasets in this "family" are unlocked automatically upon unlocking the Encryptionroot
  • What is shared among all datasets in an Encryptionroot family are: User Key, Iterations, and being treated as an "unlock family"
  • What is not explicitly shared (as they can differ) are: Master Key, Cipher, and Width (i.e, 128, 192, 256 bit)
  • **To view the Encryptionroots: zfs get -r encryptionroot Primary_Array | grep -v @
• Key File (Keystring List)
  • Exportable and importable text file (.json format and extension) that contains the 64-character HEX keystrings for each Encryptionroot in a pool
  • This is the file you save when you Export Dataset Keys (don't lose this file, and make sure it's up-to-date, and multiple copies isn't a bad idea either!)
  • Does not contain any passphrases for Encryptionroots that use a passphrase instead of HEX keystring (user must memorize those instead)
  • Access to this Key File (Keystring List) must be safeguarded and protected from unauthorized users, while also accessible to the admin user if they ever export/re-import a pool
  • The 64-character HEX keystrings within can also be arbitrarily copied and pasted when unlocking datasets
  • Losing this file (or forgetting to export a recent version of it) is equivalent to forgetting a passphrase: your encrypted data is gone forever
  • The one exception to the above rule is if your system is currently running and the datasets are still unlocked: you can still export the Key File (Keystring List) and/or make changes to the encryption options of the Encryptionroots

* Yes, I know it's technically "Password-Based Key Derivation Function", but a pass phrase is more likely used than a single pass word.
** grep -v @ will exclude snapshots from the long list of results

 

[winnielinnie]

Thanks again for helping me get this sorted :) Greatly appreciated.

Most welcome!

I forget to ask, are you on TrueNAS 12.0-U1.1? Versions 12.0-RELEASE and 12.0-U1 contain a nasty bug that can crash your system when replicating encrypted datasets. I highly recommend upgrading to -U1.1 if you hadn't done so already.

 

[Sprint]

Most welcome!

I forget to ask, are you on TrueNAS 12.0-U1.1? Versions 12.0-RELEASE and 12.0-U1 contain a nasty bug that can crash your system when replicating encrypted datasets. I highly recommend upgrading to -U1.1 if you hadn't done so already.

Yeah U1.1, heard about that bug so updated both local and remote ends fairly sharpest! :D

I'll need to re-read the above detailed info a couple of times, but I think I understand the jist, and with repeated reading, it'll hopefully make more sense, although I may come back with some questions when I re-start my offsite replications after I've rebuilt the remove server and its array (Just ordered new disks for a 3rd vdev), but I think I'm in good shape. Once the local migration is done, I have no long term plans to frequently move datasets between pools. Files will move between but that'll be done at a SMB level so up the stack away from the low level encryption.

I did use to have a Snapshot replication task, which copied snapshots of a 3rd all SSD NON-encrypted array to the Primary_Array. When I re-enable that, I'm assuming it'll create an unencrypted dataset on Primary, which is fine for what's in that dataset?

Once again, thank you for taking the time to explain in such detail, very much appreciate it :)

 

[gary_1]

That's exactly correct! Make sure to Export Dataset Keys from Primary_Array once you do all of that. (It's relevant to encrypted datasets that use a 64-character HEX keystring; yet irrelevant to those that use a passphrase since you are responsible to "remember" the passphrase).

There is one (possible) caveat to your question, however. Doing another replication, from Secondary_Array, even incremental, will reset the dataset's properties on the target (Primary_Array). But if the Secondary_Array pool will never again be used as a "source" for a send, then you don't need to worry about it. Sounds like Secondary_Array is only being used for this migration transition, after which Primary_Array will be the main pool, and thus the "source" for future replications / copies to backup pools?

Are you sure about this?

I just did a raw send of backup_pool/home to tank/home which resulted in a "home" in both locations that I could unlock with my password "1234". I then changed the tank/home dataset to inherit, so now it unlocks with the password "secret". Then I did a new incremental snapshot on backup_pool/home and sent that across via "-w" and "-I" (tried this both sending to an unlocked and locked destination dataset, both worked the same) and it kept the same "secret" password for unlocking.

I'm not certain, but I would guess the master key that's encrypted using the derived password key, was sent over during the first zfs send. Since the master key can never be changed, there's no point sending after that. So any subsequent zfs send/recv won't impact changes you've made on either side for how you unlock the master key (ie whether you use a password, a key, change the pass, inherit etc) as that never gets sent over again.

Did you see a different outcome?

In case it helps the OP (or any future readers), here's how I did my migration which was slightly different:

tank/ (legacy GELI)
backup_pool/ (native zfs password unlock "secret1")

Code:

 
zfs snap -r tank@migration
zfs send -v -R tank@migration | zfs recv -F -d -x encryption backup_pool/server-backup

Once this completes, tank/home and all other datasets are now on backup_pool/server-backup/home etc The server-backup dataset can be unlocked using "secret1" as -x ensured it inherited encryption from backup_pool.

I then destroyed tank, made a new pool with a password "secret2" and sent each dataset ie "home" etc back over one by one

Code:

zfs send -R -w backup_pool/server-backup/home@migrattion | zfs recv -F -e tank
zfs send -R -w backup_pool/server-backup/services@migrattion | zfs recv -F -e tank
...etc

After that's done you can then switch the datasets on tank to inherit so they can unlocked along with the root dataset when you use "secret2". The backup_pool continues to be unlocked with "secret1". Both passwords could be the same, I just used different here for illustration.

Note, I used "-d" during backup but "-e" during restore to avoid getting "tank/server-backup"

 

[winnielinnie]

I just did a raw send of backup_pool/home to tank/home which resulted in a "home" in both locations that I could unlock with my password "1234". I then changed the tank/home dataset to inherit, so now it unlocks with the password "secret". Then I did a new incremental snapshot on backup_pool/home and sent that across via "-w" and "-I" (tried this both sending to an unlocked and locked destination dataset, both worked the same) and it kept the same "secret" password for unlocking.

Did you also use "-R" in that test?

-R includes all properties of the dataset

I'll have to experiment again to make sure, but until then I'll take your word for it (assuming you also used "-R" in that specific test.)

Note, I used "-d" during backup but "-e" during restore to avoid getting "tank/server-backup"

Noticed, and it's a smart way of using "-d" and "-e" depending how you want your datasets structured. The zfs send/recv documentation is worded in such a way that makes you re-read it over a few times, which can confuse users on how it applies to them.

I'm not certain, but I would guess the master key that's encrypted using the derived password key, was sent over during the first zfs send.

You mean from the very first send of the legacy pool to the new pool? I don't believe any existing Master Key is used or loaded, since from what I understand, every time a new encrypted dataset is created (such as its initial creation from the first send-recv), a new Master Key is randomly generated behind the scenes to encrypt the new data. From a security standpoint, it makes no difference to the end-user, as they only need to protect/remember the User Key used to encrypt/decrypt the Master Key, and thus access their data.

What is also available when using "-x encryption" on the target is the parent dataset's User Key, assuming the parent is currently unlocked. This available User Key is used to encrypt the new Master Key (of the dataset being created), which happens to be the same User Key as inherited from the parent (i.e, Encryptionroot).

It is quite possible that ZFS / TrueNAS simply recycles the same Master Key for such a process, but once again, it makes no difference to the end-user, nor has any real security implications.)

 

[gary_1]

Yes I used "-R", one of the datasets I tried with the incremental was iocage which had several nested children too.

You mean from the very first send of the legacy pool to the new pool? I don't believe any existing Master Key is used or loaded,

What I was thinking with the master key, is that if it didn't use the same master key for the dataset that's created during the initial -R -w send, then any subsequent incremental replication to that dataset whilst locked, would receive blocks it has no way to decrypt. Given that with -w a send/recv does not decrypt and re-encrypt the sent data, the source and destination must have an underlying key in common otherwise you'd end up with a dataset that has blocks encrypted with two different underlying keys.

from what I understand, every time a new encrypted dataset is created (such as its initial creation from the first send-recv), a new Master Key is randomly generated behind the scenes to encrypt the new data.

That was my understanding too, although when you consider that you can send a locked dataset to a locked dataset on the destination, that implies the underlying key for the sent blocks and the existing blocks at the destination are the same. Otherwise re-encryption would be needed and that wouldn't be possible in a locked to locked replication scenario.

My feeling is, if you create a new dataset, it will get its own/unique master key stored encrypted with your (pass or key based) user key. If however a dataset is created as part of a zfs send (raw) it will transfer the encrypted master key, thus your source user key would be required if you wished to unlock the backup dataset. At this point either side is free to change the user key/pass or inherit on their dataset independently, it won't matter as both sides now have a master key in common. This would also mean you could send/recv whilst both sides datasets are locked without any re-encryption being required as you can send the existing encrypted blocks. The other side can't decrypt them as there's no key available, but the blocks are encrypted with the same key as all the other blocks (due to common master key) so if/when the user did unlock the dataset, all the blocks old and new are decryptable.

I've assumed a lot in the above though and may have made some mistakes, so I'd be interested in hearing anything you find out on this if you run any tests or look into it further.

Edit: To clarify, when I say common master key, I'm referring specifically to each dataset sent in a zfs send raw. e.g

tank/parent/child
backup/parent/child

if tank was replicated to backup as above, then "tank/parent" and "backup/parent" would have common master key. "tank/parent/child" and "backup/parent/child" would also have common master keys but these would be different to "parents".

If you then created on each side (manually not via a zfs send) tank/parent/child2 I would now assume you would not be able to replicate child2 across to backup/parent/child2 as the two do not share a master key so you would be blocked with a dataset already exists error?