TrueNAS encryption and replication

[DD4711]

Hello,

I'm currently struggle with the new TrueNAS encryption and the following replication for backup purpose.

My plan is to use an encrypted pool with all datasets encrypted on my TrueNAS Server. This storage should be replcated to external HDDs for backup purpose, of course encrypted. In a desaster situation the backup pool with all the datasets would be replicated back to main system. This should be done as simple as possible (replicate from storage to backup and back again with all relevant settings for the datasets). So far so good, it was no problem in FreeNAS with Geli encryption. Create 2 encrypted storages, all data in these storages are encrypted, replication works transparent regarding encryption.

I build with TrueNAS a new storage called "storage" with the new encryption and set a passphrase. Then I created datasets with encryptopn options checked "Inherit (encrypted)". "Storage" appears with a lock synbol, the datasets not. But the datasets have "encryptiopn options". So they enjoy the same encryption as the parent "storage". OK fine.

Now I want to set up the replication. Create a new pool called "ext_hdd", encrypted with passphrase.

I'm going to make some tests:

1)
Create a new poll called "ext_hdd", encrypted with passphrase.
Mark one datasets for replication
Recursive: no
Include Dataset Properties: yes (as I understand, there's now also the encryption options saved"
Full Filesystem replication: no

Doesn't work, Error:
"Destination dataset 'ext_hdd' already exists and is it's own encryption root. This configuration is not supported yet. If you want to replicate into an encrypted dataset, please, encrypt it's parent dataset."

This Message is confusing. I need to create a dataset. In this case it has an encryption, yes. This config is not supported: hm, would be probably better to use "old" Geli before implementing a new unready encryption system?!
If you want to replicate into an encrypted dataset, please, encrypt it's parent dataset: I have no idea what this message wants to tell me. I replicate into the pool root. There's no parent i can encrypt.

2)
OK, next test:
Create a new pool called "ext_hdd", NOT encrypted
Mark one datasets for replication
Recursive: no
Include Dataset Properties: yes
Full Filesystem replication: no

Doesn't work, Error:
"Unable to send encrypted dataset 'Storage/temp' to existing unencrypted or unrelated dataset 'ext_hdd'."

3)
OK, go back to encrypted ext_hdd with passphrase.
Mark one datasets for replication
Recursive: no
Include Dataset Properties: no
Full Filesystem replication: no

Destination dataset 'ext_hdd' already exists and is it's own encryption root. This configuration is not supported yet. If you want to replicate into an encrypted dataset, please, encrypt it's parent dataset.

Same Error as in 1)

4)
OK, still encrypted ext_hdd with passphrase but I mark the whole "storage" for replication.
Mark the whole "storage" with all datasets for replication
Recursive: yes
Include Dataset Properties: yes
Full Filesystem replication: no

Replication works. But every single dataset has its own lock symbol. To unlock them I need to put the passphrase in all single datasets?! This can't be the right way...

5)
In another test (I currently can't reproduce it) the result was: ext_hdd had an lock symbol and all the replicated datasets too. I then had to put into all the datasets one by one the passphrase. That was for sure not correct, too. Apart from what happens if I replicate it back in a desaster situation...



I have no ideas anymore :(
What am I douig wrong? Is this that difficult? Do I misunderstand something?

Would be very glad to get som help setting it up.

Thanks!

 

[winnielinnie]

This is why I use my method of "pseudo" root datasets, among other reasons. Purely using the web GUI to replicate and backup with native ZFS encryption has been a headache since day one, and the documentation / tooltips are not clear in their explanations. You can see others are equally confused by this:

TrueNAS 12.0 GELI to Full Drive Encryption Migration

Replication tasks can be set to "Include dataset properties" or "Full Filesystem Replication", both of which should result in a native ZFS encrypted dataset replicating to a destination and retaining its encryption. The issue you likely face, is that a GELI pool is treated as an unencrypted...

www.truenas.com

Replication Task "Full Filesystem Replication" does not send intermediate snapshots?

Up until I upgraded to TrueNAS 12.0-U3, whenever I ran a manual Replication Task with the option "Full Filesystem Replication", everything would be sent to the destination, including intermediate snapshots since the last replication. Apparently, using "Full Filesystem Replication" has always...

www.truenas.com

I found that when you replicate a natively encrypted dataset, it's preferable to "nest" it underneath your target, rather than to (intuitively, you'd assume) "match" your target. Here is what I mean:

• main_pool<--- (encrypted, keyfile) *
  • .system <--- (inherited from main_pool)
  • iocage <--- (inherited from main_pool)
  • zdataroot <--- (inheritance break, passphrase instead)
    • archive <--- (inherited from zdataroot)
    • downloads <--- (inherited from zdataroot)
    • isos <--- (inherited from zdataroot)
    • legal <--- (inherited from zdataroot)

In the above template, you would replicate zdataroot with the options "Include Dataset Properties" and "Recursive" (or simply use "Full Filesystem Replication"), and it would be nested underneath the ext_hdd root dataset of your backup pool. When it completes, it would look like so on your backup pool:

• ext_hdd<--- (encrypted, keyfile) *
  • zdataroot <--- (inheritance break, passphrase instead)
    • archive <--- (inherited from zdataroot)
    • downloads <--- (inherited from zdataroot)
    • isos <--- (inherited from zdataroot)
    • legal <--- (inherited from zdataroot)

To send it from your backup to a new pool (such as recovering from a backup), you would do the same thing in reverse: replicate zdataroot to be nested underneath your main pool's root dataset.

Unfortunately, using the GUI can lead to errors, since "how" you select your source datasets changes the behavior of the receiving side's layout. In the above example, the GUI will complain, as it will try to overwrite ext_hdd on the destination. Yet using the command-line, you can easily tell it to nest beneath ext_hdd using the -d flag, like so:

zfs send -w -R main_pool/zdataroot@backup20210531 | zfs recv -v -d -F ext_hdd

Or an incremental send:
zfs send -w -R -I main_pool/zdataroot@backup20210531 main_pool/zdataroot@backup20210607 | zfs recv -v -d -F ext_hdd

The -w flag, combined with -R, will preserve the dataset's properties, including its encryption properties, and send the stream "as is", meaning it does not need to be unlocked before/during transfer, and will remain locked upon being received on the destination, requiring the same passphrase to unlock it.



* The top-level root datasets in both pools don't even need to be encrypted, as they only serve as "placeholders" essentially. In fact, leaving them non-encrypted is safer, since you don't need to keep a separate keyfile/passphrase safe, and you won't be saving any data directly within the root dataset. Think of the root datasets as "key ring holders" and the pseudo-roots as the actual keys dangling on the keyring. It also depends on whether you want iocage and .system to inherit encryption.

 

[DD4711]

Hello @winnielinnie ,

first thanks for the really great and helpful answer and for pointing me to the right threads.

I read a lot in the linked threads. After that and your answer I think that your method is what I have to do, too. I still can't believe what's going on with this "issue". I completely ack with your points of view. I also want to use the GUI with it's build in features when using TrueNAS... but I also can't see any other soultion.

Just to sum it up: I wish an easy and transparent implementation of encryption (as it was with GELI). I think this is really nothing special. All processes accessing encrypted data shouldn't care about how the data is saved. This applies also for a replication.

Anyway.

May I ask you two things before implementing your method? Maybe you know the answers...

1) .system and iocage
These datasets are encrypted with a key saved somewhere in the TrueNAS settings / database. This is same as unencrypted. At least for iocage this is no option for me. But it shouldn't be a problem to move iocage to zdataroot so that it is also using the passphrase encryption?

2) change main_pool to passphrase in your method
What would happen if I change main_pool keyfile-unlock to passphrase-unlock? I should be nevertheless able to brake this inheritance by setting another passphrase to zdataroot?! This would encrypt my main_pool datasets like .system and iocage and would be only unlockable by the passphrase (as it was with GELI)?

 

[winnielinnie]

1) .system and iocage
These datasets are encrypted with a key saved somewhere in the TrueNAS settings / database. This is same as unencrypted. At least for iocage this is no option for me. But it shouldn't be a problem to move iocage to zdataroot so that it is also using the passphrase encryption?

I'm not sure of the layout of your pool(s) and overall system. Is your System Dataset (.system) and Jails (iocage) located on a different pool, or are they located in main_pool? If you wish to have them on main_pool, then you cannot use a passphrase on the the top-level root dataset (main_pool), since .system needs to be readily available after booting up. TrueNAS web GUI will prevent you from passphrase-protecting the top-level dataset and/or automatically re-locate .system to another data pool or boot-pool. This is why using a non-encrypted (or encrypted with a keyfile) top-level root dataset is recommended. Just make sure to backup this keyfile and keep it safe.

As for iocage, you cannot choose where to nest it with TrueNAS. It is always placed directly under the top-level root dataset of whichever pool you select from the drop-down menu under Jails > cogwheel

It would have to be its own passphrase-protected dataset. You can use the same passphrase as with zdataroot, but it will be treated as a separate "encryptionroot".

This leads into question #2. You cannot choose where to nest iocage and .system. On top of that, you cannot even passphrase-protect .system, anyways. There's no way around this unless you want to have fun hacking away and risk breaking your system.

In your situation, if you don't care about intermediary snapshots (which have a different naming schema to your Replication Task), you can "trick" the Replication Task GUI into nesting your zdataroot (and all of its children) as well as iocage (and all of its children) directly under ext_hdd. This is because when you select more than one dataset from your source (and they are parallel to each other), the background process initiated by the Replication Task will run two different replications and nest them under your selected destination. Here is what I mean:

• main_pool <--- (non-encrypted or encrypted with keyfile)
  • .system <--- (inherited from main_pool)
  • iocage <--- (inherited from main_pool)
    • jails <--- (inherited from main_pool)
    • releases <--- (inherited from main_pool)
  • zdataroot <--- (inheritance break, passphrase instead)
    • archive <--- (inherited from zdataroot)
    • downloads <--- (inherited from zdataroot)
    • isos <--- (inherited from zdataroot)
    • legal <--- (inherited from zdataroot)
  • zdataroot-playground <--- (inheritance break, passphrase instead)
    • review <--- (inherited from zdataroot-playground)
    • temporary <--- (inherited from zdataroot-playground)

Let's say you only want to backup iocage and zdataroot to your ext_hdd pool. Obviously you don't want to backup .system, and zdataroot-playground is just a temporary holding bay of sorts which is not critical nor important enough to need backups. So in your Replication Task, you configure it as normal, but you make sure to select the following two datasets as your "source", and checking the option for "Full Filesystem Replication":

• [ ] main_pool
  • [ ] .system
  • [X] iocage
    • [ ] jails
    • [ ] releases
  • [X] zdataroot
    • [ ] archive
    • [ ] downloads
    • [ ] isos
    • [ ] legal
  • [ ] zdataroot-playground
    • [ ] review
    • [ ] temporary

For the "destination" simply choose ext_hdd. (Do not select "encryption". Selecting "Full Filesystem Replication" or "Include Dataset Properties" will use a "raw" send, so your encrypted datasets will replicate "as is".) After successfully running this task, the result should look like this on your external drive:

• ext_hdd
  • iocage
    • jails
    • releases
  • zdataroot
    • archive
    • downloads
    • isos
    • legal

I believe you could passphrase-protect iocage on the source (main_pool) and it will work the same. Otherwise it will be either non-encrypted or encrypted but requiring the same 64-character HEX string to unlock it on the destination. Using Replication Tasks under the GUI will also neglect to replicate any intermediary snapshots that do not follow the same naming schema as your task. (For example, a snapshot you might have manually created named @milestone-before-big-changes.) As for the top-level root dataset ext_hdd, if you do encrypted it, keep in mind you need to be able to unlock it if you wish to access the datasets below. Or you can just leave it non-encrypted since nothing will exist on that particular dataset, as it's only a "place-holder". Always test your backup to make sure you can unlock and access the datasets and files within.

 

[DD4711]

OK, I think I got it. It is similar to your method. This is what I've tested.

My storage layout (.system is on boot-pool)

Pool name 'Storage' (encrypted with passphrase)
- dataroot (encrypted with passphrase, but not inherit from parent!)
-- dataset1 (encrypted, inherit from parent)
-- dataset2 (encrypted, inherit from parent)
- iocage (encrypted, inherit from parent)
-- 'several ipcage folder' (encrypted, inherit from parent)

Pool name 'ext_hdd' (encrypted with passphrase)
'no datasets configured'

Replication task configured through GUI:
- source: Storage/dataroot,Storage/iocage
- dest: ext_hdd
- Full Filesystem Replication: on
- Synchronize Dest Snapshots with source: on

Comment: Replication only runs with option "Full Filesystem Replication", NOT with Recursive and / or Include Dataset Properties. I read somewhere that the option "Full Filesystem Replication" does not really make a full file system replication. Someone mentioned that data is missing. I have to check that.

With this I get the exact same layout under ext_hdd as under Storage. Great.

Importing the ext_hdd results in putting in three passphrases. That's OK.

Trying to restore: seems to work the same way around. BUT: iocage dataset is not anymore inherit from parent "Storage". I need to enter the passphrase of the parent "Storage" to unlock it.


I can live with that but in my opinion this is no normal procedure... hopefully it will be sometime better...

Thanks again for your help!

 

[winnielinnie]

Comment: Replication only runs with option "Full Filesystem Replication", NOT with Recursive and / or Include Dataset Properties.

"Full Filesystem Replication" includes "Recursive" and "Dataset Properties". That's why when you select it, the other two options disappear (since they are always included in a Full Filesystem Replication.)

I read somewhere that the option "Full Filesystem Replication" does not really make a full file system replication.

It's from this bug report (submitted by another user) and I explained it in more detail in this forum thread. Basically, using the official GUI's Replication Task will skip any intermediary snapshots on the dataset(s) that do not share your included naming schema, and especially not snapshots named without a parsable timestamp.


For example, you create a Replication Task that is paired with a Periodic Snapshot and/or manually specified naming schema of:
backup_%Y%m%d-%H%M%S


During an incremental replication, any intermediary snapshots since the initial full replication that do not match the naming schema of backup_%Y%m%d-%H%M%S will be skipped. If you had made a manual snapshot simply named this-is-important, it will not be replicated to the destination (ext_hdd) when the Replication Task runs. Not only does it not start with backup_, but it has no parsable timestamp (colored in red in my example.) Only the snapshots in the following sample list will be replicated:

• backup_20210601-000000
• backup_20210608-000000
• backup_20210615-000000
• ...and so on...

You can specify as many naming schemas as needed in your Replication Task under the field "Also Include Naming Schema", but keep in mind they need to follow a standard Unix time format, such as the example above (backup_%Y%m%d-%H%M%S). That's why it's a good habit to always append your snapshot names with a timestamp (which TrueNAS does by default, unless you intentionally remove the timestamp in the name.)


Keep in mind you will also need to include all the names you've used, so that none are skipped. So for example, maybe these are the only names you use for snapshots:

• auto_%Y%m%d-%H%M%S
• backup_%Y%m%d-%H%M%S
• migrate_%Y%m%d-%H%M%S
• manual_%Y%m%d-%H%M%S

In the above, all four will need to be added to "Also Include Naming Schema".

- Synchronize Dest Snapshots with source: on

I would leave this option unchecked, personally. The tooltip warns you about a possible consequence.

Importing the ext_hdd results in putting in three passphrases. That's OK.

You don't even need to unlock anything when you import ext_hdd. You can leave its datasets locked, even when replicating from main_pool to ext_hdd. The encrypted datasets are replicated "as is". However, checking from time to time to make sure you can access your data is a good habit regardless. But as odd as it sounds, you can plug in the external drive, import ext_hdd, leave everything under ext_hdd locked, and continue like normal with your replication. Even in such a scenario where you need to go in the reverse direction, to restore from your backup (e.g, from ext_hdd to new_pool), everything can remain locked. Obviously the datasets need to be unlocked to access the data (and for services / jails to work properly), but for replication, the "raw" streams are sent "as is" and are just as fast as replicating non-encrypted to non-encrypted.

Trying to restore: seems to work the same way around. BUT: iocage dataset is not anymore inherit from parent "Storage". I need to enter the passphrase of the parent "Storage" to unlock it.

In such a scenario, this would be a "one time thing" after restoring. You would change the dataset in question (such as iocage) to "Inherit" encryption. Then it will behave like it did on the original pool.


Please be very careful. There's a greater likelihood of forever losing access to your own data due to encryption than for encryption to protect you from a malicious party. That's why it should be treated with the utmost delicate touch. Don't rush, don't trip over yourself. Practice with it freely until you get enough confidence where you don't feel like there's "still a part I don't quite understand...". It's better to make destructive missteps on a test system while practicing rather than to curse yourself after falling victim to an unforeseen mistake with your valued data.