Confused by native ZFS encryption, some observations, many questions

[winnielinnie]

Is there any "smarts" in the system to know/prevent starting a jail or service until all datasets it depends on are mounted/unlocked?

Last I tried, the Jails will not auto-start (even if set to start automatically) if their configured mount points are not available (i.e, because the dependent dataset(s) is currently locked). Upon unlocking said datasets, which makes them available for the Jail's mount points, the Jail must be manually started; it will not autostart once the mount points are available.

I had already filed a similar bug report to this, though nothing has come of it yet.

 

[gary_1]

I'm going to be doing the migration from GELI to native later this week and wanted to double check my understanding based on the info in this thread.

Currently I have a server pool, GELI encrypted. I plan to shutdown all services, make a manual/recursive snapshot of the root dataset and then replicate this over to a "backup" pool that was created with native zfs encryption. When I do a recursive replication of the server pool's root dataset I need to do so with "-x" flag to ensure encryption is applied on the backup destination (inherited from destination root dataset)?

Likewise once I've wiped the server pool and wish to recursive restore from the backup pool, I should now send with the "-w" raw option to ensure all encryption properties are carried over from the backup pool to server?

Finally, more a hypothetical question than anything, it's my understanding that it's possible to send/recv encrypted datasets that are still locked?

The zfs docs note that when using -R you must use -w for encrypted datasets, but can you not do so and if so what happens? I assume it wouldn't be possible for the destination to end up with decrypted datasets given the source was still locked, would the replication just fail or would I end up with a replicated dataset that just isn't possible to unlock?

 

[winnielinnie]

I'm going to be doing the migration from GELI to native later this week and wanted to double check my understanding based on the info in this thread.
Currently I have a server pool, GELI encrypted. I plan to shutdown all services, make a manual/recursive snapshot of the root dataset and then replicate this over to a "backup" pool that was created with native zfs encryption. When I do a recursive replication of the server pool's root dataset I need to do so with "-x" flag to ensure encryption is applied on the backup destination (inherited from destination root dataset)?

For the first-time migration, yes. With GELI, the individual ZFS datasets are not really encrypted. Upon decrypting the underlying partitions that make up the vdevs, FreeNAS 11.x and earlier treats the ZFS datasets as if they are like any other datasets: there is no encryption property. Using "-x encryption", the recv'd datasets will inherit the encryption properties of the backup pool's root dataset (passphrase / keystring, cipher, key width).


In your case, you cannot replace the root dataset of the backup pool using native encryption. This is why I use "pseudo-roots". Not only can I get around this inherent limitation, but it allows for more modular control of multiple "roots" under a single pool. If you're not going to go this route, you still need to recursively send each dataset one level underneath the root dataset of the original legacy pool. Trying to send root to root will spit out an error.

Likewise once I've wiped the server pool and wish to recursive restore from the backup pool, I should now send with the "-w" raw option to ensure all encryption properties are carried over from the backup pool to server?

That'll do it! However, they will not inherit the encryption properties of the receiver's root dataset. Without pseudo-roots, they will each be treated as separate "encryptionroots", and thus cannot be unlocked together. You'll notice multiple padlock icons, one for each "encryptionroot".

Finally, more a hypothetical question than anything, it's my understanding that it's possible to send/recv encrypted datasets that are still locked?

Yup, with the -w flag. It's a nifty perk. It's also one of the reasons why native ZFS encryption does not hide snapshot and dataset names, let alone other properties and metadata. ZFS uses records and pointers with snapshots to send/recv. This is unlike traditional methods that are file-based. If you "rename" a bunch of files under a traditional file system, and then run another rsync, it will have to copy those files all over again because they appear to be missing from the destination! With ZFS, there would be nothing to send with a subsequent snapshot and send/recv operation. Just a simple update of some metadata, and done!

The zfs docs note that when using -R you must use -w for encrypted datasets, but can you not do so and if so what happens? I assume it wouldn't be possible for the destination to end up with decrypted datasets given the source was still locked, would the replication just fail or would I end up with a replicated dataset that just isn't possible to unlock?

It won't let you. You won't even get as far as transfering anything to begin with. The key must be sent with the stream so that the destination can decrypt as it receives and then you can leave it as is, or inherit its parent's encryption. I thought about playing around with that, but then I realize it's just extra work for the same result: an encrypted dataset on another pool. I'd prefer to just leave it as is and use the same passphrase/keystring to unlock it when I need to access it.


For reference, if I ever need to restore from a backup, such as a USB drive, I can simply send the "pseudo-root(s)" back to the main pool (on internal drives). The "top-level root" datasets on my pools are just place-holders; nothing more.

 

[gary_1]

If you're using pseudo roots, do you have anything at all under the root dataset other than your pseduo-root one? Freenas doesn't auto create/manage anything that you can't change to the pseudo-root instead? For example iocage or anything along those lines?

I take it there's no way to do a restore that includes the root dataset from the backup too, since you've to make the destination pool with at least the root dataset in advance?

I suppose unlocking root + pseduo dataset on boot via a pass isn't that much more effort than unlocking just one. Certainly preferable to having to unlock several datasets all off root. A way to multi-select then unlock in one go would have solved that problem though :)

One follow up question. If you have two child datasets under the pseudo root, then they'd be unlocked when the pseudo root parent is, if they share the same key or pass? What happens if one child dataset was restored from a pool that used a different key or pass? I assume it would have to be unlocked on its own then. Is there anyway to get it to start auto-unlocking with the pseudo root by changing the pass/key used on it to the same as the pseudo root?

 

[winnielinnie]

If you're using pseudo roots, do you have anything at all under the root dataset other than your pseduo-root one? Freenas doesn't auto create/manage anything that you can't change to the pseudo-root instead? For example iocage or anything along those lines?

Optionally the hidden .system (System Dataset) and iocage (for Jails.) Other than that, nothing directly lives under my top-level root dataset except for individual pseudo-roots, and everything else nested under each one. For example:

• mainpool <--- this only serves as a place-holder, "pool identity", and determinant for .system's and iocage's inherited properties
  • .system
  • iocage
  • zrootdata1 <--- I will recursively send this to a destination/backup pool, which will place it directly under the other pool's "real" top-level root dataset*
    • archives
    • documents
    • downloads
    • homes
    • legal
    • multimedia
  • zrootdata2 <--- I will recursively send this to a destination/backup pool, which will place it directly under the other pool's "real" top-level root dataset*
    • isos
    • plexmedia
    • nextclouddata
  • zrootdata3
    • playground
    • tempdata

* I use the -d flag for my recv options.


The above example is color-coded to reflect inherited encryption. Datasets of the same color share the same encryption properties (same cipher, same key width, same passphrase/keystring), and unlocking the respective "encryptionroot" (mainpool, zrootdata1, zdootdata2, zrootdata3) automatically decrypts its children.

I take it there's no way to do a restore that includes the root dataset from the backup too, since you've to make the destination pool with at least the root dataset in advance?

That's one of the reasons why I decided to try out my idea of using pseudo-roots, and why I wish I had done so earlier. Since apparently there's no such thing as a newly created zpool without an automatically created top-level root dataset of the same name. In my ideal world, creating a zpool would result with a truly "empty" zpool: no datasets, not even a top-level root dataset. After which point, any datasets created at the top-level would each be their own root dataset, since there wouldn't be a mandatory top-level root dataset that uses the same name as the pool, which requires everything else to live underneath. Using my pseudo-roots, I get around this issue, and honestly managing my datasets feels more modular. In my example above, zrootdata1 and zrootdata2 have higher priority, and can be treated differently, as well as sent to different remote pools. Perhaps zrootdata3 is lower priority, so I never replicate it anywhere, since I don't care if I lose it.

I wish TrueNAS prompted where you want to nest .system and iocage, as perhaps I would place them under a fourth pseudo-root, who knows? I think it would make for more interesting modular control.

I suppose unlocking root + pseduo dataset on boot via a pass isn't that much more effort than unlocking just one. Certainly preferable to having to unlock several datasets all off root. A way to multi-select then unlock in one go would have solved that problem though :)

You can have the true root unlock at boot by using a keyfile/keystring instead of a passphrase. This will also allow .system and iocage to live under it. Make sure to save the .json of the pool's keystrings somewhere safe if you ever need to provide them in the future.

Unless you're referring to the backup pools where you send your pseudo-roots? In that case, you don't need to unlock anything, not even the backup pool's top-level root dataset, if you're sending a raw (-w) encrypted stream to the destination. My backup pool's top-level root dataset is using the same 64-character HEX string as my main pool's root dataset, for the sake of my own convenience. I can either copy + paste the same keystring, or load the backup-pool_keystrings.json file if I need to unlock the backup pool's root dataset.

I proposed the idea of "unlock groups" before, but it wouldn't work with the underlying ZFS implementations.

One follow up question. If you have two child datasets under the pseudo root, then they'd be unlocked when the pseudo root parent is, if they share the same key or pass?

Bingo! See my above color-coded example. Datasets that share the same color are part of the same "encryptionroot" group. Unlocking the highest level in this group (e.g, zdataroot1) unlocks all similarly-colored children.

What happens if one child dataset was restored from a pool that used a different key or pass? I assume it would have to be unlocked on its own then. Is there anyway to get it to start auto-unlocking with the pseudo root by changing the pass/key used on it to the same as the pseudo root?

You simply change its encryption properties to "Inherit". This will automatically do it for you. However, they must currently be unlocked to do this.



Manually using the same keystring or passphrase will not do it. While they "technically" may be unlocked with the same passphrase, they are treated as being residents of different encryptionroots; hence must be unlocked separately. By checking "Inherit" under the encryption properties, it's all done automatically, and simply unlocking the "encryptionroot" will unlock the rest.

In my above example, the following are all the "encryptionroots" that live on the pool. Notice the pattern:

• mainpool
• zrootdata1
• zrootdata2
• zrootdata3

The other datasets (such as archives, documents, isos, etc) will not have an "unlock" or "lock" option. Only the "encryptionroots" do. However, for some reason, TrueNAS does not allow you to manually re-lock a dataset that uses a keystring (instead of a passphrase), until you disconnect the entire pool.

 

[Patrick M. Hausen]

In my ideal world, creating a zpool would result with a truly "empty" zpool: no datasets, not even a top-level root dataset.

You can do that in ZFS and it is partly done for boot environments. The properties to use are mountpoint and canmount.
I don't think fiddling with these will play nicely with TrueNAS but on a bare FreeBSD - no problem at all.

 

[gary_1]

In my example above, zrootdata1 and zrootdata2 have higher priority, and can be treated differently, as well as sent to different remote pools. Perhaps zrootdata3 is lower priority, so I never replicate it anywhere, since I don't care if I lose it.

I've being doing this with the snapshot exclude option. I have a two recursive snapshot tasks, one that is weekly* and kept for 12m and one that is daily but only kept for 2w. Then when I replicate, I have a replication task per destination and in that I exclude some child datasets via the dataset path.

Result is I only have two snap shot tasks and one replication task for each unique destination.

I'll have another read through your posts and have a think about how I may want to structure this now. It's going to need to be slightly differently to what i was originally planning. Thanks for the help.

* I forget the exact critiera I use, but it's along those lines.

 

[gary_1]

In my above example, the following are all the "encryptionroots" that live on the pool. Notice the pattern:

• mainpool
• zrootdata1
• zrootdata2
• zrootdata3

The other datasets (such as archives, documents, isos, etc) will not have an "unlock" or "lock" option. Only the "encryptionroots" do. However, for some reason, TrueNAS does not allow you to manually re-lock a dataset that uses a keystring (instead of a passphrase), until you disconnect the entire pool.

My migration was delayed due to forgetting just how long a new hdd test takes these days. Fingers crossed, scheduled for this weekend. I've been thinking over your setup suggestions and will likely go the pseduo-root way. I really don't want too many datasets to unlock each reboot (as infrequent as they may be).

With your setup, once you have native zfs encrypted server and backup pools. Doesn't that mean you're unable to backup the "mainpool" root dataset that contains .system and iocage. As "-w" will fail? Or if you sent the root dataset snapshot to a different dataset path on the backup pool, you'd have a backup but ultimately no way to restore via a send/recv as you'd never be able to send to a root?

Given it's just .system and iocage, that might not actually be a problem as from what has been said in other threads .system isn't critical to backup and I might end up with it on the boot pool anyway, iocage can then be handled via iocage export/import.

Edit: I guess i'll figure this out soon enough. I've started the replication of each dataset over to zfs encrypted backup disk.

 

[gary_1]

Well I'm all migrated. In the end I've kept everything in the root dataset since I didn't really want to go through and re-path all my scripts. It's more of a pain during restores since you have to zfs send each dataset individually, but that's a one off if a pool experiences a total loss.

Only other real downside is that due to lack of unlock groups, having a key based auto-unlock root and a password based set of child datasets would be irritating on reboots. So I've just gone with passworded root and moved the .system dataset to my nvme boot drive. Downside is logs are no longer on an encrypted dataset, but such is life.

Thanks for the help earlier, I expect I'd have made a complete hash of this without the info in this thread as it's not all that intuitive.

 

[gary_1]

Hit a small snag (and a major one further down) when it comes to the gui replication task for backing up my newly migrated pool.

[2021/01/30 12:46:10] ERROR [replication_task__task_14] [zettarepl.replication.run] For task 'task_14' non-recoverable replication error ReplicationError("Target dataset 'backup_pool/server-backup' does not have snapshots but has data (e.g. 'media' and replication from scratch is not allowed. Refusing to overwrite existing data.")

It would appear I cannot do a simple replication send of "tank" to "backup_pool/server-backup". I had assumed that in the GUI since I ticked each child dataset of "tank" rather than "tank" itself i.e. ticked "tank/home", "tank/services" etc and then had destination as backup_pool/server-backup/ that it would result in a send/recv more like:

zfs send -v -R -w -I previoussnap tank/home@latestsnap | zfs recv -d backup_pool/server-backup
zfs send -v -R -w -I previoussnap tank/services@latestsnap | zfs recv -d backup_pool/server-backup

followed by any other ticked datasets. The above works fine from the CLI.

Instead I had to pick just one source "tank/home" and set the replication destination to "backup_pool/server-backup/home" which means I'll need a separate replication task for iocage, services and each other tank child dataset :(

The only way around that which I can see, is to delete all existing backup datasets, then the GUI will allow a single replication with multiple sources of:

Code:

tank/home,tank/iocage,tank/services,...

to a destination of "backup_pool/server-backup"

So it's possible, but to get there you have to delete an existing backup and send everything again even though the destination already has matching data and snapshots. Unless I've missed a way to configure this is the gui?

Edit: Encountered another issue. Even if I delete datasets on the backup so a gui based replication can be done. The dataset on the backup ends up with a lock icon yet cannot be unlocked by any pass in use on tank or backup. Yet doing the send with cli, all works as you'd expect. I'm not sure if I'm missing something and misusing the gui replication, but I'm considering just writing some scripts to handle replication as the gui doesn't seem to be working as I'd expect :(

Edit2: I've tried doing the replication via GUI with "include dataset properties" or with "Full filesystem replication" checked. In both cases, replication occurs and the resulting dataset cannot be unlocked. The docs don't really do a good job of explaining how the GUI relates to underlying cli commands so it's hard to get any idea of what settings should be used and what their impact will be. If I do the send/recv via cli, all works as I'd expect and can be unlocked in the gui. Of course I'd rather not have to manage backups myself, so figuring out why the gui version isn't working would be useful.

 

[winnielinnie]

I'm not sure if I'm missing something and misusing the gui replication, but I'm considering just writing some scripts to handle replication as the gui doesn't seem to be working as I'd expect :(

That's what I'm doing for the meantime, myself, since following the guide on using Replication Tasks for 12.x ends up with odd results or confusing errors. (The official guide doesn't seem to explain anything more in depth than the tooltips, nor provide real-world examples.)

There is a catch, though. You can do some serious damage with a CLI script if you don't have built-in checks, such as "do not proceed if XYZ dataset is not present, or ABC snapshot is not present, or destination is not available, or etc", as you can inadvertently initialize an incremental send that fails, then upon trying it again with everything present, the destination cannot receive the new incremental send as it lacks the previous snapshot(s) needed to compare.

I honestly thought I could use the GUI to make on-demand manual dataset replications to a destination (i.e, large WD Elements USB). I had to resort to using custom ("disabled") Cron Tasks that I manually invoke when needed.

 

[gary_1]

Well it seems the plot thickens. I've just done a CLI based send from tank/ to backup_pool and the dataset+snapshots were sent over without issue.

zfs snap -r tank/iocage@testing
zfs send -R -w tank/iocage@testing | zfs recv -v -d -F backup_pool

That gets me a backup_pool/iocage with a lock icon as expected. However, when I unlock, it prompts for the password and then refuses to unlock. "Invalid key". tank/iocage is set to inherit and tank has a password on it. backup_pool is likewise a password based pool.

There's something strange going on. The initial migration to backup_pool with -x was fine, the migration back to a fresh tank/ using -R -w was likewise fine, datasets had locks but were unlockable and could be changed over to inherit. However, now sending back to backup_pool even if datasets are deleted from it first, results in locked datasets that cannot be unlocked.

As a random test, I just changed the tank/iocage from inherit to giving it an explicit pass. Deleted the existing backup_pool/iocage and repeated the zfs send over to backup_pool again. This time the dataset unlocked. That's rather concerning.

 

[winnielinnie]

As a random test, I just changed the tank/iocage from inherit to giving it an explicit pass. Deleted the existing backup_pool/iocage and repeated the zfs send over to backup_pool again. This time the dataset unlocked. That's rather concerning.

I think I'm starting to see why I didn't bump into such issues when backing up from my main pool to my backup pool: I'm replicating over my pseudo-roots (which includes everything underneath), and each pseudo-root happens to be its own "encryptionroot". I don't believe I tried to send/recv a child dataset that is not its own "encryptionroot" (as you did, in your example of iocage.) When you "broke inheritance" for iocage in your second test, you essentially made iocage its own "encryptionroot".

I'm going to try out a similar test soon with temporary datasets.

 

[gary_1]

Yes I think that may be the difference between our cases. What's strange is when I did the initial backup -> tank migration. iocage had its own password (matching backup_pool) to unlock as you'd expect, changing it then to inherit also worked, it now was locked/unlocked with the "tank" parent dataset pass "test". However replicating after that point appears to go wrong and result in an unlockable dataset.

As a further test, the iocage that i gave its own password ("1234"), I have now switched back to inherit from tank "test" and done the zfs send once more (first deleting the backup_pool/iocage destination). backup_pool/iocage was now possible to unlock using the tank pass "test". Why that didnt work the first time around, I'm not sure but it does look like gui replication described earlier might actually be possible after all and I just ran into some bug/quirk that broke inherit?

Perhaps if I go through all the child datasets, assign them a password, then go through them all from top level down and change them to inherit, this will finally work? That's quite a few to change/test though so I'd love to see your test results on this if you're looking into it with a smaller test set.

 

[winnielinnie]

UPDATE: I just did what you tested (with iocage) and I was able to successfully unlock it on the backup pool using the same passphrase as it uses on the main pool, of which it is inheriting its encryption properties from the main pool root dataset. (Over on the backup, it's not inheriting from the backup pool's root dataset.)

I noticed that the top-level root dataset of the backup pool needed to be "unlocked" before it allowed me to attempt to unlock iocage. But other than that, it worked as expected.

 

[gary_1]

I'm struggling to think of a scenario that would allow a dataset to be unlockable by tank/ yet when replicated not be unlockable with tank's password or the password it used before being set to inherit.

I've changed my tank pass and then changed it back and tried replicating other datasets over to backup_pool and so far the smaller ones have gone over fine and unlock fine. I'll let it do a full replication and see how things turn out. At a loss to explain what I saw before with iocage and other datasets.

 

[winnielinnie]

I'm struggling to think of a scenario that would allow a dataset to be unlockable by tank/ yet when replicated not be unlockable with tank's password or the password it used before being set to inherit.

I'm trying to re-create that issue, but unable to do so. Each test I am able to successfully unlock the sent dataset on the backup destination pool.

At a loss to explain what I saw before with iocage and other datasets.

Maybe it was a fluke you happened across? I'm going to keep my eyes out if this ever happens to me.

 

[gary_1]

I'm not sure what it was, but I'm hoping when this backup completes for the larger datasets, that all can be unlocked.

Also irritating timing, but I've had a drive fault, I wouldn't be surprised if by the time this backup finishes and a replacement is put in and resilvered, if another one doesn't give up. The pool has been hammered over the last few days.

I made a couple of extra backups before starting this though as I was part expecting drive issues and mostly expecting user issues :)

 

[winnielinnie]

The pool has been hammered over the last few days.

As long as they are standard drives without SMR, I don't think replications cause any exponential wear on the drives, other than drives that might already be near failure. Consider monthly scrubs that essentially read every bit of stored data as a reference: that's a minimum of twelve "full reads" every year. That's on top of regular backups that people schedule on monthly, weekly, or even daily intervals.

I made a couple of extra backups before starting this though as I was part expecting drive issues and mostly expecting user issues :)

Always at the ready. It's best to assume you'll always need access to your backup, so you're never caught off guard.

 

[gary_1]

Pool has 6x4TB WD Reds in, I checked at the time the original SMR scandal happened to be sure they were not SMR. Even though I was in the clear, the whole episode left a bad taste in my mouth. If I have to replace a drive, I'd probably go with Seagate out of principle.

I'm not sure if the drive is actually failing or not but I don't want to do anything until the backup completes and I can pull the drive and do a short/long smart scan and see what it turns up. My comment about the pool been hammered was more, if anything had developed a fault over the last year or so, this would be the time it'd show up :)