ZFS on Linux - Native encryption

[Arwen]

The recent GITHUB pull requests for native ZFS encryption for Linux got me curious. I have my old media server just hanging around, so I set up a test. It took 10 times longer to copy and update the Linux OS than it did for my ZFS native encryption testing. (Basically I copied my new media server Linux' OS to my old media server, then made the modifications and updates. ZFS send & receive to the rescue!)

The encryption seemed to work flawless. I used a simple setup, so no encrypted root and no auto-mounting at boot. Plus, used a passphrase not key. A reboot and verification of the data worked perfectly.

All that said, I still would not use this in production. I may submit a few updates to the manual page, since it lacked a few details. But, I need to re-read it and compose my suggestion(s) carefully.

I do understand encryption to some degree, but I would not consider myself an expert. So beyond usability features, documentation and any data loss investigation, I would not be able to help.

Anyone here have straight forward questions?

Edit: Added link to Tom's video on OpenZFS encryption;

https://www.youtube.com/watch?v=frnLiXclAMo

 

[Stux]

I watched the presentation, It is an impressive piece of work.

Can you activate encryption on a pool after the fact or does it have to be created that way? If so, what happens if your pool is copies=3?

(The encryption stores its metadata in the third copy field)

Can you remove encryption? Or change the key to a clear key I guess. Since it's not ZFS style to decrypt data already on disk.

Encrypted send/receive is still coming I believe.

Defininately not pseudo-production ready until more "crypotologists" sign off on it. It's still possible to have a breaking change in the implementation if there is a major issue found.

 

[Ericloewe]

Can you activate encryption on a pool after the fact or does it have to be created that way?

It's per dataset, so at dataset creation time.

Can you remove encryption?

I'm pretty sure you'd have to send it to an unencrypted dataset.

 

[Arwen]

...
Can you activate encryption on a pool after the fact or does it have to be created that way? If so, what happens if your pool is copies=3?

(The encryption stores its metadata in the third copy field)
...

Encryption is by dataset. I don't know if you can create an encrypted pool. And yes, you HAVE to create the dataset as encrypted.

As for copies=3, I guess I would have to test that. (But, for the record, encrypted datasets WON'T have 3 copies, regardless of what copies=3 has.)

...
Can you remove encryption? Or change the key to a clear key I guess. Since it's not ZFS style to decrypt data already on disk.

Encrypted send/receive is still coming I believe.
...

No, you can not remove encryption on a dataset. You would have to copy it using a tool that was not encrypted, (so don't use encrypted send & receive).

I did not check to see if encrypted send & receive is available in this version.

Also, half the checksum space is used by the encryption. So, instead of 256 bits of checksum, you only get 128 bits. Wish encryption had been designed into ZFS from the start.

So, here are the things I have yet to test;

1. Can you create an entire pool that is encrypted? (Top level dataset?)
2. What happens when you set copies=3 during creation on an encrypted dataset? Or attempt to change it after?
3. Does this version include encrypted send and receive?

 

[Ericloewe]

I did not check to see if encrypted send & receive is available in this version.

It's not, raw send/recv is still in the works.

So, instead of 256 bits of checksum, you only get 128 bits. Wish encryption had been designed into ZFS from the start.

It's not quite as bad as it sounds. the other 128 bits are used for the MAC, which authenticates and validates integrity of the encrypted block - so you get the same effect from a different algorithm, which makes up the difference to some extent. I'll leave it to cryptographers to explain the nuances.

 

[Arwen]

Found something else I will be testing, (number 4 below). Though looks like #3 has already been answered, with no.

1. Can you create an entire pool that is encrypted? (Top level dataset?)
2. What happens when you set copies=3 during creation on an encrypted dataset? Or attempt to change it after?
3. Does this version include encrypted send and receive?
4. After deleting encrypted dataset, does the Pool feature encryption return to enabled? (From active?)

 

[Arwen]

Okay, let's start a frequently asked questions on this subject. I've bolded the answers to make them easier to read.

1. Is OpenZFS native encryption compatible with Solaris ZFS encryption in pool version 30?
   No.
2. What input keys does OpenZFS native encryption support?
   Either keyfile or passphrase.
3. Can you change the key(s) after creation?
   Yes. It's a wrapper key, so the encrypted files are not touched.
   
4. After deleting last encrypted dataset, does the Pool feature@encryption return to enabled, (from active)?
   Yes.
5. What happens when you create a child dataset of an encrypted dataset?
   The child is encrypted with the same level and key(s).
   
6. Can you create an un-encrypted child dataset of an encrypted dataset?
   No. All children must be encrypted.
   
7. Can you create an entire pool that is encrypted? (Top level dataset?)
   Yes. But, you have to use -O encryption=on ... options at pool creation time.
   
8. Can you import an encrypted pool?
   Yes. Afterwards, you have to load the key(s) of the top level dataset in order to use it.
9. Does changing copies=2 work on encrypted datasets?
   Yes.
10. Does changing copies=3 work on encrypted datasets?
    No, you get the error:
    cannot set property for 'crpytpool/test': encrypted datasets cannot have 3 copies
11. Does inheriting copies=3 on encrypted dataset creation work?
    As of 2017/08/22 it is allowed, but only 2 copies are actually created.
    Reported as issue #6557: Native encryption - Odd behavior for copies=3: inherited and at dataset creation
12. Can you use copies=3 at encrypted dataset creation?
    As of 2017/08/22 it is allowed, but only 2 copies are actually created.
    Reported as issue #6557: Native encryption - Odd behavior for copies=3: inherited and at dataset creation
13. Does zfs send allow sending encrypted datasets?
    As of 2017/08/22 it appears you can save the data stream, but not zfs receive it. Their is an issue on send/receive:
    Clarify behavior of zfs send/recv with encryption (#6547)
14. Can you unload encryption keys to make a dataset secure without exporting pool?
    Yes, it's the zfs unload-key command.
15. Can you unload encryption keys while a dataset is mounted? (Or a child dataset is mounted?)
    No.
16. Is OpenZFS' native encryption production ready?
    No.
17. Can you disable the Pool feature@encryption, (assuming no encrypted datasets)?
    No.
    
18. Can you use both passphrase and key?
    No.
    
19. Is it possible / easy to use a USB flash drive partition as a raw key?
    No, you get the following error:
    cannot create 'rpool/encrypt_test': Raw key too long (expected 32).
    Reported as issue #6556: Native encryption - request to allow raw keys on devices, (like USB flash)
20. Can you import a pool that used to have an encrypted dataset, onto a ZFS version that does not support encryption?
    Yes.
21. Can you import a pool, (perhaps R/O and not the encrypted datasets), that has encrypted datasets, onto a ZFS version that does not support encryption?
    No. Get the standard error for un-supported feature;
    status: The pool uses the following feature(s) not supported on this system:
com.datto:encryption (Support for dataset level encryption)
action: The pool cannot be imported. Access the pool on a system that supports
the required feature(s), or recreate the pool from backup.

I'll report 11 & 12 as a bugs eventually.

Whence we get more answers, and if / when it's available for FreeNAS, we can make a resource.

Edit: Fixed date & added #16, (not production ready).
Edit: Added 17, 18, 19 (with issue 6556). Reported 11 & 12 as issue 6557.
Edit: Added import tests. Basically checked on standard behaviour.

 

[Stux]

16. Is it production ready?

I suspect the answer is no :)

 

[Ericloewe]

I don't know how to force a line break.

Shift+Enter, like most test editors.

Thanks for checking some of the edge cases!

 

[danb35]

As of 2017/09/22 it appears

You have a TARDIS in Rivendell?

 

[m0nkey_]

I do :)

Code:

Welcome to FreeNAS
[root@tardis] ~#

 

[Stux]

I do :)

Code:

Welcome to FreeNAS
[root@tardis] ~#

Bigger on the inside?

 

[fracai]

Ditto, and some of my jails have admin users named "doctor".

 

[Arwen]

You have a TARDIS in Rivendell?

Sorry, it's these mortal dates I have to use with you humans. I never seem to get August's number right :).

16. Is it production ready?

I suspect the answer is no :)

Added. And of course, no it's not ready.

Shift+Enter, like most test editors.
...

FAQ updated to include answers on new line. I kept the bolding, as I think it helps clarify and separate the answers.

...
Thanks for checking some of the edge cases!

You are welcome.

 

[Arwen]

I may re-order the questions a bit.

Here are some other tests I can run, now that my test environment is already built and easy enough to use.

• Can you disable the Pool feature@encryption, (assuming no encrypted datasets)?
  
• Can you use both passphrase and key?
• Is it possible / easy to use a USB flash drive with a raw key?
• Can you use a USB flash drive raw key source, then import encrypted pool, with automatic decryption?

The last would help for those that want a physical key they can remove. Thus, after reboot, you install the USB flash drive, load the key to de-crypt the dataset and remove the USB flash drive. Any physical theft of the server or disks, would not get the encrypted data.

Plus, if you know your network is being hacked, you can manually un-load the key, (or export the pool). Again, if the physical key is not installed, a hacker / thief can't get to the encrypted data.

As for the first, their are some pool features that whence enabled, can't be disabled. Thus, may not be usable by OSes that don't have compatible ZFS pool features. So, I want to test if a pool that formerly used encrypted datasets, can be imported by an OS without OpenZFS native encryption support.

 

[Arwen]

Did some more testing, (edited the FAQ posting);

• You can't return a pool to feature@encryption disabled state.
• You can't use both passphrass and key file
• It's not possible to use a USB flash drive partition as a source for the raw key.

And reported the issues, (edited the FAQ posting);

• Native encryption - request to allow raw keys on devices, (like USB flash) #6556
• Native encryption - Odd behavior for copies=3: inherited and at dataset creation #6557

This last one is a known general properties creation / inheritance issue. Tom Caputi actually reported in the general form, though possibly for the same reason I did. But, I left my issue on a specific case in place so it can be tested whence the parent is fixed, to confirm behavior.

 

[Ericloewe]

You can't return a pool to feature@encryption disabled state.

That's normal for all ZFS features, but it doesn't break backwards compatibility.

 

[Arwen]

That's normal for all ZFS features, but it doesn't break backwards compatibility.

Yes, that is another test I would like to perform;

1. Create pool on removable drive
2. Encrypt a dataset
3. Destroy the encrypted dataset
4. Export the pool
5. Import the pool into a ZFS instance without native encryption support

If it works, then we don't care if feature@encryption is enabled or disabled.

Of course, that brings forth the test of what to do trying to import a pool with encrypted dataset and feature@encryption=active, with ZFS that does not support encryption.

 

[Ericloewe]

It'll simply fail at import time, if all is well. Definitely worth checking. FreeNAS should work well as the no-encryption guinea pig.

 

[Arwen]

Okay, ran these tests, (edited the FAQ posting);

• Can you import a pool that used to have an encrypted dataset, onto a ZFS version that does not support encryption?
  Yes.
• Can you import a pool, (perhaps R/O and not the encrypted datasets), that has encrypted datasets, onto a ZFS version that does not support encryption?
  No. Get the standard error for un-supported feature;
  status: The pool uses the following feature(s) not supported on this system:
  com.datto:encryption (Support for dataset level encryption)
  action: The pool cannot be imported. Access the pool on a system that supports
  the required feature(s), or recreate the pool from backup.