How to enable disk based encryption with FreeNAS?

[bittenoff]

I'm trying to convert our NAS to being fully encrypted and it would seem I'm missing a clue or a step somewhere. To test with, I've got a server that I can blow away the ZFS volume as many times as I choose...

To start with, the server boots and runs off a ZFS mirror (ada0p2/ada1p2) and I'm trying to create an encrypted ZFS volume on da0-da7

If I create a ZFS volume through the web interface, I have "tank" using da0p2 - da7p2.

If I then follow these steps:
zpool offline tank gptid/UUID-da0p2
geli init -B /boot/da0p2.eli -e AES-XTS -K /boot/storage.key -l 256 -s 4096 /dev/da0p2
(putting a geli restore in here makes no difference to the outcome)
geli attach -k /boot/storage.key /dev/da0p2

I can no longer bring the offline'd volume back:

Code:

zpool online tank gptid/UUID-da0p2
warning: device '/dev/gptid/UUID-da0p2' onlined, but remains in faulted state
use 'zpool replace' to replace devices that are no longer present

of note:

Code:

glabel status | grep gptid/UUID-da0p2 | wc -l
0

It would seem that the geli attach makes it otherwise impossible to get to the disk:

Code:

zpool labelclear /dev/da0p2
failed to open /dev/da0p2: Operation not permitted

Now if I do:
zpool replace tank gptid/UUID-da0p2 /dev/da0p2.eli

... it works but I'm no longer working with the UUID.

Is there a better way to do this?

 

[garm]

Why don’t you just tick the “encryption” box in the volume manager? http://doc.freenas.org/11/storage.html#encryption

In quite sure you can’t just write a geli container on top of a already existing vdev member, you need to create the geli containers on all the disks and then use those as vdevs. Atleast that is what i believe FreeNAS is doing.

http://www.daveeddy.com/2015/12/04/zfs-zpool-encryption-with-geli-on-freebsd/

 

[bittenoff]

The requirement is for disk based encryption, not volume based encryption. User error could lead to the tick box not being used. Plus there's the problem that "volume based encryption" cannot encrypt "in place data". Now that I've reached this point, next is to populate the volume with data on unencrypted disks and try it all again.

In addition to the above, a bit of hackery is required to make it boot at startup without a user being present.

In etc/fstab (/conf/base/etc/fstab), a line for each drive is required, a la:


/dev/da0p2.eli - geli ro 0 0


In /etc/rc.conf (/conf/base/rc.conf), a the following is required:

extra_netfs_types=geli
geli_enable="YES"
geli_da0p2_flags="-j /root/passphrase -k /boot/encryption/storage.key"


The "extra_netfs_types" is a trigger for mountcritlocal to ignore the "geli" filesystem types. (Of course there is no "geli" fstype ...)

There's definitely room for improvement here in supporting automated encrypted filesystem mounting with geli.

 

[garm]

The requirement is for disk based encryption, not volume based encryption. User error could lead to the tick box not being used. Plus there's the problem that "volume based encryption" cannot encrypt "in place data".

Que?
What is the difference in your mind between disks and volumes? Encryption in FreeNAS is per disk prior to them being part of a vdev.

Users shouldn’t create pools..

What do you mean with “in place”? Anything written to an encrypted pool will be encrypted at rest. If you want data encrypted while the system is up you need too create an container in a dataset any how.

 

[SweetAndLow]

The requirement is for disk based encryption, not volume based encryption. User error could lead to the tick box not being used. Plus there's the problem that "volume based encryption" cannot encrypt "in place data". Now that I've reached this point, next is to populate the volume with data on unencrypted disks and try it all again.

In addition to the above, a bit of hackery is required to make it boot at startup without a user being present.

In /etc/fstab (/conf/base/etc/fstab), a line for each drive is required, a la:

/dev/da0p2.eli - geli ro 0 0

In /etc/rc.conf (/conf/base/rc.conf), a the following is required:
extra_netfs_types=geli
geli_enable="YES"
geli_da0p2_flags="-j /root/passphrase -k /boot/encryption/storage.key"

The "extra_netfs_types" is a trigger for mountcritlocal to ignore the "geli" filesystem types. (Of course there is no "geli" fstype ...)

There's definitely room for improvement here in supporting automated encrypted filesystem mounting with geli.

You have a large misunderstanding of how things work.

 

[danb35]

The requirement is for disk based encryption, not volume based encryption.

Disk-based encryption is what's used when the "encrypt volume" checkbox is checked, and it uses geli.

User error could lead to the tick box not being used.

Then educate your users (i.e., your sysadmins who will be installing FreeNAS) to not commit this error.

Plus there's the problem that "volume based encryption" cannot encrypt "in place data".

Yes, actually, it can, though this does require some CLI-fu. The process has been detailed around here in a few places, but in short it involves taking a disk offline, wiping it, encrypting with geli, then replacing the disk. Repeat for each remaining disk.

 

[bittenoff]

Yes, actually, it can, though this does require some CLI-fu. The process has been detailed around here in a few places, but in short it involves taking a disk offline, wiping it, encrypting with geli, then replacing the disk. Repeat for each remaining disk.

Yes, I've seen it described but I've been unable to find the exact process/steps required. I've tried looking for this but my Google-fu isn't good enough so all I get is garbage from Google.

If encryption is specific to volumes then I can't see how replication of remote volumes (that aren't encrypted) to a local system will lead to data being encrypted. Similarly it isn't clear that the "Encrypted volume" flag is a property of the volume that is replicated.

What do you mean with “in place”?

The requirement is to transform unencrypted volumes into being encrypted without any downtime or needing to copy data back and forth. See below (bottom.)

What is the difference in your mind between disks and volumes?

A volume is a logical construct that is a map of various portions of a pool. In ZFS there is no direct relationship between a volume and a disk. On wait, in FreeNAS volumes are pools. Where's the baseball bat ...

You have a large misunderstanding of how things work.

More documentation would be most welcome so that I can understand things work better. As above, the goal is to transform the unencrypted into encrypted. If it takes 2 weeks to resilver all of the disks, that's fine. If it takes a 2 hour outage, that's not fine. This is just exploration in preparation for that.

The problem is here:
https://doc.freenas.org/9.3/freenas_storage.html#create-dataset

• There is no way to convert an existing, unencrypted volume. Instead, the data must be backed up, the existing pool must be destroyed, a new encrypted volume must be created, and the backup restored to the new volume.

That's not acceptable.

 

[SweetAndLow]

Yes, I've seen it described but I've been unable to find the exact process/steps required. I've tried looking for this but my Google-fu isn't good enough so all I get is garbage from Google.

If encryption is specific to volumes then I can't see how replication of remote volumes (that aren't encrypted) to a local system will lead to data being encrypted. Similarly it isn't clear that the "Encrypted volume" flag is a property of the volume that is replicated.



The requirement is to transform unencrypted volumes into being encrypted without any downtime or needing to copy data back and forth. See below (bottom.)



A volume is a logical construct that is a map of various portions of a pool. In ZFS there is no direct relationship between a volume and a disk. On wait, in FreeNAS volumes are pools. Where's the baseball bat ...



More documentation would be most welcome so that I can understand things work better. As above, the goal is to transform the unencrypted into encrypted. If it takes 2 weeks to resilver all of the disks, that's fine. If it takes a 2 hour outage, that's not fine. This is just exploration in preparation for that.

The problem is here:
https://doc.freenas.org/9.3/freenas_storage.html#create-dataset

• There is no way to convert an existing, unencrypted volume. Instead, the data must be backed up, the existing pool must be destroyed, a new encrypted volume must be created, and the backup restored to the new volume.

That's not acceptable.

https://forums.freenas.org/index.php?threads/how-to-encrypt-an-existing-raidz-or-mirror.16975/

literally the fist link in google: site:freenas.org encrypt existing volume

 

[danb35]

If encryption is specific to volumes

It isn't, as I've already told you. The disks are encrypted, and the pool is created on the encrypted disks.

Similarly it isn't clear that the "Encrypted volume" flag is a property of the volume that is replicated.

It isn't. What has suggested otherwise?

https://doc.freenas.org/9.3/freenas_storage.html#create-dataset

Admittedly the current manual says the same thing, but why are you referring to the docs for a three-year-old release, when there have been two (ignoring the late and unlamented FN10) major releases since then?

 

[bittenoff]

https://forums.freenas.org/index.php?threads/how-to-encrypt-an-existing-raidz-or-mirror.16975/

literally the fist link in google: site:freenas.org encrypt existing volume

I *hate* you ;)

I found that link several weeks ago (so says the browser link coloring) but have been unable to find it again since. Thanks.

Admittedly the current manual says the same thing, but why are you referring to the docs for a three-year-old release, when there have been two (ignoring the late and unlamented FN10) major releases since then?

Blame google. The "top rated" link to "old documentation" seems to be a common problem with google.

 

[bittenoff]

Let me add something further to this.

Doing encryption using the round-about method that I chose worked and was reasonably stable. Replicated terabytes of data, no problem.

Using the "select the checkbox" method is decidedly less stable.

As a replication destination writing to an encrypted volume, 11.0-STABLE appears to die quite regularly.

 

[wblock]

Why are you using 9.3? Or maybe just the 9.3 docs? The manual technique you are using is the same as what FreeNAS automates. GELI devices are created on the disks, then a volume (pool) is created from those encrypted devices. Except done through the GUI, the results are consistent with the FreeNAS configuration database. Done by hand, there could easily end up being a difference between what FreeNAS believes, and what actually exists.

The end result is that someone with one or more of the powered-off disks cannot read that data.

As a replication destination writing to an encrypted volume, 11.0-STABLE appears to die quite regularly.

Please give specifics. What are the error messages? As said above, command-line encryption setup could create a discrepancy.

 

[bittenoff]

Why are you using 9.3? Or maybe just the 9.3 docs?

...
Please give specifics. What are the error messages? As said above, command-line encryption setup could create a discrepancy.

Source system is 9.3 (no encryption, uptime ok), destination system is 11.0 (encrypted, panics within hours.)

What are the panic messages?

Looks like recursive zio (many screens of this.)

But it just reached a new record low - 1m1s uptime (between panics.)

Edit: Maybe this is 11.0 just being a "dot-zero" release ... on another FreeNAS-11.0 server it panic'd today with this:

Code:

Dec 14 15:58:19 freenas-1 Fatal trap 12: page fault while in kernel mode
Dec 14 15:58:19 freenas-1 WARNING: 192.168.26.8 (iqn.1991-05.com.microsoft:apm-ds-1a): tasks terminated
Dec 14 15:58:19 freenas-1 cpuid = 7; apic id = 07
Dec 14 15:58:19 freenas-1 fault virtual address	= 0x220
Dec 14 15:58:19 freenas-1 fault code		= supervisor write data, page not present
Dec 14 15:58:19 freenas-1 instruction pointer	= 0x20:0xffffffff8066e866
Dec 14 15:58:19 freenas-1 stack pointer			= 0x28:0xfffffe023b4c77c0
Dec 14 15:58:19 freenas-1 frame pointer			= 0x28:0xfffffe023b4c7830
Dec 14 15:58:19 freenas-1 code segment		= base 0x0, limit 0xfffff, type 0x1b
Dec 14 15:58:19 freenas-1 WARNING: 192.168.26.5 (iqn.1991-05.com.microsoft:apm-ds-1a): session reinstatement from different address 192.168.26.8
Dec 14 15:58:19 freenas-1			 = DPL 0, pres 1, long 1, def32 0, gran 1
Dec 14 15:58:19 freenas-1 processor eflags	= interrupt enabled, resume, IOPL = 0
Dec 14 15:58:19 freenas-1 current process		= 6119 (ctld)

 

[wblock]

Was the manual encryption setup also done on the second FreeNAS 11 system? Please give hardware details on both systems, particularly CPU, RAM, motherboard, and network card.

 

[bittenoff]

The replication destination is the only place where encryption is being done.

The replication destination was FreeNAS-11.0 (see panics above) and is now FreeNAS-11.1 (and this too has panic'd but I haven't seen the console yet.)

The panic's on the replication did not occur when I enabled encryption through manually editing rc.conf, etc.

The panic's on the replication did occur after I enabled "Volume Encryption" through the Web UI.

The screen shots quite clearly indicate that there is a recursion bug in ZFS somewhere (the kernel panic's because the kernel stack gets exhausted.) Knowing what motherboard, RAM, CPU, etc will not help solve that bug and will only help divert blame.

 

[wblock]

Knowing which hardware this is running on could certainly help eliminate that as a problem. We have had users try to run "engineering sample" processors and have mysterious problems, for instance. The reluctance to describe the hardware makes it sound even more likely to be the problem, probably the opposite of what was intended.

 

[bittenoff]

The reluctance to describe the hardware makes it sound even more likely to be the problem, probably the opposite of what was intended.

No, it just means that it is likely common off the shelf hardware. Forgive me if I don't want to bore you with the details, but posting "dmesg" output or similar will not contribute anything to the discussion except for those interested in hardware "porn" or "how big is it". It will almost certainly be responded to with "your hardware configuration is <blah>" and that is not a helpful discussion to have. Easy, yes, fruitful, no.

Note that I have been able to successfully store data encrypted after replication (see earlier post in this thread.) The problem is most definitely a software issue somewhere inside ZFS where it decides to go into a stupid amount of recursion. Pointer not being updated, field not correctly set, sequence of calls wrong, locking, etc.

What I would have expected is someone trying to replicate the problem - that would be far more informative the "what speed RAM are you using."

Should be easy enough.
Step 1) build system 1 using FreeNAS 9.3, create volume "source"
Step 2) turn on automatic snapshots of "source" and fill it with ~2TB of data over a few hours
Step 3) build system 2 using FreeNAS 11.1, create volume "destination" and enable encryption
Step 4) replicate "source" to "destination" over 1GE and wait.

 

[wblock]

I suggest you submit a bug report, but again, hardware is part of the situation, and specifics will be required.