Upgrading to TrueNAS Core, safely convert existing ZFS pools to OpenZFS with native encryption?

[winnielinnie]

I've searched across the web, these forums, and Reddit, but I couldn't find a definitive answer.

When the stable version of TrueNAS Core is released, will it have a built-in method to upgrade existing ZFS pools to OpenZFS?

What if you are currently using geli encryption? Does this imply you'll continue to unlock the disks via geli in order to access the OpenZFS pool within? Will "per-dataset encryption" with OpenZFS be agnostic to the fact that the underlying disks are encrypted with geli? (I.e., "double encryption").

My fear is that upon unlocking the geli-protected disks in order to access the current ZFS pool (non-OpenZFS), there is a high risk of data loss if you try to convert to OpenZFS via the GUI. (For example, doing things in the "wrong order" might make a userkey unusable.) Has anyone tested this yet?

 

[Gen8 Runner]

Okay everyone,
now TrueNAS Stable is out and what about this point?

I am also thinking about upgrading from FreeNAS to TrueNAS Stable, but check at the moment the options I have for my encrypted pool on FreeNAS.

Is that feature already built in? Or is the best (or only?) option to destroy the dataset and revert by backup to the old datasets and contents? Any other or better ideas?

 

[Patrick M. Hausen]

There is no conversion, really. Native FreeBSD ZFS vs. OpenZFS is mainly an implementation detail. And, yes, anything ZFS is completely agnostic to the underlying GELI encryption. GELI turns an encrypted on disk block device into a plain text block device presented to ZFS.

As long as you do not upgrade your pool, there is negligible risk in updating to TrueNAS 12. Of course using native encryption would require upgrading the pool. And of course you could use native encryption on top og GELI, because (see above) ZFS doesn't know about GELI. It's block devices all the way down.

As for a good long term strategy ... native encryption is preferrable if you want to do backups with zfs send | zfs receive, because you can send encrypted to encrypted and the target system doesn't even have to know how to decrypt. Talk about safe backups.

So I would remove the GELI encryption first. Then upgrade. Then apply native encryption to the relevant datasets.
Of course this leaves all your data unencrypted over a certain period of time and is bound to leave unencrypted data in raw disk sectors, afterwards. You might want to create a file of zero bytes taking up all the space, then removing it, once you are done.

I wrote a step by step howto for removing GELI encryption while preserving the data on a live system:

How to remove encryption from a ZFS volume (while keeping the data)

Hi, all, my first post - I have a little procedure to share. I successfully removed the geli encryption from a live ZFS pool in FreeNAS 9.1.1 with the following steps: 0. Make sure you either have a separate backup of your data, or are willing to take the risk of losing everything. Second...

www.truenas.com

 

[Patrick M. Hausen]

P.S. Re-reading my last post it does not seem perfectly clear to myself, so ...

If your question about conversion implied a conversion from GELI encryption to native - that is fundamentally impossible. It's difficult to explain, so I am repeating myself - ZFS knows zilch about GELI.

GELI turns an entire partition, also called a block device, into an encrypted partition. The data resides on the disk encrypted and is presented to ZFS in the clear.

That's why I stated that there is no conversion. New ZFS [tm] can encrypt datasets within ZFS. It is still completely oblivious of GELI.

And that's why I recommend removing GELI encryption first by the linked procedure, then do an upgrade, then encrypt individual datasets as necessary.

HTH,
Patrick

 

[winnielinnie]

And, yes, anything ZFS is completely agnostic to the underlying GELI encryption. GELI turns an encrypted on disk block device into a plain text block device presented to ZFS.

As I assumed! Though it makes me wonder how this will be worded and presented in the PDF guide for TrueNAS Core 12 (i.e, will there be two sections in the guide that address "Encryption", even though they are implemented much differently?)

Of course using native encryption would require upgrading the pool.

Am I correct to believe that after upgrading your pool, only newly created datasets in said pool can be encrypted? There is no such ZFS native "on-the-fly" encryption that re-writes the existing files in each dataset, without first having to delete the dataset entirely?

And that's why I recommend removing GELI encryption first by the linked procedure, then do an upgrade, then encrypt individual datasets as necessary.

I'll look over it, though it does feel like something could go terribly wrong! :o I do have a separate backup, yet I never had to resort to falling back on it.


This brings up another question: Let's say you safely remove GELI encryption, and all your data remains intact. No GELI encryption, ZFS pool has all your datasets. Wouldn't you still need to re-write everything in each dataset to enable native ZFS encryption? (See above.)


Another question, which I could only find partial answers elsewhere: If you were to use native ZFS encryption on the root dataset, I understand that all child datasets (basically every dataset created henceforth in this pool) is forced to inherit native encryption. Does unlocking these require the same passphrase / userkey? Will unlocking the root dataset automatically unlock all child datasets underneath?


I'm trying to get a grasp of all of this before I can feel comfortable converting to and using native ZFS encryption with TrueNAS Core 12 and newer. I don't have a spare box to play with. Trying to research this online tends to be fruitless, since it's all fairly new to FreeNAS / TrueNAS.


Much thanks! Going to look over your guide later!

 

[Patrick M. Hausen]

Am I correct to believe that after upgrading your pool, only newly created datasets in said pool can be encrypted? There is no such ZFS native "on-the-fly" encryption that re-writes the existing files in each dataset, without first having to delete the dataset entirely?

Correct.

This brings up another question: Let's say you safely remove GELI encryption, and all your data remains intact. No GELI encryption, ZFS pool has all your datasets. Wouldn't you still need to re-write everything in each dataset to enable native ZFS encryption? (See above.)

Of course.

Another question, which I could only find partial answers elsewhere: If you were to use native ZFS encryption on the root dataset, I understand that all child datasets (basically every dataset created henceforth in this pool) is forced to inherit native encryption. Does unlocking these require the same passphrase / userkey? Will unlocking the root dataset automatically unlock all child datasets underneath?

If I am not mistaken all child datasets created in an encrypted parent dataset inherit their key by default. This can be overridden per dataset if desired.

 

[Patrick M. Hausen]

Though it makes me wonder how this will be worded and presented in the PDF guide for TrueNAS Core 12 (i.e, will there be two sections in the guide that address "Encryption", even though they are implemented much differently?)

I assumed that GELI encyption will simply be dropped from the guide altogether. You cannot create new GELI encrypted systems with TN 12.0

 

[winnielinnie]

I assumed that GELI encyption will simply be dropped from the guide altogether. You cannot create new GELI encrypted systems with TN 12.0

I was not aware of this! What about existing GELI encryption from FreeNAS 11.x systems? If you cannot create new GELI encrypted disks using the GUI, are they also removing from the GUI the option to import a pool with GELI encryption underneath? Currently, when importing a pool in FreeNAS 11.x, there are some options that prompt you if it must first be decrypted (GELI underneath). Upon doing so, it also prompts you for a userkey file and/or a passphrase. This is all based on GELI encryption during a pool import. Nothing needs to be done via a terminal or shell.

 

[Patrick M. Hausen]

I expect import will work just fine and of course TN 12.0 continues to use an already GELI encrypted pool after upgrade from 11.3. But the pool creation UI only offers one encryption checkbox and that is native encryption, now.
Replacing a disk of a GELI encrypted pool will probably work, too. Hopefully so.

I am glad it's gone. It always felt out of place compared to ZFS native and it introduces too much complexity and room for failure, in my opinion.

 

[winnielinnie]

I am glad it's gone. It always felt out of place compared to ZFS native and it introduces too much complexity and room for failure, in my opinion.

Makes sense, and I agree with concept and practice of native ZFS encryption. I just don't like juggling with existing data, even if a backup exists. It's just not fun, and there is still a thing called "life" that takes up most of my time. ;) I may end up waiting to take care of other things first before I fully commit to upgrading to TrueNAS Core 12+ and doing a permanent migration away from GELI encryption.

 

[mgd]

So I would remove the GELI encryption first. Then upgrade. Then apply native encryption to the relevant datasets.

Please note, that following your guide means that the decrypted data hits the raw devices when you remove GELI. So even if you end up having a completely natively ZFS encrypted pool, your cleartext data has hit the disks in the pool in the middle of the process.

So if you ever need to RMA one of those disks, you would need to overwrite every single block (which BTW is not possible on a SSD drive) to ensure none of your data ends up at the service center. Also remember, that often a disk is RMA'ed because it is no longer working so it might not be possible to wipe the blocks.

I would therefore recommend an approach where the data is copied from the old pool using GELI encryption underneath to a new pool using native ZFS encryption. Depending on your level of paranoia for loosing your data in the process you might need one or more extra disks to do this (if, for example, you want to keep a mirror, Z2, or Z3 raid intact during the transfer). If you are not paranoid or have a backup of your data (which I would recommend anyway) and have, say, a mirror of two disks, I would recommend this procedure: 1) Degrade the mirror to a single-disk pool, 2) Create a new single-disk pool with native ZFS encryption, 3) copy your data over to the new pool, 4) destroy the original pool, 5) add the second disk to the new pool making it a two-disk mirror.

The reason for my recommendation is that most people encrypt their pools to ensure a stolen or RMA'ed disk does not contain any of their data in cleartext and can essentially be “wiped” by throwing away the encryption key. Therefore, it is essential that cleartext data has never been written to the disks.

 

[Patrick M. Hausen]

@mgd I agree in general but if you are really most concerned about the "safe disk disposal" scenario, at least in the case of spinning disks there is a simple solution: move the data to encrypted datasets and then fill the remaining space of the pool with a single file copied from /dev/zero until the pool is full, then remove the file. Yes, SSDs are a different story.

One overwrite with zeroes is enough. The "lab can read data after multiple overwrites with an STM" has long been debunked. If you are not set against a nation state adversary, one overwrite with zeroes will do. And for those adversaries, XKCD 538 still holds ;)

 

[mgd]

@Patrick-m-hausen Thank you for the reply.

I am not only concerned about the “safe disk disposal” scenario, but it is an important part of the whole picture. I agree that your solution would work perfectly on spinning disks.

And yes, that “multiple overwrites is necessary to wipe a disk” myth is difficult to get rid of. :)

That particular XKCD strip is one of my favourites.

 

[AlexGG]

move the data to encrypted datasets and then fill the remaining space of the pool with a single file copied from /dev/zero until the pool is full

You need to use /dev/random though. I think it is not possible to fill the pool with zeros because zeros will just have infinite compression; or so I suppose.

 

[Patrick M. Hausen]

You need to use /dev/random though. I think it is not possible to fill the pool with zeros because zeros will just have infinite compression; or so I suppose.

Thanks for bringing that up. Correct, unless you turn off compression.

Code:

zfs create -o compression=off tank/deleteme
dd if=/dev/zero of=/mnt/tank/deleteme/whatever bs=1m
zfs destroy tank/deleteme

Using /dev/random will result in abysmal performance for that task.

 

[mgd]

@Patrick M. Hausen Maybe you should make an edit to your guide in the #3 post in this thread to let readers understand that their new pool might contain cleartext data on the disks unless they do the wipe of free space in the pool.

After all, the pool was initially encrypted for a reason.

 

[Patrick M. Hausen]

@mgd But my guide clearly states "remove encryption while keeping the data". Of course all the data will be unencrypted on disk afterwards, that's the point.

What precisely is prone to misunderstanding about this?

 

[mgd]

@Patrick M. Hausen Sorry, I referred to the thread about removing encryption which I realised was a mistake and then I edited my post – possible after you saw it which might explain the confusion.

What I meant to refer to was post #3 in the thread we are writing in now:

So I would remove the GELI encryption first. Then upgrade. Then apply native encryption to the relevant datasets.

So I was simply suggesting editing that post adding a single line mentioning that this procedure (removing GELI encryption, then adding native ZFS encryption) will leave unencrypted data on the disks.

I know people can scroll down here to see it but maybe it makes sense to edit the above post so the casual reader sees it.

 

[Patrick M. Hausen]

I thought you suggested changing the "howto". I edited the post, thanks.