Special devices encrypted and extendable?

[Dunuin]

Hi,

I got two spare Intel S3700 200GB SSDs and added them to my main TrueNAS server running two pool (3x S3710 400GB raidz1; 4x 8TB WD white raidz1). These pools were created with FreeNAS so they are still using the legacy encryption. I updated to pools so they should be usable as fusion pool.

My idea was now to add both 200GB SSDs to the HDD pool as a mirrored special devices to offload only the metadata. If I calculated right the 200GB should be enough for metadata, even if I might upgrade later and add another 4x 8TB HDDs (64TB raw capacity * 0.3% metadata = 192GB). And if I understand it right it wouldn't be a problem if the SSDs are running out of storage because in that case the additional metadata would be stored on the HDDs again.

What I wasn'T able to find out are two things...

1.) Can someone confirm that it is possible to extend the special devices later? I maybe want to add a third SSD later to make the special device SSDs a threeway mirror.
2.) Will the special devices be encrypted too if I add them to my legacy encrypted pool?

I know that I can't remove them once added, so I want to make sure that everthing will work.

 

[sretalla]

they are still using the legacy encryption

I would deal with that before you do anything else.
@Patrick M. Hausen wrote up how to do that by basically resilvering each drive back in over itself, just not encrypted. https://www.truenas.com/community/t...om-a-zfs-volume-while-keeping-the-data.16467/

It's worth noting at this point that you're taking big risks doing that as your pool is under stress with your redundancy missing during the resilver. (one of the key reasons RAIDZ2 is recommended over 2TB disk size)

as a mirrored special devices to offload only the metadata

As long as you understand that this means that until you change your metadata, nothing will be written to the special VDEV (hence no improvement/change), fine.

If I calculated right the 200GB should be enough for metadata

Sounds OK

it wouldn't be a problem if the SSDs are running out of storage because in that case the additional metadata would be stored on the HDDs again.

True

1.) Can someone confirm that it is possible to extend the special devices later? I maybe want to add a third SSD later to make the special device SSDs a threeway mirror.

It is, and you can.

2.) Will the special devices be encrypted too if I add them to my legacy encrypted pool?

No, hence my first point.

 

[Dunuin]

I would deal with that before you do anything else.
@Patrick M. Hausen wrote up how to do that by basically resilvering each drive back in over itself, just not encrypted. https://www.truenas.com/community/t...om-a-zfs-volume-while-keeping-the-data.16467/

No, hence my first point.

Thats really annoying because I like all my data to be encrypted as much as possible...
And if I understand ZFS native encryption right it is only encrypting data but no metadata, so even if I would recreate the pool without legacy encryption but use ZFS native encryption instead, you would still see all metadata like filenames, access times and so on. I really hoped TrueNAS would use some GELI too to encrypt the full drives.
Not sure how efficient GELI is, but I compared ZFS without encyption vs ZFS with native encryption vs ZFS ontop of LUKS on my Proxmox server and enableing ZFS native encryption doubled the write amplification (and so cut the performance in half too).

 

[sretalla]

And if I understand ZFS native encryption right it is only encrypting data but no metadata, so even if I would recreate the pool without legacy encryption but use ZFS native encryption instead, you would still see all metadata like filenames, access times and so on. I really hoped TrueNAS would use some GELI too to encrypt the full drives.

Not sure what gave you that impression, but I'm not sure about it either. My understanding is that the metadata contained in/belonging to each dataset is encrypted the same as the dataset.

I see articles talking about what you mention, but I also see this:
https://github.com/illumos/illumos-gate/commit/eb633035c80613ec93d62f90482837adaaf21a0a (which is old enough that I think it may now be included)

In either case, wouldn't using the SED capability (available in almost all recently sold drives) do the job just the same in terms of providing an encrypted container for ZFS?

 

[Dunuin]

In either case, wouldn't using the SED capability (available in almost all recently sold drives) do the job just the same in terms of providing an encrypted container for ZFS?

As far as I know that wouldn't prevent someone else accessing my data who has physical access to my shutdown server because the SED key is stored unencrypted in the TrueNAS config. I like my pools to be locked until I manually type in the passphrase (or unlock it by using a passphrase via scripts using the TrueNAS API). I guess its not possible to use SED if I want to use the TrueNAS API to remotely lock and unlock the disks.

So would still be great to know if the special device gets encrypted too if the pool is using native encryption. MAybe someone else who is more into OpenZFS knows this?

 

[sretalla]

Physical access to your server is going to provide your privacy a number of problems, but I get your point.

I found this article interesting and it seems to confirm (fairly recently) that metadata is not (for the extent of the name of the datasets) protected by encryption.

A quick-start guide to OpenZFS native encryption

Learn the hows, whys, and whats of OpenZFS encryption with this short guide.

arstechnica.com

I'll leave it with you to decide how paranoid you need to be and how you want to trade that off with a system that's supported and easy to manage (and recover if needed).

EDIT:
Also very useful is this quote:

By design, ZFS encryption has several weaknesses that leak certain type of information about encrypted datasets. The first piece of information is the name and size of the file system, as well as other information available through the use of the zfs and zpool commands. I want to say it again: unlike eCryptFS, ZFS native encryption does not leak the number and size of encrypted files and folders.


ZFS encryption does not protect deduplication tables, which leaks the addresses of duplicate blocks on the disk (but not the content of individual data blocks). Finally, I will mention the CRIME (Compression Ratio Info-leak Made Easy), which is a theoretical vulnerability describing certain scenarios in which the data is compressed before encryption.

From here: https://blog.elcomsoft.com/2021/08/nas-forensics-truenas-encryption-overview/

 

[sretalla]

Or... KIMP:

The Key Management Interoperability Protocol (KMIP) is an extensible client/server communication protocol for the storage and maintenance of keys, certificates, and secret objects. KMIP on TrueNAS Enterprise is used to integrate the system within an existing centralized key management infrastructure and use a single trusted source for creating, using, and destroying SED passwords and ZFS encryption keys.

Keys can be created on a single server and then retrieved by TrueNAS. Keys wrapped within keys, symmetric, and asymmetric keys are supported. Alternately, KMIP can be used for clients to ask a server to encrypt or decrypt data without the client ever having direct access to a key. KMIP also can be used to sign certificates.

/core/uireference/system/kmip/

Although, sadly... KMIP is only available for TrueNAS Enterprise licensed systems