Encryption and TrueNAS Scale

[Skrenes]

It seems like encryption in TrueNAS is a very low priority. I've been doing an audit on TrueNAS Scale and have found that a lot of sensitive information like API keys and private keys for certificates live in the /data folder, which is never encrypted since it's a folder on the boot-pool rather than inside of the System Dataset. Furthermore, even though the System Dataset can inherit encryption from a pool's root dataset, the latter must have its decryption key in the unencrypted boot-pool, which defeats the entire "at rest" point of encryption.

It also seems that encrypting the ix-applications dataset is unsupported. So that makes two unencrypted/unprotected datasets by design with TrueNAS Scale. Why is this? Is there any roadmap that aims to address this?

It would be ideal if TrueNAS had a simple interface that request a password that when entered correctly, decrypts these datasets (with /data being inside say the System Dataset).

 

[MisterE2002]

The default they choose is the boot-pool key. This will protect of stolen data disks (without boot-pool) and you can swap failed disks without afraid of leaking data to the drive manufacturer.

You can use the "password" method but how would you expect the "ask" part? In the notification icon? At the command line at boot?

 

[garm]

What's the use case of encrypting the system? The use case for at-rest-encryption of disks in a pool is that you can send in a broken drive without having to wipe it first.. useful if the drive malfunctions.. but what is the at-rest-encryption use case for the system? when the physical security of your server ensure it cannot be accessed, all at-rest-encryption does is make sure you need to be present to boot the dammed thing, lowering physical security

 

[sretalla]

It's probably worth mentioning that most of the issues mentioned here (not sure about the iX Applications as SCALE doesn't yet have an enterprise version as such) are solved by the Enterprise edition with KIMP.

maybe something to jump on the band wagon of the request at the end of this thread:

TrueNAS Scale Free vs Enterprise

We're reluctant to support FC as SW-only without having an extensive FC lab with all HBAs and switches and clients. We would prefer that most users use iSCSI. We can provide FC via a nominal fee...it includes the physical appliance. Why are you using FC? Is it about performance or legacy...

www.truenas.com

 

[Arwen]

This has been brought up before, multiple times, over at least 5 years.

TrueNAS at rest encryption is simply not designed for all uses.

Some users are worried about (possibly extremist) law enforcement getting access to their data. Thus, they want complete encryption protection that TrueNAS does not offer.


In theory, a TrueNAS server could be built with complete data pool encryption via keys. Then the boot-pool could be encrypted with SED, (Self Encryption Device / Disk). On reboot or power cycle, the user / SysAdmin of such a server would have to enter the SED password(s) to continue booting.

TrueNAS does not need to know or support SED on the boot-pool. The BIOS would take care of that.

If the users computer BIOS does not support SEDs, then a small USB drive with EFI boot could possibly take care of that. And since the USB EFI drive would have zero private information other than the prompt code for the SED un-locking, it does not mater that this USB EFI drive & code is un-encrypted.

Again, as I said, not something TrueNAS supports directly.

 

[morganL]

This has been brought up before, multiple times, over at least 5 years.

TrueNAS at rest encryption is simply not designed for all uses.

Some users are worried about (possibly extremist) law enforcement getting access to their data. Thus, they want complete encryption protection that TrueNAS does not offer.


In theory, a TrueNAS server could be built with complete data pool encryption via keys. Then the boot-pool could be encrypted with SED, (Self Encryption Device / Disk). On reboot or power cycle, the user / SysAdmin of such a server would have to enter the SED password(s) to continue booting.

TrueNAS does not need to know or support SED on the boot-pool. The BIOS would take care of that.

If the users computer BIOS does not support SEDs, then a small USB drive with EFI boot could possibly take care of that. And since the USB EFI drive would have zero private information other than the prompt code for the SED un-locking, it does not mater that this USB EFI drive & code is un-encrypted.

Again, as I said, not something TrueNAS supports directly.

There's a tradeoff between security and reliability, but the boot pool on SED with BIOS unlocking at boot, would be most secure approach I know of. IPMI would be used to allow remote booting. With a normal system this would be very secure, but if there is any unreliability due to hardware issues or immature software, then outages would be longer.

 

[winnielinnie]

It also seems that encrypting the ix-applications dataset is unsupported.

What's the story behind that?

On TrueNAS Core 13, I have the entire iocage dataset and all of its children encrypted, inheriting the root dataset's encryption. Everything runs without a hitch, seamlessly. Why can't this be the same for SCALE and its ix-applications dataset?

 

[Skrenes]

The default they choose is the boot-pool key. This will protect of stolen data disks (without boot-pool) and you can swap failed disks without afraid of leaking data to the drive manufacturer.

You can use the "password" method but how would you expect the "ask" part? In the notification icon? At the command line at boot?

I think just about everyone using TrueNAS has their pool split across several drives, and especially with LZ4 (nevermind the fact that most files have compression such as images, videos, PDFs, etc), this would make a single sold or failed drive useless from a data leaking perspective. On the other hand, most people live in houses found in neighborhoods that have, at least from time to time, had break-ins. An entire NAS being stolen is almost infinitely more likely than just the drives comprising the data pool. If the keys are on the boot-pool (which isn't encrypted), what's the point of encryption?

As for the method of asking the password: I think the notification icon would make the most sense, which would result in the same outcome as navigating to the dataset and unlocking it. Navigating to say "credentials" would show a lock symbol indicating that secure data such as keys, credentials, and tokens (that should be in the System Dataset and not in /data) has not been decrypted.

What's the use case of encrypting the system? The use case for at-rest-encryption of disks in a pool is that you can send in a broken drive without having to wipe it first.. useful if the drive malfunctions.. but what is the at-rest-encryption use case for the system? when the physical security of your server ensure it cannot be accessed, all at-rest-encryption does is make sure you need to be present to boot the dammed thing, lowering physical security

Theft is the #1 use-case for at rest encryption. It would even protect you from a government raid because they're almost certainly going to take your NAS rather than setup camp near your closet and spend the week copying your data. You don't need to be physically at the NAS to decrypt it if it had first-class support in TrueNAS. There are even tools to decrypt LUKS drives remotely (e.g. dropbear-initramfs, which you can SSH into to decrypt your main partition, which when unlocked can use crypttab to chain-decrypt other drives/partitions). I'm not even suggesting that. Just hoping that the development team can reorganize where they store sensitive data and support encrypting ix-applications.

It's probably worth mentioning that most of the issues mentioned here (not sure about the iX Applications as SCALE doesn't yet have an enterprise version as such) are solved by the Enterprise edition with KIMP.

maybe something to jump on the band wagon of the request at the end of this thread:

TrueNAS Scale Free vs Enterprise

We're reluctant to support FC as SW-only without having an extensive FC lab with all HBAs and switches and clients. We would prefer that most users use iSCSI. We can provide FC via a nominal fee...it includes the physical appliance. Why are you using FC? Is it about performance or legacy...

www.truenas.com

Good suggestion. I'll post there and reference this. Thanks!

This has been brought up before, multiple times, over at least 5 years.

TrueNAS at rest encryption is simply not designed for all uses.

Some users are worried about (possibly extremist) law enforcement getting access to their data. Thus, they want complete encryption protection that TrueNAS does not offer.


In theory, a TrueNAS server could be built with complete data pool encryption via keys. Then the boot-pool could be encrypted with SED, (Self Encryption Device / Disk). On reboot or power cycle, the user / SysAdmin of such a server would have to enter the SED password(s) to continue booting.

TrueNAS does not need to know or support SED on the boot-pool. The BIOS would take care of that.

If the users computer BIOS does not support SEDs, then a small USB drive with EFI boot could possibly take care of that. And since the USB EFI drive would have zero private information other than the prompt code for the SED un-locking, it does not mater that this USB EFI drive & code is un-encrypted.

Again, as I said, not something TrueNAS supports directly.

I understand the frustration, but it's also frustrating that it's not considered seriously and even brushed off like it's some kind of niche thing. I guess I shouldn't be surprised when considering the parallels to HTTPS and its poor adoption until this decade, despite folks pleading its necessity since the mid 90's. Just 10 years ago, one could go to Starbucks, put their NIC in promiscuous-mode, and lift cookies to gain access to their Facebook account -- a mega cap corporation with lots of enterprise hardware/solutions. SED seems to be an enterprise solution and poorly supported by consumer hardware. It can be voltage hacked and if the hardware has a fault, the drive's data is completely lost (verses using a software master key to decrypt whatever can be recovered). It's reminiscent of the hardware RAID vs ZFS debate.

There's a tradeoff between security and reliability, but the boot pool on SED with BIOS unlocking at boot, would be most secure approach I know of. IPMI would be used to allow remote booting. With a normal system this would be very secure, but if there is any unreliability due to hardware issues or immature software, then outages would be longer.

Agreed. But entering a single password into a web interface after a reboot is an inconvenience I think most would happily put up with.... kind of like having locks on your doors.

What's the story behind that?

On TrueNAS Core 13, I have the entire iocage dataset and all of its children encrypted, inheriting the root dataset's encryption. Everything runs without a hitch, seamlessly. Why can't this be the same for SCALE and its ix-applications dataset?

Exactly! I'm not asking for a novel approach to encryption. I'm simply asking, why does TrueNAS not properly take advantage of a well-supported ZFS feature? I expected some technical reasons and perhaps a link to a road map, but it seems even at the community level, we're still debating the merits of encryption .

 

[Arwen]

You've brought up some good points for small business and home users.

I wonder if a separate "/boot" partition could be employed on the boot device, that then un-locked "/" and switched to the newly un-locked "/" file system. It can prompt for a password on the console. This takes care of potentially un-reliable SED, (and trusting vendors to actually have both security clues and good firmware programmers). Plus, it takes care of your comment about loss of a block not damaging everything.


The Enterprise users tend to have decent security on their data centers, so theft is less likely. (One data center I visited regularly had a tall iron fence, guards, ID checks, pre-approval for visit, access to specific locations only, etc...)

 

[winnielinnie]

I wonder if a separate "/boot" partition could be employed on the boot device, that then un-locked "/" and switched to the newly un-locked "/" file system. It can prompt for a password on the console.

Since desktop Linux distros have always done that. I don't see why it can't be implemented with TrueNAS + ZFS.

 

[morganL]

Since desktop Linux distros have always done that. I don't see why it can't be implemented with TrueNAS + ZFS.

The desktop has a local user and very low reliability needs...

 

[Skrenes]

You've brought up some good points for small business and home users.

I wonder if a separate "/boot" partition could be employed on the boot device, that then un-locked "/" and switched to the newly un-locked "/" file system. It can prompt for a password on the console. This takes care of potentially un-reliable SED, (and trusting vendors to actually have both security clues and good firmware programmers). Plus, it takes care of your comment about loss of a block not damaging everything.


The Enterprise users tend to have decent security on their data centers, so theft is less likely. (One data center I visited regularly had a tall iron fence, guards, ID checks, pre-approval for visit, access to specific locations only, etc...)

Yeah, it's also my understanding that when using Enterprise with KMIP, the /data folder has no keys. As I had mentioned though, one could use dropbear-initramfs that allows one to unlock the LUKS partition remotely (through SSH), but I think if the sensitive data in the /data folder simply existed in System Dataset, it would go a long way.

The desktop has a local user and very low reliability needs...

LUKS and dropbear-initramfs are rock solid and supported by Debian stable going back many years. But again, while FDE is great, I think some thought into the placement of tokens, keys/credentials, would go a long way. I'm still deciding if I do this with TrueNAS or stick to what I have (Debian + LUKS/ZFS + Docker). I don't like the idea of cowboying TrueNAS.

 

[morganL]

Yeah, it's also my understanding that when using Enterprise with KMIP, the /data folder has no keys. As I had mentioned though, one could use dropbear-initramfs that allows one to unlock the LUKS partition remotely (through SSH), but I think if the sensitive data in the /data folder simply existed in System Dataset, it would go a long way.


LUKS and dropbear-initramfs are rock solid and supported by Debian stable going back many years. But again, while FDE is great, I think some thought into the placement of tokens, keys/credentials, would go a long way. I'm still deciding if I do this with TrueNAS or stick to what I have (Debian + LUKS/ZFS + Docker). I don't like the idea of cowboying TrueNAS.

I'd suggest focussing on the specific security issue you are trying to resolve.... if there's no good solution, then a suggestion (via report-a-bug) would make sense. It will require a new major release, so won't be a short turn-around.