SOLVED How to best resolve this warning? "datasets are not encrypted but are within an encrypted dataset" [22.12.3]

[sheps]

After updating to 22.12.3, I have the following warning:

The following datasets are not encrypted but are within an encrypted dataset: 'coolpool/backups/ix-applications, dualpool/ix-applications' which is not supported behaviour and may lead to various issues.

I see in the release notes that this probably stemmed from a bug, as I don't remember deliberately having my dataset as unencrypted.

I suppose I'd need to copy my ix-applications dataset to an encrypted one, right? Could someone recommend best practices? Thank you.

 

[winnielinnie]

Can you post the output of:

Code:

zfs list -r -t filesystem -o name,encryption,encroot poolname

I suppose I'd need to copy my ix-applications dataset to an encrypted one, right? Could someone recommend best practices? Thank you.

If the "ix-applications" dataset works like "iocage" in Core, you might be able to move it off the pool (to another pool) then back on, in which it will inherit the encryption properties of the root dataset?

But I'm not sure how feasible this is on SCALE.

 

[sheps]

Thanks for the reply. Here's the output.

Code:

admin@chonky[~]$ sudo zfs list -r -t filesystem -o name,encryption,encryptionroot coolpool
NAME                                                                                                 ENCRYPTION   ENCROOT
coolpool                                                                                             aes-256-gcm  coolpool
coolpool/.system                                                                                     aes-256-gcm  coolpool
coolpool/.system/configs-8f179c8648fc4419af075a5cf26c19f8                                            aes-256-gcm  coolpool
coolpool/.system/cores                                                                               aes-256-gcm  coolpool
coolpool/.system/ctdb_shared_vol                                                                     aes-256-gcm  coolpool
coolpool/.system/glusterd                                                                            aes-256-gcm  coolpool
coolpool/.system/rrd-8f179c8648fc4419af075a5cf26c19f8                                                aes-256-gcm  coolpool
coolpool/.system/samba4                                                                              aes-256-gcm  coolpool
coolpool/.system/services                                                                            aes-256-gcm  coolpool
coolpool/.system/syslog-8f179c8648fc4419af075a5cf26c19f8                                             aes-256-gcm  coolpool
coolpool/.system/webui                                                                               aes-256-gcm  coolpool
coolpool/applications                                                                                aes-256-gcm  coolpool
coolpool/applications/drawio                                                                         aes-256-gcm  coolpool
coolpool/applications/kiwix                                                                          aes-256-gcm  coolpool
coolpool/backups                                                                                     aes-256-gcm  coolpool
coolpool/backups/fractal                                                                             aes-256-gcm  coolpool
coolpool/backups/home                                                                                aes-256-gcm  coolpool/backups/home
coolpool/backups/home/shervin                                                                        aes-256-gcm  coolpool/backups/home/shervin
coolpool/backups/ix-applications                                                                     off          -
coolpool/backups/ix-applications/catalogs                                                            off          -
coolpool/backups/ix-applications/default_volumes                                                     off          -
coolpool/backups/ix-applications/docker                                                              off          -
coolpool/backups/ix-applications/k3s                                                                 off          -
coolpool/backups/ix-applications/k3s/kubelet                                                         off          -
coolpool/backups/ix-applications/releases                                                            off          -
coolpool/backups/ix-applications/releases/cert-manager                                               off          -
coolpool/backups/ix-applications/releases/cert-manager/charts                                        off          -
coolpool/backups/ix-applications/releases/cert-manager/volumes                                       off          -
coolpool/backups/ix-applications/releases/cert-manager/volumes/ix_volumes                            off          -
coolpool/backups/ix-applications/releases/cloudflared                                                off          -
coolpool/backups/ix-applications/releases/cloudflared/charts                                         off          -
coolpool/backups/ix-applications/releases/cloudflared/volumes                                        off          -
coolpool/backups/ix-applications/releases/cloudflared/volumes/ix_volumes                             off          -
coolpool/backups/ix-applications/releases/drawio                                                     off          -
coolpool/backups/ix-applications/releases/drawio/charts                                              off          -
coolpool/backups/ix-applications/releases/drawio/volumes                                             off          -
coolpool/backups/ix-applications/releases/drawio/volumes/ix_volumes                                  off          -
coolpool/backups/ix-applications/releases/jellyfin                                                   off          -
coolpool/backups/ix-applications/releases/jellyfin/charts                                            off          -
coolpool/backups/ix-applications/releases/jellyfin/volumes                                           off          -
coolpool/backups/ix-applications/releases/jellyfin/volumes/ix_volumes                                off          -
coolpool/backups/ix-applications/releases/jellyfin/volumes/pvc-92d63cb7-9616-4be5-9969-c1039de3de9d  off          -
coolpool/backups/ix-applications/releases/lidarr                                                     off          -
coolpool/backups/ix-applications/releases/lidarr/charts                                              off          -
coolpool/backups/ix-applications/releases/lidarr/volumes                                             off          -
coolpool/backups/ix-applications/releases/lidarr/volumes/ix_volumes                                  off          -
coolpool/backups/ix-applications/releases/lidarr/volumes/pvc-faa3be5a-71ec-4a6c-9d2f-e42849114bb2    off          -
coolpool/backups/ix-applications/releases/pihole                                                     off          -
coolpool/backups/ix-applications/releases/pihole/charts                                              off          -
coolpool/backups/ix-applications/releases/pihole/volumes                                             off          -
coolpool/backups/ix-applications/releases/pihole/volumes/ix_volumes                                  off          -
coolpool/backups/ix-applications/releases/pihole/volumes/pvc-57a145a3-f410-43bd-ad30-13d1649894da    off          -
coolpool/backups/ix-applications/releases/pihole/volumes/pvc-97e28735-ce24-4daf-9ac1-59c7bb246de9    off          -
coolpool/backups/ix-applications/releases/prowlarr                                                   off          -
coolpool/backups/ix-applications/releases/prowlarr/charts                                            off          -
coolpool/backups/ix-applications/releases/prowlarr/volumes                                           off          -
coolpool/backups/ix-applications/releases/prowlarr/volumes/ix_volumes                                off          -
coolpool/backups/ix-applications/releases/prowlarr/volumes/pvc-bb1721c7-7fc4-43ad-a13e-844341317ce4  off          -
coolpool/backups/ix-applications/releases/radarr                                                     off          -
coolpool/backups/ix-applications/releases/radarr/charts                                              off          -
coolpool/backups/ix-applications/releases/radarr/volumes                                             off          -
coolpool/backups/ix-applications/releases/radarr/volumes/ix_volumes                                  off          -
coolpool/backups/ix-applications/releases/radarr/volumes/pvc-97e72aa2-32d5-473b-91b7-a9213508c1e8    off          -
coolpool/backups/ix-applications/releases/readarr                                                    off          -
coolpool/backups/ix-applications/releases/readarr/charts                                             off          -
coolpool/backups/ix-applications/releases/readarr/volumes                                            off          -
coolpool/backups/ix-applications/releases/readarr/volumes/ix_volumes                                 off          -
coolpool/backups/ix-applications/releases/readarr/volumes/pvc-50eec42d-ca4f-459f-baa4-84e153f0e139   off          -
coolpool/backups/ix-applications/releases/scale                                                      off          -
coolpool/backups/ix-applications/releases/scale/charts                                               off          -
coolpool/backups/ix-applications/releases/scale/volumes                                              off          -
coolpool/backups/ix-applications/releases/scale/volumes/ix_volumes                                   off          -
coolpool/backups/ix-applications/releases/sonarr                                                     off          -
coolpool/backups/ix-applications/releases/sonarr/charts                                              off          -
coolpool/backups/ix-applications/releases/sonarr/volumes                                             off          -
coolpool/backups/ix-applications/releases/sonarr/volumes/ix_volumes                                  off          -
coolpool/backups/ix-applications/releases/sonarr/volumes/pvc-2688839c-41a3-4d1b-b88d-3f6911459ac5    off          -
coolpool/backups/ix-applications/releases/syncthing                                                  off          -
coolpool/backups/ix-applications/releases/syncthing/charts                                           off          -
coolpool/backups/ix-applications/releases/syncthing/volumes                                          off          -
coolpool/backups/ix-applications/releases/syncthing/volumes/ix_volumes                               off          -
coolpool/backups/ix-applications/releases/syncthing/volumes/ix_volumes/ix-syncthing_config           off          -
coolpool/backups/ix-applications/releases/traefik                                                    off          -
coolpool/backups/ix-applications/releases/traefik/charts                                             off          -
coolpool/backups/ix-applications/releases/traefik/volumes                                            off          -
coolpool/backups/ix-applications/releases/traefik/volumes/ix_volumes                                 off          -
coolpool/backups/macos                                                                               aes-256-gcm  coolpool
coolpool/backups/macos/megan                                                                         aes-256-gcm  coolpool
coolpool/backups/macos/shervin                                                                       aes-256-gcm  coolpool
coolpool/data                                                                                        aes-256-gcm  coolpool
coolpool/data/3d-printing                                                                            aes-256-gcm  coolpool
coolpool/data/bungalow                                                                               aes-256-gcm  coolpool
coolpool/data/games                                                                                  aes-256-gcm  coolpool
coolpool/multimedia                                                                                  aes-256-gcm  coolpool
coolpool/syncthing                                                                                   aes-256-gcm  coolpool

 

[winnielinnie]

The heck is going on with SCALE?

Why did the dataset "ix-applications" NOT inherit the encryption properties of your pool's root dataset?

@morganL: What is this blasphemy?

On the one hand "unencrypted datasets beneath an encrypted parent/root is unsupported", yet on the other hand the "ix-applications" dataset is created as an unencrypted dataset... beneath an encrypted root dataset?

I don't use SCALE, but I'm assuming the user is not prompted about encrypting the "ix-applications" dataset (or inheriting such a property) upon first-time setup of "Apps"?

 

[sheps]

The only guidance I see is this blurb in the release notes.

This release fixes a bug with dataset encryption where it was possible to create an encrypted storage pool or dataset and unencrypted datasets within that pool or dataset. Beginning with 22.12.3, it is no longer possible to create an unencrypted dataset when the storage pool or dataset is created with encryption active. Datasets created in this manner are not affected by this fix. If the original intention was for the dataset to be encrypted, please migrate any data from the unencrypted dataset to a new encrypted dataset.

I believe ix-applications created itself when I first installed an app, and I assumed it would be encrypted given the hierarchy (and didn't notice otherwise).

 

[winnielinnie]

I assumed it would be encrypted given the hierarchy

Any reasonable person would have assumed the same, including myself.

This is how it operates on Core (for the "iocage" dataset.)

 

[sheps]

I'm not sure how to migrate my dataset, but I was considering zfs send/receive. Do you think this is appropriate or is there a nicer way?

 

[winnielinnie]

I'm the wrong person to ask. I only use Core, and must rely on second-hand experience from SCALE users.

I don't believe it's as seamless as "just send/recv off the pool to another pool, then back to the original pool, and your Apps will continue to work as if nothing happened." (There's K3s that runs under-the-hood, which might not take kindly to pulling the rug from underneath its feet as you move datasets around on-the-cuff.)

Someone who uses SCALE can hopefully chime in with the appropriate method.

 

[morganL]

I'm not sure how to migrate my dataset, but I was considering zfs send/receive. Do you think this is appropriate or is there a nicer way?

Its a warning... and not desirable behaviour.

I assume there was nothing special about how the iX-applications dataset was created?

In any case, feel free to report this as a bug.... if you have time we'll work out how best to resolve.

 

[winnielinnie]

If there's a recommendation not to use unencrypted datasets underneath an encrypted parent or encrypted root dataset...

Plus, TrueNAS SCALE (and Core) will no longer allow you to create an unencrypted dataset underneath an encrypted dataset...

Then how come the "ix-applications" dataset was created as an unencrypted dataset underneath an encrypted parent upon the first use of an App?

Is this no longer true? (e.g, Had @sheps used a fresh installation of SCALE 22.12.3, would his "ix-applications" dataset been encrypted (inherited) upon the first use of an App?)

Can you see how this sends mixed messages?

 

[sheps]

I assume there was nothing special about how the iX-applications dataset was created?

Nothing special of which I'm aware. If memory recalls, the ix-application dataset was automatically created after installing my first application, which was syncthing from the truenas catalog.

This post and this other post from a week ago also mention the bug.

@sammael , @pookNast - did either of you two fix the issue behind your "unencrypted dataset within an encrypted pool"? If so, I'd appreciate some insight into how to best fix it. Thanks!

 

[morganL]

If there's a recommendation not to use unencrypted datasets underneath an encrypted parent or encrypted root dataset...

Plus, TrueNAS SCALE (and Core) will no longer allow you to create an unencrypted dataset underneath an encrypted dataset...

Then how come the "ix-applications" dataset was created as an unencrypted dataset underneath an encrypted parent upon the first use of an App?

Is this no longer true? (e.g, Had @sheps used a fresh installation of SCALE 22.12.3, would his "ix-applications" dataset been encrypted (inherited) upon the first use of an App?)

Can you see how this sends mixed messages?

I think the unanswered question is when was the ix-applications dataset created... I assume it was prior to 22.12.3. If not that is a bug.

We acknowledge that the problem of unencrypted datasets within encrypted datasets existed... everything worked, except for replication.

 

[sheps]

I think the unanswered question is when was the ix-applications dataset created... I assume it was prior to 22.12.3. If not that is a bug.

Yes, prior to 22.12.3. My setup should have been around March of this year, so I might have been on 22.12.1.

 

[winnielinnie]

I think the unanswered question is when was the ix-applications dataset created... I assume it was prior to 22.12.3.

Then why was it explicitly created as an unencrypted dataset beneath an encrypted root dataset? Why wouldn't it inherit the encryption properties, in the same vein that the "iocage" dataset on Core does?

 

[notsure]

I think the unanswered question is when was the ix-applications dataset created... I assume it was prior to 22.12.3. If not that is a bug.

We acknowledge that the problem of unencrypted datasets within encrypted datasets existed... everything worked, except for replication.

I have attempted to recreate the ix-applications volume under 22.12.3.1 and it's still created as unencrypted with encrypted parent.

Looks like a bug to me.

 

[Ataraxia]

Just chiming in, new user and hit this same issue during the initial setup on a new environment.

Steps to reach the error were: installed from 22.12.3.1 iso today (~3hrs ago), ran through initial setup (set up admin and password rather than used root or set up later option), reboot when prompted, navigate to the web UI's IP from another box, built initial dataset in the web UI, went to apps in the web UI, selected that dataset to be used for apps when prompted to select a dataset. This is prior to installation of any app from the catalog, it appears to be happening during the initial setup to make room for installing apps from the catalog.

 

[morganL]

I have attempted to recreate the ix-applications volume under 22.12.3.1 and it's still created as unencrypted with encrypted parent.

Looks like a bug to me.

Agreed.. can you report-a-bug and provide the NAS ticket ID.

 

[notsure]

Agreed.. can you report-a-bug and provide the NAS ticket ID.

Sure, here you go: https://ixsystems.atlassian.net/browse/NAS-122617

Thanks!

 

[Kris Moore]

The decision for ix-applications to be created unencrypted was intentional:

https://github.com/truenas/middleware/pull/9954

Bottom line is, ZFS encryption should be used for pretty static datasets. It's not great for parent datasets where lots of dynamic datasets are coming and going frequently, which is the case for ix-applications due to how containers are checked out and snapped.

 

[morganL]

The decision for ix-applications to be created unencrypted was intentional:

https://github.com/truenas/middleware/pull/9954

Bottom line is, ZFS encryption should be used for pretty static datasets. It's not great for parent datasets where lots of dynamic datasets are coming and going frequently, which is the case for ix-applications due to how containers are checked out and snapped.

So, if this is the case, we need to clearly indicate that the ix-application dataset should be created in an unencrypted pool or parent dataset,

So lets report as a docs issue.....