Decryption of ZFS Pool not working anymore

[Sapp]

Hi,

my system is as follows:

Proxmox hosting several VMs, one of these is TrueNAS. I passed through the PCIE Controllers directly. The datasets are encrypted by using a passphrase. Was working flawlessly until today - my Server had an unexpected outage (I cannot figure out what went wrong, but it went unresponsive and came back to life after forced reboot).

All VMs came up as usual including TrueNAS - so far so good. Then I wanted to decrypt the Datasets: Passphrase is wrong. I compared password manager, local copy etc.: it is correct.

Do you know what might went wrong? I think it is quite unlikely that the passphrase of all datasets over multiple pools and disks is corrupted.

Any ideas what could cause that?

 

[sretalla]

I passed through the PCIE Controllers directly

What controller?

Any ideas what could cause that?

Pool metadata corruption?

 

[Sapp]

What controller?

The SSD and SATA controller:

Code:

00:17.0 SATA controller: Intel Corporation Cannon Lake PCH SATA AHCI Controller (rev 10)
0a:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe SSD Controller SM981/PM981/PM983

Pool metadata corruption?

According to pool status, it seems fine.

Code:

  pool: boot-pool
 state: ONLINE
status: Some supported and requested features are not enabled on the pool.
        The pool can still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
        the pool may no longer be accessible by software that does not support
        the features. See zpool-features(7) for details.
  scan: scrub repaired 0B in 00:00:03 with 0 errors on Sat Jul  8 03:45:05 2023
config:

        NAME        STATE     READ WRITE CKSUM
        boot-pool   ONLINE       0     0     0
          sda3      ONLINE       0     0     0

errors: No known data errors

  pool: ssd
 state: ONLINE
status: Some supported and requested features are not enabled on the pool.
        The pool can still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
        the pool may no longer be accessible by software that does not support
        the features. See zpool-features(7) for details.
  scan: scrub repaired 0B in 00:00:02 with 0 errors on Sun Jun 25 00:00:03 2023
config:

        NAME                                    STATE     READ WRITE CKSUM
        ssd                                     ONLINE       0     0     0
          4327697a-a14e-11ed-b6d2-152f576bb940  ONLINE       0     0     0

errors: No known data errors

  pool: storage-pool
 state: ONLINE
status: Some supported and requested features are not enabled on the pool.
        The pool can still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
        the pool may no longer be accessible by software that does not support
        the features. See zpool-features(7) for details.
  scan: scrub repaired 0B in 00:17:58 with 0 errors on Sun Jun 25 00:17:59 2023
config:

        NAME                                      STATE     READ WRITE CKSUM
        storage-pool                              ONLINE       0     0     0
          raidz1-0                                ONLINE       0     0     0
            08afcbef-a07c-11ed-a161-5911b2b13be6  ONLINE       0     0     0
            08ba5858-a07c-11ed-a161-5911b2b13be6  ONLINE       0     0     0
            08b505f4-a07c-11ed-a161-5911b2b13be6  ONLINE       0     0     0
        cache
          08353f18-a07c-11ed-a161-5911b2b13be6    ONLINE       0     0     0

errors: No known data errors

I didnt upgrade to not cause any more issues.

 

[sretalla]

can you get it to unlock with the API?

midclt call pool.dataset.unlock tank/data/encrypted '{ "key_file": false,"recursive": false,"datasets": [{"name" : “tank/data/encrypted” , "passphrase" : “somepassword”}]}’

 

[Sapp]

Hi, just tried that, but it just responds with some number (I assume a request counter?) and the dataset is still locked.

Code:

admin@truenas[~]$ midclt call pool.dataset.unlock storage-pool/Development '{ "key_file": false,"recursive": false,"datasets": [{"name" : "storage-pool/Development" , "passphrase" : "<some pass>"}]}'
2453

 

[sretalla]

(I assume a request counter?)

That's a Job.ID.
You should be able to see it with its status in the Jobs list (top right of the UI... picture of a clipboard/tasklist).

 

[Sapp]

That's a Job.ID.
You should be able to see it with its status in the Jobs list (top right of the UI... picture of a clipboard/tasklist).

Ah, got it. It doesn't indicate any errors, but as said not unlocked still.

 

[Davvo]

If I'm not wrong this situation is precisely the reason you need a SLOG with PLP. Try to search about this is in the forum... Pretty sure @jgreco was part of such a thread.

The following might be useful.

Some insights into SLOG/ZIL with ZFS on FreeNAS

What is the ZIL? POSIX provides a facility for the system or an application to make sure that data requested to be written is actually committed to stable storage: a synchronous write request. Upon completion of a sync write request, the underlying filesystem is supposed to guarantee that a...

www.truenas.com

 

[Sapp]

If I'm not wrong this situation is precisely the reason you need a SLOG with PLP. Try to search about this is in the forum... Pretty sure @jgreco was part of such a thread.

The following might be useful.

Some insights into SLOG/ZIL with ZFS on FreeNAS

What is the ZIL? POSIX provides a facility for the system or an application to make sure that data requested to be written is actually committed to stable storage: a synchronous write request. Upon completion of a sync write request, the underlying filesystem is supposed to guarantee that a...

www.truenas.com

Okay, I was thinking that the SLOG basically just protects the integrity of the stored data (i.e., corrupted data blocks due to interrupted transactions). Could that also corrupt the metadata sector while not updating any meta information. I don't know how ZFS' layout is like, but I would assume they store it in the first/last sector and don't update them frequently (and at once) - do they?

 

[sretalla]

Ah, got it. It doesn't indicate any errors, but as said not unlocked still.

You see that is says "success" though, right?

Are you absolutely sure (noting that the command you used was not recursive, so only that top-level dataset would be unlocked)?

 

[sretalla]

At the very least, the lack of error there tells me the password you're using is still OK. (you can try with a deliberately incorrect one to see if it still reports success or not).

Also note: if the real status is unlocked, unlock may fail due to that.

 

[Sapp]

You see that is says "success" though, right?

Are you absolutely sure (noting that the command you used was not recursive, so only that top-level dataset would be unlocked)?

Yes, that's what is confusing me even more. It still does not show up though. Neither it is mounted (/mnt/ssd/Kubernetes is empty) nor it is shown in the GUI:

Also note: if the real status is unlocked, unlock may fail due to that.

When repeatetly calling the unlock API call, it still succeedes.

 

[winnielinnie]

Why not skip the API, GUI, and middleware for now to see if you can at least unlock and access the datasets using the command-line?

Code:

zfs list -r -t filesystem -o name,encroot,encryption,keyformat,keylocation,keystatus storage-pool

Then try to do a "dry run" to unlock the enryptionroot, which I'm assuming is "Development". Does it say "wrong passphrase"?

Code:

zfs load-key -n storage-pool/Development

If it looks good, try to unlock it for real this time:

Code:

zfs load-key storage-pool/Development

Then check the status again:

Code:

zfs list -r -t filesystem -o name,encroot,encryption,keyformat,keylocation,keystatus storage-pool

The above commands will not automatically mount the dataset(s). This is only a test to see if you can indeed decrypt the encrypted dataset.


You can "re-lock" the dataset like so:

Code:

zfs unload-key storage-pool/Development

 

[Sapp]

Why not skip the API, GUI, and middleware for now to see if you can at least unlock and access the datasets using the command-line?

Code:

zfs list -r -t filesystem -o name,encroot,encryption,keyformat,keylocation,keystatus storage-pool

Then try to do a "dry run" to unlock the enryptionroot, which I'm assuming is "Development". Does it say "wrong passphrase"?

Code:

zfs load-key -n storage-pool/Development

If it looks good, try to unlock it for real this time:

Code:

zfs load-key storage-pool/Development

Then check the status again:

Code:

zfs list -r -t filesystem -o name,encroot,encryption,keyformat,keylocation,keystatus storage-pool

The above commands will not automatically mount the dataset(s). This is only a test to see if you can indeed decrypt the encrypted dataset.


You can "re-lock" the dataset like so:

Code:

zfs unload-key storage-pool/Development

Hey, thanks for the reply. Already tried that (forgot to mention that, sorry!).
ZFS also states that the key is invalid:

Code:

admin@truenas[~]$ sudo zfs load-key -n storage-pool/Development
Enter passphrase for 'storage-pool/Development':
Key load error: Incorrect key provided for 'storage-pool/Development'.

So it seems like something on the ZFS-layer has been messed up

 

[winnielinnie]

What about the other commands? To get an overview of the layout and encryptionroot hierarchy.

 

[sretalla]

OK, so via the API, it seems to unlock... so can you open the next level of datasets and see if they are still locked?

Also does the command from @winnielinnie show it's unlocked?

zfs list -r -t filesystem -o name,encroot,encryption,keyformat,keylocation,keystatus storage-pool

 

[Sapp]

What about the other commands? To get an overview of the layout and encryptionroot hierarchy.

Sure.

Code:

admin@truenas[~]$ sudo zfs list -r -t filesystem -o name,encroot,encryption,keyformat,keylocation,keystatus storage-pool
NAME                                                           ENCROOT                             ENCRYPTION   KEYFORMAT   KEYLOCATION  KEYSTATUS
storage-pool                                                   storage-pool                        aes-256-gcm  hex         prompt       available
storage-pool/.system                                           storage-pool                        aes-256-gcm  hex         none         available
storage-pool/.system/configs-cadb1ce96f8c4a01a65be3ef8f5cb996  storage-pool                        aes-256-gcm  hex         none         available
storage-pool/.system/cores                                     storage-pool                        aes-256-gcm  hex         none         available
storage-pool/.system/ctdb_shared_vol                           storage-pool                        aes-256-gcm  hex         none         available
storage-pool/.system/glusterd                                  storage-pool                        aes-256-gcm  hex         none         available
storage-pool/.system/rrd-cadb1ce96f8c4a01a65be3ef8f5cb996      storage-pool                        aes-256-gcm  hex         none         available
storage-pool/.system/samba4                                    storage-pool                        aes-256-gcm  hex         none         available
storage-pool/.system/services                                  storage-pool                        aes-256-gcm  hex         none         available
storage-pool/.system/syslog-cadb1ce96f8c4a01a65be3ef8f5cb996   storage-pool                        aes-256-gcm  hex         none         available
storage-pool/.system/webui                                     storage-pool                        aes-256-gcm  hex         none         available
storage-pool/Development                                       storage-pool/Development            aes-256-gcm  passphrase  prompt       unavailable
storage-pool/Kubernetes                                        storage-pool/Kubernetes             aes-256-gcm  passphrase  prompt       unavailable
storage-pool/Kubernetes/Active                                 storage-pool/Kubernetes/Active      aes-256-gcm  passphrase  prompt       unavailable
storage-pool/Kubernetes/Backup                                 storage-pool/Kubernetes/Backup      aes-256-gcm  passphrase  prompt       unavailable
storage-pool/Personal Data                                     storage-pool/Personal Data          aes-256-gcm  passphrase  prompt       unavailable
storage-pool/Personal Data/Maurice                             storage-pool/Personal Data/Maurice  aes-256-gcm  passphrase  prompt       unavailable
storage-pool/Personal Data/Maurice/maurice                     storage-pool/Personal Data/Maurice  aes-256-gcm  passphrase  none         unavailable
storage-pool/Personal Data/Shanine                             storage-pool/Personal Data/Shanine  aes-256-gcm  passphrase  prompt       unavailable
storage-pool/iocage                                            storage-pool                        aes-256-gcm  hex         none         available
storage-pool/iocage/download                                   storage-pool                        aes-256-gcm  hex         none         available
storage-pool/iocage/images                                     storage-pool                        aes-256-gcm  hex         none         available
storage-pool/iocage/jails                                      storage-pool                        aes-256-gcm  hex         none         available
storage-pool/iocage/log                                        storage-pool                        aes-256-gcm  hex         none         available
storage-pool/iocage/releases                                   storage-pool                        aes-256-gcm  hex         none         available
storage-pool/iocage/templates                                  storage-pool                        aes-256-gcm  hex         none         available
storage-pool/ix-applications                                   -                                   off          none        none         -
storage-pool/ix-applications/catalogs                          -                                   off          none        none         -
storage-pool/ix-applications/default_volumes                   -                                   off          none        none         -
storage-pool/ix-applications/docker                            -                                   off          none        none         -
storage-pool/ix-applications/k3s                               -                                   off          none        none         -
storage-pool/ix-applications/k3s/kubelet                       -                                   off          none        none         -
storage-pool/ix-applications/releases                          -                                   off          none        none         -

Code:

admin@truenas[~]$ sudo zfs unload-key storage-pool/Development Key unload error: Key already unloaded for 'storage-pool/Development'.

 

[winnielinnie]

I understand the need for privacy and ambiguity, which can be violated by sharing dataset names and such. But in cases like these, it really helps to "unblind" those trying to help. Giving us a full overview of the layout and status (in the command-line, not a screenshot) will minimize the chances of going off on wild goose chases.

You can always use "fake" names in the output that you share on here.

EDIT: Posted at the same time.

 

[winnielinnie]

This shows it was not unlocked.

storage-pool/Development storage-pool/Development aes-256-gcm passphrase prompt unavailable

Does your pool's history show any change to the dataset's key?

Code:

zpool history storage-pool | grep change

An alternative means to check (run this command as well):

Code:

zpool history storage-pool | grep set | grep key

 

[Sapp]

Does your pool's history show any change to the dataset's key?

Code:

zpool history storage-pool | grep change

An alternative means to check:

Code:

zpool history storage-pool | grep set | grep key

First command doesn't find any results.

Second just greps what has been set when setting up TrueNAS:

Code:

admin@truenas[~]$ sudo zpool history storage-pool | grep set | grep key
2023-01-30.09:56:50  zfs set keylocation=prompt storage-pool
2023-01-30.09:59:12  zfs set keylocation=prompt storage-pool/Development
2023-01-30.10:00:05  zfs set keylocation=prompt storage-pool/Personal Data
2023-01-30.10:00:40  zfs set keylocation=prompt storage-pool/Backup
2023-01-30.10:07:54  zfs set keylocation=prompt storage-pool/Personal Data/Maurice
2023-01-30.10:08:10  zfs set keylocation=prompt storage-pool/Personal Data/Shanine
2023-01-30.10:12:12  zfs set keylocation=prompt storage-pool/Development/Dev
2023-01-31.02:20:47  zfs set keylocation=prompt storage-pool/Kubernetes
2023-01-31.02:24:20  zfs set keylocation=prompt storage-pool/Kubernetes/Backup
2023-01-31.02:24:33  zfs set keylocation=prompt storage-pool/Kubernetes/Postgres
2023-01-31.11:08:19  zfs set keylocation=prompt storage-pool/Kubernetes/Active