SOLVED (help pls) Problem with decryption after disk swap

[elpatron80]

Hi there,
I'm asking you guys for a little help. My situation is:

TrueNAS 12.0 installed on 1x SSD 128GB
Pool1 was a MIRROR on 2x HDD WD 1TB
Pool1 ENCRYPTED

Few days ago I've changed Pool1 disks from 2xWD (mentioned above) to 2x Seagate Ironwolf 4TB.
Pool has been resilvered without any issues (via GUI).

Today I updated TrueNAS 12 to 12u1 (config file saved with passwd seed before upgrade).
And my SSD (with boot-pool) crashed... some CRC errors and cannot load GUI.
So I wiped that SSD with another PC and set it back to TrueNAS and performed clean installation of TrueNAS 12, then restored config file.

From that moment I cannot get in to Pool1 (encrypted) volume. The status is LOCKED and I cannot unlock it with password, nor with GELI recovery key...

I tried this: https://www.truenas.com/community/threads/pool-unlock-fails-with-geli-key.82670/post-572371

And in terminal everything seems to be good but it is not.

I performed:
geli attach -k /root/2020-10-07_pool_Pool1_encryption.key /dev/gptid/b2e57fc4-34e5-11eb-9574-7071bcc181ef Enter passphrase:
OK - no errors
geli attach -k /root/2020-10-07_pool_Pool1_encryption.key /dev/gptid/b2e57fc4-34e5-11eb-9574-7071bcc181ef Enter passphrase:
OK - no errors

zpool import
pool: Pool1
id: 8155427189674573108
state: ONLINE
action: The pool can be imported using its name or numeric identifier.
config:

Pool1 ONLINE
mirror-0 ONLINE
gptid/23a8f85f-34d3-11eb-a5d2-7071bcc181ef.eli ONLINE
gptid/b2e57fc4-34e5-11eb-9574-7071bcc181ef.eli ONLINE

zpool status
pool: boot-pool
state: ONLINE
config:

NAME STATE READ WRITE CKSUM
boot-pool ONLINE 0 0 0
ada2p2 ONLINE 0 0 0

errors: No known data errors

BUT: In GUI I can see Pool1 status: OFFLINE (it seems to be mounted with all options available, but it's offline and I cannot access data).

Another thing:
sqlite3 /data/freenas-v1.db 'select sed.encrypted_provider from storage_encrypteddisk as sed join storage_volume sv on sv.id = sed.encrypted_volume_id where sv.vol_name = "Pool1";'
gptid/23a8f85f-34d3-11eb-a5d2-7071bcc181ef
gptid/b2e57fc4-34e5-11eb-9574-7071bcc181ef

But in /dev/gptid/ I see:
23a8f85f-34d3-11eb-a5d2-7071bcc181ef
872a103f-3b02-11eb-b671-7071bcc181ef
8725f046-3b02-11eb-b671-7071bcc181ef
b2e57fc4-34e5-11eb-9574-7071bcc181ef

And when I tried the same with 2 other disks I got an error message:
geli attach -k /root/2020-10-07_pool_Pool1_encryption.key /dev/gptid/b2e57fc4-34e5-11eb-9574-7071bcc181ef
Enter passphrase:
geli: Wrong key for gptid/b2e57fc4-34e5-11eb-9574-7071bcc181ef.
geli: There was an error with at least one provider.

I don't have any other geli keys... I have only encryption key and recovery key exported when the previous disks (2xWD) were in game. And GUI didn't give any info to make a backup of new (?) encryption keys...

Any suggestions what should I do? Will be very grateful.

 

[elpatron80]

In GUI when I'm trying to Unlock pool normally, I get a message like this:
-----

FAILED
[EFAULT] Pool could not be imported: 2 devices failed to decrypt.

Error: concurrent.futures.process._RemoteTraceback:
"""
Traceback (most recent call last):
File "/usr/local/lib/python3.8/concurrent/futures/process.py", line 239, in _process_worker
r = call_item.fn(*call_item.args, **call_item.kwargs)
File "/usr/local/lib/python3.8/site-packages/middlewared/worker.py", line 91, in main_worker
res = MIDDLEWARE._run(*call_args)
File "/usr/local/lib/python3.8/site-packages/middlewared/worker.py", line 45, in _run
return self._call(name, serviceobj, methodobj, args, job=job)
File "/usr/local/lib/python3.8/site-packages/middlewared/worker.py", line 39, in _call
return methodobj(*params)
File "/usr/local/lib/python3.8/site-packages/middlewared/worker.py", line 39, in _call
return methodobj(*params)
File "/usr/local/lib/python3.8/site-packages/middlewared/schema.py", line 977, in nf
return f(*args, **kwargs)
File "/usr/local/lib/python3.8/site-packages/middlewared/plugins/zfs.py", line 371, in import_pool
self.logger.error(
File "libzfs.pyx", line 391, in libzfs.ZFS.__exit__
File "/usr/local/lib/python3.8/site-packages/middlewared/plugins/zfs.py", line 362, in import_pool
raise CallError(f'Pool {name_or_guid} not found.', errno.ENOENT)
middlewared.service_exception.CallError: [ENOENT] Pool 8155427189674573108 not found.
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/middlewared/plugins/pool_/encryption_freebsd.py", line 272, in unlock
await self.middleware.call('zfs.pool.import_pool', pool['guid'], {
File "/usr/local/lib/python3.8/site-packages/middlewared/main.py", line 1238, in call
return await self._call(
File "/usr/local/lib/python3.8/site-packages/middlewared/main.py", line 1203, in _call
return await self._call_worker(name, *prepared_call.args)
File "/usr/local/lib/python3.8/site-packages/middlewared/main.py", line 1209, in _call_worker
return await self.run_in_proc(main_worker, name, args, job)
File "/usr/local/lib/python3.8/site-packages/middlewared/main.py", line 1136, in run_in_proc
return await self.run_in_executor(self.__procpool, method, *args, **kwargs)
File "/usr/local/lib/python3.8/site-packages/middlewared/main.py", line 1110, in run_in_executor
return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs))
middlewared.service_exception.CallError: [ENOENT] Pool 8155427189674573108 not found.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/middlewared/job.py", line 361, in run
await self.future
File "/usr/local/lib/python3.8/site-packages/middlewared/job.py", line 397, in __run_body
rv = await self.method(*([self] + args))
File "/usr/local/lib/python3.8/site-packages/middlewared/schema.py", line 973, in nf
return await f(*args, **kwargs)
File "/usr/local/lib/python3.8/site-packages/middlewared/plugins/pool_/encryption_freebsd.py", line 286, in unlock
raise CallError(msg)
middlewared.service_exception.CallError: [EFAULT] Pool could not be imported: 2 devices failed to decrypt.

 

[elpatron80]

Is it possible that after disks swap new encryption keys has been generated?
If yes - why there was any info about that (and no tip to make backup of new keys)?
If no - is there any way to make it work (ex. insert new disks ID to db?) or should I put old drives back (WD) instead of IronWolfs and start over?

 

[gary_1]

On pre 12 versions of truenas, if you resilver a disk on a pool that was unlocked with your regular key, the new disk will only be possible to be unlocked with the regular key. The recovery key will not exist for it (or is different, either way won't work). If on the other hand, you'd unlocked the pool using the recovery key, then resilvered, I believe the reverse is true and the new disk can only be unlocked in the future with the recovery key. That's why the manual suggests you rekey after resilvering and download both the new regular and recover keys as it allows all drives in the pool to regain access to the same two keys.

One of the two keys you have should work though if you ran into that issue.

That said, I have no idea if anything has changed under version 12+ for geli.

Are you sure the recovery key or regular key you have is actually up to date. Have you used them recently to unlock the pool? Or were you allowing it to auto-unlock on boot?