Unable to unencrypt pool after upgrading to 11.3 RELEASE

[Fierce]

Looking for guidance. I upgraded from 11.2-U7 to 11.3 RELEASE and found I could not un-encrypt my pool (named "Pool") with the passphrase or the key, geli.key. Immediately I reverted back to 11.2-U7, but still could not un-encrypt the pool. Next I exported the pool thinking that if I re-import it I might be able to un-encrypt it (perhaps unwisely). When attempting to import the pool, the pool would not show up in the import dialog.

I next moved to another prior release (don't remember which one) where the encrypted pool was already showing in the dashboard (didn't need to be imported). Upon attempting to unlock it I got this error message:

Code:

Error: concurrent.futures.process._RemoteTraceback:
"""
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/concurrent/futures/process.py", line 239, in _process_worker
    r = call_item.fn(*call_item.args, **call_item.kwargs)
  File "/usr/local/lib/python3.7/site-packages/middlewared/worker.py", line 95, in main_worker
    res = loop.run_until_complete(coro)
  File "/usr/local/lib/python3.7/asyncio/base_events.py", line 579, in run_until_complete
    return future.result()
  File "/usr/local/lib/python3.7/site-packages/middlewared/worker.py", line 51, in _run
    return await self._call(name, serviceobj, methodobj, params=args, job=job)
  File "/usr/local/lib/python3.7/site-packages/middlewared/worker.py", line 43, in _call
    return methodobj(*params)
  File "/usr/local/lib/python3.7/site-packages/middlewared/worker.py", line 43, in _call
    return methodobj(*params)
  File "/usr/local/lib/python3.7/site-packages/middlewared/schema.py", line 964, in nf
    return f(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/middlewared/plugins/zfs.py", line 382, in import_pool
    zfs.import_pool(found, found.name, options, any_host=any_host)
  File "libzfs.pyx", line 369, in libzfs.ZFS.__exit__
  File "/usr/local/lib/python3.7/site-packages/middlewared/plugins/zfs.py", line 380, in import_pool
    raise CallError(f'Pool {name_or_guid} not found.', errno.ENOENT)
middlewared.service_exception.CallError: [ENOENT] Pool 17998721979278846804 not found.
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/middlewared/plugins/pool.py", line 1656, in unlock
    'cachefile': ZPOOL_CACHE_FILE,
  File "/usr/local/lib/python3.7/site-packages/middlewared/main.py", line 1127, in call
    app=app, pipes=pipes, job_on_progress_cb=job_on_progress_cb, io_thread=True,
  File "/usr/local/lib/python3.7/site-packages/middlewared/main.py", line 1074, in _call
    return await self._call_worker(name, *args)
  File "/usr/local/lib/python3.7/site-packages/middlewared/main.py", line 1094, in _call_worker
    return await self.run_in_proc(main_worker, name, args, job)
  File "/usr/local/lib/python3.7/site-packages/middlewared/main.py", line 1029, in run_in_proc
    return await self.run_in_executor(self.__procpool, method, *args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/middlewared/main.py", line 1003, in run_in_executor
    return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs))
middlewared.service_exception.CallError: [ENOENT] Pool 17998721979278846804 not found.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/middlewared/job.py", line 349, in run
    await self.future
  File "/usr/local/lib/python3.7/site-packages/middlewared/job.py", line 386, in __run_body
    rv = await self.method(*([self] + args))
  File "/usr/local/lib/python3.7/site-packages/middlewared/schema.py", line 960, in nf
    return await f(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/middlewared/plugins/pool.py", line 1668, in unlock
    raise CallError(msg)
middlewared.service_exception.CallError: [EFAULT] Pool could not be imported: 2 devices failed to decrypt.

I then returned to 11.3-RELEASE and found the encrypted pool showing in the dashboard here as well. However, the dashboard did not look like the 11.3 - RELEASE it looks like this: ???

Next I tried suggested queries from another post on this forum. I admit I don't know what they do, but they did seem un-intrusive.

Code:

root@FreeNAS:~ # grep -A 2 "geli attach" /var/log/*"
Unmatched '"'.

Code:

root@FreeNAS:~ # sqlite3 /data/freenas-v1.db 'select vol_encryptkey from storage_volume where vol_name = "Pool";'
8ddf8b1e-faef-488b-80bf-7cab4485d271

Finally,

Code:

root@FreeNAS:~ # ls /data/geli
8ddf8b1e-faef-488b-80bf-7cab4485d271.key

First and foremost, whatever help I can get to unlock the encrypted pool WOULD BE GREATLY APPRECIATED. While I have tried to solve my FreeNAS problems on my own, this one is beyond my capabilities.

Lastly, what's with the 11.3-RELEASED desktop I have?

Thanks,

 

[Samuel Tai]

Have you tried your recovery key in 11.2-U7?

Unfortunately, the 11.3 upgrade REQUIRES any encrypted pools to have passphrases removed BEFORE upgrading.

 

[Fierce]

Have you tried your recovery key in 11.2-U7?

Yes, no luck.

Unfortunately, the 11.3 upgrade REQUIRES any encrypted pools to have passphrases removed BEFORE upgrading.

Damn. Is the pool lost at this point? Will need to read up on this.

 

[Samuel Tai]

If neither your passphrase nor your recovery key work, then, yes, the pool is lost.

 

[Samuel Tai]

Note, the recovery key isn’t the same as geli.key.

 

[Fierce]

So if I go back to 11.2-U7 is it like nothing happened, and it should work? Or has the encryption been damaged by upgrading?

 

[Samuel Tai]

If you can find your recovery key, you may be able to get your pool back in 11.2-U7.

So if I go back to 11.2-U7 is it like nothing happened, and it should work? Or has the encryption been damaged by upgrading?

Your attempts to export/import an encrypted pool may have borked your pool. I can't say for certain if this is the case.

 

[winnielinnie]

Unfortunately, the 11.3 upgrade REQUIRES any encrypted pools to have passphrases removed BEFORE upgrading.

I have two encrypted pools, both with passphrases, that I created back in 11.1, upgraded to 11.2, and eventually upgraded to 11.3.

I never changed userkey0 (geli.key), userkey1 ("recovery key"), nor the passphrase (that I used with userkey0).

The passphrase has remained untouched, never removed, never modified, from 11.1 to 11.2 to 11.3.

 

[Samuel Tai]

I have two encrypted pools, both with passphrases, that I created back in 11.1, upgraded to 11.2, and eventually upgraded to 11.3.

I never changed userkey0 (geli.key), userkey1 ("recovery key"), nor the passphrase (that I used with userkey0).

The passphrase has remained untouched, never removed, never modified, from 11.1 to 11.2 to 11.3.

And you're one of the lucky ones, probably because your system dataset isn't on either.

 

[Fierce]

I'm not ready to give up yet. There must be steps to diagnose more closely.

 

[winnielinnie]

Did you ever backup / copy userkey0 (geli.key) for the encrypted pool? This is what happens when you select Padlock Icon > Encryption Key / Passphrase > Download Encryption Key

 

[winnielinnie]

And you're one of the lucky ones, probably because your system dataset isn't on either.

This is correct. I have a separate (small) pool that serves as a dedicated residence for system-dataset, logs, and jails. (I consider it my "sacrificial" pool, since no valuable data lives on it.)

I guess I inadvertently lucked out by doing it that way.

 

[Samuel Tai]

Did you ever backup / copy userkey0 (geli.key) for the encrypted pool? This is what happens when you select Padlock Icon > Encryption Key / Passphrase > Download Encryption Key

Which doesn't help in this case. What's needed is the Recovery key.

 

[Fierce]

Did you ever backup / copy userkey0 (geli.key) for the encrypted pool? This is what happens when you select Padlock Icon > Encryption Key / Passphrase > Download Encryption Key

Yes, I have the original passphrase and original geli.key. I'm not clear on whether I have a "recovery key".

Samuel, how do you know that I have the system dataset on this pool?

 

[winnielinnie]

Yes, I have the original passphrase and original geli.key. I'm not clear on whether I have a "recovery key".

It's generated and downloaded with Pools > Padlock Icon > Recovery Key while your pool is unlocked.

It is something you have to deliberately download to your client machine (laptop, desktop, phone), as it is not saved on your FreeNAS server, as opposed to userkey0, which is stored under /data/geli/

userkey0 is stored under /data/geli/ (sometimes known as the "Encryption Key"), and may or may not be used in conjunction with a passphrase to unlock the encrypted pool via the masterkey (unchangeable, generated randomly when the pool is initially created.) This userkey0 can also be downloaded for safe-keeping if desired (and recommended!)

userkey1 is only generated and downloaded "on demand" by the user, also known as the "Recovery Key", which is used on its own (without a passphrase) if the user forgets their passphrase or somehow cannot access userkey0.

masterkey is the actual key that is used to decrypt and encrypt data to and from the pool. It is stored in the pool's metadata, yet inaccessible ("scrambled") until the user provides their userkey0, userkey1, or userkey 0 + passphrase to "unscramble" it, and this in turn is used to "unlock" your pool. (This metadata can also be downloaded and stored for safe-keeping, yet it must be done in the shell, as there is no such GUI option for it yet. This is usually irrelevant, unless you want to insure yourself against the metadata corrupting, in which your entire pool will forever be inaccessible.)

FreeNAS has a strange naming convention, especially with builds in the past where "Recovery Key" was used in the GUI for places that referred to userkey0, and other times as userkey1!

If only it worked like LUKS does in Linux. In my opinion, the naming convention with LUKS is much more intuitive. You can use up to 8 "keyslots" and each keyslot can be either a passphrase or a key file. (You can have passphrases only, without requiring any key files at all, so there's no fear of "losing" a mandatory key file, as opposed to geli on FreeNAS being a nervous experience.)

Example:
keyslot0: created with passphrase
keyslot1: created with passphrase
keyslot2: created with key file
keyslot3: unused
keyslot4: unused
keyslot5: unused
keyslot6: unused
keyslot7: unused

Forgot the passphrase for keyslot1? No problem, as long as you remember the passphrase for keyslot0 or have the key file for keyslot2! Lost the key file for keyslot2? No problem, as long as you remember the passphrase for keyslot0 or keyslot 1!

 

[Fierce]

Thanks for this detail. In the past I unlocked the pool with the valid passphrase or geli.key I have. These are the two things the system asks for to unlock the pool. So why do they not work now?

 

[Samuel Tai]

Because you blindly exported/imported your pool, and invalidated your keys and passphrases.

 

[Fierce]

OK, I didn't understand that point. So what is the default file name of the recovery key (userkey1 ) under 11.2?

 

[winnielinnie]

OK, I didn't understand that point. So what is the default file name of the recovery key (userkey1 ) under 11.2?

It's nowhere on your server. It gets "generated" as a brand new userkey1 every time you want to create/download it. Each time you create/download a Recovery Key, the previous one is invalidated, thus only the most recently downloaded one can be used.

If you're lucky, you might have created/downloaded a Recovery Key (userkey1) in the past at one point and saved it to your desktop or laptop computer. It's likely to have the extension .geli, but I forget the default name it uses. I give mine unique descriptive file names so I know exactly what they are used for.

Here's an example in my case:

• mainpool_userkey0_also-requires-passphrase_20180601.key
• mainpool_userkey1_no-passphrase_will-invalidate-userkey0-emergencies-only_20180601.key
• mediapool_userkey0_also-requires-passphrase_20180915.key
• mediapool_userkey1_no-passphrase_will-invalidate-userkey0-emergencies-only_20180915.key

I save all my keys on a secure USB flash stick, including key files I use on encrypted LUKS (Linux) containers. I use the .key file name extension just because I prefer its universal meaning. The numbers at the very end of the file name is the date in which it was created. I even write in reminders to be safe so that I know that using userkey1 (Recovery Key) will essentially overwrite the previous userkey0, and thus for my encrypted pool userkey1 is now the "new" userkey0, but without an accompanying passphrase. Another reason why I prefer LUKS on Linux over geli on FreeNAS. There's no risk of overwriting key slots, unless you explicitly do so. Read it again. If you unlock an encrypted pool with your Recovery Key (userkey1), your original Encryption Key (userkey0) is now invalidated, and the Recovery Key (userkey1) is now the "new" Encryption Key (userkey0).

 

[Fierce]

Again, I do have the key(s) downloaded to my PC. So that I know which key is which, I'd like to know the default name of these keys.