Restore from USB failure - Encrypted pool

[longroad]

I restarted the system yesterday and it wouldn't boot to the USB drive. Ran a diagnostics on it and there were a few bad sectors. No problem I thought as I had a backup config file and was back up and running on the latest version. Previously on 11.2, restored to 11.3.

I had 3 pools, two restored okay. The important pool was encrypted and did not restore, it is showing as locked and is visible but not accessible. When I try and unlock it in the GUI it gives the error below.

This led me to another thread here which talks about a bug within Freenas that a pool did not unlock properly.

What I am looking to do:

Unlock the encrypted pool. Pool is mirrored dual 8TB drive setup.

What I have tried:
Unlocking through GUI - Passphrase guessing not successful
Cloning the failed USB drive using PartedMagic without success. It is visible in the tools and I can see the partitions but when I attempt to do the cloning it will end early or give an error.

Problem:
I can not locate a .key file on my local from the time of setup. I was able to go into the GUI and download the pool encryption key but I don't know if that helps me. I don't remember setting up a passcode. I understand now the serious problem as I don't have a key or passphrase. My assumption was that it was exported in the config file, huge error on my part.

Should I try and restore to freenas 11.2, would that make any difference? Any other ideas on recovering the USB, key, or other methods? Appreciate any help from the community.

Code:

Error: concurrent.futures.process._RemoteTraceback:
"""
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/concurrent/futures/process.py", line 239, in _process_worker
    r = call_item.fn(*call_item.args, **call_item.kwargs)
  File "/usr/local/lib/python3.7/site-packages/middlewared/worker.py", line 97, in main_worker
    res = loop.run_until_complete(coro)
  File "/usr/local/lib/python3.7/asyncio/base_events.py", line 579, in run_until_complete
    return future.result()
  File "/usr/local/lib/python3.7/site-packages/middlewared/worker.py", line 53, in _run
    return await self._call(name, serviceobj, methodobj, params=args, job=job)
  File "/usr/local/lib/python3.7/site-packages/middlewared/worker.py", line 45, in _call
    return methodobj(*params)
  File "/usr/local/lib/python3.7/site-packages/middlewared/worker.py", line 45, in _call
    return methodobj(*params)
  File "/usr/local/lib/python3.7/site-packages/middlewared/schema.py", line 965, in nf
    return f(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/middlewared/plugins/zfs.py", line 390, in import_pool
    'Failed to mount datasets after importing "%s" pool: %s', name_or_guid, str(e), exc_info=True
  File "libzfs.pyx", line 369, in libzfs.ZFS.__exit__
  File "/usr/local/lib/python3.7/site-packages/middlewared/plugins/zfs.py", line 380, in import_pool
    raise CallError(f'Pool {name_or_guid} not found.', errno.ENOENT)
middlewared.service_exception.CallError: [ENOENT] Pool 16213477825591501306 not found.
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/middlewared/plugins/pool.py", line 1661, in unlock
    'cachefile': ZPOOL_CACHE_FILE,
  File "/usr/local/lib/python3.7/site-packages/middlewared/main.py", line 1131, in call
    app=app, pipes=pipes, job_on_progress_cb=job_on_progress_cb, io_thread=True,
  File "/usr/local/lib/python3.7/site-packages/middlewared/main.py", line 1078, in _call
    return await self._call_worker(name, *args)
  File "/usr/local/lib/python3.7/site-packages/middlewared/main.py", line 1098, in _call_worker
    return await self.run_in_proc(main_worker, name, args, job)
  File "/usr/local/lib/python3.7/site-packages/middlewared/main.py", line 1033, in run_in_proc
    return await self.run_in_executor(self.__procpool, method, *args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/middlewared/main.py", line 1007, in run_in_executor
    return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs))
middlewared.service_exception.CallError: [ENOENT] Pool 16213477825591501306 not found.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/middlewared/job.py", line 349, in run
    await self.future
  File "/usr/local/lib/python3.7/site-packages/middlewared/job.py", line 386, in __run_body
    rv = await self.method(*([self] + args))
  File "/usr/local/lib/python3.7/site-packages/middlewared/schema.py", line 961, in nf
    return await f(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/middlewared/plugins/pool.py", line 1673, in unlock
    raise CallError(msg)
middlewared.service_exception.CallError: [EFAULT] Pool could not be imported: 2 devices failed to decrypt.

 

[Samuel Tai]

To mount a GELI-encrypted zpool, you'll need either the passphrase or the recovery key. If you didn't save the recovery key, you're out of luck. There's 2 areas which FreeNAS asks to save the recovery key:

In the pool management GUI, clicking the lock icon yields a pulldown menu for Recovery Key, which will all you to set and download a recovery key.

Alternatively, during the configuration save, you can opt to save both the encryption and recovery keys.

This is a straight TAR file, and both keys are saved in the geli folder inside the TAR. Unfortunately, the keys are labeled by UUID, so you'll have to try each in turn to see which is the recovery key.

Also, it's good practice once the encrypted zpools are mounted to reset the encryption and recovery keys, so you know they'll be fresh for the next boot.

 

[Samuel Tai]

Needless to say, the safety and security of those keys are paramount. I keep 2 copies of both the encryption/recovery keys and the saved configuration:

1. A BitLocker-encrypted thumb drive, for which I save the unlock key in OneDrive and have set to auto-open on insertion.
2. My Personal Vault in OneDrive, which unlocks using the Google Authenticator.

 

[Apollo]

If you have previous boot environment on your USB at 11.2 or prior, then I would try starting from those instead.
Even if USB key has bad sectors, as long has you are able to get access to gui, you should be able to download the config file from it, and hopefully the seeds for the encryptions and then install the same version of Freenas on a new boot drive and restore the config.

 

[longroad]

I have attempted to extract as much data as possible from the usb drive using dd on PartedMagic and extracted 99.96% but the cloned drive will not boot and will not register in FreeBSD as a mountable pool.

I rolled back the operating system to 11.2 but I get no additional boot environments in the GUI.

When installing 11.2 I inserted the cloned damaged USB but it did not give me the option to upgrade so no luck there either.

I have not been able to locate a .key file on my local. I'm in a bad spot now, this is a major failure on my part.

 

[Apollo]

If you are able to access CLI via SSH or through web UI shell, then go to the following folder:

/data/geli

This is where the GELI keys are/should be located.
Make sure you are logged in as root.

 

[longroad]

SOLVED!!

I was able to extract the geli key from the damaged boot drive after trying a number of different avenues.

The method that worked for me was the following:

1. Made a linux boot disk that had the DDRESCUE utility
2. Made 2 copies of the original USB
3. Back in windows I made disk images from the two USB clones
4. Used Klennet ZFS Recovery on the disk images. Neither the original drive or the cloned USB drives worked for some reason.
5. Extracted the GELI key and unlocked the drive.