SOLVED Can't mount encrypted pool

[_noble_six_]

Hello folks.

I've build a home NAS server for FreeNAS some time ago

• it has 4 data disks
• 1 slog device (if I name it correctly)
• an my boot OS drive is M2 in M2-to-USB case mounter inside internal USB.
• FreeNAS-11.3-U1

I created an encrypted pool using all data disks. I have all relevant keys and recovery key with passphrase.

It was all working alright until I was not playing with is for some months being busy.
There was power outages, NAS went down and now I am trying to bring it back.

It didn't boot on it's own and I forgot how was the boot drive configured.
I tried to boot from a device, supposed to be slog and it aid this is data disk, no OS here.
I was not sure (didn't remember) about the situation with M2-to-USB disk and was wondering if it ever booted successfully, or I am missing some other USB stick.

So I decided maybe I just reinstall FreeNAS on the available 5-th disk (which wa proably an slog device. but I dont remember) and then use my keys to import the pool.
After fresh installation of FreeNAS on the available 5-th disk the OS booted, but I couldn't mount the pool. Some brief googling showed that I may be missing some geli keys, so fresh install was not an option, as I understood.

I decided to find the disk which was supposeed to be the boot one.
Tried to connect the supposed boot disk directly to M2 and it worked.
I have booted my FreeNAS installation as it used to be. I think now I must have all my geli keys in place.

I now see my pool as being locked. Trying to unlock it with a passphrase or a recovery key fails. Below is the log output.

What other details can I provide to sort this issue?

Code:

Error: concurrent.futures.process._RemoteTraceback:
"""
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/concurrent/futures/process.py", line 239, in _process_worker
    r = call_item.fn(*call_item.args, **call_item.kwargs)
  File "/usr/local/lib/python3.7/site-packages/middlewared/worker.py", line 95, in main_worker
    res = loop.run_until_complete(coro)
  File "/usr/local/lib/python3.7/asyncio/base_events.py", line 579, in run_until_complete
    return future.result()
  File "/usr/local/lib/python3.7/site-packages/middlewared/worker.py", line 51, in _run
    return await self._call(name, serviceobj, methodobj, params=args, job=job)
  File "/usr/local/lib/python3.7/site-packages/middlewared/worker.py", line 43, in _call
    return methodobj(*params)
  File "/usr/local/lib/python3.7/site-packages/middlewared/worker.py", line 43, in _call
    return methodobj(*params)
  File "/usr/local/lib/python3.7/site-packages/middlewared/schema.py", line 965, in nf
    return f(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/middlewared/plugins/zfs.py", line 382, in import_pool
    zfs.import_pool(found, found.name, options, any_host=any_host)
  File "libzfs.pyx", line 369, in libzfs.ZFS.__exit__
  File "/usr/local/lib/python3.7/site-packages/middlewared/plugins/zfs.py", line 382, in import_pool
    zfs.import_pool(found, found.name, options, any_host=any_host)
  File "libzfs.pyx", line 870, in libzfs.ZFS.import_pool
libzfs.ZFSException: one or more devices is currently unavailable
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/middlewared/plugins/pool.py", line 1660, in unlock
    'cachefile': ZPOOL_CACHE_FILE,
  File "/usr/local/lib/python3.7/site-packages/middlewared/main.py", line 1127, in call
    app=app, pipes=pipes, job_on_progress_cb=job_on_progress_cb, io_thread=True,
  File "/usr/local/lib/python3.7/site-packages/middlewared/main.py", line 1074, in _call
    return await self._call_worker(name, *args)
  File "/usr/local/lib/python3.7/site-packages/middlewared/main.py", line 1094, in _call_worker
    return await self.run_in_proc(main_worker, name, args, job)
  File "/usr/local/lib/python3.7/site-packages/middlewared/main.py", line 1029, in run_in_proc
    return await self.run_in_executor(self.__procpool, method, *args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/middlewared/main.py", line 1003, in run_in_executor
    return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs))
libzfs.ZFSException: ('one or more devices is currently unavailable',)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/middlewared/job.py", line 349, in run
    await self.future
  File "/usr/local/lib/python3.7/site-packages/middlewared/job.py", line 386, in __run_body
    rv = await self.method(*([self] + args))
  File "/usr/local/lib/python3.7/site-packages/middlewared/schema.py", line 961, in nf
    return await f(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/middlewared/plugins/pool.py", line 1672, in unlock
    raise CallError(msg)
middlewared.service_exception.CallError: [EFAULT] Pool could not be imported: 1 devices failed to decrypt.

 

[_noble_six_]

forgot to say - the error message title says
[EFAULT] Pool could not be imported: 1 devices failed to decrypt.

is it one of the disks failing to decrypt?
or could it be slog missing (i used it to reinstall FreeNAS)

can my pool be recovered?

 

[Samuel Tai]

Yes, it's complaining about your missing SLOG device. You'll need to manually import your GELI-encrypted pool.

1. First, run geli attach -k /path/to/recovery/key <GELI partition> for each of your 4 data disks. The GELI partition is usually partition 2, so for example, if your disks are ada0-3, then try ada0p2, ada1p2, etc.
2. Once each partition is decrypted, then run zpool import -f -m -R /mnt <name of your pool>

 

[_noble_six_]

Yes, it's complaining about your missing SLOG device. You'll need to manually import your GELI-encrypted pool.

1. First, run geli attach -k /path/to/recovery/key <GELI partition> for each of your 4 data disks. The GELI partition is usually partition 2, so for example, if your disks are ada0-3, then try ada0p2, ada1p2, etc.
2. Once each partition is decrypted, then run zpool import -f -m -R /mnt <name of your pool>

Thank you, Sir.
I will try this guide tomorrow, will need to google these mentioned commands so that I understand whats going on.

But do you say/think my pool is reparable?

Will this manual import permanently fix my pool and it be healthy?

I ask because you propose to mount it as something like

Code:

/mnt 

which seems like temporary mount point for... data recovery?

Or is this a one time recovery access to save data and it wont work normally any more?

Afterwards will my pool be displayed in GUI once again?

 

[Samuel Tai]

I think your pool is fine. Once it's imported, it will display in the GUI. At that point, you can remove the SLOG, and the pool will be healthy.

Note, you should reset the keys, set a passphrase, and create a new recovery key after you've imported the pool and removed the missing SLOG device.

As for /mnt, that's not the mount point, that's the altroot for the pool. The pool will actually mount at /mnt/<name of pool>, which is where FreeNAS expects it.

 

[_noble_six_]

I think your pool is fine. Once it's imported, it will display in the GUI. At that point, you can remove the SLOG, and the pool will be healthy.

Note, you should reset the keys, set a passphrase, and create a new recovery key after you've imported the pool and removed the missing SLOG device.

As for /mnt, that's not the mount point, that's the altroot for the pool. The pool will actually mount at /mnt/<name of pool>, which is where FreeNAS expects it.

Dear Samuel, your guide worked like a charm!
Just one remark - somehow recovery key was not accepted, so I had to use my regular encryption key + passphrase (which was accepted).
So yes, I made geli attach for all 4 disks (using second partitions) and then zpool import was able to build the pool back.
Then the pool showed up in GUI as unlocked and degraded.
I was able to remove missing SLOG which made the pool healthy.

Pardon me my last question - what about resetting the keys?
- I remember doing this sort of operation long ago when creating pool and making myself a training. Than is a risky operation. no?
- Given the fact I used my regular encryption key to fix the pool, is it mandatory to change the keys, or I can live with the current key? It seems to be ok.
- If changing the key is mandatory now - where should I start with (I dont see anything about keys change in the GUI at this point).Maybe a link to howto?

Meanwhile is it safe to reboot the system now? Will the pool unlock normally after restart?

Thank you very much for you precise answer. You made my day, I was so worried to fix the pool but we did it!

 

[Samuel Tai]

Any time you change your pool, you need to update the keys, passphrase, and recovery keys. The old key matches the old topology. Please don't reboot until you've accomplished this.

Are you using the legacy UI or the new UI? The procedure is different in each.

 

[_noble_six_]

I am not sure which UI is this but probably legacy, since I booted from the old boot drive.
Here is a menu to the right, it opens over lock icon. Which menu item should I go for?
All of them one by one?

 

[Samuel Tai]

That's the new UI. Select Reset Keys, then Encryption Key/Passphrase, and finally Recovery Key in turn.

 

[_noble_six_]

That's the new UI. Select Reset Keys, then Encryption Key/Passphrase, and finally Recovery Key in turn.

All done.
Keys and passphrase renewed successfully.
Pool locks/unlocks normally now.
Thank you.

Now how can I express my gratitude? Cant find any "raise karma" button here. Any options to upvote your profile? :))

 

[Samuel Tai]

You're welcome. You've already Liked all my replies. That's the best that can be done in this forum.