SOLVED HELP! No access to encrypted pool after boot failure

[IceBoosteR]

Hi folks,

another time that encryption with FreeNAS drives me crazy. Try to keep long story short.
Had mirrored boot pool. I came back after some weeks and both USB thumb drives died. I could not boot the machine anymore. Could save a backup from the database though. Installed fresh 11.2-U5 and uploaded my conig from 11.1-U7 - no issue.

But I am unable to open my encrypted pools. Of course I do have the geli.key, the recovery.key and the passphrase - but no luck via GUI nor CLI.
Messages like:

• geli: Wrong key for da6p2
• Importing RED [1922545619372436668] failed with: cannot import '1922545619372436668': no such pool available
• geli: Cannot read metadata from /dev/da6p1: Invalid argument
• ...

I do also savely eyported/disconnected the pool. Trying to import it again throws errors....

Code:

Error: Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/middlewared/job.py", line 332, in run
    await self.future
  File "/usr/local/lib/python3.6/site-packages/middlewared/job.py", line 365, in __run_body
    rv = await self.middleware.run_in_thread(self.method, *([self] + args))
  File "/usr/local/lib/python3.6/site-packages/middlewared/main.py", line 1006, in run_in_thread
    return await self.loop.run_in_executor(executor, functools.partial(method, *args, **kwargs))
  File "/usr/local/lib/python3.6/concurrent/futures/thread.py", line 56, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/local/lib/python3.6/site-packages/middlewared/schema.py", line 668, in nf
    return f(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/middlewared/plugins/disk.py", line 253, in decrypt
    raise CallError(f'The following devices failed to attach: {", ".join(failed)}')
middlewared.service_exception.CallError: [EFAULT] The following devices failed to attach: gptid/50245289-8164-11e8-a571-1866da308b0d, gptid/4f4d1954-8164-11e8-a571-1866da308b0d, gptid/4e769436-8164-11e8-a571-1866da308b0d, gptid/4db11db0-8164-11e8-a571-1866da308b0d, gptid/4cf26b2f-8164-11e8-a571-1866da308b0d, gptid/4c2219f0-8164-11e8-a571-1866da308b0d, gptid/4b56f034-8164-11e8-a571-1866da308b0d, gptid/4a5608c3-8164-11e8-a571-1866da308b0d, gptid/48be4b72-8164-11e8-a571-1866da308b0d, gptid/498e87dd-8164-11e8-a571-1866da308b0d

I have actually NO guess how to move on. I assume my data is still there. Maybe I can recover the data from the broken thumb drives? Need to figure out on which system though. Anyhow any idea how to manually import would be AWESOME!

Thank you

 

[PhiloEpisteme]

I have actually NO guess how to move on. I assume my data is still there. Maybe I can recover the data from the broken thumb drives? Need to figure out on which system though. Anyhow any idea how to manually import would be AWESOME!

Sorry to hear you're having trouble. You won't be able to recover your data unless you can manage to unlock the drives. To get started can you please tell me all of the drives attached to your system, which pools they belong to, and whether that pool was encrypted or not?

The reason I'm asking for the above information is because I'd like to suggest that you go through each drive one by one and verify that your keys work for those disks.

geli: Wrong key for da6p2

This is telling you that the key/passphrase combination that you used is wrong for that drive or pool. I know when I've tested encryption myself I've run into this, it was either because I was trying to use the wrong disk combinations or I was using an out-of-date key/passphrase.

 

[IceBoosteR]

Hi there,
thanks for helping me out.
I do have a bunch of disks. The pool does have 10x 4TB WD Reds which are:
ada2
ada3
da0-7

I have tried to unlock some of the disks by using
geli attach -k /path-to-backed-up-key/geli.key

I haven't had replaced my hard drives for quite a wile, so I am assuming that the key is up-to-date. Rebooting was also no issue. Please point me to any information you need.
Cheers!

 

[IceBoosteR]

Update: I think I made it!!!!!!!

Update soon with progress

 

[Heracles]

Hey IceBooster,

Good for you that you recovered your data.

About pool encryption, I suggest you go read this thread I created on the subject...

Encryption will turn to a self-inflicted ransomware much more often than being of any use... Risk is so high for a so low benefit, to do it is asking for trouble.

 

[IceBoosteR]

Hey IceBooster,

Good for you that you recovered your data.

About pool encryption, I suggest you go read this thread I created on the subject...

Encryption will turn to a self-inflicted ransomware much more often than being of any use... Risk is so high for a so low benefit, to do it is asking for trouble.

Hi,
actually I do use FreeNAS for quite some years and had my experiences with encryption. And yes I was thinking about the fact how encryption is actually a downside. Even when you think about the disk replacement. A little hickup from your system while it's resilvering and you haven't downloaded the key and set the passphrase and data is gone. Yeah I know about it....
Aynhow I agree with you but for the moment I use it ;)

 

[IceBoosteR]

So actually I made it, data is back - doing a full backup to my backup box.
My boot drives died and FreeNAS was not able to recover from it. The recovery key and the geli.key which were provided by the GUI were useless for me, absolutely useless. I have mounted the dead boot drive and was able to access the system partition thanks to this page:
https://gmpreussner.com/reference/recovering-freenas-configuration-from-zfs-boot-drive (Hey author, if you read this- take the biggest "Thank You" you could imagine)

I was hacking around on the CLIfor hours and was glad it worked. I could recover the geli.key(s) from /data/geli and saved them. Then I detached all pools and could import the pool with an issue, the systme crashed. After the reboot I was able to decrypt the pool.

That was the looooooong story (8 hours) in short....

Was really happy to recover from the failure. Now I do boot from an SSD.

-IceBoosteR

 

[PhiloEpisteme]

That was the looooooong story (8 hours) in short....

Was really happy to recover from the failure. Now I do boot from an SSD.

Very happy to hear it worked out for you. It sounds like the key you had originally was simply the wrong key, luckily you were able to get the correct key. :)