SOLVED Boot Drive Gone / Encrypted Disks / Attempt to Import + Unlock Fail [ESXi]

[svtkobra7]

• Config: FreeNAS-9.10.2-U4 on VMware ESXi 6.0 U3. 1 zPool = "Tank1" = 8 2-disk mirror vdevs (reference signature for all details), encrypted, no passphrase.
  
• Problem: Boot environment gone, attempted to import with recent config backup + recovery key (/data/GELI not available to restore).
• Attempt to resolve:

1. Prior to importing backup config (no volume shown) ... Import Volume = > Decrypt Disks using Recovery Key = 7 disks error due to "Missing -p flag" + 9 disks error due to "Wrong key"


2. Import backup config (volume shown as locked) ... Attempt to unlock = Error due to no keyfile in /data/geli


3. Detach Volume, reboot, still no luck.

I believe if I had the /data/GELI keyfiles I would be OK, but I believe there is a way to rebuilt them somehow? Thank you for taking the time to read.

Code:

[root@FreeNAS] ~# zpool status
  pool: freenas-boot
 state: ONLINE
  scan: none requested
config:
  NAME  STATE  READ WRITE CKSUM
  freenas-boot  ONLINE  0  0  0
  da0p2  ONLINE  0  0  0
errors: No known data errors

Code:

[root@FreeNAS] ~# gpart show
=>  34  16777149  da0  GPT  (8.0G)
  34  1024  1  bios-boot  (512K)
  1058  6  - free -  (3.0K)
  1064  16776112  2  freebsd-zfs  (8.0G)
  16777176  7  - free -  (3.5K)
=>  34  11721045101  da1  GPT  (5.5T)
  34  94  - free -  (47K)
  128  11721045000  1  freebsd-zfs  (5.5T)
  11721045128  7  - free -  (3.5K)
=>  34  11721045101  da2  GPT  (5.5T)
  34  94  - free -  (47K)
  128  11721045000  1  freebsd-zfs  (5.5T)
  11721045128  7  - free -  (3.5K)
=>  34  11721045101  da3  GPT  (5.5T)
  34  94  - free -  (47K)
  128  11721045000  1  freebsd-zfs  (5.5T)
  11721045128  7  - free -  (3.5K)
=>  34  11721045101  da4  GPT  (5.5T)
  34  94  - free -  (47K)
  128  11721045000  1  freebsd-zfs  (5.5T)
  11721045128  7  - free -  (3.5K)
=>  34  11721045101  da5  GPT  (5.5T)
  34  94  - free -  (47K)
  128  11721045000  1  freebsd-zfs  (5.5T)
  11721045128  7  - free -  (3.5K)
=>  34  11721045101  da6  GPT  (5.5T)
  34  94  - free -  (47K)
  128  11721045000  1  freebsd-zfs  (5.5T)
  11721045128  7  - free -  (3.5K)
=>  34  11721045101  da7  GPT  (5.5T)
  34  94  - free -  (47K)
  128  11721045000  1  freebsd-zfs  (5.5T)
  11721045128  7  - free -  (3.5K)
=>  34  11721045101  da8  GPT  (5.5T)
  34  94  - free -  (47K)
  128  11721045000  1  freebsd-zfs  (5.5T)
  11721045128  7  - free -  (3.5K)
=>  34  11721045101  da9  GPT  (5.5T)
  34  94  - free -  (47K)
  128  11721045000  1  freebsd-zfs  (5.5T)
  11721045128  7  - free -  (3.5K)
=>  34  11721045101  da10  GPT  (5.5T)
  34  94  - free -  (47K)
  128  11721045000  1  freebsd-zfs  (5.5T)
  11721045128  7  - free -  (3.5K)
=>  34  11721045101  da11  GPT  (5.5T)
  34  94  - free -  (47K)
  128  11721045000  1  freebsd-zfs  (5.5T)
  11721045128  7  - free -  (3.5K)
=>  34  11721045101  da12  GPT  (5.5T)
  34  94  - free -  (47K)
  128  11721045000  1  freebsd-zfs  (5.5T)
  11721045128  7  - free -  (3.5K)
=>  34  11721045101  da13  GPT  (5.5T)
  34  94  - free -  (47K)
  128  11721045000  1  freebsd-zfs  (5.5T)
  11721045128  7  - free -  (3.5K)
=>  34  11721045101  da14  GPT  (5.5T)
  34  94  - free -  (47K)
  128  11721045000  1  freebsd-zfs  (5.5T)
  11721045128  7  - free -  (3.5K)
=>  34  11721045101  da15  GPT  (5.5T)
  34  94  - free -  (47K)
  128  11721045000  1  freebsd-zfs  (5.5T)
  11721045128  7  - free -  (3.5K)
=>  34  11721045101  da16  GPT  (5.5T)
  34  94  - free -  (47K)
  128  11721045000  1  freebsd-zfs  (5.5T)
  11721045128  7  - free -  (3.5K)

 

[Jailer]

Without the encryption key you might as well consider your data gone.

 

[svtkobra7]

Without the encryption key you might as well consider your data gone.

It would be in a .vmdk I exported I believe, but an attempt to add that export as a new VM fail "Line 1: Could not parse the document: 'no element found'". Any thoughts on how to repair this .vmdk file for that purpose?

Thanks for your prompt reply.

 

[rs225]

What has happened to the boot drive?

And what is an exported vmdk, and what is giving you an error with it? ESXi or FreeNAS?

 

[Jailer]

If extracting the encryption key to decrypt the pool were that easy it would kind of defeat the purpose of encryption in the first place. If you don't have the key backed up your pool is gone and you're not getting it back.

 

[svtkobra7]

What has happened to the boot drive?

I wiped it thinking I had the necessary components to put everything back together. The worst reason ever, I know, but I wanted a fresh start (I realize there are other ways to accomplish) from the datastore level (.vmdk - see below) up. To put that in scope, the first time I installed FreeNAS was the first time I ran ESXi (a couple months back) and I thought with some more experience under my belt, I could get rid of some of the "gremlins" that were present (likely from initially configuring with a lack of knowledge).

And what is an exported vmdk, and what is giving you an error with it? ESXi or FreeNAS?

VMware disk file. From vSphere Web Client I attempted to create a new VM using the previously exported files (disk-0.vmdk + FreeNAS.ovf) and I receive that error when attempting to add the VM. The same occurs in VMware workstation. I had been "exporting" the VM thinking I was backing them up, having no clue that they were worthless (corrupt?) backups. I believe that error is somehow related to "passthrough" of the LSI HBA to FreeNAS.

 

[svtkobra7]

If extracting the encryption key to decrypt the pool were that easy it would kind of defeat the purpose of encryption in the first place.

If I had a working VM image I could reload, I would think it could be mounted and I could navigate to /data/GELI and pull the files or otherwise SSH into that instance powered on. As the encryption we are referring to here is to mitigate physical theft (right?), possession of the disk file is nearly the same, except that theft would be virtual. Further, in the later instance, you would need a key to SSH so if a threat has that, security (and by extension encryption) has already been compromised. So not sure I entirely agree that it would defeat the purpose, but regardless ...

If you don't have the key backed up your pool is gone and you're not getting it back.

I'm afraid that would be the feedback I would get. I spent hours researching and thought I read that

Code:

geli attach [-dprv] [-j passfile] [-k keyfile] prov

could be used to recreate the keyfile, provided other parts were on hand, recovery key, password. I believe the recovery key + passphrase gets you to the /data/GELI keyfile (I may be mistaken).
At any rate, thank you for your feedback (even though it wasn't what I wanted to hear, although expected).

I assume no last ditch efforts to try, just go ahead and create a new volume, and learn from my mistake?

 

[rs225]

How big is the boot vmdk? Is it a passthru? Is the underlying drive wiped?

 

[Jailer]

I assume no last ditch efforts to try, just go ahead and create a new volume, and learn from my mistake?

You're more than welcome to stick around for a bit and have some other more knowledgeable members chime in with their opinions.

 

[svtkobra7]

How big is the boot vmdk?

Thin provisioned = 1.34 GB (of allocated 8 GB).

Is it a passthru?

No, those 2 SSDs are SATA to mobo (and not pass through), but the 16 HGST HDDs are passthrough via LSI HBA.

Is the underlying drive wiped?

Yes :(

 

[rs225]

Then the boot vmdk is a file? And how big is that file?

 

[svtkobra7]

You're more than welcome to stick around for a bit and have some other more knowledgeable members chime in with their opinions.

I think with 2,135 posts and counting, you certainly qualify as knowledgeable! Further, you provided the answer that I expected and that aligns with the user documentation. I don't doubt you are correct.

With that "last ditch" comment, I meant an option that might work 1% of the time (or the like). Perhaps I misinterpreted the posts I read, but I thought there was some hope if at least the recovery key was possessed. That's all.

Again, thank you for taking the time to review and provide feedback.

 

[svtkobra7]

Then the boot vmdk is a file? And how big is that file?

Correct. 1.34 GB. Are you thinking PhotoRec / carving?

 

[rs225]

If you actually have 1.34GB of vmdk, then I would think you just need to get that into the ESXi datastore, and then add a hard drive to a VM, specifying that vmdk as the existing drive, and you should be able to boot it.

Aren't vmdk usually made up of a couple of files? Maybe you're missing that part. Even so, the larger part is the important part.

 

[svtkobra7]

of vmdk, then I would think you just need to get that into the ESXi datastore, and then add a hard drive to a VM, specifying that vmdk as the existing drive, and you should be able to boot it.

Tried creating new VM, with blank hard disk, then swapping in the .vmdk ("backup") for the one that was created (before powering it on). It erred - something about SCSI device 0:0 ... I'm happy to do it again and report back with the exact error (frankly I don't know ESXi well enough to know if this would work, but it seems logical if we separately came to the same "theory").

Aren't vmdk usually made up of a couple of files? Maybe you're missing that part. Even so, the larger part is the important part.

Absolutely correct, I believe the .ovf file specifies the config for re-import and the .vmdk is the actual (virtual) hard disk. As you actually "import" the .ovf (but ESXi calls both files), I bet the issue is the darn passthrough HBA (I could never export from the vSphere Client, only the vSphere Web Client as the Client always whined about the passthrough device. Point being, I bet the .vmdk is actually in tact.

 

[rs225]

If I were doing it, I would delete the hard disk that is automatically created, and then add a hard disk (existing) and then select the vmdk. But you have to copy the vmdk file into a datastore visible to ESXi.

 

[svtkobra7]

If I were doing it, I would delete the hard disk that is automatically created, and then add a hard disk (existing) and then select the vmdk. But you have to copy the vmdk file into a datastore visible to ESXi.

Yes, that is precisely what I did. Result = that error.

 

[rs225]

Can you get head -c1536 disk-0.vmdk | strings? It should have a contained spec for the vmdk, and maybe something in there will look abnormal.

What is the full error when you try to import the ovf?

 

[rs225]

I think your OVF file may be corrupt. It seems small. But the vmdk is most likely in a compressed stream format. It can be changed into a usable format with vmkfstools, and possibly other ways I don't know. If you can run command line on your ESXi, you need to do vmkfstools -i disk-0.vmdk -d thin usable.vmdk and the result should be able to attach to a VM.

 

[Dice]

It erred - something about SCSI device 0:0

This might've been caused from a different controller options in the VM.
There is an option to run virtualized drives on different types of controllers. I believe default is depending on what VM template is used. IIRC FreeBSD template doesn't give you a SATA controller, but for example Windows templates do.
The SCSI error has plagued people before to get FreeNAS running on ESXi judging from PM's I recieved.
With some luck, that lack of configuration may be what causes your vmdk import to not be accepted.

[*]Create a new VM
[*]Make sure there is a SATA controller listed (else add it "add another device")
[*]Change the type of controller underlying the virtualized drive. ("Virtual device node")
[*]Attempt the same vmdk maneuver.

Now that I think of it, you should probably replicate your previous configuration. If that was using IDE or what not and it worked - do it.
Else, try the above.

I think your OVF file may be corrupt. It seems small.

Mine are all <2kb too.