SOLVED Unable to import encrypted pool after boot device failure (Recovery key does not require passphase)

[vjz224]

Yet another import failure of an encrypted pool (I read the other posts I swear).

USB boot device failed, rebuilding in VMware VM (PCI Passthrough). All hardware is the same with the addition of a hard drive, installed ESXi 6.5 U2, installed FreeNAS VM.

Unable to import encrypted volume from the GUI. It does show all 4 disks, I provide the key (not the recovery), and passphase. Error: The following disks failed to attach: gptid/5ade1c3d-b3af-11e7-805d-ac1f6b1a7c12, gptid/5b816450-b3af-11e7-805d-ac1f6b1a7c12, gptid/5c21dc85-b3af-11e7-805d-ac1f6b1a7c12, gptid/5a37a31d-b3af-11e7-805d-ac1f6b1a7c12

zpool import and zpool import -f return to cursor (no action).

Hardware:
Supermicro SYS-5028D-TN4T
8 CPUs (Cores) Intel(R) Xeon(R) CPU D-1541 @ 2.10GHz
16GB RAM
4 x 10TB Ironwolf Drives
128GB SSD (newly added for ESXi install, VM datastore)

VM:
2 CPU
8GB RAM
16GB vDisk
PCI Passthrough of the SATA controller

zpool status:

Code:

root@freenas:~ # zpool status
  pool: freenas-boot
 state: ONLINE
  scan: none requested
config:

		NAME		STATE	 READ WRITE CKSUM
		freenas-boot  ONLINE	   0	 0	 0
		  da0p2	 ONLINE	   0	 0	 0

errors: No known data errors

geli list:

Code:

root@freenas:~ # geli list
Geom name: mirror/swap0.eli
State: ACTIVE
EncryptionAlgorithm: AES-XTS
KeyLength: 128
Crypto: hardware
Version: 7
Flags: ONETIME
KeysAllocated: 4
KeysTotal: 4
Providers:
1. Name: mirror/swap0.eli
   Mediasize: 2147483648 (2.0G)
   Sectorsize: 512
   Mode: r1w1e0
Consumers:
1. Name: mirror/swap0
   Mediasize: 2147483648 (2.0G)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r1w1e1

Geom name: mirror/swap1.eli
State: ACTIVE
EncryptionAlgorithm: AES-XTS
KeyLength: 128
Crypto: hardware
Version: 7
Flags: ONETIME
KeysAllocated: 4
KeysTotal: 4
Providers:
1. Name: mirror/swap1.eli
   Mediasize: 2147483648 (2.0G)
   Sectorsize: 512
   Mode: r1w1e0
Consumers:
1. Name: mirror/swap1
   Mediasize: 2147483648 (2.0G)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r1w1e1

glabel list:

Code:

root@freenas:~ # glabel list
Geom name: da0p1
Providers:
1. Name: gptid/5ebab0c1-5946-11e8-86cb-000c291587b3
   Mediasize: 524288 (512K)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 20480
   Mode: r0w0e0
   secoffset: 0
   offset: 0
   seclength: 1024
   length: 524288
   index: 0
Consumers:
1. Name: da0p1
   Mediasize: 524288 (512K)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 20480
   Mode: r0w0e0

Geom name: ada1p2
Providers:
1. Name: gptid/5ade1c3d-b3af-11e7-805d-ac1f6b1a7c12
   Mediasize: 9998683774976 (9.1T)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r0w0e0
   secoffset: 0
   offset: 0
   seclength: 19528679248
   length: 9998683774976
   index: 0
Consumers:
1. Name: ada1p2
   Mediasize: 9998683774976 (9.1T)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r0w0e0

Geom name: ada2p2
Providers:
1. Name: gptid/5b816450-b3af-11e7-805d-ac1f6b1a7c12
   Mediasize: 9998683774976 (9.1T)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r0w0e0
   secoffset: 0
   offset: 0
   seclength: 19528679248
   length: 9998683774976
   index: 0
Consumers:
1. Name: ada2p2
   Mediasize: 9998683774976 (9.1T)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r0w0e0

Geom name: ada3p2
Providers:
1. Name: gptid/5c21dc85-b3af-11e7-805d-ac1f6b1a7c12
   Mediasize: 9998683774976 (9.1T)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r0w0e0
   secoffset: 0
   offset: 0
   seclength: 19528679248
   length: 9998683774976
   index: 0
Consumers:
1. Name: ada3p2
   Mediasize: 9998683774976 (9.1T)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r0w0e0

Geom name: ada4p2
Providers:
1. Name: gptid/5a37a31d-b3af-11e7-805d-ac1f6b1a7c12
   Mediasize: 9998683774976 (9.1T)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r0w0e0
   secoffset: 0
   offset: 0
   seclength: 19528679248
   length: 9998683774976
   index: 0
Consumers:
1. Name: ada4p2
   Mediasize: 9998683774976 (9.1T)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r0w0e0

camcontrol devlist:

Code:

root@freenas:~ # camcontrol devlist
<VMware Virtual disk 2.0>		  at scbus2 target 0 lun 0 (pass0,da0)
<NECVMWar VMware SATA CD00 1.00>   at scbus3 target 0 lun 0 (cd0,pass1)
<SuperMicro SSD SOB20R>			at scbus33 target 0 lun 0 (pass2,ada0)
<ST10000VN0004-1ZD101 SC60>		at scbus34 target 0 lun 0 (pass3,ada1)
<ST10000VN0004-1ZD101 SC60>		at scbus35 target 0 lun 0 (pass4,ada2)
<ST10000VN0004-1ZD101 SC60>		at scbus36 target 0 lun 0 (pass5,ada3)
<ST10000VN0004-1ZD101 SC60>		at scbus37 target 0 lun 0 (pass6,ada4)

I have high confidence in the encryption key, recovery key, and passphase. Hard drives were fine before the failure as well.

Any ideas?

 

[vjz224]

Ah Ha! Minor detail. Recovery key does not require the passphase. Imported in the GUI just fine.

Going to rekey now just in case the main key on file was off...

 

[svtkobra7]

Ah Ha! Minor detail. Recovery key does not require the passphase. Imported in the GUI just fine.

Going to rekey now just in case the main key on file was off...

Since you are being extra cautious (I would be and am too with encryption), you might consider back up of your GELI metadata JIC. It shouldn't ever be needed, but saved my arse once (granted that was before I really understood FreeNAS). Just a thought.

Sent from my Pixel 2 using Tapatalk