SOLVED Problem with onlining encrypted drive

[bkvamme]

Hi all,

I have an encrypted storage pool which spans over three RAID-Z2 vdevs.

During some cable management in my chassis, one of the harddrives was not properly attached to the backplane, and thus was not discovered by FreeNAS when I unlocked the pool. This resulted in the device being unavaliable, and the pool being in a degraded state.

I am now having trouble attaching the drive to the pool again. After properly attaching the drive, I have tried the following:

1. Locking the pool and unlocking it again
2. Rebooting FreeNAS and unlocking the pool
3. Offlining the device, and onlining it again in the console

When checking /dev/gptid, the device is listed (19c89a0f-5111-11e4-9f6f-000c29e6c5f0), but is not unlocked (19c89a0f-5111-11e4-9f6f-000c29e6c5f0.eli is missing).

So offlining/onlining the device would obviously not work, as I cannot online the unlocked volume, as FreeNAS has not unlocked it. My guess is that FreeNAS does not even attempt to unlock the drive, as it believes that it is missing, and needs to be replaced.

Is there a way to manually unlock the drive and mount it to 19c89a0f-5111-11e4-9f6f-000c29e6c5f0.eli, and thus be able to online it again?

Or should I try to clear the fault with:

Code:

zpool clear sPool 19c89a0f-5111-11e4-9f6f-000c29e6c5f0.eli

And then attempt to lock/unlock the pool again?

Appreciate your help, would prefer not to resilver when the drive is not faulty.

EDIT: Just tried to manually unlock the drive using:

Code:

geli attach -k /tmp/Z1D37JBN.eli /dev/gptid/19c89a0f-5111-11e4-9f6f-000c29e6c5f0

Where /tmp/Z1D37JBN.eli is the drive encryption key backed up using Dusan's script (https://forums.freenas.org/index.ph...gle-freenas-primary-storage.17316/#post-93073). It prompted me for my passphrase, but gave the following after I entered the passphrase to unlock the drive.

Code:

geli: Wrong key for gptid/19c89a0f-5111-11e4-9f6f-000c29e6c5f0.

I also tried to clear the pool for errors and lock/unlock the pool, but it still refused to unlock my drive.

 

[bkvamme]

Great success.

After some sleep and a good cup of coffee, I managed to online the drive again. Turns out I had to use the pool key to manually unlock the drive (which makes sense). Here's how I did it.

Parameters:
Pool: sPool
Pool Keyfile: /tmp/geli.key
Disconnected drive: /dev/gptid/19c89a0f-5111-11e4-9f6f-000c29e6c5f0.eli
Disconnected encrypted drive: /dev/gptid/19c89a0f-5111-11e4-9f6f-000c29e6c5f0

1. Offline the drive in the pool

Code:

zpool offline sPool /dev/gptid/19c89a0f-5111-11e4-9f6f-000c29e6c5f0.eli

2. Manually decrypt the drive using GELI

Code:

geli attach -k /tmp/geli.key /dev/gptid/19c89a0f-5111-11e4-9f6f-000c29e6c5f0

3. Online the drive in the pool

Code:

zpool online sPool /dev/gptid/19c89a0f-5111-11e4-9f6f-000c29e6c5f0.eli

The disk is now added back to the storage pool, and will be resilvered.

Hope this can be of help to others.

 

[cyberjock]

That tends to imply you offlined the drive at some point in the past. A disk isn't "offlined" permanently unless someone actually took it offline.

 

[bkvamme]

As I mentioned in the OP, I tried to offline the disk, online it again, and then lock/unlock the pool, but it still did not unlock automatically.

When I rebooted the server, I had not done anything except trying to lock/unlock the pool in order to get the drive recognized by FreeNAS.

The disk was only listed as UNAVAIL, but it appeared to be stuck in that status. If the code does not have anything of this sort, I guess I must have been unlucky.

Oh well. 15 minutes of resilvering later, the pool was restored back to normal.

 

[bkvamme]

Hi,

It appears that I am not done with this issue. After a server reboot, the same drive refused to be decrypted and online when unlocking the pool in the WebUI, and listed the drive as unavailable.

I checked the "View disks" tab before unlocking, and the disk was listed there.

Manually decrypting and onlining the drive resolved the issue this time aswell.

When I powered the freenas box off, I just chose shutdown in the WebUI.

I assume that freenas attempts to decrypt all drives currently in the pool when I unlock the pool, but it appears that it skips the drive that was unavailable earlier.

I assume that exporting/importing the pool again would resolve this issue, but what could be the source of this problem?

I don't reboot the server that often, so this is a minor issue for me.

 

[cyberjock]

It is possible. In the event that you replace a disk you are supposed to rekey the pool per the disk replacement procedure for encrypted disks.

 

[bkvamme]

Okay, thanks for the help.

A detach/import did wonders, now it unlocks it properly again.