Can't Re-Key Encryption After Replace/Re-Silver

[Syris]

My pool is set up as follows:

RAID-Z2-0
(6)3TB drives
RAID-Z2-1
(5)3TB Drives
(1)4TB Drive(This is the new disk that replaced a 3TB and just finished re-silvering)
RAID-Z2
(6)4TB Drives

After re-silvering takes place I' try to follow the steps from the freeNAS manual here:
http://doc.freenas.org/9.3/freenas_storage.html#replacing-an-encrypted-drive\

However when attempting to re-key the pool I get the following error:
"Error: Unable to set key: [MiddlewareError: Unable to set passphrase on gptid/d1a6e94d-9f9f-11e4-b99f-000c2934e0ad: geli: Cannot open gptid/d1a6e94d-9f9f-11e4-b99f-000c2934e0ad: No such file or directory. ]"

Part of the blame lies with me I suspect b/c I removed the old 3TB drive with out off lining it first, but I figured it would just simulate a HDD dying. I've replaced like 3 other hard drives that have failed (this one didn't fail but is being replaced one at a time to extend the vdev) and never had this problem despite not being able to mark them offline.

Also how the hell do you lose access to your whole pool if you reboot before re-key (warning according to freeNAS man.)
Any help is greatly appreciated.

Found this Thread:
https://forums.freenas.org/index.ph...ive-unable-to-set-key-geli-cannot-open.14554/

Seems he found a way to remove info for old drive, but I cant make heads or tales of the commands he was using.

[root@freenas] /data# sqlite3 /data/freenas-v1.db "select * from storage_disk;"

for me trying that get a y/n/e/a/prompt and replying yes gets permissions denied.

Edit #2: Finally got it figured out.

FIRST USE:

Code:

sqlite3 /data/freenas-v1.db "select * from storage_encrypteddisk;"

Then Identify the one giving the error and use

Code:

sqlite3 /data/freenas-v1.db "delete from storage_encrypteddisk where id=3;"

and replace the '3' in "where id=3" with the correct number of the offending entry.

 

[m0nkey_]

Before replacing a drive, you must always off-line it prior to physically removing it. Also, by using encryption you're adding an unnecessary level of complication creating issues like this if you don't follow the documentation to the letter. I would highly recommend backing up your data if you still have access to it before proceeding any further.

 

[Syris]

Also, by using encryption you're adding an unnecessary level of complication creating issues like this if you don't follow the documentation to the letter. I would highly recommend backing up your data if you still have access to it before proceeding any further.

All is good , I have re-keyed and rebooted and everything is fine. Your correct that I should have off-line'd the drive first, but if you treat my removal as a drive that died, then it would have behaved the same way. There is actually a bug report filed for this reason.

 

[m0nkey_]

SMART should give an opportunity to identify a failing drive and to replace it before it takes it last platter revolution, making sure correct use of off-lining a disk is used :)

Glad to know you managed to sort it out.

 

[Syris]

Not trying to be condescending at all (have no idea your exp level) ,but just so your aware I have personally had a drive in a FreeNAS system die out-right without any S.M.A.R.T. errors. That said in my exp your usually right that errors will show before drive gives up the ghost. Regardless I have definitely learned my lesson about making sure to offline in the future. :)

 

[soulburn]

My pool is set up as follows:

RAID-Z2-0
(6)3TB drives
RAID-Z2-1
(5)3TB Drives
(1)4TB Drive(This is the new disk that replaced a 3TB and just finished re-silvering)
RAID-Z2
(6)4TB Drives

After re-silvering takes place I' try to follow the steps from the freeNAS manual here:
http://doc.freenas.org/9.3/freenas_storage.html#replacing-an-encrypted-drive\

However when attempting to re-key the pool I get the following error:
"Error: Unable to set key: [MiddlewareError: Unable to set passphrase on gptid/d1a6e94d-9f9f-11e4-b99f-000c2934e0ad: geli: Cannot open gptid/d1a6e94d-9f9f-11e4-b99f-000c2934e0ad: No such file or directory. ]"

Part of the blame lies with me I suspect b/c I removed the old 3TB drive with out off lining it first, but I figured it would just simulate a HDD dying. I've replaced like 3 other hard drives that have failed (this one didn't fail but is being replaced one at a time to extend the vdev) and never had this problem despite not being able to mark them offline.

Also how the hell do you lose access to your whole pool if you reboot before re-key (warning according to freeNAS man.)
Any help is greatly appreciated.

Found this Thread:
https://forums.freenas.org/index.ph...ive-unable-to-set-key-geli-cannot-open.14554/

Seems he found a way to remove info for old drive, but I cant make heads or tales of the commands he was using.

[root@freenas] /data# sqlite3 /data/freenas-v1.db "select * from storage_disk;"

for me trying that get a y/n/e/a/prompt and replying yes gets permissions denied.

Edit #2: Finally got it figured out.

FIRST USE:

Code:

sqlite3 /data/freenas-v1.db "select * from storage_encrypteddisk;"

Then Identify the one giving the error and use

Code:

sqlite3 /data/freenas-v1.db "delete from storage_encrypteddisk where id=3;"

and replace the '3' in "where id=3" with the correct number of the offending entry.

Thank you for this! You saved me today as I just ran into this issue. The difference for my particular scenario was that the drive was offlined by FreeNAS automatically according to the GUI and this problem persisted. I created another topic to ask a related question which you can reference here if you'd like. Thanks again!

 

[erre]

My pool is set up as follows:

RAID-Z2-0
(6)3TB drives
RAID-Z2-1
(5)3TB Drives
(1)4TB Drive(This is the new disk that replaced a 3TB and just finished re-silvering)
RAID-Z2
(6)4TB Drives

After re-silvering takes place I' try to follow the steps from the freeNAS manual here:
http://doc.freenas.org/9.3/freenas_storage.html#replacing-an-encrypted-drive\

However when attempting to re-key the pool I get the following error:
"Error: Unable to set key: [MiddlewareError: Unable to set passphrase on gptid/d1a6e94d-9f9f-11e4-b99f-000c2934e0ad: geli: Cannot open gptid/d1a6e94d-9f9f-11e4-b99f-000c2934e0ad: No such file or directory. ]"

Part of the blame lies with me I suspect b/c I removed the old 3TB drive with out off lining it first, but I figured it would just simulate a HDD dying. I've replaced like 3 other hard drives that have failed (this one didn't fail but is being replaced one at a time to extend the vdev) and never had this problem despite not being able to mark them offline.

Also how the hell do you lose access to your whole pool if you reboot before re-key (warning according to freeNAS man.)
Any help is greatly appreciated.

Found this Thread:
https://forums.freenas.org/index.ph...ive-unable-to-set-key-geli-cannot-open.14554/

Seems he found a way to remove info for old drive, but I cant make heads or tales of the commands he was using.

[root@freenas] /data# sqlite3 /data/freenas-v1.db "select * from storage_disk;"

for me trying that get a y/n/e/a/prompt and replying yes gets permissions denied.

Edit #2: Finally got it figured out.

FIRST USE:

Code:

sqlite3 /data/freenas-v1.db "select * from storage_encrypteddisk;"

Then Identify the one giving the error and use

Code:

sqlite3 /data/freenas-v1.db "delete from storage_encrypteddisk where id=3;"

and replace the '3' in "where id=3" with the correct number of the offending entry.

Thank you very much!

This should be in the guide. This has happened 2 times for me. But it has been about 1 year in between, so I had forgetting how to fix it.

In my case, I had a faulty SATA power connector, and dropped 2 drives that wasn't faulty. No possibility to offline the disk first.