Expanding a raidz2 pool and encryption (Panic attack)

[Fr33nasnoobi3]

So HOPEFULLY this is fixable and i didn't find a bug.... So i had a 6 drive raidz2 (6x4tb drives) setup working fine for a few years it was running out of space and i expanded it with an additional 6x6tb drives ok everything worked fine and dandy for a few months fast forward to me trying to update to freenas 10 to check it out i installed freenas 10 (formated) my boot drives thinking i would just be able to import my volume and go on as normal.... This is where things went south come to find out still a bug (from what i read) with 10 and the importing of encypted volumes sigh* no biggie i think ill just reinstall 9.10 so i do that and go to import the volume like i have done before and.....

• The following disks failed to attach: gptid/723c9664-e2aa-11e6-81a3-d050992ecffc, gptid/b6362ee9-9e0c-11e5-b673-d050992ecffc, gptid/5d085582-5328-11e4-b620-d050992ecffc, gptid/5c97b2be-5328-11e4-b620-d050992ecffc, gptid/737f80ad-e2aa-11e6-81a3-d050992ecffc, gptid/5e4a6066-5328-11e4-b620-d050992ecffc, gptid/6e92dd63-e2aa-11e6-81a3-d050992ecffc, gptid/6d5a535f-e2aa-11e6-81a3-d050992ecffc, gptid/5de1d537-5328-11e4-b620-d050992ecffc, gptid/5eb1d8e2-5328-11e4-b620-d050992ecffc, gptid/7103a362-e2aa-11e6-81a3-d050992ecffc, gptid/6fc79c28-e2aa-11e6-81a3-d050992ecffc

OH CRAP!!! and if i remember right this is the same error it gave me in freenas 10 when trying to import (before i read there was an issue with it) Bug #19568

So my main question is.... (i mean besides how do i fix this for the love of god all my data is gone) when u expand a volume like i did does it re-key the geil.key file? becuase the file i have downloaded was last modified a day after i got my hard drives from amazon so i assume it asked me to re-download the key or i just did it out of habit. not sure slept a few times since jan 23rd.

So anyone know how i get this damn thing back online? :'( please help!

 

[SweetAndLow]

output of zpool status, zpool import and hardware specs please.

 

[m0nkey_]

That isn't good. Just to add to @SweetAndLow request, also include the output of camcontrol devlist.

Your best course of recovery is to restore your data from backups.

 

[Fr33nasnoobi3]

output of zpool status, zpool import and hardware specs please.

Hardware specs

https://www.amazon.com/gp/product/B00HIDQG6E/ref=oh_aui_detailpage_o00_s01?ie=UTF8&psc=1

https://www.amazon.com/gp/product/B008EMA5VU/ref=oh_aui_detailpage_o00_s01?ie=UTF8&psc=1


[root@freenas ~]# zpool status
pool: freenas-boot
state: ONLINE
scan: none requested
config:

NAME STATE READ WRITE CKSUM
freenas-boot ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
da0p2 ONLINE 0 0 0
da1p2 ONLINE 0 0 0

errors: No known data errors

zpool import displays nothing and im to stupid to figure out how to do it from the command line.... cant seem to get a usb key to mount with the geil.key file on it to try....


[root@freenas ~]# camcontrol devlist
<ST4000DM000-1F2168 CC52> at scbus0 target 0 lun 0 (pass0,ada0)
<ST6000VN0041-2EL11C SC61> at scbus1 target 0 lun 0 (pass1,ada1)
<ST4000DM000-1F2168 CC54> at scbus2 target 0 lun 0 (pass2,ada2)
<ST4000DM000-1F2168 CC54> at scbus3 target 0 lun 0 (pass3,ada3)
<ST4000DM000-1F2168 CC52> at scbus4 target 0 lun 0 (pass4,ada4)
<ST4000VN000-1H4168 SC43> at scbus5 target 0 lun 0 (pass5,ada5)
<Marvell Console 1.01> at scbus9 target 0 lun 0 (pass6)
<ST6000VN0041-2EL11C SC61> at scbus10 target 0 lun 0 (pass7,ada6)
<ST6000VN0041-2EL11C SC61> at scbus11 target 0 lun 0 (pass8,ada7)
<ST6000VN0041-2EL11C SC61> at scbus12 target 0 lun 0 (pass9,ada8)
<ST6000VN0041-2EL11C SC61> at scbus13 target 0 lun 0 (pass10,ada9)
<ST4000VN000-1H4168 SC46> at scbus14 target 0 lun 0 (pass11,ada10)
<ST6000VN0041-2EL11C SC61> at scbus15 target 0 lun 0 (pass12,ada11)
<Kingston DataTraveler SE9 PMAP> at scbus17 target 0 lun 0 (pass13,da0)
<Kingston DataTraveler SE9 PMAP> at scbus18 target 0 lun 0 (pass14,da1)
<Kingston DataTraveler 3.0 PMAP> at scbus19 target 0 lun 0 (da2,pass15)

 

[Fr33nasnoobi3]

That isn't good. Just to add to @SweetAndLow request, also include the output of camcontrol devlist.

Your best course of recovery is to restore your data from backups.

[root@freenas ~]# camcontrol devlist
<ST4000DM000-1F2168 CC52> at scbus0 target 0 lun 0 (pass0,ada0)
<ST6000VN0041-2EL11C SC61> at scbus1 target 0 lun 0 (pass1,ada1)
<ST4000DM000-1F2168 CC54> at scbus2 target 0 lun 0 (pass2,ada2)
<ST4000DM000-1F2168 CC54> at scbus3 target 0 lun 0 (pass3,ada3)
<ST4000DM000-1F2168 CC52> at scbus4 target 0 lun 0 (pass4,ada4)
<ST4000VN000-1H4168 SC43> at scbus5 target 0 lun 0 (pass5,ada5)
<Marvell Console 1.01> at scbus9 target 0 lun 0 (pass6)
<ST6000VN0041-2EL11C SC61> at scbus10 target 0 lun 0 (pass7,ada6)
<ST6000VN0041-2EL11C SC61> at scbus11 target 0 lun 0 (pass8,ada7)
<ST6000VN0041-2EL11C SC61> at scbus12 target 0 lun 0 (pass9,ada8)
<ST6000VN0041-2EL11C SC61> at scbus13 target 0 lun 0 (pass10,ada9)
<ST4000VN000-1H4168 SC46> at scbus14 target 0 lun 0 (pass11,ada10)
<ST6000VN0041-2EL11C SC61> at scbus15 target 0 lun 0 (pass12,ada11)
<Kingston DataTraveler SE9 PMAP> at scbus17 target 0 lun 0 (pass13,da0)
<Kingston DataTraveler SE9 PMAP> at scbus18 target 0 lun 0 (pass14,da1)
<Kingston DataTraveler 3.0 PMAP> at scbus19 target 0 lun 0 (da2,pass15)

 

[SweetAndLow]

When you try and use the GUI to import the pool does it take your key and passphrase or your recovery key?

Sent from my Nexus 5X using Tapatalk

 

[Fr33nasnoobi3]

It doesnt say its wrong but it does say ...

• The following disks failed to attach: gptid/723c9664-e2aa-11e6-81a3-d050992ecffc, gptid/b6362ee9-9e0c-11e5-b673-d050992ecffc, gptid/5d085582-5328-11e4-b620-d050992ecffc, gptid/5c97b2be-5328-11e4-b620-d050992ecffc, gptid/737f80ad-e2aa-11e6-81a3-d050992ecffc, gptid/5e4a6066-5328-11e4-b620-d050992ecffc, gptid/6e92dd63-e2aa-11e6-81a3-d050992ecffc, gptid/6d5a535f-e2aa-11e6-81a3-d050992ecffc, gptid/5de1d537-5328-11e4-b620-d050992ecffc, gptid/5eb1d8e2-5328-11e4-b620-d050992ecffc, gptid/7103a362-e2aa-11e6-81a3-d050992ecffc, gptid/6fc79c28-e2aa-11e6-81a3-d050992ecffc

 

[SweetAndLow]

Is this with the recovery key or normal key? Sounds like you will be restoring from backup because you can't decrypt your pool.

Sent from my Nexus 5X using Tapatalk

 

[Fr33nasnoobi3]

honestly up until a day or 2 ago when i started researching this i didn't know there was a recovery key i always downloaded the key and saved it.... so does that error mean the key is corrupted?

 

[SweetAndLow]

Key isn't corrupt you just messed something up when creating the key or adding the vdev.

Sent from my Nexus 5X using Tapatalk

 

[Fr33nasnoobi3]

So what did i do wrong if everything is truly gone? i used the gui for everything it expanded the data set fine everything was working correctly and im about 99% sure i downloaded another copy of the key...

 

[SweetAndLow]

So what did i do wrong if everything is truly gone? i used the gui for everything it expanded the data set fine everything was working correctly and im about 99% sure i downloaded another copy of the key...

I have no clue what you did. That is a question you have to ask yourself. I can tell you that the key you are trying to use to unlock the volume is probably the wrong one. You should just test it yourself but I did it for you. When I encrypted a pool then added a vdev and did not rekey the pool like required. I could not import the pool using the old key and got the exact error you are seeing. So I suspect you have the original encryption key from the first pool and you did not rekey the system or you are accidentally using the older key.