rfang9524
Cadet
- Joined
- Jan 3, 2019
- Messages
- 3
I had the exact same issue. I though of doing the solutions posted above, but having to buy 20 large drives would be quite expensive. Luckily, I was able fix it after hours and hours of troubleshooting. Hope this helps anyone that had the same issue.
TLDR;
What Happened
I purchased 4 new hard drive and added to the RAID as a new vdev. I restarted my server a few hours later. Upon startup I could decrypt the pool anymore. It will prompt with an error stating four of the (new) drives couldn't be decrypted. I had the recovery key, encryption key, and password from the day before, but none of them worked. I remember I did not issue or received a new key since. After hours and hours of research and trial and error, I was finally able to decrypt the pool.
Important Things to Note:
Here's What I Did
TLDR;
- Use geli to attach and decrypt each drive one at a time using the recovery key
- Use geli to attach the drives that failed to decrypt previously using the encryption key
- Import the pool and remount/relocate if necessary
What Happened
I purchased 4 new hard drive and added to the RAID as a new vdev. I restarted my server a few hours later. Upon startup I could decrypt the pool anymore. It will prompt with an error stating four of the (new) drives couldn't be decrypted. I had the recovery key, encryption key, and password from the day before, but none of them worked. I remember I did not issue or received a new key since. After hours and hours of research and trial and error, I was finally able to decrypt the pool.
Important Things to Note:
- The encrypted partition is on p2 (i.e. /dev/sda#p2). Thus, you can't just specify /dev/sda#
- da8-da11 are the new drives that had issues decrypting
- After the pool is mounted, I had to remount it using zfs to the correct location on the filesystem.
Here's What I Did
Code:
root@freenas:~ # geli attach -p -k pool_RAID_recovery\ \(2\).key /dev/da0p2
root@freenas:~ # geli status
Name Status Components
da0p2.eli ACTIVE da0p2
root@freenas:~ # geli attach -p -k pool_RAID_recovery\ \(2\).key /dev/da1p2
root@freenas:~ # geli attach -p -k pool_RAID_recovery\ \(2\).key /dev/da2p2
root@freenas:~ # geli status
Name Status Components
da0p2.eli ACTIVE da0p2
da1p2.eli ACTIVE da1p2
da2p2.eli ACTIVE da2p2
root@freenas:~ # geli attach -p -k pool_RAID_recovery\ \(2\).key /dev/da3p2
root@freenas:~ # geli attach -p -k pool_RAID_recovery\ \(2\).key /dev/da4p2
root@freenas:~ # geli attach -p -k pool_RAID_recovery\ \(2\).key /dev/da5p2
root@freenas:~ # geli attach -p -k pool_RAID_recovery\ \(2\).key /dev/da6p2
root@freenas:~ # geli attach -p -k pool_RAID_recovery\ \(2\).key /dev/da7p2
root@freenas:~ # geli attach -p -k pool_RAID_recovery\ \(2\).key /dev/da8p2
geli: Wrong key for da8p2.
root@freenas:~ # geli attach -k pool_RAID_encryption\ \(3\).key -p /dev/da8p2
root@freenas:~ # geli attach -k pool_RAID_encryption\ \(3\).key -p /dev/da9p2
root@freenas:~ # geli attach -k pool_RAID_encryption\ \(3\).key -p /dev/da10p2
root@freenas:~ # geli attach -k pool_RAID_encryption\ \(3\).key -p /dev/da11p2
root@freenas:~ # geli status
Name Status Components
da0p2.eli ACTIVE da0p2
da1p2.eli ACTIVE da1p2
da2p2.eli ACTIVE da2p2
da3p2.eli ACTIVE da3p2
da4p2.eli ACTIVE da4p2
da5p2.eli ACTIVE da5p2
da6p2.eli ACTIVE da6p2
da7p2.eli ACTIVE da7p2
da8p2.eli ACTIVE da8p2
da9p2.eli ACTIVE da9p2
da10p2.eli ACTIVE da10p2
da11p2.eli ACTIVE da11p2
root@freenas:~ # geli attach -p -k pool_RAID_recovery\ \(2\).key /dev/da12p2
root@freenas:~ # geli attach -p -k pool_RAID_recovery\ \(2\).key /dev/da13p2
root@freenas:~ # geli attach -p -k pool_RAID_recovery\ \(2\).key /dev/da14p2
root@freenas:~ # geli attach -p -k pool_RAID_recovery\ \(2\).key /dev/da15p2
root@freenas:~ # geli attach -p -k pool_RAID_recovery\ \(2\).key /dev/da16p2
root@freenas:~ # geli attach -p -k pool_RAID_recovery\ \(2\).key /dev/da17p2
root@freenas:~ # geli attach -p -k pool_RAID_recovery\ \(2\).key /dev/da18p2
root@freenas:~ # geli attach -p -k pool_RAID_recovery\ \(2\).key /dev/da19p2
root@freenas:~ # zpool import
pool: RAID
id: 628020232310xxxxxx
state: ONLINE
action: The pool can be imported using its name or numeric identifier.
config:
RAID ONLINE
raidz1-0 ONLINE
da12p2.eli ONLINE
da13p2.eli ONLINE
da14p2.eli ONLINE
da19p2.eli ONLINE
raidz1-1 ONLINE
da18p2.eli ONLINE
da16p2.eli ONLINE
da17p2.eli ONLINE
da15p2.eli ONLINE
raidz1-2 ONLINE
da3p2.eli ONLINE
da5p2.eli ONLINE
da6p2.eli ONLINE
da7p2.eli ONLINE
raidz1-3 ONLINE
da0p2.eli ONLINE
da1p2.eli ONLINE
da2p2.eli ONLINE
da4p2.eli ONLINE
raidz1-4 ONLINE
da11p2.eli ONLINE
da8p2.eli ONLINE
da9p2.eli ONLINE
da10p2.eli ONLINE
cache
nvd0p1
root@freenas:~ # zpool import -a
root@freenas:~ # zpool status -v
pool: RAID
state: ONLINE
scan: resilvered 2.57T in 0 days 11:39:28 with 0 errors on Sun Mar 7 17:32:58 2021
config:
NAME STATE READ WRITE CKSUM
RAID ONLINE 0 0 0
raidz1-0 ONLINE 0 0 0
da12p2.eli ONLINE 0 0 0
da13p2.eli ONLINE 0 0 0
da14p2.eli ONLINE 0 0 0
da19p2.eli ONLINE 0 0 0
raidz1-1 ONLINE 0 0 0
da18p2.eli ONLINE 0 0 0
da16p2.eli ONLINE 0 0 0
da17p2.eli ONLINE 0 0 0
da15p2.eli ONLINE 0 0 0
raidz1-2 ONLINE 0 0 0
da3p2.eli ONLINE 0 0 0
da5p2.eli ONLINE 0 0 0
da6p2.eli ONLINE 0 0 0
da7p2.eli ONLINE 0 0 0
raidz1-3 ONLINE 0 0 0
da0p2.eli ONLINE 0 0 0
da1p2.eli ONLINE 0 0 0
da2p2.eli ONLINE 0 0 0
da4p2.eli ONLINE 0 0 0
raidz1-4 ONLINE 0 0 0
da11p2.eli ONLINE 0 0 0
da8p2.eli ONLINE 0 0 0
da9p2.eli ONLINE 0 0 0
da10p2.eli ONLINE 0 0 0
cache
nvd0p1 ONLINE 0 0 0
errors: No known data errors
pool: freenas-boot
state: ONLINE
scan: scrub repaired 0 in 0 days 04:26:38 with 0 errors on Fri Mar 5 08:15:47 2021
config:
NAME STATE READ WRITE CKSUM
freenas-boot ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
da21p2 ONLINE 0 0 0
da20p2 ONLINE 0 0 0
errors: No known data errors
root@freenas:/ # zfs set mountpoint=/mnt/RAID RAID
