Encrypted pool access after reinstall&configuration import

[asmodeus]

Hello,


I'm recovering from a corrupted boot volume (replacing USB stick with SSD). I find myself in a bit of a pickle as I'm struggling to access my encrypted volume.


Here's what I did:


1. Reinstalled FreeNAS to USB stick

2. Decrypted disks using import volume with my key and passphrase

3. Restored my config from a copy on the decrypted pool

4. Shut down, removed USB stick, installed SSD

4. Reinstalled FreeNAS to newly installed SSD

5. Imported config, reboot

6. The unlock volume step does not take my passphrase ("Error: Volume could not be imported: 8 devices failed to decrypt")

7. Rebooted back from USB stick - cannot decrypt the pool with passphrase there either.

The log has a few interesting messages:

Code:

May 22 15:16:29 nas manage.py: [middleware.exceptions:37] [MiddlewareError: Unable to geli attach gptid/ccb0b7fd-d7b8-11e9-9dd3-0cc47a82d3e8: geli: Cannot open keyfile /data/geli/662c4bf5-09cd-42cd-b9ad-71c8dac5dbbf.key: No such file or directory.
]
May 22 15:16:29 nas manage.py: [middleware.notifier:1340] [MiddlewareError: Unable to geli attach gptid/ccb0b7fd-d7b8-11e9-9dd3-0cc47a82d3e8: geli: Cannot open keyfile /data/geli/662c4bf5-09cd-42cd-b9ad-71c8dac5dbbf.key: No such file or directory.
]
May 22 15:16:29 nas manage.py: [middleware.exceptions:37] [MiddlewareError: Unable to geli attach gptid/926b7cdd-d9f8-11e9-9dd3-0cc47a82d3e8: geli: Cannot open keyfile /data/geli/662c4bf5-09cd-42cd-b9ad-71c8dac5dbbf.key: No such file or directory.
]
May 22 15:16:29 nas manage.py: [middleware.notifier:1340] [MiddlewareError: Unable to geli attach gptid/926b7cdd-d9f8-11e9-9dd3-0cc47a82d3e8: geli: Cannot open keyfile /data/geli/662c4bf5-09cd-42cd-b9ad-71c8dac5dbbf.key: No such file or directory.
]
May 22 15:16:29 nas manage.py: [middleware.exceptions:37] [MiddlewareError: Unable to geli attach gptid/69ac090b-9a47-11e9-98a7-0cc47a82d3e8: geli: Cannot open keyfile /data/geli/662c4bf5-09cd-42cd-b9ad-71c8dac5dbbf.key: No such file or directory.
]
May 22 15:16:29 nas manage.py: [middleware.notifier:1340] [MiddlewareError: Unable to geli attach gptid/69ac090b-9a47-11e9-98a7-0cc47a82d3e8: geli: Cannot open keyfile /data/geli/662c4bf5-09cd-42cd-b9ad-71c8dac5dbbf.key: No such file or directory.
]
May 22 15:16:29 nas manage.py: [middleware.exceptions:37] [MiddlewareError: Unable to geli attach gptid/498db3e4-98f5-11e9-98a7-0cc47a82d3e8: geli: Cannot open keyfile /data/geli/662c4bf5-09cd-42cd-b9ad-71c8dac5dbbf.key: No such file or directory.
]
May 22 15:16:29 nas manage.py: [middleware.notifier:1340] [MiddlewareError: Unable to geli attach gptid/498db3e4-98f5-11e9-98a7-0cc47a82d3e8: geli: Cannot open keyfile /data/geli/662c4bf5-09cd-42cd-b9ad-71c8dac5dbbf.key: No such file or directory.
]
May 22 15:16:29 nas manage.py: [middleware.exceptions:37] [MiddlewareError: Unable to geli attach gptid/a6d234d0-9de2-11e9-bf30-0cc47a82d3e8: geli: Cannot open keyfile /data/geli/662c4bf5-09cd-42cd-b9ad-71c8dac5dbbf.key: No such file or directory.
]
May 22 15:16:29 nas manage.py: [middleware.notifier:1340] [MiddlewareError: Unable to geli attach gptid/a6d234d0-9de2-11e9-bf30-0cc47a82d3e8: geli: Cannot open keyfile /data/geli/662c4bf5-09cd-42cd-b9ad-71c8dac5dbbf.key: No such file or directory.
]
May 22 15:16:29 nas manage.py: [middleware.exceptions:37] [MiddlewareError: Unable to geli attach gptid/66ff29af-dd81-11e9-9dd3-0cc47a82d3e8: geli: Cannot open keyfile /data/geli/662c4bf5-09cd-42cd-b9ad-71c8dac5dbbf.key: No such file or directory.
]
May 22 15:16:29 nas manage.py: [middleware.notifier:1340] [MiddlewareError: Unable to geli attach gptid/66ff29af-dd81-11e9-9dd3-0cc47a82d3e8: geli: Cannot open keyfile /data/geli/662c4bf5-09cd-42cd-b9ad-71c8dac5dbbf.key: No such file or directory.
]
May 22 15:16:30 nas manage.py: [middleware.exceptions:37] [MiddlewareError: Unable to geli attach gptid/3f78a642-dbbc-11e9-9dd3-0cc47a82d3e8: geli: Cannot open keyfile /data/geli/662c4bf5-09cd-42cd-b9ad-71c8dac5dbbf.key: No such file or directory.
]
May 22 15:16:30 nas manage.py: [middleware.notifier:1340] [MiddlewareError: Unable to geli attach gptid/3f78a642-dbbc-11e9-9dd3-0cc47a82d3e8: geli: Cannot open keyfile /data/geli/662c4bf5-09cd-42cd-b9ad-71c8dac5dbbf.key: No such file or directory.
]
May 22 15:16:30 nas manage.py: [middleware.exceptions:37] [MiddlewareError: Unable to geli attach gptid/83d40d4d-9f0e-11e9-bf30-0cc47a82d3e8: geli: Cannot open keyfile /data/geli/662c4bf5-09cd-42cd-b9ad-71c8dac5dbbf.key: No such file or directory.
]
May 22 15:16:30 nas manage.py: [middleware.notifier:1340] [MiddlewareError: Unable to geli attach gptid/83d40d4d-9f0e-11e9-bf30-0cc47a82d3e8: geli: Cannot open keyfile /data/geli/662c4bf5-09cd-42cd-b9ad-71c8dac5dbbf.key: No such file or directory.
]
May 22 15:16:30 nas manage.py: [middleware.notifier:3518] Importing tank [16833427744088962482] failed with: cannot import '16833427744088962482': no such pool available
May 22 15:16:30 nas manage.py: [middleware.exceptions:37] [MiddlewareError: Volume could not be imported: 8 devices failed to decrypt]
May 22 15:16:30 nas manage.py: [rollbar:1265] Got unexpected status code from Rollbar api: 403
Response:
{
"err": 1,
"message": "access token not found: caf06383cba14d5893c4f4d0a40c33a9"
}
May 22 15:16:30 nas manage.py: [rollbar:1097] Exception while posting item ApiError(u'access token not found: caf06383cba14d5893c4f4d0a40c33a9',)
Traceback (most recent call last):
File "/usr/local/lib/python2.7/site-packages/rollbar/__init__.py", line 1095, in _send_payload
_post_api('item/', payload, access_token=access_token)
File "/usr/local/lib/python2.7/site-packages/rollbar/__init__.py", line 1138, in _post_api
return _parse_response(path, SETTINGS['access_token'], payload, resp)
File "/usr/local/lib/python2.7/site-packages/rollbar/__init__.py", line 1274, in _parse_response
raise ApiError(json_data.get('message') or 'Unknown error')
ApiError: access token not found: caf06383cba14d5893c4f4d0a40c33a9

8. The import volume/decrypt disk step shows an empty list of drives to decrypt.

I was thinking to detach the drives and attempt import/decrypt again, but before that I wanted to have a second pair of eyes on this please?

I have a geli key, a geli recovery key, the volume passphrase and a passphrase for the geli key.


Thank you,

Achim

 

[Samuel Tai]

You skipped a step between 3 and 4. Once your config is back, you need to reset the keys and regenerate a new recovery key, as restoring the config overrode the in-memory keys at that time.

Start over at step 1. Your original recovery key should work.

 

[asmodeus]

Thanks, I thought I must have missed something. How do I start over though? Currently the import volume/decrypt disks step does not list any drives to decrypt. Is a "detach drives" the action that would allow me to do that?

 

[Samuel Tai]

No, don't detach drives. That takes them out of the pool, and would make the pool permanently unrecoverable.
When I said start over, I meant really start over with a fresh install of FreeNAS, and then import the pool.

 

[asmodeus]

Thank you Samuel, you made my day :) That worked.
Is setting a new key required? I've downloaded the geli key, set a new passphrase and added a new recovery key.

 

[asmodeus]

I take that back :( This worked until I re-imported my configuration and rebooted. I am now again unable to unlock the volume, attempting to do so results in the same console error:

Code:

May 22 15:16:30 nas manage.py: [rollbar:1265] Got unexpected status code from Rollbar api: 403
Response:
{
"err": 1,
"message": "access token not found: caf06383cba14d5893c4f4d0a40c33a9"
}
May 22 15:16:30 nas manage.py: [rollbar:1097] Exception while posting item ApiError(u'access token not found: caf06383cba14d5893c4f4d0a40c33a9',)
Traceback (most recent call last):
File "/usr/local/lib/python2.7/site-packages/rollbar/__init__.py", line 1095, in _send_payload
_post_api('item/', payload, access_token=access_token)
File "/usr/local/lib/python2.7/site-packages/rollbar/__init__.py", line 1138, in _post_api
return _parse_response(path, SETTINGS['access_token'], payload, resp)
File "/usr/local/lib/python2.7/site-packages/rollbar/__init__.py", line 1274, in _parse_response
raise ApiError(json_data.get('message') or 'Unknown error')
ApiError: access token not found: caf06383cba14d5893c4f4d0a40c33a9

Any ideas

 

[Samuel Tai]

Start over. Again, when you upload your configuration, you MUST rekey and set a new recovery key. This is because you've uploaded an older key, which doesn't match the current key in memory, and have to tell FreeNAS those are now invalid.

 

[asmodeus]

Thank you for explaining the part on re-keying. I went ahead with a re-key but I'm still struggling:

1. Start over - fresh install to SSD, booted from SSD.
2. Import volume using geli key and passphrase.
3. Re-keyed, downloaded key, set new passphrase, set recovery key, downloaded recovery key.
4. Imported previous config and rebooted
5. Still unable to unlock the volume.

When I attempt to do this on the UI, I get the previous error on access token not found.
I also tried to unlock the volume using the REST API and got a different error (see screenshot below)

 

[Samuel Tai]

Your order is wrong. Rekey AFTER importing the config. I've been emphasizing this three times now.

 

[Samuel Tai]

Again, rekey AFTER importing, but BEFORE rebooting.

 

[asmodeus]

Thanks for sticking with me, import config, then pool, then re-key.

I was able to unlock the volume using the recovery key. I'll do the re-key now and hope to be in good shape then.

 

[nas_newbie]

Hi, I am having a near identical issue to asmodeus, where I am getting an import error due to my encrpyted pool needing to be re-keyed as described in #7.

However, in following the steps:

1. Fresh install to SSD, boot from SSD.
2. Import volume using geli key and passphrase.
3. Import previous config
4. Re-key, download key, set new passphrase, set recovery key, download recovery key.
5. reboot

I have an issue at step 3. FreeNAS is forcing me to reboot after config upload. I can import the pool just fine using my geli.key and passphrase, so I think the pool is ok, it's just that I cannot do the re-key step before I am forced to reboot.

Is there a way to force a config upload without a reboot? I am currently trying to recover a config based on FreeNAS 9.10.2-U5, if that makes a difference.

Thank you in advance.

 

[Samuel Tai]

Please don't resurrect necro threads; rather start a new thread. No, it's not possible to forestall the reboot after the config upload. You'll need to use the recovery key to import the pool after the reboot, and then rekey again.

 

[nas_newbie]

Hi Samuel,

Sorry, I read the rules and saw "bumping" was ok.. I'll start a new thread in the future. If you would prefer we move this to a new thread please let me know.

My issue prevails after using the recovery key on the pool in the GUI. Steps I followed exactly are:

1. fresh install FreeNAS 9.10.2-U5 to SSD
2. import volume using geli.key and passphrase (it shows an ZFS ID number in the import wizard for my pool)
3. import backup config (triggering an initial reboot to upload the backup .db, followed by a second automatic reboot after the "database upgrade" has finished.)
4. unlock volume with recovery key

I stall at step 4. I get the same "no such pool available" error whether I use the recovery key or my passphrase on the "unlock" GUI.

I will note, that it seems that the ZFS ID it is looking for is the same one that was found in step 2, so I dont understand what the mis-match is... maybe I am mis-understanding the error, its the same as in #8 but with my own pool/ID.

 

[Samuel Tai]

If you didn't rekey, then use your geli.key and passphrase.

 

[nas_newbie]

Sorry, can you clarify? The "Unlock" dialogue only accepts the passphrase itself or the recovery key.. is there somewhere else I should be trying to use the actual geli.key file? I thought that was only for step 2 where I imported the volume before my config upload?

Just to ensure I understand, to re-key and have FreeNAS accept that I have re-keyed my pool, I must have my config uploaded and pool unlocked before any subsequent reboot occurs. However, in this case since FreeNAS is rebooting after my config is loaded I don't see how mechanically it's possible, therefore I think I must be missing something.

Thank you for your patience.

 

[Samuel Tai]

Unfortunately, the last time I ran 9.10 with GELI was a couple years ago, so my recollection is a bit fuzzy, sorry. Be that as it may, I believe after the step 2 import, you should rekey and create a new recovery key. Then import the backup config, which triggers the reboot. The config at this point thinks the pool has the key before your step 2 rekey, so even if you kept the same passphrase, it won't unlock the pool, as that passphrase won't work with the restored keys from the config upload. When unlocking the pool after the reboot, use your most recent recovery key and passphrase. Then once the pool is unlocked, perform another rekey and recovery key generation.

 

[nas_newbie]

No problem, thanks for digging through the cobwebs for a random internet stranger!

I am happy to you inform you that your recollection was correct!

For anyone who may stumble upon this in the future, here are the steps I used to resolve my issue:

1. fresh install FreeNAS 9.10.2-U5 to SSD
2. import volume using geli.key and passphrase
3. re-key volume, set passphrase, download key, download recovery-key
4. import backup config
5. after automatic reboot, unlock volume with recovery key
6. re-key volume, set passphrase, download key, download recovery-key
7. reboot

Thank you so much Samuel, I greatly appreciate your effort and patience. This really helped me out, and I hope it helps others who may stumble upon this issue in the future.