Volume won't unlock

[pletopia]

Please help. I'm not sure what happened but my encrypted Volume will not unlock anymore.

Environment:

Software Version: FreeNAS-11.1-U5 (8e2a858a1)
Request Method: GET
Request URL: http://192.168.5.105/admin/storage/volumestatus/datagrid/?id=2&dojo.preventCache=1529617252265


Traceback:
File "/usr/local/lib/python3.6/site-packages/django/core/handlers/exception.py" in inner
42. response = get_response(request)
File "/usr/local/lib/python3.6/site-packages/django/core/handlers/base.py" in _legacy_get_response
249. response = self._get_response(request)
File "/usr/local/lib/python3.6/site-packages/django/core/handlers/base.py" in _get_response
178. response = middleware_method(request, callback, callback_args, callback_kwargs)
File "./freenasUI/freeadmin/middleware.py" in process_view
162. return login_required(view_func)(request, *view_args, **view_kwargs)
File "/usr/local/lib/python3.6/site-packages/django/contrib/auth/decorators.py" in _wrapped_view
23. return view_func(request, *args, **kwargs)
File "./freenasUI/freeadmin/options.py" in wrapper
210. return self._admin.admin_view(view)(*args, **kwargs)
File "/usr/local/lib/python3.6/site-packages/django/utils/decorators.py" in _wrapped_view
149. response = view_func(request, *args, **kwargs)
File "/usr/local/lib/python3.6/site-packages/django/views/decorators/cache.py" in _wrapped_view_func
57. response = view_func(request, *args, **kwargs)
File "./freenasUI/freeadmin/site.py" in inner
142. return view(request, *args, **kwargs)
File "./freenasUI/freeadmin/options.py" in datagrid
961. context.update(self.get_datagrid_context(request=request))
File "./freenasUI/storage/admin.py" in get_datagrid_context
415. pool = notifier().zpool_parse(volume.vol_name)
File "./freenasUI/middleware/notifier.py" in zpool_parse
3386. parse = zfs.parse_status(name, doc, res)
File "./freenasUI/middleware/zfs.py" in parse_status
761. status = data.split('config:')[1]

Exception Type: IndexError at /admin/storage/volumestatus/datagrid/
Exception Value: list index out of range

Everything was working fine until I removed another volume from the machine, imported a new volume that I created on a 2nd FreeNas system. That new volume is working fine after import but one of the 2 original volumes on this machine will not unlock anymore.

 

[pletopia]

My thought was to remove the volume from the system and then add it back in using the saved key. Any thoughts on that? I hope I didn't lose 25GB of data.

 

[dlavigne]

Were you able to resolve this?

 

[pletopia]

I did not as I didn’t attempt my remove and then add back the volume. Unfortunately I’m moving cross country and my server has been packaged up and I won’t have access to it for a couple weeks until I move into my new place. If people have a thought of what to do to fix this before I get it back I’m all ears otherwise mid July I’ll try to do what I posted above.

 

[pletopia]

Okay, I finally was able to unpack everything and start up the FreeNAS machine. It still will not unlock. Something I noticed when I was trying to unlock the volume was there was an error which I failed to copy/paste or screenshot but it said "failed to decrypt 1 drive". I just looked at the SMART data for all the drives on that zpool and they passed, even after running fresh short tests on them manually.

Maybe before doing anything drastic I should run extended tests on them? Or anyone have any other thoughts?

 

[pletopia]

All the drives passed short, conveyance and long tests ...

 

[pro lamer]

All the drives passed short, conveyance and long tests ...

Badblocks too? Long smart is recommended after badblocks (or badblocks-similar)

Sent from my mobile phone

 

[pletopia]

I did not test badblocks.

badblocks also offers a non-destructive read-write test that (in theory) shouldn't damage any existing data, but if you do choose to run it on a production drive and suffer data loss, on your own head be it:
Code:
badblocks -ns /dev/adaX

whats the chance of it wiping my data ?

 

[Ericloewe]

About as likely as you mistyping that, which is fairly dangerous, as a corollary of Murphy's Law.

 

[Ericloewe]

"failed to decrypt 1 drive"

Surely your pool has enough redundancy to deal with a single failure, right? Unlock the disks and manually import the pool in the CLI to get to your data. We can work from there.

 

[pletopia]

Yes, its a RAIDZ2.

Unlock it through CLI instead of the web gui? Cause everytime I try to in the web gui it fails ...

 

[Ericloewe]

Unlock it through CLI instead of the web gui? Cause everytime I try to in the web gui it fails ...

Go through the process and stop the wizard after unlocking the disks. They should still be unlocked.

You could do it in the CLI, but you'll have to read the man pages for GELI.

 

[pletopia]

I'm not sure what you mean by "stop the wizard after unlocking the disks". Normally I go here ..



And Select the Volume I want, in this case the bad one is James.T.Kirk. After I enter passkey and hit enter I get an error ...



And then this is how my screen looks.

 

[pletopia]

This is what I see on the monitor hooked up to the machine when I try to unlock the volume.


poems by George Hitchcock

 

[Ericloewe]

What's the output of zpool import, at that stage?

 

[pletopia]

root@StarTrek:~ # zpool import
pool: James.T.Kirk
id: 6709121945907512084
state: UNAVAIL
status: One or more devices are missing from the system.
action: The pool cannot be imported. Attach the missing
devices and try again.
see: http://illumos.org/msg/ZFS-8000-3C
config:

James.T.Kirk UNAVAIL insufficient replicas
raidz2-0 UNAVAIL insufficient replicas
3320284236721174246 UNAVAIL cannot open
11172670065561495953 UNAVAIL cannot open
gptid/2f796ff2-f0c7-11e6-9933-bc5ff4fcef41.eli ONLINE
8580267007091217866 UNAVAIL cannot open
gptid/31a13970-f0c7-11e6-9933-bc5ff4fcef41.eli ONLINE
gptid/32d3f8e5-f0c7-11e6-9933-bc5ff4fcef41.eli ONLINE
gptid/3461c330-f0c7-11e6-9933-bc5ff4fcef41.eli ONLINE
gptid/3570a267-f0c7-11e6-9933-bc5ff4fcef41.eli ONLINE


Are you telling me 3 drives crapped out at the same time?

 

[Ericloewe]

Okay, three disks missing. That raises questions...

At this point, I recommend you read up on GELI and try to manually unlock all the disks, one at a time. Try both the normal key + password and the recovery key.

 

[pletopia]

What if I removed the pool and imported it again using the backed up geli.key file or the geli_recovery.key as I thought to do in my 2nd post above?

What are your thoughts on that Ericloewe

 

[Ericloewe]

The pool isn't even imported at this point, there's nothing to "remove". You can try the recovery key from the GUI, though.

 

[pletopia]

Its back !!!!
One of the drives died and needs replacement I guess but the data is there.

Very strange that 1 drive dying prevented two others from loading up.