SOLVED Cannot Unlock Volume

[NAiLs]

Hi Everyone -

First and foremost, I must say that I'm relatively "new" to FreeNAS. I have been using it for long enough to know my way around the GUI, but I do not know all the tricks behind it, especially CLI.

So earlier this year, I rebuilt my FreeNAS box, because my old hardware was aged and I had 2 of 4 drives fail on me. Fortunately I was able to recover 95% of my data before that second drive failed. So I've got new-ish (re-purposed) hardware and 5 brand new disks. I had minor hiccups with both of my flash drives corrupting at the same time, but fortunately that was all in testing phases and I lost nothing. Now that I have a good working flash drive again, I've built my NAS back up and have been using it successfully since February 2018. I chose to encrypt my disks, but I'm now finding out that many folks do not for various reasons.

I am currently using FreeNAS 11.1-U1. I have intentions of upgrading to the latest soon, but now I'm struck with a problem before I can. I have not applied any updates since building in February.

I had a power outage late last week, FreeNAS server came back up as it has in the past, but I did not notice that my volume was unavailable until last night. I went to Storage -> Volumes and saw my volume, but it said it was locked. Never had to unlock it before, but whatever, no big deal... right? I tried to unlock it, but it just gave me an error of "Volume Failed Unlocked" as soon as I click YES on the 'Do you want to unlock this volume' dialogue. I am never asked for my key or anything, just a dialogue that says that, followed by the error.

So, reading around on the forums here, I noticed that a lot of folks just have to detatch the volume and just import it again. I see the guide tell you to do such, as well. I made sure the two boxes were NOT checked, so my data wouldn't wipe. Well, I go to import the volume, but I see nothing listed. I go to Import Disks, and I see nothing listed there, too. Heck, even if I go to the Volume Manager, it tells me there are no disks to use. I'm assuming 'zpool status' is the only way to list disks, but I can't see anything and I'm guessing it's because the disks are encrypted.

I reloaded my FreeNAS configuration from February (didn't make any major changes since) and my volume is there again, but I still cannot unlock it. I continue to get the 'Volume Failed Unlocked" message on the top of the GUI.

Here is what the console shows after I attempt to unlock the volume:
Oct 8 22:01:18 System uwsgi: [middleware.notifier:2524] Importing VolumeName [12275454876993067764] failed with: cannot import '12275454876993067764': no such pool available

zpool status
pool: freenas-boot
state: ONLINE
scan: scrub repaired 0 in 0 days 00:01:06 with 0 errors on Sun Oct 7 03:46:06
2018
config:

NAME STATE READ WRITE CKSUM
freenas-boot ONLINE 0 0 0
da0p2 ONLINE 0 0 0

errors: No known data errors

zpool import does nothing - outputs no text.

I'm stumped.... I can't find any way to get my data/volume to unlock. I have restarted the box several times, as well, to no avail.

I am using a RAID-Z2 volume that is encrypted. I DO have the geli.key backed up, as well as my configuration.

FWIW:
Intel Core-i7 4770k
16GB Non-ECC RAM
MSI Z97 Gaming 5
5x 4TB HGST NAS Drives
FreeNAS 11.1-U1

One of these days, I do plan on getting hardware that supports ECC RAM, as I have a ton of it set aside, but for now this is what I've got.

I appreciate your help in advance! My apologies for being a complete newb! Let me know if there is something else that I can run to provide more information.

-Brian

 

[Chris Moore]

If it says there are no disks, and you had a power hit, you may have some kind of fault with the hardware such that the disks are not available. Have you checked on the disks? When was the last time you checked the SMART status of them?
When you go to to Storage ‣ Volumes ‣ View Disks, what do you see? You can throw a screenshot in, it might help.

 

[Chris Moore]

PS. Have you looked at the manual: http://doc.freenas.org/11/storage.html#import-volume

 

[NAiLs]

If it says there are no disks, and you had a power hit, you may have some kind of fault with the hardware such that the disks are not available. Have you checked on the disks? When was the last time you checked the SMART status of them?
When you go to to Storage ‣ Volumes ‣ View Disks, what do you see? You can throw a screenshot in, it might help.

I'm seeing the disks in the hardware monitor, but nowhere else within FreeNAS. I'll grab a screenshot of that screen in a few hours. I will pull up the BIOS to triple check, but I'm pretty sure I should be fine on the hardware. Power outages are pretty rare where I live, so I was shocked to see all of my stuff reset when I came home from work that day.

I do not remember SMART, but I'm pretty sure the last time I checked all was well.

Have scoped out the manual, yes. 8.1.9 Note #2 is what gave me the ideas for detaching the volume, then I ran into several forum posts where people had success with that.

 

[Ericloewe]

ZRAID2

RAID-Z2.

I DO have the geli.key backed up, as well as my configuration.

You need to be more specific. There are two keys - the first one requires a password along with it, the second one does not. Is anything logged by GELI when trying to unlock the disks?

 

[NAiLs]

RAID-Z2.


You need to be more specific. There are two keys - the first one requires a password along with it, the second one does not. Is anything logged by GELI when trying to unlock the disks?

Sorry...


For the keys, I was only provided one key when I downloaded them. I do not recall adding a passphrase, so I would not have a key for a password, correct?

Regardless, I can't even get to the point to add the keys to unlock the volume. This is where I'm stuck. For the output when I try to unlock, yes, I that is on my original post.


Here is what the console shows after I attempt to unlock the volume:
Oct 8 22:01:18 System uwsgi: [middleware.notifier:2524] Importing VolumeName [12275454876993067764] failed with: cannot import '12275454876993067764': no such pool available

 

[Chris Moore]

For the keys, I was only provided one key when I downloaded them. I do not recall adding a passphrase, so I would not have a key for a password, correct?

That is bad. There are two keys, the regular key and the recovery key.

 

[Chris Moore]

I'm seeing the disks in the hardware monitor

I am not sure what this hardware monitor is, but if FreeNAS isn't seeing a pool to import, it makes me wonder if it is seeing the disks that the pool is on.

I'll grab a screenshot of that screen in a few hours.

I am guessing that you are away from where the system is?

 

[NAiLs]

That is bad. There are two keys, the regular key and the recovery key.

I don't recall having another option to back-up keys. I have that geli.key backed up in multiple places and don't have any other files with it, aside from my configuration file. Perhaps I missed a step. Hmm.... Is geli.key the regular key or the recovery key?

In all honesty, encryption is going if I/we can figure out how to get my data back.

I am not sure what this hardware monitor is, but if FreeNAS isn't seeing a pool to import, it makes me wonder if it is seeing the disks that the pool is on.

I am guessing that you are away from where the system is?

This is under Reporting -> Disks. I see all of them listed there.

Yeah, I'm away at the moment, but should be home about 5pm Central. I typed up everything last night and waited for my account to be enabled.

 

[NAiLs]

Ok I was mistaken with the disks part... I was not seeing them, but now I do. I re-seated all the SATA cables on my drives and they're working fine now. My motherboard wasn't seeing them in the first place, so that got me to re-seat the drives.

So, now that I see the disks, I still cannot unlock my volume. I still get the same error of 'Volume Failed Unlocked.'

Oct 9 17:50:56 System uwsgi: [middleware.notifier:2524] Importing VolumeName [12275454876993067764] failed with: cannot import '12275454876993067764': no such pool

See attached of my disks.

*EDIT* FWIW - I just ran a short SMART test on all drives and they all completed without error.

 

[Chris Moore]

Ok I was mistaken with the disks part... I was not seeing them, but now I do. I re-seated all the SATA cables on my drives and they're working fine now. My motherboard wasn't seeing them in the first place, so that got me to re-seat the drives.

So, now that I see the disks, I still cannot unlock my volume. I still get the same error of 'Volume Failed Unlocked.'

Oct 9 17:50:56 System uwsgi: [middleware.notifier:2524] Importing VolumeName [12275454876993067764] failed with: cannot import '12275454876993067764': no such pool

See attached of my disks.

View attachment 26031

View attachment 26032

*EDIT* FWIW - I just ran a short SMART test on all drives and they all completed without error.

Did you try to export / import again, now that the disks are being seen?

 

[NAiLs]

Did you try to export / import again, now that the disks are being seen?

Well I'll be.... yep, that did it! I was hesitant to do it again.

So in the end, my stupid motherboard not seeing ANY drive was the root cause. Detaching the volume and importing it ended up resolving my issue.

Sooo.... encryption, I assume I'm stuck with it unless I redo the server? I should probably create a recovery key, too, eh?

One last thing not related that I'm super curious about. I moved over a ton of data from one of my other servers (2 drives that are mirrored, but both are starting to go), which is just a bunch of random sized files. The size on disk is huge. Size is 53.5GB and size on disk is 67.2GB. Is there a way to say what the block size it for FreeNAS?

Thank you again to everyone!!

 

[m0nkey_]

Sooo.... encryption, I assume I'm stuck with it unless I redo the server?

Correct. For a home user, Encryption shouldn't be used, because as you saw it's a real pain in the ass when things start to go wrong.

I should probably create a recovery key, too, eh?

If you don't plan to rebuilding without encryption, absolutely.

 

[Ericloewe]

Is there a way to say what the block size it for FreeNAS?

You can check what's set for the dataset in question. It's only a maximum, so it's a good idea to set it to 1M if it's extremely sequential stuff that doesn't get edited somewhere in the middle, like videos.

 

[danb35]

Sooo.... encryption, I assume I'm stuck with it unless I redo the server?

It's possible to remove encryption on the pool from the command line, but messy. In short: offline a disk, destroy the GELI object, repartition the disk, then replace the offline (encrypted) disk with the (non-encrypted) disk. Then repeat for all remaining disks.

 

[pro lamer]

then replace the offline (encrypted) disk with the (non-encrypted) disk. Then repeat for all remaining disks.

I guess it needs to resilver before moving on to next disk?

Sent from my mobile phone

 

[danb35]

I guess it needs to resilver before moving on to next disk?

Correct.