Import encrypted volume after detach

[El Under]

Hi there,

After trying to find information about this for several days I am ready to admit that I can not do this by myself.
I am sure this is an easy cherry for some of you, so I am handing over the fate of my my data into more capable hands...

The situation:
I had 3 geli-encrypted drives that were initialized in FreeNAS 9.x using ZFS and were running just fine in single configuration, no raid.
My plan was to add a 4th, which I did. After the next boot, one of the 3 disks did not decrypt anymore.
Some research indicated that this issue might be fixed with detaching and reimporting the drive.
I detached it just to find out that it now does not appear in the import volume dialog. It is listed in the [Storage>Volume>View Disks] though.
To check if my import skills are just not good enough, I created the 4th encrypted drive and detached it. Same problem. Gives me hope that the drive is not forfeit though.
I guess my importing skills suck.

ada0 and ada1 are running as intended
ada2 is the drive that had the decryption trouble and ada3 is the new drive (both were detached)

Here is the gpart list output

Code:

Geom name: ada0
modified: false
state: OK
fwheads: 16
fwsectors: 63
last: 5860533134
first: 34
entries: 128
scheme: GPT
Providers:
1. Name: ada0p1
  Mediasize: 2147483648 (2.0G)
  Sectorsize: 512
  Stripesize: 4096
  Stripeoffset: 0
  Mode: r1w1e1
  rawuuid: 0cd54fbc-3cd7-11e3-a009-94de80048cac
  rawtype: 516e7cb5-6ecf-11d6-8ff8-00022d09712b
  label: (null)
  length: 2147483648
  offset: 65536
  type: freebsd-swap
  index: 1
  end: 4194431
  start: 128
2. Name: ada0p2
  Mediasize: 2998445412352 (2.7T)
  Sectorsize: 512
  Stripesize: 4096
  Stripeoffset: 0
  Mode: r0w0e0
  rawuuid: 0ce7df3d-3cd7-11e3-a009-94de80048cac
  rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b
  label: (null)
  length: 2998445412352
  offset: 2147549184
  type: freebsd-zfs
  index: 2
  end: 5860533127
  start: 4194432
Consumers:
1. Name: ada0
  Mediasize: 3000592982016 (2.7T)
  Sectorsize: 512
  Stripesize: 4096
  Stripeoffset: 0
  Mode: r1w1e2

Geom name: ada1
modified: false
state: OK
fwheads: 16
fwsectors: 63
last: 5860533134
first: 34
entries: 128
scheme: GPT
Providers:
1. Name: ada1p1
  Mediasize: 2147483648 (2.0G)
  Sectorsize: 512
  Stripesize: 4096
  Stripeoffset: 0
  Mode: r1w1e1
  rawuuid: 8af59959-4007-11e3-82ae-94de80048cac
  rawtype: 516e7cb5-6ecf-11d6-8ff8-00022d09712b
  label: (null)
  length: 2147483648
  offset: 65536
  type: freebsd-swap
  index: 1
  end: 4194431
  start: 128
2. Name: ada1p2
  Mediasize: 2998445412352 (2.7T)
  Sectorsize: 512
  Stripesize: 4096
  Stripeoffset: 0
  Mode: r0w0e0
  rawuuid: 8b0b0b1d-4007-11e3-82ae-94de80048cac
  rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b
  label: (null)
  length: 2998445412352
  offset: 2147549184
  type: freebsd-zfs
  index: 2
  end: 5860533127
  start: 4194432
Consumers:
1. Name: ada1
  Mediasize: 3000592982016 (2.7T)
  Sectorsize: 512
  Stripesize: 4096
  Stripeoffset: 0
  Mode: r1w1e2

Geom name: ada3
modified: false
state: OK
fwheads: 16
fwsectors: 63
last: 5860533134
first: 34
entries: 128
scheme: GPT
Providers:
1. Name: ada3p1
  Mediasize: 2147483648 (2.0G)
  Sectorsize: 512
  Stripesize: 4096
  Stripeoffset: 0
  Mode: r0w0e0
  rawuuid: 5305cd6e-3b24-11e7-a332-94de80048cac
  rawtype: 516e7cb5-6ecf-11d6-8ff8-00022d09712b
  label: (null)
  length: 2147483648
  offset: 65536
  type: freebsd-swap
  index: 1
  end: 4194431
  start: 128
2. Name: ada3p2
  Mediasize: 2998445412352 (2.7T)
  Sectorsize: 512
  Stripesize: 4096
  Stripeoffset: 0
  Mode: r0w0e0
  rawuuid: 531820c0-3b24-11e7-a332-94de80048cac
  rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b
  label: (null)
  length: 2998445412352
  offset: 2147549184
  type: freebsd-zfs
  index: 2
  end: 5860533127
  start: 4194432
Consumers:
1. Name: ada3
  Mediasize: 3000592982016 (2.7T)
  Sectorsize: 512
  Stripesize: 4096
  Stripeoffset: 0
  Mode: r0w0e0

Geom name: da0
modified: false
state: OK
fwheads: 255
fwsectors: 63
last: 30867422
first: 34
entries: 128
scheme: GPT
Providers:
1. Name: da0p1
  Mediasize: 524288 (512K)
  Sectorsize: 512
  Stripesize: 0
  Stripeoffset: 17408
  Mode: r0w0e0
  rawuuid: 5ffe34c7-fa36-11e4-889b-6cf049054279
  rawtype: 21686148-6449-6e6f-744e-656564454649
  label: (null)
  length: 524288
  offset: 17408
  type: bios-boot
  index: 1
  end: 1057
  start: 34
2. Name: da0p2
  Mediasize: 15803572224 (15G)
  Sectorsize: 512
  Stripesize: 0
  Stripeoffset: 544768
  Mode: r1w1e2
  rawuuid: 60059b3d-fa36-11e4-889b-6cf049054279
  rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b
  label: (null)
  length: 15803572224
  offset: 544768
  type: freebsd-zfs
  index: 2
  end: 30867415
  start: 1064
Consumers:
1. Name: da0
  Mediasize: 15804137472 (15G)
  Sectorsize: 512
  Mode: r1w1e3

Could anyone run me through the steps that I have to do to import both drives using the shell? I am handing out karma points!
Thanks in advance and let me know which further output is needed!

Regards
elunder

 

[styr]

Hello. Would you be able to provide some more information? I am curious about the problem, although I may not be the one with any solution.

I had 3 geli-encrypted drives that were initialized in freenas 9.x using ZFS and were running just fine in single configuration, no raid.
My plan was to add a 4th, which I did. After the next boot, one of the 3 disks did not decrypt anymore.

You mean stripe across one drive each, independently?
Are you using the same key for all drives, or different keys for each drive?
Could you share the actual error message from when the decryption fails?

Some research indicated that this issue might be fixed with detatching and reimporting the drive.

ada2 is the drive that had the decryption trouble and ada3 is the new drive (both were detatched)

When you mention detach and reimport, do you mean GELI detach, and ZFS import, or do you mean something else?

Here's hoping you get a fix! I just think it may need more information to get there. Best of luck!

 

[El Under]

First of all, thanks for the response!
My biggest problem is that I am not familiar with UNIX based systems albeit being an engineer that works in the software industry... I might stumble over some terms here and there.

You mean stripe across one drive each, independently?
Are you using the same key for all drives, or different keys for each drive?
Could you share the actual error message from when the decryption fails?

All drives are individual zpools and are encrypted with their own key. I have the key files, the passphrase and the backup keys safely tucked away.
I could not afford the additional disks for a raid when I set up the system, which is why I used "just a bunch of disks".
I can not reproduce the error now, but the message was "unable to decrypt". Not much to go on, I know!

To clarify, here is the code that my init script is running to make the drives available:

Code:

geli attach -j /mnt/remote/pass.txt -k /mnt/remote/disk1.key /dev/ada0p2
geli attach -j /mnt/remote/pass.txt -k /mnt/remote/disk2.key /dev/ada3p2
geli attach -j /mnt/remote/pass.txt -k /mnt/remote/disk3.key /dev/ada2p2
zpool import -R /mnt Disk1
zpool import -R /mnt Disk2
zpool import -R /mnt Disk3

When you mention detach and reimport, do you mean GELI detach, and ZFS import, or do you mean something else?

I used [Storage>Volumes>Detach Volume] in the frontend. Whatever that is. And as I mentioned earlier, I did this with a brand new working disk as well, and have the same issue.
I might just not know how to properly attach my volumes after this detach.

If anyone could list the steps needed to do this using the shell, I could report the error messages that occurred along the way. I guess there must be something wrong, otherwise the frontend would let me import the volumes.

Thanks again,
elunder

 

[styr]

OK, good info. What is the output of geli status when the error happens?

A more detailed error message might be nice, but maybe not possible depending on what's wrong--I'm not sure why decryption would just fail unless the data or key were somehow corrupted, which could happen with a single-disk stripe zpool. That would be rather frustrating, although I am not sure if there is a one-size-fits-all command-line solution, short of some monster script that tests for every possible case (possible feature request?). Hopefully more light can shine on this.

 

[IceBoosteR]

Hi,
after a reboot it is possible that the drive letters changes. This means, ada0 is changed after a reboot to ada3, and ada1 is maybe ada0.
So you are trying to decrypt a volume with the wrong key, and you'll get the "unable to decrypt" message.
That would explain your problem, and in my eyes a possible answer to your problem.

You could also have a look into /var/log/messages for some more details of the error.

IceBoosteR

 

[El Under]

after a reboot it is possible that the drive letters changes

That could be it.
In the [Storage>View Disks] pane, I can see all the disks that I have attached. And the list changes if I remove the SATA cables and reboot so it seems to be updated.
In the terminal however, "gpart list" for example does not list partitions of all drives.
After detatching the drive using the UI, what do I have to do to access it again in the terminal?
In order to try to decrypt a drive I must know its handle...
What I am trying hard (and failing) to say is:
I know a drive that is listed as ada0 in [Storage>View Disks] and in "gpart list" and I can decrypt and mount it using:

Code:

geli attach -j /mnt/remote/pass1.txt -k /mnt/remote/disk1.key /dev/ada0p2
zpool import -R /mnt Disk1

The drive that disappeared after I added the new drive (and possibly swapped some DATA cables) is also listed in [Storage>View Disks] but not in "gpart list".
If i try the same code with the associated password, key and disk handle (which is visible in [Storage>View Disks]) I get the error:
geli: Cannot open /dev/ada2p2: No such file or directory.
Is there a step I have to undertake to reverse the detatch (ie. make the disk known to the system)?

Thanks again,
elunder

 

[IceBoosteR]

Well, thats really strange.

Could you navigate to /dev and do an

Code:

ls -lisa

I am interested in the listed devices, maybe this could bring us a step further.
But maybe other guys has better ideas ;)

IceBoosteR

 

[El Under]

First of all, thank you IceBoosteR for your continuous help!
Here is the output of ls -lisa in /dev

Code:

total 5
  2 1 dr-xr-xr-x  9 root  wheel  512 Jun 24 11:40 .
  4 2 drwxr-xr-x  19 root  wheel  26 Jun 24 11:41 ..
 40 0 crw-r--r--  1 root  wheel  0x28 Jun 24 11:40 acpi
104 0 crw-r-----  1 root  operator  0x68 Jun 24 11:40 ada0
105 0 crw-r-----  1 root  operator  0x69 Jun 24 11:40 ada0p1
106 0 crw-r-----  1 root  operator  0x6a Jun 24 11:40 ada0p2
107 0 crw-r-----  1 root  operator  0x6b Jun 24 11:40 ada1
113 0 crw-r-----  1 root  operator  0x71 Jun 24 11:40 ada1p1
114 0 crw-r-----  1 root  operator  0x72 Jun 24 11:40 ada1p2
108 0 crw-r-----  1 root  operator  0x6c Jun 24 11:40 ada2
109 0 crw-r-----  1 root  operator  0x6d Jun 24 11:40 ada3
115 0 crw-r-----  1 root  operator  0x73 Jun 24 11:40 ada3p1
116 0 crw-r-----  1 root  operator  0x74 Jun 24 11:40 ada3p2
 42 0 crw-rw-r--  1 root  operator  0x2a Jun 24 11:40 apm
 41 0 crw-rw----  1 root  operator  0x29 Jun 24 11:40 apmctl
 44 0 crw-------  1 root  wheel  0x2c Jun 24 11:40 atkbd0
 16 0 crw-------  1 root  kmem  0x10 Jun 24 11:40 audit
 15 0 crw-------  1 root  wheel  0xf Jun 24 11:40 auditpipe
 22 0 crw-------  1 root  wheel  0x16 Jun 24 11:41 bpf
 23 0 lrwxr-xr-x  1 root  wheel  3 Jun 24 11:40 bpf0 -> bpf
101 1 dr-xr-xr-x  2 root  wheel  512 Jun 24 11:40 cam
  9 0 crw-------  1 root  wheel  0x9 Jun 24 11:42 console
 35 0 crw-------  1 root  wheel  0x23 Jun 24 11:40 consolectl
 25 0 crw-r-----  1 root  kmem  0x19 Jun 24 11:40 cpuctl0
 26 0 crw-r-----  1 root  kmem  0x1a Jun 24 11:40 cpuctl1
 51 0 crw-rw-rw-  1 root  wheel  0x33 Jun 24 11:40 crypto
 17 0 crw-rw-rw-  1 root  wheel  0x11 Jun 24 11:40 ctty
110 0 crw-r-----  1 root  operator  0x6e Jun 24 11:40 da0
117 0 crw-r-----  1 root  operator  0x75 Jun 24 11:40 da0p1
118 0 crw-r-----  1 root  operator  0x76 Jun 24 11:40 da0p2
  4 0 crw-------  1 root  wheel  0x4 Jun 24 11:40 devctl
  5 0 crw-------  1 root  wheel  0x5 Jun 24 11:40 devctl2
 80 0 cr--r--r--  1 root  wheel  0x50 Jun 24 11:40 devstat
133 1 dr-xr-xr-x  2 root  wheel  512 Jun 24 11:41 dtrace
126 0 lrwxr-xr-x  1 root  wheel  11 Jun 24 11:41 dumpdev -> /dev/ada1p1
  1 0 dr-xr-xr-x  2 root  wheel  512 Jun 24 11:40 fd
 38 0 crw-------  1 root  wheel  0x26 Jun 24 11:40 fido
 18 0 crw-rw-rw-  1 root  wheel  0x12 Jun 24 11:40 full
  6 0 crw-r-----  1 root  operator  0x6 Jun 24 11:40 geom.ctl
103 0 crw-------  1 root  wheel  0x67 Jun 24 11:41 ggctl
125 1 dr-xr-xr-x  2 root  wheel  512 Jun 24 11:40 gptid
 43 0 crw-r--r--  1 root  wheel  0x2b Jun 24 11:40 hpet0
 27 0 crw-------  1 root  wheel  0x1b Jun 24 11:40 io
 39 0 crw-------  1 root  wheel  0x27 Jun 24 11:40 iscsi
 45 0 lrwxr-xr-x  1 root  wheel  6 Jun 24 11:40 kbd0 -> atkbd0
 12 0 lrwxr-xr-x  1 root  wheel  7 Jun 24 11:40 kbd1 -> kbdmux0
 11 0 crw-------  1 root  wheel  0xb Jun 24 11:40 kbdmux0
 10 0 crw-------  1 root  wheel  0xa Jun 24 11:40 klog
 14 0 crw-r-----  1 root  kmem  0xe Jun 24 11:40 kmem
 54 0 crw-------  1 root  wheel  0x36 Jun 24 11:40 mdctl
 13 0 crw-r-----  1 root  kmem  0xd Jun 24 11:40 mem
 36 0 crw-------  1 root  kmem  0x24 Jun 24 11:40 nfslock
 19 0 crw-rw-rw-  1 root  wheel  0x13 Jun 24 11:44 null
 81 0 crw-------  1 root  operator  0x51 Jun 24 11:40 pass0
 82 0 crw-------  1 root  operator  0x52 Jun 24 11:40 pass1
 83 0 crw-------  1 root  operator  0x53 Jun 24 11:40 pass2
 84 0 crw-------  1 root  operator  0x54 Jun 24 11:40 pass3
 85 0 crw-------  1 root  operator  0x55 Jun 24 11:40 pass4
 21 0 crw-r--r--  1 root  wheel  0x15 Jun 24 11:40 pci
 24 0 crw-rw-rw-  1 root  wheel  0x18 Jun 24 11:40 ptmx
135 1 dr-xr-xr-x  2 root  wheel  512 Jun 24 11:44 pts
  7 0 crw-r--r--  1 root  wheel  0x7 Jun 24 11:41 random
 98 0 crwx------  1 root  wheel  0x62 Jun 24 11:40 rdma_cm
 99 1 dr-xr-xr-x  2 root  wheel  512 Jun 24 11:40 reroot
 34 0 crw-------  1 root  wheel  0x22 Jun 24 11:40 snp
 33 0 lrwxr-xr-x  1 root  wheel  4 Jun 24 11:40 stderr -> fd/2
 29 0 lrwxr-xr-x  1 root  wheel  4 Jun 24 11:40 stdin -> fd/0
 31 0 lrwxr-xr-x  1 root  wheel  4 Jun 24 11:40 stdout -> fd/1
 37 0 crw-------  1 root  wheel  0x25 Jun 24 11:40 sysmouse
 86 0 crw-------  1 root  wheel  0x56 Jun 24 11:42 ttyv0
 87 0 crw-------  1 root  wheel  0x57 Jun 24 11:42 ttyv1
 88 0 crw-------  1 root  wheel  0x58 Jun 24 11:42 ttyv2
 89 0 crw-------  1 root  wheel  0x59 Jun 24 11:42 ttyv3
 90 0 crw-------  1 root  wheel  0x5a Jun 24 11:42 ttyv4
 91 0 crw-------  1 root  wheel  0x5b Jun 24 11:42 ttyv5
 92 0 crw-------  1 root  wheel  0x5c Jun 24 11:42 ttyv6
 93 0 crw-------  1 root  wheel  0x5d Jun 24 11:42 ttyv7
 94 0 crw-------  1 root  wheel  0x5e Jun 24 11:40 ttyv8
 95 0 crw-------  1 root  wheel  0x5f Jun 24 11:40 ttyv9
 96 0 crw-------  1 root  wheel  0x60 Jun 24 11:40 ttyva
 97 0 crw-------  1 root  wheel  0x61 Jun 24 11:40 ttyvb
 47 0 crw-------  1 root  wheel  0x2f Jun 24 11:40 ufssuspend
 56 0 lrwxr-xr-x  1 root  wheel  9 Jun 24 11:40 ugen0.1 -> usb/0.1.0
 58 0 lrwxr-xr-x  1 root  wheel  9 Jun 24 11:40 ugen1.1 -> usb/1.1.0
 60 0 lrwxr-xr-x  1 root  wheel  9 Jun 24 11:40 ugen2.1 -> usb/2.1.0
 62 0 lrwxr-xr-x  1 root  wheel  9 Jun 24 11:40 ugen3.1 -> usb/3.1.0
 77 0 lrwxr-xr-x  1 root  wheel  9 Jun 24 11:40 ugen3.2 -> usb/3.2.0
 64 0 lrwxr-xr-x  1 root  wheel  9 Jun 24 11:40 ugen4.1 -> usb/4.1.0
 66 0 lrwxr-xr-x  1 root  wheel  9 Jun 24 11:40 ugen5.1 -> usb/5.1.0
 68 0 lrwxr-xr-x  1 root  wheel  9 Jun 24 11:40 ugen6.1 -> usb/6.1.0
  8 0 lrwxr-xr-x  1 root  wheel  6 Jun 24 11:40 urandom -> random
102 1 dr-xr-xr-x  2 root  wheel  512 Jun 24 11:40 usb
 52 0 crw-rw----  1 root  uucp  0x34 Jun 24 11:40 usbctl
 53 0 crw-------  1 root  operator  0x35 Jun 24 11:40 xpt0
 20 0 crw-rw-rw-  1 root  wheel  0x14 Jun 24 11:40 zero
 48 0 crw-rw-rw-  1 root  operator  0x30 Jun 24 11:40 zfs

 

[El Under]

ada0p2 and ada1p2 can be attached using geli attach.
If I try to attach ada3 I get the following error:

Code:

geli attach -j /root/password.txt -k /root/keyfile.key /dev/ada3p2
geli: Cannot read metadata from /dev/ada3p2: Invalid argument.

I am wondering, why are there no partitions on ada2? Did this happen due to the detatch?
Do I have to announce the partitions somehow so I can use geli attach on them?

Thanks for the help!

 

[SweetAndLow]

ada0p2 and ada1p2 can be attached using geli attach.
If I try to attach ada3 I get the following error:

Code:

geli attach -j /root/password.txt -k /root/keyfile.key /dev/ada3p2
geli: Cannot read metadata from /dev/ada3p2: Invalid argument.

I am wondering, why are there no partitions on ada2? Did this happen due to the detatch?
Do I have to announce the partitions somehow so I can use geli attach on them?

Thanks for the help!

gpart show I think is the command that will show you the partitions. Did you ever modify this pool using the zpool commands? If you don't do it though the GUI you can create a disk without any partitions.

Sent from my Nexus 5X using Tapatalk

 

[El Under]

Nope. I did everything through the UI and the last thing I did was Detach.

 

[IceBoosteR]

For me it looks like the MBR/GPT table from ada2 is broken or missing. Because ada2 is listed as a raw device.
For ada3 maybe the same applies, the MBR/GPT could be corrupted, as the system is not able to read the data.
But at this level, it is only dangerous half-knowledge....

Edit: What you could try: Don't import ada2p2, try ada2

 

[El Under]

Edit: What you could try: Dont import ada2p2, try ada2

Import? When I geli attach ada2 directly I get the same error message...

 

[IceBoosteR]

Import? When I geli attach ada2 directly I get the same error message...

I meant attach ;)

But mybe the MBR/GPT is really faulty...

 

[El Under]

On both disks, that are now behaving exaclty the same way? I doubt it.