Accidentally marked ZFS drives as "Unconfigured Good"

[Ibes]

I was using a new RAID card and trying to get the JBOD drives to show up in FreeNAS, but accidentally marked them as unconfigured good.

Now that I have reflashed the RAID card with IT mode, the drives show up in the O/S; however, they seem to be missing something that FreeNAS expects, because they don't even appear in the "Disks" list when I try to import them.

They still show up as partitioned with FreeBSD ZFS, so I'm not 100% sure what changed that's causing FreeNAS to not recognize/accept them as valid disks for a zpool.

 

[Ibes]

Code:

+ ls /dev/gptid/
2a0a_rest_of_uuid_obscured
2a28_rest_of_uuid_obscured
3a8a_rest_of_uuid_obscured
578c_rest_of_uuid_obscured
9588_rest_of_uuid_obscured
9588_rest_of_uuid_obscured.eli
c55f_rest_of_uuid_obscured
c9df_rest_of_uuid_obscured
caed_rest_of_uuid_obscured
caed_rest_of_uuid_obscured.eli
f4ff_rest_of_uuid_obscured
+ cat geli.attach.sh
#!/usr/bin/env bash

for i in /dev/gptid/*; do
  geli attach -p -k /data/geli/uuid_obscured.key $i
done
+ ./geli.attach.sh
geli: Cannot read metadata from /dev/gptid/2a0a_rest_of_uuid_obscured: Invalid argument.
geli: Cannot read metadata from /dev/gptid/2a28_rest_of_uuid_obscured: Invalid argument.
geli: Cannot read metadata from /dev/gptid/3a8a_rest_of_uuid_obscured: Invalid argument.
geli: Cannot read metadata from /dev/gptid/578c_rest_of_uuid_obscured: Invalid argument.
geli: Cannot access gptid/9588_rest_of_uuid_obscured (error=1).
geli: Cannot read metadata from /dev/gptid/9588_rest_of_uuid_obscured.eli: Invalid argument.
geli: Cannot read metadata from /dev/gptid/c55f_rest_of_uuid_obscured: Invalid argument.
geli: Cannot read metadata from /dev/gptid/c9df_rest_of_uuid_obscured: Invalid argument.
geli: Cannot access gptid/caed_rest_of_uuid_obscured (error=1).
geli: Cannot read metadata from /dev/gptid/caed_rest_of_uuid_obscured.eli: Invalid argument.
geli: Cannot read metadata from /dev/gptid/f4ff_rest_of_uuid_obscured: Invalid argument.

Contents of geli.attach.sh

Code:

#!/usr/bin/env bash

for i in /dev/gptid/*; do
  geli attach -p -k /data/geli/uuid_obscured.key $i
done

 

[Ibes]

Any other tools anyone can recommend tinkering with to try repairing the drives?

My guess is that something minor was changed like an identifier or address offset on the drives, but it doesn't seem like anything permanent has happened to make the data inaccessible.

 

[Ibes]

`gpart list vtbd#` on all of the `bad` drives is showing a Mode for the big partition of: `Mode: r0w0e0` whereas the `good` drives are showing `Mode: r1w1e2`.

Additionally, the `Consumers` section shows `Mode: r0w0e0` on the `bad` drives and `Mode: r2w2e5` for the `good` drives.

Anyone know how to change these?

 

[Allan Jude]

Any other tools anyone can recommend tinkering with to try repairing the drives?

My guess is that something minor was changed like an identifier or address offset on the drives, but it doesn't seem like anything permanent has happened to make the data inaccessible.

Tools:
first, get the list of devices: `camcontrol devlist`

Then, check the partition tables: `gpart show`

Likely, the disk is actually slightly larger now that it is not part of the RAID, since the RAID controller usually keeps a bit of metadata at the end of the disk.

Which disks appear to be working, and which do not?

On a disk that is not working: `zdb -l /dev/adaX` and each partition: `zdb -l /dev/adaXpY`

This prints the ZFS label information, which is stored at the stand and end of each disk. It's presence and/or offset might help us determine how we need to modify the partition table to match where your old data is.

 

[Ibes]

`gpart show`

Code:

=>		34  7814037101  vtbd1  GPT  (3.6T)
		  34		  94		 - free -  (47K)
		 128	 4194304	  1  freebsd-swap  (2.0G)
	 4194432  7809842696	  2  freebsd-zfs  (3.6T)
  7814037128		   7		 - free -  (3.5K)

=>		34  7814037101  vtbd2  GPT  (3.6T)
		  34		  94		 - free -  (47K)
		 128	 4194304	  1  freebsd-swap  (2.0G)
	 4194432  7809842696	  2  freebsd-zfs  (3.6T)
  7814037128		   7		 - free -  (3.5K)

=>		34  7814037101  vtbd3  GPT  (3.6T)
		  34		  94		 - free -  (47K)
		 128	 4194304	  1  freebsd-swap  (2.0G)
	 4194432  7809842696	  2  freebsd-zfs  (3.6T)
  7814037128		   7		 - free -  (3.5K)

=>		34  7814037101  vtbd4  GPT  (3.6T)
		  34		  94		 - free -  (47K)
		 128	 4194304	  1  freebsd-swap  (2.0G)
	 4194432  7809842696	  2  freebsd-zfs  (3.6T)
  7814037128		   7		 - free -  (3.5K)

=>		34  7814037101  vtbd5  GPT  (3.6T)
		  34		  94		 - free -  (47K)
		 128	 4194304	  1  freebsd-swap  (2.0G)
	 4194432  7809842696	  2  freebsd-zfs  (3.6T)
  7814037128		   7		 - free -  (3.5K)

=>		34  7814037101  vtbd6  GPT  (3.6T)
		  34		  94		 - free -  (47K)
		 128	 4194304	  1  freebsd-swap  (2.0G)
	 4194432  7809842696	  2  freebsd-zfs  (3.6T)
  7814037128		   7		 - free -  (3.5K)

=>		40  7814037088  vtbd7  GPT  (3.6T)
		  40		  88		 - free -  (44K)
		 128	 4194304	  1  freebsd-swap  (2.0G)
	 4194432  7809842688	  2  freebsd-zfs  (3.6T)
  7814037120		   8		 - free -  (4.0K)

All of /dev/vtbd# and /dev/vtbd#p# return when I run `zdb -l ${DEV}`:

Code:

--------------------------------------------
LABEL 0
--------------------------------------------
failed to unpack label 0
--------------------------------------------
LABEL 1
--------------------------------------------
failed to unpack label 1
--------------------------------------------
LABEL 2
--------------------------------------------
failed to unpack label 2
--------------------------------------------
LABEL 3
--------------------------------------------

5 and 7 are the working ones, 1-4 and 6 are the non-working ones.

 

[Ibes]

What I find interesting about that ^ is that there isn't much consistent difference between those partition tables. Is there anything else that could have been impacted by having marked those drives Unconfigured Good?

 

[Ibes]

Found this thread:
https://www.reddit.com/r/freebsd/comments/3nujw3/zfs_corruption_after_upgrade_from_101_to_102/

Which seemed to suggest checking this kern.geom.conftxt could help:

da0 is the boot device, so we can ignore it
da1 is a spare drive that is not formatted, so we can ignore it.
da2-da8 are the relevant ones (I switched from virtio to scsi hoping for some change in behavior to no avail)

I can't really tell what I'm looking at yet, but I'm going to keep studying up on this output to try to figure it out.

Code:

# sysctl kern.geom.conftxt
kern.geom.conftxt: 0 DISK cd0 636065792 2048 hd 0 sc 0
1 LABEL iso9660/FreeNAS 636065792 2048 i 0 o 0
0 DISK da8 4000787030016 512 hd 255 sc 63
1 PART da8p2 3998639456256 512 i 2 o 2147549184 ty freebsd-zfs xs GPT xt 516e7cba-6ecf-11d6-8ff8-00022d09712b
2 LABEL gptid/958880b5-f26f-11e7-9fcd-001fc631d2ca 3998639456256 512 i 0 o 0
1 PART da8p1 2147483648 512 i 1 o 65536 ty freebsd-swap xs GPT xt 516e7cb5-6ecf-11d6-8ff8-00022d09712b
2 MIRROR mirror/swap0 2147483648 512
3 ELI mirror/swap0.eli 2147483648 512
0 DISK da7 4000787030016 512 hd 255 sc 63
1 PART da7p2 3998639460352 512 i 2 o 2147549184 ty freebsd-zfs xs GPT xt 516e7cba-6ecf-11d6-8ff8-00022d09712b
2 LABEL gptid/c9dfc9a4-ca96-11e4-aecf-60a44cab7de5 3998639460352 512 i 0 o 0
1 PART da7p1 2147483648 512 i 1 o 65536 ty freebsd-swap xs GPT xt 516e7cb5-6ecf-11d6-8ff8-00022d09712b
2 MIRROR mirror/swap0 2147483648 512
3 ELI mirror/swap0.eli 2147483648 512
0 DISK da6 4000787030016 512 hd 255 sc 63
1 PART da6p2 3998639460352 512 i 2 o 2147549184 ty freebsd-zfs xs GPT xt 516e7cba-6ecf-11d6-8ff8-00022d09712b
2 LABEL gptid/caed716a-c98f-11e4-8004-60a44cab7de5 3998639460352 512 i 0 o 0
1 PART da6p1 2147483648 512 i 1 o 65536 ty freebsd-swap xs GPT xt 516e7cb5-6ecf-11d6-8ff8-00022d09712b
2 MIRROR mirror/swap1 2147483648 512
3 ELI mirror/swap1.eli 2147483648 512
0 DISK da5 4000787030016 512 hd 255 sc 63
1 PART da5p2 3998639460352 512 i 2 o 2147549184 ty freebsd-zfs xs GPT xt 516e7cba-6ecf-11d6-8ff8-00022d09712b
2 LABEL gptid/f4ffa429-c9fe-11e4-8004-60a44cab7de5 3998639460352 512 i 0 o 0
1 PART da5p1 2147483648 512 i 1 o 65536 ty freebsd-swap xs GPT xt 516e7cb5-6ecf-11d6-8ff8-00022d09712b
2 MIRROR mirror/swap1 2147483648 512
3 ELI mirror/swap1.eli 2147483648 512
0 DISK da4 4000787030016 512 hd 255 sc 63
1 PART da4p2 3998639460352 512 i 2 o 2147549184 ty freebsd-zfs xs GPT xt 516e7cba-6ecf-11d6-8ff8-00022d09712b
2 LABEL gptid/3a8a731c-c919-11e4-8004-60a44cab7de5 3998639460352 512 i 0 o 0
1 PART da4p1 2147483648 512 i 1 o 65536 ty freebsd-swap xs GPT xt 516e7cb5-6ecf-11d6-8ff8-00022d09712b
2 MIRROR mirror/swap2 2147483648 512
3 ELI mirror/swap2.eli 2147483648 512
0 DISK da3 4000787030016 512 hd 255 sc 63
1 PART da3p2 3998639460352 512 i 2 o 2147549184 ty freebsd-zfs xs GPT xt 516e7cba-6ecf-11d6-8ff8-00022d09712b
2 LABEL gptid/c55f1938-cb2a-11e4-aecf-60a44cab7de5 3998639460352 512 i 0 o 0
1 PART da3p1 2147483648 512 i 1 o 65536 ty freebsd-swap xs GPT xt 516e7cb5-6ecf-11d6-8ff8-00022d09712b
2 MIRROR mirror/swap2 2147483648 512
3 ELI mirror/swap2.eli 2147483648 512
0 DISK da2 4000787030016 512 hd 255 sc 63
1 PART da2p2 3998639460352 512 i 2 o 2147549184 ty freebsd-zfs xs GPT xt 516e7cba-6ecf-11d6-8ff8-00022d09712b
2 LABEL gptid/2a288391-ccbf-11e4-b67e-60a44cab7de5 3998639460352 512 i 0 o 0
1 PART da2p1 2147483648 512 i 1 o 65536 ty freebsd-swap xs GPT xt 516e7cb5-6ecf-11d6-8ff8-00022d09712b
2 LABEL gptid/2a0ab97e-ccbf-11e4-b67e-60a44cab7de5 2147483648 512 i 0 o 0
0 DISK da1 4000787030016 512 hd 255 sc 63
0 DISK da0 34359738368 512 hd 255 sc 63
1 PART da0p2 34359169024 512 i 2 o 544768 ty freebsd-zfs xs GPT xt 516e7cba-6ecf-11d6-8ff8-00022d09712b
1 PART da0p1 524288 512 i 1 o 20480 ty bios-boot xs GPT xt 21686148-6449-6e6f-744e-656564454649
2 LABEL gptid/578c1200-b3c4-11e8-a177-cdfb72a4bc64 524288 512 i 0 o 0

 

[Ibes]

Real fast way to see if your metadata is gone forever:

Code:

# This gets you your media size (mine was 3998639460352)
gpart list da2 | grep Mediasize

# The number of bytes per block
BLOCK_SIZE=512
# The number of blocks on the media
BLOCK_COUNT=$(( 3998639460352 / ${BLOCK_SIZE} ))
# The number of blocks to skip (count - 1)
let SKIP=$(( ${BLOCK_COUNT} - 1))

# dump the entire last block (256B) then pipe it into grep to see if "GEOM::ELI" is present
dd if=/dev/da2p2 bs=${BLOCK_SIZE} skip=${SKIP} 2>&1 | grep GEOM::ELI

Example:

Code:

freenas# dd if=/dev/da2p2 bs=${BLOCK_SIZE} skip=${SKIP} 2>&1 | grep GEOM::ELI
freenas# dd if=/dev/da4p2 bs=${BLOCK_SIZE} skip=${SKIP} 2>&1 | grep GEOM::ELI
Binary file (standard input) matches

This shows da2p2 does not contain the expected metadata at the offset, but da4p2 does.

Hope this helps someone at least save some time, even though it's bad news if you have the same bad luck I did.