Possible to reattach a disconnected drive to an empty pool?

[christian83]

Hi! I have disconnected a drive from a mirrored pool and removed the drive that I was not supposed to remove. I also formatted that drive by accident. So, my situation is that I can see an empty pool in the web interface, but I have no drives to attach.

Is there a way to reattach the drive that was disconnected?

Thank you

 

[Ericloewe]

If the pool is empty, it might be more practical to just wipe everythind and start over. Not that it should be unrecoverable, but it's typically faster to start from scratch.

 

[christian83]

If the pool is empty, it might be more practical to just wipe everythind and start over. Not that it should be unrecoverable, but it's typically faster to start from scratch.

Yeah, but i dont have a backup.. , else I would have..

 

[Ericloewe]

Well, then it's not really an empty pool, is it?
What's the output of zpool status and lsblk?

 

[christian83]

Well, then it's not really an empty pool, is it?
What's the output of zpool status and lsblk?

Yeah it is, so, I disconnected one of the drives, that I was supposed to move to another computer and setup a backup on that machine.
Hovewer I took the wrong drive and reformatted and also have put some files on it, so that drive I really cant do anything with.

But the other drive, that was disconnected, I cant readd because it doesent show that it has been part of the pool.

The pool is visible in the web-ui, but not when I do
`zpool status`
`lsblk` shows the drive that is disconnected.

 

[Ericloewe]

I'm not sure I'm following this, so let's try to clear this up:

1. You had a mirror pool with two disks
2. You detached one of the disks from the pool
3. You then physically disconnected the other disk, offlining the pool, and effectively erased it
4. The first disk is still in the machine

Is that what happened?

 

[christian83]

I'm not sure I'm following this, so let's try to clear this up:

1. You had a mirror pool with two disks
2. You detached one of the disks from the pool
3. You then physically disconnected the other disk, offlining the pool, and effectively erased it
4. The first disk is still in the machine

Is that what happened?

Yes, as far as I can remember this is what happend..

 

[Ericloewe]

In that case, try zpool import poolname.

 

[christian83]

In that case, try zpool import poolname.

`cannot import 'name': no such pool available`

So, I guess i must have done something to the drive..
That sucks, lesson learned I guess.. if there are no other way to restore the drive

 

[Ericloewe]

Hold on, we're not there yet, what does zpool import (with no arguments) say?

 

[christian83]

Hold on, we're not there yet, what does zpool import (with no arguments) say?

no pools available to import

 

[samarium]

maybe zpool thinks it is a deleted pool?

zpool import -D

zpool import -D -d /dev/device-in-question

General suggestion, have a list of drive serial number and postion in the case taped to the interior of the case, and cross check before during and after swap operations.

 

[christian83]

maybe zpool thinks it is a deleted pool?

zpool import -D

zpool import -D -d /dev/device-in-question

General suggestion, have a list of drive serial number and postion in the case taped to the interior of the case, and cross check before during and after swap operations.

zpool import -D - no pools available to import
zpool import -D -d /dev/sdd - cannot import '/dev/sdd': no such pool available

I must have done something I cant remember, I was in a process swapping drives etc to organize..
Yeah I know, I do have that.. And I checked the serial before removing, but I was a little distracted while doing so.. and somehow mixed the seirals.

 

[samarium]

zpool import -D - no pools available to import
zpool import -D -d /dev/sdd - cannot import '/dev/sdd': no such pool available

I must have done something I cant remember, I was in a process swapping drives etc to organize..
Yeah I know, I do have that.. And I checked the serial before removing, but I was a little distracted while doing so.. and somehow mixed the seirals.

I guess you will pay more attention swapping drives next time.

Maybe -d should be -d /dev/disk/by-id rather than the specific device, in which case it would not be useful, as the default should have found it.

I suggest you double check /dev/sdd is the correct device, as this can change across reboots, maybe ls -l /dev/disk/by-id/ | grep /sdd then try zdb -l /dev/sdd1 and see if zdb returns anything interesting like a pool structure. You could try this on any of your devices that isn't in a pool, and any that are in a pool should return information relating to the vdev to which they belong, and will show you what a normal zdb -l looks like. Maybe you will find your missing device. If you go thru each device on the system, and zdb can't find a label on the device that matches the pool in question, then I can't think of anything else to try.

 

[christian83]

I guess you will pay more attention swapping drives next time.

Maybe -d should be -d /dev/disk/by-id rather than the specific device, in which case it would not be useful, as the default should have found it.

I suggest you double check /dev/sdd is the correct device, as this can change across reboots, maybe ls -l /dev/disk/by-id/ | grep /sdd then try zdb -l /dev/sdd1 and see if zdb returns anything interesting like a pool structure. You could try this on any of your devices that isn't in a pool, and any that are in a pool should return information relating to the vdev to which they belong, and will show you what a normal zdb -l looks like. Maybe you will find your missing device. If you go thru each device on the system, and zdb can't find a label on the device that matches the pool in question, then I can't think of anything else to try.

Yes I will.

sdd is the correct device.

zdb -l /dev/sdd1 : cannot open '/dev/sdd1': No such file or directory
zdb -l /dev/sdd : failed to unpack label 0 failed to unpack label 1 failed to unpack label 2 failed to unpack label 3

When running that command on another functional device I get almost the same error:
zdb -l /dev/sdb : failed to unpack label 0 failed to unpack label 1 failed to unpack label 2 failed to unpack label 3
zdb -l /dev/sdb1 : failed to unpack label 0 failed to unpack label 1 failed to unpack label 2 failed to unpack label 3

 

[samarium]

Please post to following:

Code:

# sgdisk -p /dev/sdd
# sgdisk -p /dev/sdb
# zpool list -vL

 

[christian83]

sgdisk -p /dev/sdd

Code:

Warning: Partition table header claims that the size of partition table
entries is 0 bytes, but this program  supports only 128-byte entries.
Adjusting accordingly, but partition table may be garbage.
Warning: Partition table header claims that the size of partition table
entries is 0 bytes, but this program  supports only 128-byte entries.
Adjusting accordingly, but partition table may be garbage.
Creating new GPT entries in memory.
Disk /dev/sdd: 35156656128 sectors, 16.4 TiB
Model: TOSHIBA MG09ACA1
Sector size (logical/physical): 512/4096 bytes
Disk identifier (GUID): 82D91365-8611-424B-8D2A-12EDC5F1D5C4
Partition table holds up to 128 entries
Main partition table begins at sector 2 and ends at sector 33
First usable sector is 34, last usable sector is 35156656094
Partitions will be aligned on 2048-sector boundaries
Total free space is 35156656061 sectors (16.4 TiB)

Number  Start (sector)    End (sector)  Size       Code  Name

sgdisk -p /dev/sdb

Code:

Disk /dev/sdb: 5860533168 sectors, 2.7 TiB
Model: ST3000VN0001-1SF
Sector size (logical/physical): 512/4096 bytes
Disk identifier (GUID): 0D2E03D7-BEAF-4460-AB89-4A52078729AB
Partition table holds up to 128 entries
Main partition table begins at sector 2 and ends at sector 33
First usable sector is 34, last usable sector is 5860533134
Partitions will be aligned on 128-sector boundaries
Total free space is 221 sectors (110.5 KiB)

Number  Start (sector)    End (sector)  Size       Code  Name
   1             128         4194304   2.0 GiB     8200  
   2         4194432      5860533134   2.7 TiB     BF01  

zpool list -vL

Code:

NAME         SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
aaa         3.62T  1.76T  1.86T        -         -     0%    48%  1.00x    ONLINE  /mnt
  mirror-0  3.62T  1.76T  1.86T        -         -     0%  48.6%      -    ONLINE
    sdh2    3.64T      -      -        -         -      -      -      -    ONLINE
    sde2    3.64T      -      -        -         -      -      -      -    ONLINE
bbb         2.72T   482G  2.25T        -         -     0%    17%  1.00x    ONLINE  /mnt
  mirror-0  2.72T   482G  2.25T        -         -     0%  17.3%      -    ONLINE
    sdb2    2.73T      -      -        -         -      -      -      -    ONLINE
    sdc2    2.73T      -      -        -         -      -      -      -    ONLINE
boot-pool     31G  9.32G  21.7G        -         -     3%    30%  1.00x    ONLINE  -
  sda3      31.5G  9.32G  21.7G        -         -     3%  30.1%      -    ONLINE
bucket-1     928G  13.9G   914G        -         -     0%     1%  1.00x    ONLINE  /mnt
  sdf2       930G  13.9G   914G        -         -     0%  1.49%      -    ONLINE
bucket-2     928G   890G  37.9G        -         -    87%    95%  1.00x    ONLINE  /mnt
  sdg2       930G   890G  37.9G        -         -    87%  95.9%      -    ONLINE

Thank you for your help and time, but I guess this drive is toast..
Time to rebuild i guess

 

[samarium]

Drive looks like toast I agree, however you may be able to just re-partition it.

In my experiments, zpool detach pool device left the partitioning intact, and thus zdb could read the zfs label on sdX1, but I wasn't using webui, and it might have been more thorough, or you might have done something yourself.

Looking at your zpool list -v output and sgdisk -p /dev/sdb, it seems all your disks are using partition 2 for the vdev member device, so I guess that you have left the default system->swap setting enabled, and webui carved off a chunk, usually 2GB unless you changed it, off the front of the disk as sdX1 to use for swap, which is consitent with the GPT on sdb.

So if you re-partitioned the disk like it was partitioned before, then sdd2 might still have your ZFS label.
I would be reticent about actually writing to the disk unless you have already made a copy with dd or ddrescue, or some other disk imaging tool?

Try zdb -l /dev/sdb2 so you know what good zdb output looks like.

You could try taking the start of sdb2 from sgdisk -p, and using dd and zdb to check.

Code:

# dd if=/dev/sdd bs=512 iseek=4194432 count=2048 of=/tmp/img.sdd2
# zdb -l /tmp/img.sdd2

If that doesn't work, maybe you have a different partition offset, so you might need to hunt around, or just grab a bigger chunk and start earlier, and use strings to see if you get something like zdb -l output.

Code:

# strings /tmp/img.sdd2 | less
# ### or even
# dd if=/dev/sdd bs=1M | strings | grep metaslab_shift

but the later is almost guaranteed to work unless somehow you have wiped the entire drive, although it would tell you that, unless you are using encryption? I don't know enough about encryption to know for sure how it impacts on disk structure, but since it seems to allow certain metadata to be read I'm guessing not much.

You could loop in 1M chunks until you get a hit, at least then you would know roughtly where in the disk it started, and could use that to help inform the partition table reconstruction, allowing for the apparent 128 sector offset, after you have imaged the drive.

 

[samarium]

I also found these relevant links:

https://unix.stackexchange.com/questions/564640/recover-import-zfs-pool-from-single-old-mirror
https://github.com/openzfs/zfs/issues/4187
https://thr3ads.net/zfs-discuss/2008/04/351635-recovering-data-from-a-dettach-mirrored-vdev
https://gist.github.com/jjwhitney/baaa63144da89726e482
https://github.com/andrewc12/openzfs/commit/2b517ae76321b7295d4508ca222a7a6aa7588497

So a prereq for recovering the data is determining finding the label so zdb can see it, and for that we need to reconstruct the partition table.

 

[christian83]

I also found these relevant links:

https://unix.stackexchange.com/questions/564640/recover-import-zfs-pool-from-single-old-mirror
https://github.com/openzfs/zfs/issues/4187
https://thr3ads.net/zfs-discuss/2008/04/351635-recovering-data-from-a-dettach-mirrored-vdev
https://gist.github.com/jjwhitney/baaa63144da89726e482
https://github.com/andrewc12/openzfs/commit/2b517ae76321b7295d4508ca222a7a6aa7588497

So a prereq for recovering the data is determining finding the label so zdb can see it, and for that we need to reconstruct the partition table.

Thank you for the info, I will continue investigate when I have some time over.
At the moment I think I will put the disk to the side, and try to get my nas going again.

Do I need the original truenas instance with the "empty pool" or would I have equal chance to rescue some files even if I would rebuild my truenas system?