Cannot re-import my pool after erasing a drive that was added via the GUI...

[Joel Jamison]

Hello, I have a pool that I was trying to upgrade from 6 x 3TB drives to 6 x 4TB drives by replacing one drive at a time. I have used ZFS on a Mac (OS X) for 6 years and have been using FreeNAS for the last 2 years. I've done this same type of replacement/upgrade move many times through the years but the recent move to Corral has me more than a little lost.



When I added the new 4TB drive, I read that I should not use zpool commands with Corral so I used the GUI to drag the new drive over to the pool I wanted to swap the drive with. The GUI would not let me replace a drive directly so I tried adding it to the pool first. The GUI seemed to show the new (4TB) drive as a separate VDEV as it showed my 6 drives in one block and the new drive in another. I tried swapping the drive from there but it would not work. I then tried using the console (“unix”) to replace the drive, it said the device was in use, I even tried using the zpool replace <pool> <old> <new> command but none of this worked.



I tried booting up FreeNAS with that drive removed but the pool would not come up. I was thinking that I needed to erase that new drive then perhaps I would be able to use the FreeNAS "unix" console to replace a drive (it was saying there were no valid replicas when I tried without erasing the drive.) I used the console to detach the pool, went into disk>da6 and erased the drive, then tried to import the pool back in but it isn't working. I see the message "Additional devices are known to be part of this pool, though their exact configuration cannot be determined." when running a "zpool import"



From the console session or an ssh session, if I try to import the pool using the "zpool import <poolname>" command I get the same message "cannot import 'zpool': one or more devices is currently unavailable"



When I do a "gpart show" or a "glabel status" I see the original 6 drives in that pool listed but not the recently erased 4TB drive. If I look for the drive specifically I get "gpart: No such geom: da6." so it looks like I need to as the partitions (slices 1 and 2) if I want the drive to show up with those commands. When I use "camcontrol devlist" I see the drive listed there.

[root@freenas] ~# camcontrol devlist



<ATA TOSHIBA DT01ACA3 ABB0> at scbus0 target 0 lun 0 (pass0,da0)



<ATA TOSHIBA DT01ACA3 ABB0> at scbus0 target 1 lun 0 (pass1,da1)



<ATA TOSHIBA DT01ACA3 ABB0> at scbus0 target 2 lun 0 (pass2,da2)



<ATA TOSHIBA DT01ACA3 ABB0> at scbus0 target 3 lun 0 (pass3,da3)



<ATA TOSHIBA DT01ACA3 ABB0> at scbus0 target 4 lun 0 (pass4,da4)



<ATA TOSHIBA DT01ACA3 ABB0> at scbus0 target 5 lun 0 (pass5,da5)



<INTEL SSDSA2M040G2GC 2CV102HD> at scbus6 target 0 lun 0 (pass6,ada0)



<ATA ST4000DM005-2DP1 0001> at scbus8 target 22 lun 0 (da6,pass7)



At this point I feel like I have 3 options to pursue,

1.) try to find a way to import the pool with the 6 original drives (not using the 4TB drive) so that I can proceed with the 3TB to 4TB swaps again.

2.) find a way to undo the erase command on the da6 drive (this is why I have not added the partitions yet) to restore the situation I was in with the separate VDEV, then try to sort things out from there.

3.) roll back to FreeNAS 9.10 where the zpool commands seem to be the way to get things done and try the same import commands from there.



Thank you in advance for your guidance, Joel

 

[danb35]

You shouldn't be using Corral; it's dead.

The GUI would not let me replace a drive directly so I tried adding it to the pool first.

Then you're done. Once you add the disk to the pool, it cannot be removed. Since you removed it and erased it, your data is gone. Restore from backup.

 

[Joel Jamison]

Ah, that is bad news. My backup is not complete (old, I know, bad administration on my part.) Is there a way to replace the drive that is missing? The pool was RAIDZ2 so it should be able to survive loosing one drive.

I exported the pool, then erased the drive, that would be like exporting the pool then having a drive failure. Sorry, I know this is a common problem with lazy administration, just hoping I don't have to start all over again. Thanks, Joel

 

[Robert Trevellyan]

The pool was RAIDZ2 so it should be able to survive loosing one drive.

Unfortunately, at the end your pool was made of 2 vdevs, one 6-disk RAIDZ2 and one single-disk stripe. Removing and wiping the single-disk vdev destroyed the pool (the failure of any vdev destroys a pool).

bad administration on my part

To be fair, you were led by the hand to your doom by Corral, which was released before it was ready. There was no way to replace a disk via the GUI.

 

[Joel Jamison]

Sad Christmas... I only used the Corral console to erase the 2 partitions on that second VDEV with the disk>da6>erase command, I'm fishing now on how I can use BSD magic to restore those partitions. Perhaps gpart?
Thanks for the responses... Joel

 

[danb35]

The best guess, or hope, I can give is that ZFS retained enough of the transaction history to be able to roll back to before the disk was added. It's a slim hope. See what zpool import -f -F -n poolname returns.

 

[Joel Jamison]

Thanks Dan, it was worth a try but no luck:
^C[root@freenas] ~# zpool import -f -F -n zpool

[root@freenas] ~# zpool status

pool: freenas-boot

state: ONLINE

scan: none requested

config:


NAME STATE READ WRITE CKSUM

freenas-boot ONLINE 0 0 0

ada0p2 ONLINE 0 0 0


errors: No known data errors

[root@freenas] ~#

 

[rs225]

If you only removed the partitions, but didn't do a full erase, then you may be in luck. Somebody will have to give you advice on how to re-create the partition table though. (And somebody who knows the Corral commands will have to determine if they only removed the partitions.)

 

[Joel Jamison]

Thanks RS, some hope is better than no hope. I worry because the "help erase" says that it does securely erase. Hopefully someone has some insights. I've tried doing the gpart recover and am currently trying "gpart restore -l /dev/da6" but it isn't looking good.

If anyone has any suggestions please let me know. Thanks again, Joel

 

[rs225]

The answer on the type of erase is in how long it took. Instant, means just partition table. An hour+ means it was complete.

 

[joeschmuck]

Yes, the hope is very slim but if you could restore the partition table then you might be okay. The trick is figuring out the values to put into the partition table. There are some product which can be purchased but that doesn't mean they will be able to get your data back.

You need to think about how important that data is to you and how much you are willing to do to get it back in terms of time to learn/read and financial cost. I'm not trying to stop you, just making this subject realistic. I truley hope you are able to get all your data back and if you come up with a way to restore your system after this type of situation (I know of none to date) then we all want a tutorial on how to do it.

Best of luck!

 

[Joel Jamison]

Good, then there is hope. It was instant.

 

[Joel Jamison]

Thanks Joe, in looking into how to recover the partitions there do seem to be some paid solutions. I'm willing to try them if I can't get some thing else going. Hopefully it won't come to that. I'll keep you posted either way. Thanks again, Joel

 

[joeschmuck]

I must have lost my train of thought but there are free program out there as well but I have no idea if they would work either. Regardless, it's an uphill battle.

If you plan to recover your pool, power off your FreeNAS system until you are ready to try a recovery process. There is no need to leave it running.

 

[rs225]

I would do a gpart show and look at the output for all of your drives. I suspect the partition table is similar across all drives and then it is just a matter of creating a similar one on the problem drive.

 

[Joel Jamison]

yup, I seem to recall 2 partitions like the other ones. Hopefully they can be restored:

[root@freenas] ~# gpart show

=> 34 78165293 ada0 GPT (37G)

34 6 - free - (3.0K)

40 1024 1 bios-boot (512K)

1064 78164256 2 freebsd-zfs (37G)

78165320 7 - free - (3.5K)


=> 34 5860533101 da0 GPT (2.7T)

34 94 - free - (47K)

128 4194304 1 freebsd-swap (2.0G)

4194432 5856338696 2 freebsd-zfs (2.7T)

5860533128 7 - free - (3.5K)


=> 34 5860533101 da1 GPT (2.7T)

34 94 - free - (47K)

128 4194304 1 freebsd-swap (2.0G)

4194432 5856338696 2 freebsd-zfs (2.7T)

5860533128 7 - free - (3.5K)


=> 34 5860533101 da2 GPT (2.7T)

34 94 - free - (47K)

128 4194304 1 freebsd-swap (2.0G)

4194432 5856338696 2 freebsd-zfs (2.7T)

5860533128 7 - free - (3.5K)


=> 34 5860533101 da3 GPT (2.7T)

34 94 - free - (47K)

128 4194304 1 freebsd-swap (2.0G)

4194432 5856338696 2 freebsd-zfs (2.7T)

5860533128 7 - free - (3.5K)


=> 34 5860533101 da4 GPT (2.7T)

34 94 - free - (47K)

128 4194304 1 freebsd-swap (2.0G)

4194432 5856338696 2 freebsd-zfs (2.7T)

5860533128 7 - free - (3.5K)


=> 34 5860533101 da5 GPT (2.7T)

34 94 - free - (47K)

128 4194304 1 freebsd-swap (2.0G)

4194432 5856338696 2 freebsd-zfs (2.7T)

5860533128 7 - free - (3.5K)


Nothing listed for da6 which is the 4TB drive that is causing all the fuss. Joel

 

[diedrichg]

I have two programs burned on a CD sitting on my shelf (that tells you how old they are). They are called PING and Active Partition Recovery. Check if they still exist and if they might help.

 

[rs225]

Here is something to try:

Code:

Do not try this at home.

The output should hopefully show the freebsd-zfs starting on xxx, and have a very small amount of free space after it.
You can then try a zpool import and see what shows up as potentially importable.

 

[Joel Jamison]

Ah, no luck there either:
^

Code:

magic commands omitted

[root@freenas] ~# zpool import zpool

cannot import 'zpool': one or more devices is currently unavailable

[root@freenas] ~# zpool import -f -F -n zpool
[root@freenas] ~# zpool status zpool
cannot open 'zpool': no such pool

 

[rs225]

You can try this to see if anything turns up, but it presumes the partition table is pretty close and that there was no erase and also that hexdump exists.
Do not try this at home.
You would be looking for a section that will have some strings like 'slab' and 'vdev' and 'delphix'