impossible import zpool state UNAVAIL

[alexfax]

Hi, sorry for my english ... i'm italian.
i made a mistake
i try to substitute an hdd on my zfs pool but accidentally Accidentally added a disk to a ZFS RaidZ pool instead of replacing failing one.

1 i hade remove a hdd ada3p2 (is not broken but i must replace it)
2 i have insert a new hdd (but accidentally added a disk to a ZFS RaidZ (with GUI) pool instead of replacing failing one)
3 in the hdd management of the zpool under the 8 hdd of raidz1-0 appeared one hdd "stripe"
4 i tried to wipe this hdd


And now i'm in this situation
please i haven't a backup of file i need to recover files
please please

I tried to connect all 9 hdd but the result is the same

Code:

root@freenas /dev]# zpool import                                               
   pool: ZFS_BASE                                                               
     id: 13233542194602196282                                                   
  state: UNAVAIL                                                                
 status: One or more devices are missing from the system.                       
 action: The pool cannot be imported. Attach the missing                        
        devices and try again.                                                  
   see: http://www.sun.com/msg/ZFS-8000-6X                                      
 config:                                                                        
                                                                                
        ZFS_BASE                                        UNAVAIL  missing device 
          raidz1-0                                      ONLINE                  
            gptid/30736404-2873-11e2-a9fa-0023cdf1dc06  ONLINE                  
            gptid/30e28ffe-2873-11e2-a9fa-0023cdf1dc06  ONLINE                  
            gptid/31538beb-2873-11e2-a9fa-0023cdf1dc06  ONLINE                  
            gptid/31c95fae-2873-11e2-a9fa-0023cdf1dc06  ONLINE                  
            gptid/322c9cde-2873-11e2-a9fa-0023cdf1dc06  ONLINE                  
            gptid/32913474-2873-11e2-a9fa-0023cdf1dc06  ONLINE                  
            gptid/81204f2e-2d17-11e2-9828-0023cdf1dc06  ONLINE                  
            gptid/33c0920a-2873-11e2-a9fa-0023cdf1dc06  ONLINE                  
                                                                                
        Additional devices are known to be part of this pool, though their      
        exact configuration cannot be determined.                               
[root@freenas /dev]#

thanks, bye

 

[paleoN]

4 i tried to wipe this hdd

This was foolish.

And now i'm in this situation
please i haven't a backup of file i need to recover files

Which makes this rather dumb.


First, install FreeNAS-8.3.0-RELEASE to a flash drive. Then from an SSH session as root:

Code:

camcontrol devlist

gpart show

glabel status

Try the following zpool command:

Code:

zpool import -f -R /mnt rdonly=on ZFS_BASE

 

[alexfax]

camcontrol devlist

Code:

[root@freenas] ~# camcontrol devlist
<TRANSCEND 20110519>               at scbus0 target 0 lun 0 (pass0,ada0)
<WDC WD20EARS-00MVWB0 51.0AB51>    at scbus2 target 0 lun 0 (pass1,ada1)
<WDC WD20EARS-00MVWB0 51.0AB51>    at scbus3 target 0 lun 0 (pass2,ada2)
<WDC WD20EARS-00MVWB0 50.0AB50>    at scbus4 target 0 lun 0 (pass3,ada3)
<HL-DT-ST DVD-ROM GDRH10N 0F04>    at scbus5 target 0 lun 0 (pass4,cd0)
<WDC WD20EARX-008FB0 51.0AB51>     at scbus6 target 0 lun 0 (pass5,ada4)
<WDC WD20EARS-00MVWB0 51.0AB51>    at scbus6 target 1 lun 0 (pass6,ada5)
<WDC WD20EARS-00MVWB0 51.0AB51>    at scbus7 target 0 lun 0 (pass7,ada6)
<WDC WD20EARS-00MVWB0 51.0AB51>    at scbus7 target 1 lun 0 (pass8,ada7)
<WDC WD20EARX-00PASB0 51.0AB51>    at scbus8 target 0 lun 0 (pass9,ada8)
<WDC WD20EARS-00MVWB0 51.0AB51>    at scbus9 target 0 lun 0 (pass10,ada9)
[root@freenas] ~#

gpart show

Code:

[root@freenas] ~# gpart show
=>     63  7831089  ada0  MBR  (3.8G)
       63  1930257     1  freebsd  [active]  (942M)
  1930320       63        - free -  (31k)
  1930383  1930257     2  freebsd  (942M)
  3860640     3024     3  freebsd  (1.5M)
  3863664    41328     4  freebsd  (20M)
  3904992  3926160        - free -  (1.9G)

=>        34  3907029101  ada1  GPT  (1.8T)
          34          94        - free -  (47k)
         128     4194304     1  freebsd-swap  (2.0G)
     4194432  3902834696     2  freebsd-zfs  (1.8T)
  3907029128           7        - free -  (3.5k)

=>        34  3907029101  ada2  GPT  (1.8T)
          34          94        - free -  (47k)
         128     4194304     1  freebsd-swap  (2.0G)
     4194432  3902834696     2  freebsd-zfs  (1.8T)
  3907029128           7        - free -  (3.5k)

=>      0  1930257  ada0s1  BSD  (942M)
        0       16          - free -  (8.0k)
       16  1930241       1  !0  (942M)

=>        34  3907026988  ada3  GPT  (1.8T)
          34          94        - free -  (47k)
         128     4194304     1  freebsd-swap  (2.0G)
     4194432  3902832584     2  freebsd-zfs  (1.8T)
  3907027016           6        - free -  (3.0k)

=>        34  3907029101  ada5  GPT  (1.8T)
          34          94        - free -  (47k)
         128     4194304     1  freebsd-swap  (2.0G)
     4194432  3902834696     2  freebsd-zfs  (1.8T)
  3907029128           7        - free -  (3.5k)

=>        34  3907029101  ada6  GPT  (1.8T)
          34          94        - free -  (47k)
         128     4194304     1  freebsd-swap  (2.0G)
     4194432  3902834696     2  freebsd-zfs  (1.8T)
  3907029128           7        - free -  (3.5k)

=>        34  3907029101  ada7  GPT  (1.8T)
          34          94        - free -  (47k)
         128     4194304     1  freebsd-swap  (2.0G)
     4194432  3902834696     2  freebsd-zfs  (1.8T)
  3907029128           7        - free -  (3.5k)

=>        34  3907029101  ada8  GPT  (1.8T)
          34          94        - free -  (47k)
         128     4194304     1  freebsd-swap  (2.0G)
     4194432  3902834696     2  freebsd-zfs  (1.8T)
  3907029128           7        - free -  (3.5k)

=>        34  3907029101  ada9  GPT  (1.8T)
          34          94        - free -  (47k)
         128     4194304     1  freebsd-swap  (2.0G)
     4194432  3902834696     2  freebsd-zfs  (1.8T)
  3907029128           7        - free -  (3.5k)

[root@freenas] ~#

glabel status

Code:

[root@freenas] ~# glabel status
                                      Name  Status  Components
                             ufs/FreeNASs3     N/A  ada0s3
                             ufs/FreeNASs4     N/A  ada0s4
gptid/32913474-2873-11e2-a9fa-0023cdf1dc06     N/A  ada1p2
gptid/30736404-2873-11e2-a9fa-0023cdf1dc06     N/A  ada2p2
                            ufs/FreeNASs1a     N/A  ada0s1a
gptid/33c0920a-2873-11e2-a9fa-0023cdf1dc06     N/A  ada3p2
gptid/31538beb-2873-11e2-a9fa-0023cdf1dc06     N/A  ada5p2
gptid/30e28ffe-2873-11e2-a9fa-0023cdf1dc06     N/A  ada6p2
gptid/31c95fae-2873-11e2-a9fa-0023cdf1dc06     N/A  ada7p2
gptid/81204f2e-2d17-11e2-9828-0023cdf1dc06     N/A  ada8p2
gptid/322c9cde-2873-11e2-a9fa-0023cdf1dc06     N/A  ada9p2
[root@freenas] ~#

i try to execute the command zpool import -f -R /mnt rdonly=on ZFS_BASE

but this is the result.

Code:

[root@freenas] ~# zpool import -f -R /mnt rdonly=on ZFS_BASE
cannot import 'rdonly=on': no such pool available
[root@freenas] ~#

the WD20EARX on ada4 is the "wipe hdd" and the foreign..
I made a virtual machine with vmware and i have created the same situation.
I can do all the tests you want
Thanks for your help

 

[paleoN]

Code:

[root@freenas] ~# zpool import -f -R /mnt rdonly=on ZFS_BASE
cannot import 'rdonly=on': no such pool available
[root@freenas] ~#

Typo on my part. The corrected import command:

Code:

zpool import -f -R /mnt -o rdonly=on ZFS_BASE

The rdonly=on is a mount option and you need the -o.

 

[alexfax]

hi, thanks!
The result of your comand:

Code:

[root@freenas] ~# zpool import -f -R /mnt -o rdonly=on ZFS_BASE
cannot import 'ZFS_BASE': one or more devices is currently unavailable
[root@freenas] ~#

unfortunately it is still not accessible!

 

[cyberjock]

Yeah. It looks like you lost all of your data then. :(

If you read my guide you'll see that you did exactly what my guide says you shouldn't do. Of course, you lost your data by wiping that one drive. Slides 22 and 23 I believe. :(

 

[alexfax]

I realized my mistake!
My hope exists because I did a wipe of a new disk and it is a stripe disk .
The 8 disks of the zpool are all perfect!
I did wipe a disk without data ...
I "just" have to say to the zpool to ignore this disk!

 

[cyberjock]

You can't ask it to "ignore" a disk. The disk had file system data put on it the second you added it to the zpool. You really have no options, unless you have $10k+ and can find a data recovery company that can do ZFS recovery.

This is an example of why you have to have a good attention to detail when administering servers. :(

I'm sorry about your data.

 

[paleoN]

My hope exists because I did a wipe of a new disk and it is a stripe disk .
The 8 disks of the zpool are all perfect!
I did wipe a disk without data ...

I don't have much hope. Your 8 disks aren't perfect anymore, because they are actually 9 now. The disk did have metadata on it. The bigger mistake was wiping that disk without having any backups.

Still let's try the drastic rewind of the pool:

Code:

zpool import -fFX -R /mnt ZFS_BASE

 

[alexfax]

hi,
i try with

Code:

zpool import -fFX -R /mnt ZFS_BASE

on ssh session nothing happens, but in the monitor connected to the nas
I made a picture


my hope decrease every day.... :(
thanks...

at the same time I'm trying to recover deleted files from the backup hdd (deleted files one month ago) I try to save as much as possible
....if we do not find a solution

 

[cyberjock]

Promise yourself you'll start doing backups of the data that is most important to you.

Back in 1998 I gave a 600MB(oh yeah. I was a roller!) hard drive to a friend so he could burn a CD of the data for backups. I didn't own a CD burner at the time. I handed the drive to him and he put it in an anti-static bag. As he dropped the hard drive in the bag it fell through the bag 4 feet to the floor. Drive never worked again. :(

Needless to say, I was very unhappy since that was the only copy of the data I had.

 

[jgreco]

Sad story. Funny in the other direction: the guys at MEI in 1992 had a Quantum ProDrive 105S (100MB SCSI) they somehow knocked off a lab bench while running (I never did get a good answer as to how this happened, considering the cabling involved) which promptly landed on the floor and spun down. They donated it to us here, because they all had shell accounts on our public access UNIX system, and I spent some time with the drive which was mostly just fine except for a bunch of new bad sectors. That drive eventually became the Sun's boot drive, and because the Sun 3 series had no upgrade path, and we kept the system running, it continued to run just fine for many years. We killed it (the system, not the drive) in 2011, I think.

 

[paleoN]

There are only a few things left to try.

1. Choose "6. Escape to loader prompt" during boot.
2. Type the following:
   Code:

   set vfs.zfs.recover=1
   boot -s

3. Attempt to import pool again.
4. Possibly, if it even works on 9.1, after booting like above:
   Code:

   zdb -e -bcsvL <poolname>

If that doesn't help you can try:

1. booting a live FreeBSD 9.1-RC3 based CD/DVD and importing.
2. Try importing the pool from a Solaris/Illumos based distro.
3. Attempt to run ZFS forensics scrollback script.
   
   You can use the following to try and guess a txg before you added the stripe.

   Code:

   zdb -lu /dev/adaX

   
   There's this line, "This script should work on linux and solaris." Seems odd no mention of FreeBSD. Of course you could always manually use dd and zero out the appropriate uberblocks. This could possibly allow the import with -F or -X to succeed, depending on why you got the panic.​

 

[paleoN]

One last thing. I've been assuming you no longer have the zpool.cache file from when you added the stripe:

Code:

zdb -U /data/zfs/zpool.cache

If you did you could use it to try and recreate a ZFS label on ada4. Which might allow one of the import commands to work.

 

[alexfax]

thank you, you're very kind
in the first solution
after reboot I get an error in loading
(I am attaching a screenshot of the virtual machine, it is the same in physical machine)


I do not know the commands that you gave me.
I stopped ... I do not know how to continue ....
now I'm trying with solution 2
thanks again for the help

 

[paleoN]

I do not know the commands that you gave me.
I stopped ... I do not know how to continue ....

Press <ENTER>/<RETURN> to get to the shell.

 

[alexfax]

I continued with the code

Code:

zdb -e -bcsvL ZFS_BASE

but the result is the same


now i'm trying with the ZFS forensics scrollback script

 

[lancee]

Did you have any luck with the forensics script? Having same issue now...

 

[alexfax]

I'm sorry..
I haven't good news..
I have try with to fix the situation in many ways but without success.
as last option i format all (lost all data).:(