PLZ HELP - ZPOOL STATUS UNKNOWN

[Amu W Ramappa]

Our CIFS share quit working and I went to restart the server and then the status of the pool was UNKNOWN on the WEBGUI. So, went into the SSH and try to do an zfs import and this is the message I get:

Code:

[root@nas1] ~# zpool import
  pool: storage
    id: 10069954888908849131
  state: FAULTED
status: The pool metadata is corrupted.
action: The pool cannot be imported due to damaged devices or data.
        The pool may be active on another system, but can be imported using
        the '-f' flag.
  see: http://www.sun.com/msg/ZFS-8000-72
config:
 
        storage                                        FAULTED  corrupted data
          mirror-0                                      ONLINE
            gptid/eedc2471-273c-11e2-979a-1c6f658d67db  ONLINE
            gptid/ef3b1d66-273c-11e2-979a-1c6f658d67db  ONLINE
          mirror-1                                      ONLINE
            gptid/0b800f5c-273d-11e2-979a-1c6f658d67db  ONLINE
            gptid/0bce2118-273d-11e2-979a-1c6f658d67db  ONLINE
          mirror-2                                      ONLINE
            gptid/4ae6f8ea-e90a-11e2-bf4c-1c6f658d67db  ONLINE
            gptid/4b17e483-e90a-11e2-bf4c-1c6f658d67db  ONLINE
          mirror-3                                      ONLINE
            gptid/871a6868-e90a-11e2-bf4c-1c6f658d67db  ONLINE
            gptid/87428542-e90a-11e2-bf4c-1c6f658d67db  ONLINE

I tried to do zpool import -f storage and here's the output of that command:

Code:

[root@nas1] ~# zpool import -f storage
cannot import 'storage': I/O error
        Destroy and re-create the pool from
        a backup source.

Here's my version of freenas:

Code:

[root@nas1] ~# uname -a
FreeBSD nas1.baisedu.org 8.3-RELEASE-p7 FreeBSD 8.3-RELEASE-p7 #1 r249203M: Sat Apr  6 09:28:27 PDT 2013    root@build.ixsystems.com:/tank/home/jpaetzel/fn8.3/freenas/os-base/amd64/tank/home/jpaetzel/fn8.3/freenas/FreeBSD/src/sys/FREENAS.amd64  amd64

Could someone please help me resolve this issue? Our school is at a stall because of this. :(
Thanks!

 

[cyberjock]

WOW. That sucks really badly. My first guess is your data is gone. Recreate your pool from backup like the message says.

Second guess is that I do offer recovery services, but I don't have any solid info on your chances of success since you included no hardware at all.

 

[Amu W Ramappa]

Thanks for the reply cyberjock. I didn't have any backups because all the drives were mirrored and didn't think i needed a separate backup. Now I am regretting this decision.

Here's the gpart information, at this point I can get any help I can to recover our school's data.

Code:

[root@freenas] /mnt# zpool import -f
  pool: storage
    id: 10069954888908849131
  state: FAULTED
status: One or more devices are missing from the system.
action: The pool cannot be imported. Attach the missing
        devices and try again.
        The pool may be active on another system, but can be imported using
        the '-f' flag.
  see: http://illumos.org/msg/ZFS-8000-3C
config:
 
        storage                                        FAULTED  corrupted data
          mirror-0                                      ONLINE
            gptid/eedc2471-273c-11e2-979a-1c6f658d67db  ONLINE
            gptid/ef3b1d66-273c-11e2-979a-1c6f658d67db  ONLINE
          mirror-1                                      ONLINE
            gptid/0b800f5c-273d-11e2-979a-1c6f658d67db  ONLINE
            gptid/0bce2118-273d-11e2-979a-1c6f658d67db  ONLINE
          mirror-2                                      ONLINE
            gptid/4ae6f8ea-e90a-11e2-bf4c-1c6f658d67db  ONLINE
            gptid/4b17e483-e90a-11e2-bf4c-1c6f658d67db  ONLINE
          mirror-3                                      ONLINE
            gptid/871a6868-e90a-11e2-bf4c-1c6f658d67db  ONLINE
            gptid/87428542-e90a-11e2-bf4c-1c6f658d67db  ONLINE
          mirror-4                                      DEGRADED
            gptid/04b43a43-87dd-11e3-b1ac-1c6f658d67db  ONLINE
            15948730246039370594                        UNAVAIL  cannot open
[root@freenas] /mnt# zpool import -f storage
cannot import 'storage': I/O error
        Destroy and re-create the pool from
        a backup source.
[root@freenas] /mnt# gpart show
=>        34  5860533101  da2  GPT  (2.7T)
          34          94      - free -  (47k)
        128    8388608    1  freebsd-swap  (4.0G)
    8388736  5852144399    2  freebsd-zfs  (2.7T)
 
=>        34  1953525101  da3  GPT  (931G)
          34          94      - free -  (47k)
        128    8388608    1  freebsd-swap  (4.0G)
    8388736  1945136399    2  freebsd-zfs  (927G)
 
=>        34  5860533101  da4  GPT  (2.7T)
          34          94      - free -  (47k)
        128    8388608    1  freebsd-swap  (4.0G)
    8388736  5852144399    2  freebsd-zfs  (2.7T)
 
=>        34  1953525101  da5  GPT  (931G)
          34          94      - free -  (47k)
        128    8388608    1  freebsd-swap  (4.0G)
    8388736  1945136399    2  freebsd-zfs  (927G)
 
=>        34  5860533101  da8  GPT  (2.7T)
          34          94      - free -  (47k)
        128    8388608    1  freebsd-swap  (4.0G)
    8388736  5852144399    2  freebsd-zfs  (2.7T)
 
=>        34  5860533101  da9  GPT  (2.7T)
          34          94      - free -  (47k)
        128    8388608    1  freebsd-swap  (4.0G)
    8388736  5852144399    2  freebsd-zfs  (2.7T)
 
=>        34  5860533101  da10  GPT  (2.7T)
          34          94        - free -  (47k)
        128    8388608    1  freebsd-swap  (4.0G)
    8388736  5852144399    2  freebsd-zfs  (2.7T)
 
=>        34  5860533101  da11  GPT  (2.7T)
          34          94        - free -  (47k)
        128    8388608    1  freebsd-swap  (4.0G)
    8388736  5852144399    2  freebsd-zfs  (2.7T)
 
=>        34  1953525101  da12  GPT  (931G)
          34          94        - free -  (47k)
        128    8388608    1  freebsd-swap  (4.0G)
    8388736  1945136399    2  freebsd-zfs  (927G)
 
=>      63  31266753  da14  MBR  (14G)
        63  1930257    1  freebsd  [active]  (942M)
  1930320        63        - free -  (31k)
  1930383  1930257    2  freebsd  (942M)
  3860640      3024    3  freebsd  (1.5M)
  3863664    41328    4  freebsd  (20M)
  3904992  27361824        - free -  (13G)
 
=>      0  1930257  da14s1  BSD  (942M)
        0      16          - free -  (8.0k)
      16  1930241      1  !0  (942M)
 
[root@freenas] /mnt#

 

[cyberjock]

Well, it looks like your server was neglected for too long. One disk is missing(UNAVAIL).

RAID is not and has never been a substitute for backups. RAID gives higher uptime. That's it.

What's the story with the missing disk? When did it fail? Do you know when it failed? And I'd still like to know the server specs.

 

[Amu W Ramappa]

It's a custom build. I have an AMD Desktop motherboard with 16GB of memory and AMD Phenom(tm) II X2 555 Processor. The RAID card is Areca ARC-1882iX-24. All the drives are set as "Pass Thru" on the card and they mirror is managed by FreeNAS. I have no idea when that drive failed. It didn't even show on the SMART status as it's going to fail. :(

 

[cyberjock]

Well, right now I'd consider your data gone. If you want to do data recovery I do offer data recovery services. PM me for details and cost.

As a rule, desktop motherboards and non-ECC RAM are a disaster waiting to happen. In any case, unless you can magically get that disk that's offline to start working you aren't likely to see your data again easily. I might be able to bring it back, but that depends on a few factors.

 

[leenux_tux]

Amu,

Have you looked at the configuration screens for the RAID card, do all of the disks appear in the listings ? Not sure if you have checked this but could it be a faulty power cable or SATA cable ?

 

[Amu W Ramappa]

Amu,

Have you looked at the configuration screens for the RAID card, do all of the disks appear in the listings ? Not sure if you have checked this but could it be a faulty power cable or SATA cable ?

Yes the drives show up on the RAID card. I tried moving the two drives from mirror-4 to a different bay and I still get the same error.
I am thinking of taking both the drives out and clone them to a different 1TB drives and then try with the new drives. You think this will work?

 

[leenux_tux]

IMHO..... I have no idea if what your suggesting will work. It all depends on what has caused the error. Perhaps other forum members have tried this before ?

What I would advise though, if you are going to attempt this, and this depends on what systems you have available, is that if you use something like "clonezilla" to clone each of the drives to a file, create a FreeNAS Virtual Machine (in something like VirtualBox which is free), and clonezilla the disk images back to virtual disks; essentially doing a partial Physical2Virtual (P2V)conversion. I know this works, but for your drives, the only way to find out is to test it.

Of course this all depends on the amount of disk capacity you have spare on another system