Help needed for recovering data from zpool i can't import!!

[djoole]

Hi,

The worst thing that could happen to me just happened : i can't access very important data from a disk.

I made a zpool with a single disk (jayjay, da1), where i backup data from another zpool (zepool).
Then i destroyed and recreated the zpool zepool (8-disks raidz2), and rsynced data from jayjay to zpool.

Durning transfer, I/O device failure happened, da1 disconnected (i should mentionned i mounted the drive on a USB3 dock). This kind of error happened before, during the backup process from zepool to jayjay, and i had to mount the dock to a USB2 dock in order to have transfer without error.

So i shut the computer down, plug the drive in a USB2 port, and boot.

Problem is the jayjay zpool is not mounted.

So i try the command zpool import, and here it is :

Code:

[root@nas] ~# zpool import
  pool: jayjay
    id: 7649169962979289867
 state: FAULTED
status: The pool metadata is corrupted.
action: The pool cannot be imported due to damaged devices or data.
        The pool may be active on another system, but can be imported using
        the '-f' flag.
   see: http://www.sun.com/msg/ZFS-8000-72
config:

        jayjay      FAULTED  corrupted data
          da1p2     ONLINE

So i try zpool import -f jayjay :

Code:

[root@nas] ~# zpool import -f jayjay
cannot import 'jayjay': I/O error

Please, tell me there is another way to access, even partly, the data from this disk

 

[ProtoSD]

It's not very likely you'll be able to recover unless someone else has an idea. The 'corrupted metadata' is the worst problem. You *might* try 'zpool import -fD jayjay' or take the drive out of the USB enclosure and connect it directly to your system and then try to import. You might also try a Live boot CD with ZFS v28 on it, there's a link in this post: Live FreeBSD ZFS boot CD

 

[djoole]

Hi, thanks for answering, although your answer is not what i expected :(

zpool import -fD jayjay gives nothing :

Code:

[root@nas] ~# zpool import -fD jayjay
cannot import 'jayjay': no such pool available

I'm downloading Solaris 11 Express live CD (ZFS v30), and in the meantime, i'll try to plug the drive directyl on a SATA port of the NAS (i'll have to degrade zepool, but i don't care, i haven't been able to put some data in it yet :( )

 

[djoole]

If this can beof any help, here are the messages logs when i try to import jayjay :

Code:

Oct 16 00:31:48 nas root: ZFS: checksum mismatch, zpool=jayjay path=/dev/ada0p2 offset=1889789869568 size=1024
Oct 16 00:31:48 nas root: ZFS: checksum mismatch, zpool=jayjay path=/dev/ada0p2 offset=225654211584 size=1024
Oct 16 00:31:48 nas root: ZFS: checksum mismatch, zpool=jayjay path=/dev/ada0p2 offset=741066910208 size=1024
Oct 16 00:31:48 nas root: ZFS: checksum mismatch, zpool=jayjay path=/dev/ada0p2 offset=1889789869568 size=1024
Oct 16 00:31:48 nas root: ZFS: checksum mismatch, zpool=jayjay path=/dev/ada0p2 offset=225654211584 size=1024
Oct 16 00:31:48 nas root: ZFS: checksum mismatch, zpool=jayjay path=/dev/ada0p2 offset=741066910208 size=1024
Oct 16 00:31:48 nas root: ZFS: zpool I/O failure, zpool=jayjay error=86
Oct 16 00:31:48 nas root: ZFS: failed to load zpool jayjay

Edit : I'm pissed. How can it possible that metadata has been corrupted while i was only reading from that pool!!

 

[ProtoSD]

I don't blame you for being pissed, I would be too. I hope you have better luck with the Solaris 11 Express live CD, it gave me lots of errors every time I tried to do any simple ZFS command. There is another thread where several of us tried to help someone with corrupt metadata and had no luck either.

 

[djoole]

I'm on Solaris 11 Express, and zpool import command returns strictly nothing.

zdb -l /dev/dsk//c7t0d0p0 returns :

Code:

jack@solaris:~# zdb -l /dev/dsk/c7t0d0p0
--------------------------------------------
LABEL 0
--------------------------------------------
failed to unpack label 0
--------------------------------------------
LABEL 1
--------------------------------------------
failed to unpack label 1
--------------------------------------------
LABEL 2
--------------------------------------------
failed to unpack label 2
--------------------------------------------
LABEL 3
--------------------------------------------
failed to unpack label 3

This smells so bad...

I've found this article talking about deleting Uberblocks to be able to import a corrupted zpool.
I'll try i have nothing left to lose.
http://www.solarisinternals.com/wiki/index.php/ZFS_forensics_scrollback_script

 

[ProtoSD]

Try the other CD, I got useless results from the Solaris CD too. I'll take a look at that other link.

 

[djoole]

You're link is broken :)
I guess it's easy to find anyway.

I've launched a script on Solaris that will take time to finish.

I'm off to bed and tomorrow i'll have a try with FreeBSD!

 

[ProtoSD]

You're link is broken :)
I guess it's easy to find anyway.

I've launched a script on Solaris that will take time to finish.

I'm off to bed and tomorrow i'll have a try with FreeBSD!

Sorry about the link, I fixed it. It's better to deal with problems like this after some good sleep. Good luck

 

[djoole]

Finally i didn't go to sleep (couldn't, too much value in the lost data), i managed to find a FreeBSD Live cd with ZFS v28 (mfsBSD).

And protosd, it was the right move, as you advised me.

ZFS v28 seems to be able to roolback to the previous state before data had been corrupted.

After a zpool import (which find the broken pool), it told me as would be able to recover the pool with zpool import -fF jajay.
And it worked like a charm!!!!

Disk is back in the NAS (still on USB dock, no more SATA ports available), and rsyncing, this time in USB2 (i'll never use again USB3 in FreeNAS 8)

Thanks again for you support protosd :)

 

[Milhouse]

And hopefully your next purchase will be a backup solution? :)

 

[djoole]

Yes ;)
I'm going to add the 2 TB drive that will come back from Seagate RMA in my desktop PC, and i'll dedicate it to sync the sensitive content of my NAS.