Recovering pool which crashes OS

[stangri]

So I have (thru running a Nas4Free for about a week over the pool I've created with FreeNAS 7 a while ago) a pool which so far has crashed every OS I've tried to import it with: FreeNAS 8, OpenIndiana, Nas4Free and an old custom build of FreeNAS 7 with zfs 28 support.

I've had most success trying to restore it with FreeNAS 8.3.1 image tho, so hopefully someone can help me.

I've read up on the zfs import problems and the most common solutions I found were to try to mount it read-only (also causes kernel panic) or trying to recover the pool with zdb. I'm having a problem with zdb which hopefully someone can help me with.

While I can get listing of disks with zdb:

Code:

# zdb -l /dev/ada3
ZFS NOTICE: Prefetch is disabled by default if less than 4GB of RAM is present;
            to enable, add "vfs.zfs.prefetch_disable=0" to /boot/loader.conf.
ZFS filesystem version 5
zvol_init:1709[1]: ZVOL Initialized.
ZFS storage pool version 28
--------------------------------------------
LABEL 0
--------------------------------------------
    version: 28
    name: 'DATA'
    state: 0
    txg: 9766163
    pool_guid: 9522190198077791104
    hostname: ''
    top_guid: 252819746167074421
    guid: 8951831291888754281
    vdev_children: 1
    vdev_tree:
        type: 'raidz'
        id: 0
        guid: 252819746167074421
        nparity: 1
        metaslab_array: 23
        metaslab_shift: 31
        ashift: 12
        asize: 8001576501248
        is_log: 0
        children[0]:
            type: 'disk'
            id: 0
            guid: 2027052211370973537
            path: '/dev/ada0'
            phys_path: '/dev/ada0'
            whole_disk: 0
            DTL: 4102
        children[1]:
            type: 'disk'
            id: 1
            guid: 764366381652363129
            path: '/dev/ada1'
            phys_path: '/dev/ada1'
            whole_disk: 0
            DTL: 4101
        children[2]:
            type: 'disk'
            id: 2
            guid: 7379136512656866183
            path: '/dev/ada2'
            phys_path: '/dev/ada2'
            whole_disk: 0
            DTL: 4100
        children[3]:
            type: 'disk'
            id: 3
            guid: 8951831291888754281
            path: '/dev/ada3'
            phys_path: '/dev/ada3'
            whole_disk: 1
            DTL: 1330
--------------------------------------------
LABEL 1
--------------------------------------------
    version: 28
    name: 'DATA'
    state: 0
    txg: 9766163
    pool_guid: 9522190198077791104
    hostname: ''
    top_guid: 252819746167074421
    guid: 8951831291888754281
    vdev_children: 1
    vdev_tree:
        type: 'raidz'
        id: 0
        guid: 252819746167074421
        nparity: 1
        metaslab_array: 23
        metaslab_shift: 31
        ashift: 12
        asize: 8001576501248
        is_log: 0
        children[0]:
            type: 'disk'
            id: 0
            guid: 2027052211370973537
            path: '/dev/ada0'
            phys_path: '/dev/ada0'
            whole_disk: 0
            DTL: 4102
        children[1]:
            type: 'disk'
            id: 1
            guid: 764366381652363129
            path: '/dev/ada1'
            phys_path: '/dev/ada1'
            whole_disk: 0
            DTL: 4101
        children[2]:
            type: 'disk'
            id: 2
            guid: 7379136512656866183
            path: '/dev/ada2'
            phys_path: '/dev/ada2'
            whole_disk: 0
            DTL: 4100
        children[3]:
            type: 'disk'
            id: 3
            guid: 8951831291888754281
            path: '/dev/ada3'
            phys_path: '/dev/ada3'
            whole_disk: 1
            DTL: 1330
--------------------------------------------
LABEL 2
--------------------------------------------
    version: 28
    name: 'DATA'
    state: 0
    txg: 9766163
    pool_guid: 9522190198077791104
    hostname: ''
    top_guid: 252819746167074421
    guid: 8951831291888754281
    vdev_children: 1
    vdev_tree:
        type: 'raidz'
        id: 0
        guid: 252819746167074421
        nparity: 1
        metaslab_array: 23
        metaslab_shift: 31
        ashift: 12
        asize: 8001576501248
        is_log: 0
        children[0]:
            type: 'disk'
            id: 0
            guid: 2027052211370973537
            path: '/dev/ada0'
            phys_path: '/dev/ada0'
            whole_disk: 0
            DTL: 4102
        children[1]:
            type: 'disk'
            id: 1
            guid: 764366381652363129
            path: '/dev/ada1'
            phys_path: '/dev/ada1'
            whole_disk: 0
            DTL: 4101
        children[2]:
            type: 'disk'
            id: 2
            guid: 7379136512656866183
            path: '/dev/ada2'
            phys_path: '/dev/ada2'
            whole_disk: 0
            DTL: 4100
        children[3]:
            type: 'disk'
            id: 3
            guid: 8951831291888754281
            path: '/dev/ada3'
            phys_path: '/dev/ada3'
            whole_disk: 1
            DTL: 1330
--------------------------------------------
LABEL 3
--------------------------------------------
    version: 28
    name: 'DATA'
    state: 0
    txg: 9766163
    pool_guid: 9522190198077791104
    hostname: ''
    top_guid: 252819746167074421
    guid: 8951831291888754281
    vdev_children: 1
    vdev_tree:
        type: 'raidz'
        id: 0
        guid: 252819746167074421
        nparity: 1
        metaslab_array: 23
        metaslab_shift: 31
        ashift: 12
        asize: 8001576501248
        is_log: 0
        children[0]:
            type: 'disk'
            id: 0
            guid: 2027052211370973537
            path: '/dev/ada0'
            phys_path: '/dev/ada0'
            whole_disk: 0
            DTL: 4102
        children[1]:
            type: 'disk'
            id: 1
            guid: 764366381652363129
            path: '/dev/ada1'
            phys_path: '/dev/ada1'
            whole_disk: 0
            DTL: 4101
        children[2]:
            type: 'disk'
            id: 2
            guid: 7379136512656866183
            path: '/dev/ada2'
            phys_path: '/dev/ada2'
            whole_disk: 0
            DTL: 4100
        children[3]:
            type: 'disk'
            id: 3
            guid: 8951831291888754281
            path: '/dev/ada3'
            phys_path: '/dev/ada3'
            whole_disk: 1
            DTL: 1330

when I run:

Code:

zdb -e -bcsvL 9522190198077791104

or

Code:

zdb -e -bcsvL DATA

I get the following error message:

Code:

zdb: can't open 'DATA': No such file or directory

I hope it's a something simple to get zdb working at this point, but I'm at loss at what to try.
Any help please?

 

[cyberjock]

Sorry but I don't have anything. If it crashes every OS you've tried that's naturally not a good sign.

Have you tried to do rollbacks of your zpool?

What exactly happened that lead to you starting this adventure?

I assume you don't have backups since you are going to such lengths to recover the data...

 

[stangri]

Have you tried to do rollbacks of your zpool?

If you mean "import -F" then yes, I have tried it, still crashes everything. If there's any other way to rollback transactions, then I don't know about them so please do advise.

I do have backups of most of the important stuff, but I'm not mentally prepared to give up on all of it, it was about 86% full with 4x3Tb raidz pool.

As for how I came to this point -- I've been running FN7 for years but decided to try N4F9 (for better acpi support of my particular box within newer version of FreeBSD) and after about a week of it, it crashed and kept crashing upon reboot when trying to bring pool online.

I also have tried switching to a different box (exactly same model/config tho) and trying to get yet another, completely different box to try.

 

[cyberjock]

I'd recommend you go to the Nas4Free forums and see if you can get help there. They are likely to have a better understanding of what is going on with your zpool than anyone here.

I believe that the rollback is -X. I can't remember if it works on FreeNAS or not and I have no clue about NAS4Free. I experimented with it for about a week but decided I liked FreeNAS more.

 

[stangri]

I'd recommend you go to the Nas4Free forums and see if you can get help there. They are likely to have a better understanding of what is going on with your zpool than anyone here.

I've tried, one of the first suggestions I've got from a moderator there was to run fsck. ;) So I came here.

I'll try -X and post here how did it go.

 

[paleoN]

I hope it's a something simple to get zdb working at this point, but I'm at loss at what to try.

You will have better luck with zdb & mfsBSD.

 

[stangri]

Still same issue with zdb:

Code:

mfsbsd# zdb -e -bcsvL 9522190198077791104
zdb: can't open 'DATA': No such file or directory

Still kernel panics with:

Code:

mfsbsd# zpool import -F -f -o readonly=on -R /mnt DATA

OK, now with the mfsbsd-9 there's a little progress:

Code:

root@mfsbsd:/root # zdb -e -bcsvL 9522190198077791104
Assertion failed: (offset < dn->dn_datablksz (0x200 < 0x0)), file /usr/src/cddl/lib/libzpool/../../../sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dbuf.c, line 469.
Abort (core dumped)
root@mfsbsd:/root # zdb -e -bcsvL -AAA 9522190198077791104
Bus error (core dumped)

But as zdb doesn't fix anything, still kernel panics with:

Code:

root@mfsbsd:/root # zpool import -F -X

 

[cyberjock]

What is the output of zpool import?

 

[stangri]

In zfs debug mode it lists information about all 4 drives in the pool before causing kernel panic, in the normal mode it just causes kernel panic before giving any output.

 

[cyberjock]

what? A regular "zpool import" that should list pools that are importable and not actually import anything panics?

I'm talking "zpool import" with no extra parameters or anything.

 

[stangri]

Ah, right.

No parameters just causes kernel panic (just tried in the debug mode in mfsbsd).

 

[cyberjock]

What about in FreeNAS? I'm not too familiar with mfsbsd and I'm not sure if there's any differences between mfsbsd.

 

[stangri]

I'm pretty sure I've tried zpool import without any parameters in FreeNAS 8 as well with the same result.

 

[cyberjock]

darn. I'd try doing a "zpool import" with each disk missing in sequence to see if you have lucked out and one of the disks is corrupted and causing the problems.

What is the exact configuration of your pool? 4 vdevs of 4 disks in RAIDZ1s?

 

[stangri]

First of all -- thanks, I appreciate all the help and suggestions.
I have already tried first just booting FreeNAS 7 with each drive removed separately, still kernel panics. I have also tried importing the pool with each drive removed, but haven't tried zpool import with no parameters. Will try that later tonight.
Yeah, it's 4 disks in raidz.

 

[cyberjock]

Just a note.. all of the commands I'm recommending should be performed with FreeNAS 8.3.0 or 8.3.1. I wasn't around for FreeNAS 7 and I'm not familiar with how it behaves. I have gotten alot of experience with recovering zpools in FreeNAS lately as many people have made alot of big mistakes. If you don't have a USB stick with FreeNAS 8.3.0 or 8.3.1 I'd appreciate it if you could make one for the purposes of my requests ;) 8.3.0 and 8.3.1 seem to work better for recovering zpools than the older versions.

 

[stangri]

Yep, I've switched to a FreeNAS 8.3.1 usb stick yesterday when my attempts to recover the pool wit FN7/N4F were fruitless. Now I'm thinking that maybe if I switched to FN8 a while ago I wouldn't have had this problem in a first place.

I will try zpool import with just 3 out of 4 disks with FN8.

 

[stangri]

Just tried FreeNAS 8.3.1 in a totally different box with three drives only (first without #4, then without #3). Both times zpool import (no parameters) caused fatal traps.

 

[cyberjock]

Darn. Sorry, I don't have any other ideas. :(

Sent you a PM.

 

[stangri]

I was suspicious it's the lack of RAM so I've tried "zpool import" with the FreeNAS 8 ran from the usb stick in my friend's computer which has 16Gb RAM -- same thing: kernel panic.