Need a Freenas expert to help get us out of this mess

[Durkatlon]

Does someone know what exactly leads ZFS to declare a disk as "Failed"? Can a ZFS RAIDZ operate properly if more than 1 of the constituent disks has a significant number of bad blocks on it (in the sense of having bad CRCs, not physically defective, i.e. the result of a sector level copy from a mechanically failed disk)?

By the way, if Joel's going to go the clone route, I would suggest doing any and all recovery attempts on the clones, stashing the original drives away. You don't want to give the recovery folks an excuse to blame failure to get to your files on anything you did to the drives after they were done with them.

 

[joelsawyer]

FYI, I scrounged up 4 spare 2TB Hitachi drives and I'm now cloning each disk with ddrescue. (just started disk 1, so I've still got some waiting to do)

When I get all four drives cloned, what should my next step be? 'zpool import -f'?

 

[ProtoSD]

what should my next step be? 'zpool import -f'?

That's what I would try and depending on how that turns out a 'zpool scrub'. The scrub should go over all the data comparing checksums etc. and doing error correction. If you just try and copy data without a scrub you might just get garbage, or a mixture of something that appears ok, but corrupt where you may not notice it. The scrub will tell you more specifically where there are problems, but it can also take about as long as cloning a disk.

 

[joelsawyer]

Ok. I'll let you know when I get to do a 'zpool inport -f' and report on what I get back.

 

[Milhouse]

Ok. I'll let you know when I get to do a 'zpool inport -f' and report on what I get back.

Make sure you try the above command on your *scrounged* (cloned) disks and not your originals (which should be sitting in a safe place). :)

 

[ProtoSD]

Also remember when you do a 'zpool import -f' from the command line you need to manually create a mount point (mkdir /mnt/tank or whatever the name of your pool was).

 

[joelsawyer]

I successfully cloned all four drives. Stuck them back in, booted up to using FreeNAS 8.0.1 RC2. Ran 'zpool import -f' from the command line, but got the same error message as before.

What should my next step be?

 

[ProtoSD]

Joel, that's another big disappointment. Here's what I would try. I haven't used this or tried it myself, but the the "Solaris Express 11 Live CD", there's an x86 version part way down the page in the first link below. The second link tells you a little bit about what it is/does.

Like a mentioned toward the beginning of this thread, there was a mention of some recovery tools on that CD. It also may be able to import where FreeNAS can't. That's where *I* would begin, hopefully some of the others following this thread will have some other suggestions.

http://www.oracle.com/technetwork/server-storage/solaris11/downloads/index.html

http://download.oracle.com/docs/cd/E19963-01/html/820-7799/explore.html

EDIT: Sorry, you'll have to sign up for a ridiculous Oracle account before it'll let you download. It's just a hassle, but you can still download it. I'm grabbing it now to see if I can give you some tips on using it.

 

[ProtoSD]

Well, I've been playing around with the Solaris Express Live CD and I'm sorry to report I can't find any type of recovery tools. To make things even worse when I try to import the four disks from my test pool I get a 'core dump' and it doesn't seem to find my pool.

I hate to lead you down the wrong path, but unless there's a Solaris expert that can jump in I don't think it's worth wasting time on.

My next idea might be to try a newer version of FreeBSD which has much more recent ZFS tools. I don't know, does anyone know if there's a FreeBSD live CD? I'll have to take a look.

One other thing you might try with FreeNAS is doing a 'zpool clear -f' and then trying the 'zpool import -f' again...

EDIT: And yet another bad suggestion, it doesn't look like the FreeBSD Live CD has ZFS. Probably need a full install of FreeBSD to get the ZFS stuff.... !#$%

 

[Durkatlon]

The Installation DVD can be booted into "Fixit" mode. You can then issue a couple of commands to get ZFS loaded into the kernel. See:

http://forums.freenas.org/showthread.php?1716-Retrieve-Data-Issue&p=7937&viewfull=1#post7937

 

[joelsawyer]

protosd, thanks for checking those options out for me. I was just about to load Solaris and fumble about.

Durkatlon, do you suspect that FreeBSD's fixit mode will have any chance of recovering my pool?

 

[Durkatlon]

Durkatlon, do you suspect that FreeBSD's fixit mode will have any chance of recovering my pool?

To be quite honest I don't think any of us here on the forum will be able to help you get your data back. If it were me I would try to get my money back from the place you spent it at, and look for someone with deep ZFS knowledge to see if they have any tools that might be able to retrieve useful data off your drives. I would think there would have to be forensic tools, but they are probably not easily available.

 

[survive]

Hi joelsawyer,

There is a freebsd boot CD that has ZFS v28 available here:

http://mfsbsd.vx.sk/

It might have better luck importing your volume, just be sure you don't update the ZFS version if you get lucky!

-Will

 

[joelsawyer]

Thanks for the tip, survive. I downloaded and burned 8.2-RELEASE-p2 i386 with ZFSv28, booted it and ran 'zpool import -f'. Sadly, it gave a similar error.

 

[freeflow]

Hi

Have you taken time to read up on the zpool import command. There are a number of options you can try, e.g. the -F and the -D flags. But I do suggest you take some time to do the reading first. Just google 'zpool import man'.

 

[joelsawyer]

freeflow, I appreciate the subtle nod to RTFM. I did read the manual, but since I'm far from an expert on these sorts of things, the only flag that seemed appropriate for me to run was -f (which I've tried several times and on various OSes). I even tried the -d flag as you mentioned, but it says there are no pools available to import (because to my knowledge, no pools were ever deleted).

 

[Durkatlon]

Are you getting any more support from the data recovery place?

 

[joelsawyer]

no, I'm unable to reach them. Last I talked to the rep was on Monday, and he said he's still waiting to hear back from their development team to see if they can build a tool to extract the data.

 

[ProtoSD]

Taking a look at that screenshot makes me confused. I thought you had FOUR disks, that screenshot only shows THREE. The missing disk could make all the difference.

 

[freeflow]

Not -d, -D