Need a Freenas expert to help get us out of this mess

[ProtoSD]

Is this referring to snapshots that FreeNAS seemed to automatically be generating?

No, it's referring to the entire pool / *everything*. Here's a link:

http://dlc.sun.com/osol/docs/content/ZFSADMIN/gbbwl.html

If you were at the point of throwing in the towel and didn't want to pursue further data recovery, you could *try* the 'zpool import -f' and if that worked, do a 'zpool scrub', but that's a one-way path which could either make things worse, or *possibly* get you back online.

Was your original pool a Z1 or Z2?

 

[louisk]

Just wondering, was the old freenas box using ZFS or UFS? Your original post didn't make it clear, and it will effect the data recovery process.

 

[joelsawyer]

it was ZFS

 

[joelsawyer]

If you were at the point of throwing in the towel and didn't want to pursue further data recovery, you could *try* the 'zpool import -f' and if that worked, do a 'zpool scrub', but that's a one-way path which could either make things worse, or *possibly* get you back online.

Thanks, I did notice those commands while poking around today. I don't think I'm to that point yet. We've already put $6000 to get to this point. I want to be positive there's not other alternative before I pull the trigger.

 

[ProtoSD]

Joel,

I found a mention of something that could be an option in this post. It involves importing your pool as 'read-only' on a Solaris Express system. http://mail.opensolaris.org/pipermail/zfs-discuss/2011-January/046933.html

This link is interesting also:
http://utcc.utoronto.ca/~cks/space/blog/solaris/ZFSLosingPoolsWays

Are there any Solaris users out there that can confirm this possibility? @Jgreco, are you around?

 

[joelsawyer]

I'm completely in the dark when it comes to Solaris. Is this an os that I would install on another machine? or on the same box?

 

[ProtoSD]

I'm a little in the dark with Solaris also, that's why I yelled out to @Jgreco, I think he's pretty familiar with it, though I'm not sure. I know you can download the installation CD, but it would need to be installed to a hard drive. I'll run on a regular PC, you could get a spare hard drive and install it on that in your troubled system. The version of ZFS in FreeNAS is a bit older than in Solaris, but that post implied there were some tools that *might* help you. I know you're in a desperate situation, so I was just trying to find some possible options.

 

[joelsawyer]

Trust me, I'm deeply grateful for any help you can give!

@Jgreco, any idea if this will work?

 

[Durkatlon]

If this is mission critical data and you've already tossed thousands of dollars at the problem, I would not go boot into an unknown operating system and execute esoteric commands in hopes that you can get it to work. I would contact the recovery place and ask them if you can send the drives AND the enclosure to them and have them put back a working box for you.

Also, you probably should soon start thinking through a strategy to prevent this from happening again. Remember: "RAID is not backup". Words to live by.

 

[Milhouse]

Was your original pool a Z1 or Z2?

Looked like Z1 in one of the screenshots, this would also corroborate the catastrophic data loss with 2 out of 4 disks failing.

Would OpenIndiana serve as an alternative to Solaris, I think this will install to USB though as Durkatlon sagely suggests - exercise caution.

Perhaps if joelsawyer has 4x 2TB disks spare (or can buy them - a small sum compared to what has already been spent) then dd'ing (cloning) the content from the recovered disks would be a good idea then all (potentially) destructive actions can be performed on the cloned disks - if it all goes horribly wrong, just re-clone the data from the originals and try something else...

 

[ProtoSD]

Milhouse, I was tempted to suggest the same thing using ddrescue. Of course after spending that much money I would be cautious too. I'm not sure if joelsawyer is tech savvy enough to try that, but it would sure beat throwing more money to a place so eager to take it.

 

[Durkatlon]

Yes, if I paid 6 grand to a place like that I'd want my FILES back, not a collection of drives that you can maybe piece back together into a working system. I can't understand what they did to "recover" the files, if they cannot give you a bunch of new disks with the files on them. So far you've paid them an exorbitant sum and have gotten nothing.

 

[Milhouse]

Milhouse, I was tempted to suggest the same thing using ddrescue. Of course after spending that much money I would be cautious too. I'm not sure if joelsawyer is tech savvy enough to try that, but it would sure beat throwing more money to a place so eager to take it.

Yes, but talking joel through the required commands should be easy enough considering he has a working FreeNAS system - just need to make sure he gets his source and destination disks correct!

Without cloned disks he's looking at a one-shot deal whatever course of action he takes - even sending them away for further data recovery - and given the money spent so far that doesn't sound like being an acceptable risk. So the first thing I would do is clone those disks...

 

[joelsawyer]

Thanks for the suggestions guys.

I finally got a call from a rep at KrollOnTrack today. He confirmed that 2 drives mechanically failed and that they did a sector copy of the recovered data onto two brand new drives.

I also voiced my frustration that I was originally told we would be getting FILES back, not a bunch of drives still in an array. His response was that if this were a "normal" RAID configuration (like RAID5 or RAID10) that it wouldn't be an issue, but that because it's RAID-Z, it's something they haven't run across before so would have to reverse-engineer a tool to extract and reassemble the actual files.

If this is the case, is there another data recovery place that HAS dealt with RAID-Z before and has a tool to simply extract the data?

As for cloning the disks... I've never done it before, but I'm confident that with the right guidance I can make it happen. We already have three 2TB disk at the office that I purchased on the day we discovered the failure. I attempted to replace three of the drives I thought had died, hoping to rebuild the array... but obviously that failed. So buying one more 2TB drive isn't a stretch. What software would you recommend I use to clone the drives? I'm assuming this is something I'll need to run from my main computer and not the FreeNAS box.

 

[Milhouse]

I finally got a call from a rep at KrollOnTrack today. He confirmed that 2 drives mechanically failed and that they did a sector copy of the recovered data onto two brand new drives.

Quite possibly had to remove the platters from the mechanically dead disks and install them in donor drives using a clean room. In which case I wonder why they could only recover 98% of the data, unless the platters were also damaged?

I also voiced my frustration that I was originally told we would be getting FILES back, not a bunch of drives still in an array. His response was that if this were a "normal" RAID configuration (like RAID5 or RAID10) that it wouldn't be an issue, but that because it's RAID-Z, it's something they haven't run across before so would have to reverse-engineer a tool to extract and reassemble the actual files.

That's a really, really poor excuse from Kroll - I find it hard to believe Kroll haven't "run across" ZFS before, considering it's likely to be fairly widely used in enterprises, ie. their bread & butter business (or so I'd imagine).

As for cloning the disks... I've never done it before, but I'm confident that with the right guidance I can make it happen. We already have three 2TB disk at the office that I purchased on the day we discovered the failure. I attempted to replace three of the drives I thought had died, hoping to rebuild the array... but obviously that failed. So buying one more 2TB drive isn't a stretch. What software would you recommend I use to clone the drives? I'm assuming this is something I'll need to run from my main computer and not the FreeNAS box.

You should be able to use your FreeNAS system, you just need console access. What you need to do is identify the disks - I would suggest connecting one "original" disk (source) and one "new" (destination) at a time, boot the system then use "dmesg" to check which is which (or run "smartctl -a /dev/daX" to obtain the serial number for each disk).

Obviously, correctly identifying the source and destination disks is of paramount importance - if you're not sure, do nothing and ask here!

So, once you have identified the disks, you just need to use the dd command to transfer the data, eg:

Code:

dd if=/dev/daX of=/dev/daY bs=1M

where /dev/daX is the source and /dev/daY is the destination. The cloning of each disk will take several hours, possibly up to 12 hours for 2TB disks, but at least you'll have your original disks as backups should it all go wrong.

 

[Durkatlon]

How is a set of RAID component drives that have only been 98% recovered going to ever form a working array again?

 

[Milhouse]

How is a set of RAID component drives that have only been 98% recovered going to ever form a working array again?

Admittedly it is a grim starting point, I suppose it all depends on whether the missing 2% data is crucial to the sanity of the array (which, unfortunately, looks to be the case so far). Or maybe ZFS has some magic that will allow the array to be imported under duress (isn't metadata dual redundant? Although in a worst case/bad luck scenario the user could lose both copies...).

Maybe it will import with "-f", and all of the data will be readable. Or none of it. Or somewhere in between... :)

 

[ProtoSD]

Assuming after cloning he's able to import with -f, wouldn't a scrub be the first thing to run?
I thought scrubbing supposedly compares / reconstructs data using some ZFS algorithm?

How is a set of RAID component drives that have only been 98% recovered going to ever form a working array again?

Like I just asked, I thought this was the beauty of ZFS... That missing 2% could be in one spot (bad), spread out enough to allow the parity etc. to rebuild the data, or in an unused area of the disk. It seems likely that even if -f works, there could be some missing or corrupt files.

Although dd will work great to clone the disk and is the easiest to guide Joel through, if you don't know about ddrescue (there are 2 versions, one of them is better/newer), it's nice because you can interrupt the clone and resume it. It's also nice because if it hits a spot it has trouble reading, it skips it and goes on copying (logging the bad spot), and then comes backs later and tries to re-read trouble spots. There's more to it than that, but it's a great tool if you haven't tried it.

 

[Milhouse]

Hopefully the data on these non-failed (x2) and new (x2, courtesy of Kroll) disks should be 100% readable, even if 2% of it is garbage! :)

If the OP wants to take the ddrescue route, booting from the SystemRescueCD would probably be the best approach (and it's always a handy LiveCD to have lying around!)

ddrescue documentation is here.

 

[joelsawyer]

I think the original 98% figure may be incorrect. It what I was told initially, but the Kroll rep today mentioned that there was a 99.9% success rate when the read from the new drives, so let's hope it's the higher number.

I'll burn the LiveCD, get 4 new drives tomorrow and try to clone all four drives using ddrescue. I'll let you know how it goes.