Need a Freenas expert to help get us out of this mess

[joelsawyer]

I'm an idiot who didn't make a separate backup. Now that that's out of the way, I need some help getting our FreeNAS server working.

A few weeks ago, we had two drives out of a four drive array die on us. Our FreeNAS box had been working flawlessly for 6 months prior.

After talking with several local IT shops (none of which were familiar with FreeNAS), we determined that there were multiple drive failures. We sent all four drives in for data recovery. $6000 later they said that 98% of the data has been recovered and transfered to two new drives.

We received all four "good" drives today and was told all we had to do was stick them back in and fire up FreeNAS. The problem is it doesn't work. It's unable to mount our original volume. Looking at the GIU, I see this error when going to the Storage section:

"Error getting available space"

Can anyone help? I have no clue what to do next. Is it possible to get this volume running again? Willing to pay for some support from someone familiar with FreeNAS 8.

 

[ProtoSD]

What exact version of FreeNAS are you using?

This problem has been covered here and you should be able to get your volume/data back assuming that the data recovery place really did recover everything and the data/partition structure has been preserved.

I would use the latest version of FreeNAS, 8.01 RC2, and just use the 'Auto Import' from the GUI. You could try importing it from the command line, but it will complain about the mount point being missing and if you create that manually it will be gone after you reboot. There are problems with the Auto Import on version 8, as you discovered, even with the betas. RC2 is the best way to go. Just start with a new flash disk (keep the old one for now), and do a fresh install from CD to the new flash drive. After you get it installed, setup the network so you can get to the GUI and just go straight to Auto Import. If that works, you can export your volume, plug in the old key (flash disk with FreeNAS) and try to import again, or just stick with RC2. It's also a more reliable way to confirm that you're going to be able to mount your volume and look for your data.

 

[joelsawyer]

I'm using FreeNAS 8.0 Release.

I'll go get a new Compact Flash card, install 8.1 RC2, boot into the GUI and press "Auto Import". My complete lack of understanding how this works made me wary of clicking any of those buttons for fear of ruining the whole lot.

The only think I'm unclear on though is... "plug in the old key". What key?

 

[ProtoSD]

"plug in the old key". What key?

Sorry, I assumed you were booting from a USB flash drive (key). Compact flash is fine, or if you have a USB flash drive around (2GB or more), you can use that.

Are you using a 64bit processor / AMD64 version of FreeNAS? (Not important, just curious)

Here's the link for 8.01, you want RC2, but if you have a 64bit processor make sure you get the AMD64 version.

http://sourceforge.net/projects/freenas/files/FreeNAS-8.0.1/

Jumping ahead and assuming the import works, it's a good idea to run a scrub. I'll cover that after the import works.

How did you have your disks configured? Raid-z1 or z2? (again not really important)
How much data did you have / how full was your volume?

 

[joelsawyer]

According to the System Information, I'm using FreeNAS-8.0-RELEASE-amd64

Am I correct in assuming that once I do a fresh install, and the AUTO Import Volume works, that I should be able to just pull all my data off?

 

[ProtoSD]

Am I correct in assuming that once I do a fresh install, and the AUTO Import Volume works, that I should be able to just pull all my data off?

Yes, but...

It's probably ok to login to the console and 'cd' to where your data is and have a look, but before I pulled any data off I'd do a scrub which will verify all of your data and structure etc. is ok.

I'll stick around here and watch for your replies and when we get to that point I'll tell you how to do the scrub after you have a look at your data.

 

[joelsawyer]

I installed FreeNAS 8.0.1 RC2 on a new CF card, plugged in all four drives, booted it up, went straight to the GUI, clicked "Auto Import Volume" under Storage and got this error message:

"Sorry, an error occurred"

Now what?

 

[joelsawyer]

Oh, looking over to the monitor plugged into my FreeNAS box, there's a message there that might give a clue:

ZFS NOTICE: Prefetch is disabled by default on i386 -
- to enable,
add "vfs.zfs.prefetch_disable=0" to /boot/loader.conf.
ZFS WARNING: Recommend minimum kmem_size is 512MB; expect unstable behavior.
Consider turning vm.kmem_size and vm.Kmem_size_max in /boot/loader.conf.
ZFS filesystem version 4
ZFS storage pool version 15

 

[ProtoSD]

Gotta love the FreeNAS error messages... ;-)

Looks like we'll need to work from the command line at the console. How experienced are you at using the command line?

You get there by choosing #9 from the menu at the console.

It would be helpful if you could post the results of the command 'gpart show', however there's no easy way to copy & paste them from the console and the output could be long. That would be really ideal, but lets just try an import from the console to see if we can get a better idea of what the error was the GUI saw.

So go to the console, choose #9 on the menu, type 'su' (without the quotes) and enter your admin password. Actually if that password doesn't work, you need to go to the GUI and change the password for 'root' and then go back to the console and try the 'su' again with the password you just set.

If it works, you should see a '#' prompt.

Type 'zpool import' and post back the results. I need to leave in about 1/2 hour and will be gone for over an hour. Lets see what we can discover before that.

If anyone else is following the thread, please feel free to help out.

 

[ProtoSD]

Something you said makes me wonder:

We sent all four drives in for data recovery. $6000 later they said that 98% of the data has been recovered and transfered to two new drives.

Did they possibly just recover the data from all 4 of the disks and put it on the 2 new ones? Probably not, but it wasn't clear.

 

[joelsawyer]

What I was told was that they recovered data from just two of the four drives. They haven't been really clear on what exactly they did.

 

[joelsawyer]

I'm pretty basic when it comes to command line stuff. Here's a screenshot of what I can see from doing a 'gpart show':

 

[ProtoSD]

The output of the 'gpart show' command would be really helpful and help answer what they did a little better.

Do you have a FAT or FAT32 formatted USB flash disk?

If you can put that in, we can mount it and dump the output of dmesg etc. to that, then you can dismount it, pull it out and put it in your windows machine or whatever, and copy the output off the key and paste it here.

I need to leave now, sorry to have to stop for now.

If anyone else out there wants to help, please jump in and I'll follow up when I get back.

 

[ProtoSD]

Wow, ok, great, you must have posted at the same time as I was. Great screen shot, things look ok. What happens when you do the 'zpool import' (no quotes)?

 

[joelsawyer]

Here's what zpool import comes up with:

 

[Durkatlon]

I am highly skeptical that they gave you back 4 drives that together form a FreeNAS array. You probably got the 2 original drives that are not broken, and then 2 more drives that contain your data (in non-redundant form). You should contact the data recovery shop to see what exactly they did.

 

[joelsawyer]

I'll get on that and ask them exactly what they did.

 

[ProtoSD]

I'm was a little skeptical like Durk, but the screen shot from 'gpart show' shows a partition table like FreeNAS/ZFS would create, not just a flat out single partition with data dumped to it. It would be nice to know exactly what they did before doing anything else.

From the Sun ZFS website:

Pool metadata – ZFS requires a certain amount of data to be parsed to open a pool and access datasets. If this data is corrupted, the entire pool or complete portions of the dataset hierarchy will become unavailable.

If the damage is in pool metadata that damage prevents the pool from being opened, then you must restore the pool and all its data from backup.

 

[joelsawyer]

Still waiting to hear back from Kroll (the company that did our data recovery), but I have a question about their "Plan B" they offered. When asked if we could get the data from these drives returned back to us in a no-RAID format (just the various files on one disk), they said they couldn't do it and that it would require them to custom build a "tool" to do so and hinted at a cost in the $10,00+ range.

Is that normal? Is there no other way to extract the data off of these drives?

Hoping to hear back from them tomorrow specifically about your question, Durkatlon.

 

[joelsawyer]

If the damage is in pool metadata that damage prevents the pool from being opened, then you must restore the pool and all its data from backup.

Is this referring to snapshots that FreeNAS seemed to automatically be generating?