RAIDZ Won't rebuild on single drive fail

[jovialwoodchuck]

Hi all, I've been using FreeNas for almost a year now, and it's been great. I started out with 4 2TB drives in RaidZ, then awhile later added a 3TB drive and used the GUI expand method. This worked perfectly, and raised me from 5.3TB available storage to around 8TB. That was a few months ago, and everything has been working great.

It's currently running in an old desktop I had, hooked directly to the SATA ports on the motherboard. Yeah, not the fastest thing in the world, but it works. I recently decided to upgrade to some better hardware, so I transferred it to a new case setup. During that process, through general stupidity on my part, the newest drive (3TB) was dropped, and is no longer usable. No problem right? This is RaidZ, so a single drive failure won't be an issue? Wrong. I got everything set back up, and it would not import. I've tried everything I can think of, but no go.

I've since hooked everything back up in the original case/hardware setup, in case the new hardware was the issue, but I'm still having the same issue. The only thing I'm not positive on is which drive was hooked to which SATA port, but that shouldn't matter right?

My critical data is backed up elsewhere, but there's still a great deal on here I'd prefer to not lose if possible. Any input would be greatly appreciated. I've included a screenshot of what zpool import does.

 

[cyberjock]

Ok, if you started off with a RAIDZ of drives and then later added a single drive, you had no redundancy from that single drive failing. Read my presentation(link in my sig) and you'll understand better. This is a common noobie mistake, and unfortunately there is no "fix" except to create a new zpool and restore from backups(assuming you have backups). In your case, it sounds like you have backups, but not everything that you'd like to have. I actually wrote my presentation hoping other people wouldn't lose their data this way. It's a lame mistake that can be easily avoided, isn't always known if you didn't do all of your homework, and the consequences can be disastrous.

Which drive was hooked up to which SATA port doesn't matter so long as the controller its attached to actually works in FreeNAS. If you have an Intel and Brand X and Brand X isn't compatible and you move a hard drive to it, then you'd have problems. But if Brand X also is compatible it doesn't matter at all.

Also, -f forces a mount, which is borderline irresponsible under most circumstances and shouldn't be done until every other possible option has been exhausted. I generally tell people that when they're ready to click the "create new zpool" and kiss their data goodbye, then try -f. Because -f can be just as devastating to any data that might still exist. In your case it won't mount without that missing disk anyway so I don't think any harm was done. If the "missing" disk is being detected on boot up and is in good working order then the partition information on the single drive may have been corrupted leading to the disk being "lost" from the zpool. Again though, there is no fix for that problem you since the disk has no redundancy.

Can you post the output of the following with [code][/code] tags from an SSH session as root?

Code:

zpool status -v

camcontrol devlist

glabel status

gpart show

Don't go crazy doing anything silly yet. There still may be a chance to get your zpool working.

 

[jovialwoodchuck]

Thanks very much for the information. Back when I set this up, I tested the redundancy, and it worked great. I didn't realize that expanding the RaidZ would remove that redundancy, so stupid me for not discovering that ahead of time.

The drive that failed doesn't even recognize in BIOS anymore, and makes the lovely click of death sounds, so it's completely fried. I suspect that means I'm hosed with any possible recovery, but I've pasted the output of your requested commands below, just in case. While my absolutely critical stuff has dual backups, I just discovered that the backup for some of the still-fairly-important stuff is corrupt (sigh), so I'm very interested in recovering whatever I can, even if it's a limited recovery.

Thanks again.

 

[jovialwoodchuck]

As a side note, I am willing to pay or purchase software to recover what can be recovered. I see that there are commercial software packages available that do this for most other types of Raid, but do you know if there's anything available for RaidZ?

 

[cyberjock]

Ok. I checked out your post. It does look like you are probably screwed. If you really want your data back you could contact one of those data recovery services, but expect to pay about $2000 at the minimum.

I'd wait until you get a response from one of the other senior guys before giving up all hope and reformatting the drives. I could be missing something or there may be other options you can try.

The adding a single drive thing has bitten A LOT of people. Too many people make the jump to RAID5 = RAIDZ1, and it pretty much does. But there was never any guarantee for OCE(online capacity expansion). I remember 10+ years ago if you had a RAID controller that even supported OCE you were very very lucky and spent alot of money on the RAID card. Today everyone pretty much has it. One person(and this shocked the heck out of me) posted and said he added a single 3TB disk to his RAIDZ1 and was shocked to find that despite his zpool being fairly full it took less than 10 seconds to add the disk and that it couldn't have possibly reorganized his data that fast. By noticing something I wouldn't expect many people to notice he saved himself alot of headaches later.

Not sure how you do your backups, but I highly recommend you read up on ZFS replication and snapshots. It is so handy for backups with zpools I can't even explain it. If you use it you get the reliability of ZFS for your backups and can have as many snapshots as you can so long as you have enough space on your backup drive/zpool.

 

[jovialwoodchuck]

Ok, so let me just make sure I understand what happened correctly:

(1) I started with 4 disks in RaidZ, which *did* give me a redundancy where 1 drive could fail.

(2) I added another disk, and used the FreeNAS GUI expand option. If I understand correctly, this striped them, in Raid 0?

All I've done since then is add data to the array. Could this conceivably mean that the original RaidZ array is more or less intact, if I can find some software that will access it? If so, would you happen to know of any that might possibly work? I've found several software packages that work in Raid 0, 5, etc, but none have RaidZ listed.

 

[cyberjock]

Step 1 is correct. But when you added the extra disk you added it as its own vdev. If you lose a vdev due to insufficient replicas then you lose the zpool attached to it. In your case, you lost the 5th disk since it was added by itself. One way to look at it is a RAID0 of the two vdevs.

Once you click add, even if you add no more data, then realize you made a mistake seconds later, it is already too late. Once its added to the zpool the file system requires both the RAIDZ1 plus the single disk to work. Now that the single disk has failed you have no way to mount the zpool because you are missing a vdev.

AFAIK, and other people have not been successful either, there is no recovery tools for ZFS at all. I think this is because ZFS is an enterprise class solution. In an enterprise situation you'd have religious backups and you'd be better off creating a new zpool and doing a backup restore than trying to do any kind of data recovery(this can be time consuming) and there is no guarantee that the data will be intact. In essence, I think there is no tool because there is almost no market for one.

Your only 2 options are to not do recovery and accept the data loss or call a company that will disassemble your drive and recover your data. AFAIK there aren't any companies that actually recover ZFS drives, but if you ask them to do a sector copy from the old drive to a new drive you have a chance that you will be able to mount your zpool after you plug in the "new" drive. Depending on the damage to your old disk you may not even be able to recover any data. For instance, some drives have had head crashes that scratch the disk platters to hell and obliterate large quantities of data. These companies(from personal experience) usually charge a minimum fee(often $1500 to $2500 USD) regardless of if any data is recovered and then an additional fee if they recover any data.

When I was considering FreeNAS as a storage solution at home I actually considered the possibility of recovery if something went horribly wrong. In my mind I took at as "NTFS = chance for recovery and ZFS = no chance at all".