Drive Failure Question

[Magneto417x]

So I built a brand new FreeNas with 6 drives set-up in Raid Z1. I moved (cut and pasted) some data from old FreeNas. Yesterday the horror happened I had a double drive failure on new box and now I'm left with wondering is there any hope of recovering the data. Any help would be be much appreciated.

 

[mav@]

With two drive failed in RAIDZ1 you already have incomplete data. Each ZFS data block already lost 1/5th part of it. What's left are random bits and pieces, which ZFS itself won't even try to use.

Didn't you have any snapshots on the old FreeNAS to restore deleted data there?

 

[Magneto417x]

Unfortunately I didn't have any snapshots which is why I'm in such a pickle. I'm hoping since no data was written to the old that it may be possible to recover it somehow.

 

[mav@]

Theoretically it may be possible. ZFS is copy-on-write, and each transaction group is written in the new place, while there is a cyclical list of the root pointers. AFAIK there is possibility to import pool in recovery mode, reverting back on some number of transaction groups, though I am not sure how far back is that possible if system wasn't shut down immediately after delete. I've mostly heard that method to be used in case of pool corruptions, where working state is somewhere not so far back.

 

[Magneto417x]

The delete was performed on Friday.

 

[mav@]

Modern FreeNAS stores internal system dataset with periodic stats on the main data pool, so there is always some writes. I'm afraid that at least root pointers were overwritten long ago if system was running all the way since Friday. That probably makes recovery much more problematic, if ever possible.

 

[Magneto417x]

This is most depressing news :-(

 

[mav@]

You may try to google recipes like this: https://lists.freebsd.org/pipermail/freebsd-hackers/2013-July/043131.html , but on my systems all transaction groups I can see are within only several minutes.

 

[Magneto417x]

I still cant believe I had 2 drives die at the exact same time :-( I will never buy Seagate again

 

[Dice]

Sad news.

I've a couple of questions on the situation before this event occured.
1. What hardware are you running?
2.Did you perform the adviced burn-in procedures?
Such as:
https://forums.freenas.org/index.php?threads/building-burn-in-and-testing-your-freenas-system.17750/
and
https://forums.freenas.org/index.php?threads/how-to-hard-drive-burn-in-testing.21451/page-14
3. Did you setup SMART tests - email/notifications?
4. What was your SCRUB schedule?

 

[Z300M]

I still cant believe I had 2 drives die at the exact same time :-( I will never buy Seagate again

No matter what brand of drive you pick, there are some online reviews that say they're terrible and others that say they're wonderful. E.g., there are NewEgg reviewers who will never buy Seagate again, will never buy WD again, will never buy Hitachi/HGST again, will never buy Toshiba again (have I missed any?).

Are you sure the drives themselves have failed? What about connections to the drives from PSU and from drive controller? Did you disconnect and reconnect them before blaming the drives. When I had an apparent failure of three drives at once (on returning from vacation when the system was powered off), replacing the PSU and remaking all the connections -- both data and power -- got everything running just fine again.

 

[Magneto417x]

Absolutely certain the drives are dead. I had notifications set-up and the new FreeNas box was online and running for about 2 weeks before I moved live data to it. It was exactly 2 days after I suffered the double drive failure. I have replaced the 2 dead Seagate drives with Western Digital Enterprise drives and today after 2 days I now have the following error and not sure what to do with it exactly.

Device /dev/ada5, 4 Currently unreable (pending) sectors

 

[Dice]

Device /dev/ada5, 4 Currently unreable (pending) sectors

that is definitely another drive to replace...

 

[Magneto417x]

That's a brand new drive I cant believe this run of terrible luck.........

 

[Dice]

HAve a look at the "burn in" thread.
I'd be really keen to follow those steps in detail if I was in your situation..

 

[Z300M]

That's a brand new drive I cant believe this run of terrible luck.........

If you have *apparent* failures of three drives from two different manufacturers, I suggest that you check the power supply, disk controller, and cables.

 

[Dice]

I suggest that you check the power supply, disk controller, and cables.

Which is the implicit aim of he burn in methodology, in the shadow of stressing hdds.