Save some data after 2 dead HDDs in RAIDZ1 ?

[Altharion]

Hi and thanks in advance for your help,

I am in trouble after 2 dead HDDs in my RAIDZ1 on FreeNAS 9.1 (I know I should have used RAIDZ2, didn’t know 3 years ago when I built this NAS, and I will do if I recover at least some of my 10 years of data…)

I have (or had ?) a 4 disk RAIDZ1. A few days ago, one disk (ada2) was having errors, so I stopped all, bought a new disk, put it in and began resilvering from ada0, ada1 and ada3.

At 80%, yesterday evening, I’ve been delighted to see errors on ada1 as well. I bought ANOTHER disk this evening and… I don’t know what to do.

With ada0+faulty ada1+new ada2 (80%)+ada3 : freeNAS won’t import the pool because of ada1
With ada0+new disk+new ada2 (80%)+ada3 : freeNAS won’t import the pool because it has only 2 disks out of 4
With ada0+new disk+old faulty ada2+ada3 : same thing

I know that I won’t recover everything, but do I have a chance at least to get back these 80% that are on my new ada2 ? I’d like FreeNAS to stop trying to finish its resilvering, consider itself with 3 online disks (ada0, 2 and 3) and try to recreate ada1 on the new disk based on these 80%. Is it possible ?

Thanks a lot for your help

 

[cyberjock]

You are what we call "screwed". You have to finish the current rebuild before you can do another disk replacement. Now you see why RAIDZ1 is strongly discouraged and I have that link in my sig on why it is pretty irresponsible to do RAIDZ1 at all.

Your options are pretty much kiss your data goodbye or call a data recovery expert. I will warn you that you will spend about $2500 before you are even shown if recovery is likely. If they find your data is recoverable in any form you'll pay more than $20k for it. These numbers aren't made up and are first-hand numbers as I helped someone contact a data recovery expert.

 

[Altharion]

Thanks for your answer. Yes I see that now, I wish I read it 3 years ago when I built this NAS...

This is what I figured out, I have to find a way to get my ada1 working in order to finish the current rebuild. If I don't get a correct ada1, I'm screwed.

I'm currently trying to replicate my faulty ada1 on a new disk with recoverdisk, I'll see where it goes. I'll certainly not spend €20k for these data, I don't have them and it's not THAT important, just personal data.

 

[cyberjock]

Promise you'll never do RAIDZ1 again! I hate these kinds of threads. :(

 

[Altharion]

NEVER !
RAIDZ2 with 6 disks for optimal disk usage / performance, that's right ?
Of course I've just read about someone who lost 3 disks at the same time...

 

[cyberjock]

Yes. RAIDZ2 with 6 disks is optimal. :)

 

[cyberjock]

Some questions because I'm always curious as to what exactly happened when someone loses their pool:

Did you setup the FreeNAS emailing function?
Did you setup SMART monitoring? If so, on what frequency?
Did you setup SMART testing? If so, what schedule?
Did you setup regular pool scrubs? If so, on what schedule?
If you setup SMART monitoring, did you set a temperature threshold? If so, what did you set it at?
When did you first identify that the first disk was bad? When do you think it actually went bad?

 

[Altharion]

Well, I "deserved" what happened because I didn't really care about it : no monitoring, SMART testing once in a while but not scheduled, no scrubbing... Of course, I've learned my lesson, I'll be much more careful next time.

I identified the 1st bad disk last sunday when I saw an empty directory via my AFP share. When I understood the problem, I turned the server off and ordered a new disk. It may have been that bad the day before but not much earlier. But a pool scrub may have shown me way before that trouble was on its way...

Good news, the "recoverdisk" has finished. I forgot to write the progress in a log file, so because of a timeout on my SSH window I don't know how it went, but I'm sure that it has stopped properly (no process, no disk activity). I've put the system back together with my "recoverdisked ada1", FreeNAS has accepted it and the ongoing rebuild has restarted where it was. Of course there are a lot of data errors and I will probably lose some data, but I have good hope to recover most of it.

 

[DrKK]

Good luck sir. Let us know.

 

[Altharion]

The rebuild has finished yesterday !
As anticipated, zpool status -v shows a lot of errors, but most of them are .AppleDouble files (OS X Finder meta-informations) and my TimeMachine backup. I'll recreate it later from my disk, so I don't care. Of course there is still some lost data, but it's not that bad compared to what I feared 2 days ago !
I've launched a scrub this morning to try to recover some more data, still 13h to go...