5 drives all failed on zfs pool

[Spannerman]

Evening all, just experienced what may be a total disaster with my FreeNAS system. I hope this is the appropriate place to share and ask for advice. I've simultaneously lost 5 drives from my pool! I'm using FreeNAS 9.2, with what was a total of 16 disks in two sets of 8 with dual parity. All was well until this evening apart from a loose SATA data cable on one of my Icy Docks. This dock holds the 5 disks. So I bought a new Icy Dock and tonight swapped it out for a later model. The later model uses the SATA style power cables instead of the Molex 4 pins so I'd also bought another "suitable" cable to use with the new dock, and my P/S has a suitable spare socket. Everything swapped, powered back up, and pool status unknown due to 5 missing drives... Short version: After a while rechecking, I found my new power cable was wired internally incorrectly and seems to have fried the drive capability on all 5 of those drives, although I've only separately checked one so far as I'm now concerned about total data loss and feeling like a large whisky! It may be that some are ok, but I'm assuming not. They are quite new Toshiba 3TB drives so I plan to write to Toshiba to see if they can help in any way, perhaps by cloning the failed drives onto new ones? Of course this will take time, but I would like to try and save the data in the correct ZFS format if possible.

My question is whether anyone has had experience of this being successful or not? Or is there a better way? Or am I completely screwed? Server powered down for now whilst I go looking for the whisky. Appreciate any pointers or advice. Thanks,

 

[no_connection]

Maybe the controller board could be replaced.
Although it seems not as easy as just replacing the board.

I stumbled across these guys as they have a few interesting videos of HDD and other recovery.
http://www.youtube.com/user/hddrecoveryservices
They are quite informative at times so they might give you some advice what to do.

 

[Starpulkka]

How you checked and came conclusion of incorrect wiring? <= This really intrests me.

Over voltage cases seagates have a component that takes the hit for saving your data so perhaps its too early to start listening kill music. (still today i dont understand why oh why people need home a hot swap cages.) Ill leave drive "repair" information on some one who needs money. Edit: ok if you lucky you can kind of "bypass" that component and drive will be alive again but you dont have overvoltage fuse. But go to expert if you want to be sure. (as i would first double check your power cables and psu for not losing anymore hdd's)
^_^

 

[PenalunWil]

I had to buy an additional modular power cable x 4 Molex (type3 on the original Cosair RM450 PSU) as only one came with the PSU. I needed it as I didn't have enough molex connector for my two Icy Boxes and the case fan controller. I ordered an additional one from Scan Computers which was advertised as a Cosair Type3 modular cable ( http://www.scan.co.uk/products/cors...peripheral-cable-with-4-connectors-cp-8920112 ). I compared the photograph with the cable I had and it all looked all okay. However when I received it I noticed the 3 x 2 block at the end was wired/pinned differently from the original Cosair RM450 cable, they do not match. So I re-pinned it to match the one I had. Thank God I did by the sounds of the above thread. Surely all Corsair Type3 modular cables should be identically wired???

 

[Spannerman]

My scenario was almost exactly as PenalunWil mentions. I made the stupid assumption that the SATA power cable was wired correctly, and only discovered the variation after the disaster when comparing the pins with a voltmeter on another live power supply. First I checked a Molex style cable I had, with the SATA/Molex converters added, then I compared that to the "special" cable. Different pin patterns....!!! I really wish I'd just stuck with the Molex converters, but too late now. Anyway I will check out the possibility of changing controller boards, so thanks for the link no_connection. If its possible to do this I will post the results, and since I currently live in SW China hopefully I can find a way to source parts too.

 

[Spannerman]

For further information, here is an update on the pool, even though I still have more checks to make. As I write I have recovered my pool (wooohooo ) , apart from two drives which are not working - so only 2 of the 5 in the Icydock were unuseable. So thankful for dual parity.....:) It looks like the faulty cable was responsible for taking out the two drives, some circuit damage on the Icydock and finally that particular connection on my power supply popped a fusible link, luckily leaving the rest of the PSU functional, and probably saving the other 3 drives. So I have put in two replacement drives which are re-silvering now, whilst I decide what to do with the old units, as it would be good to have them available again, as hot spares for example.

Prompted by no_connection above I did some research online and, assuming there is no physical damage, it looks like on these drives I can replace the control boards along with swapping the original error correction ROM and theoretically the drive should revive itself. I have found a local source in China for Toshiba control boards, and 3 are on their way to me now. I'd considered doing this even before I had checked each drive individually, but had thought it was a little risky if I screwed it up. Now that the pool is functional, but degraded, I can go ahead and do this less stressfully. If I'd wanted, a company in the US (Donor Drives: http://www.donordrives.com/ ) has offered to look at the drives for me but given the distance and time factors so far I haven't done that. But they did respond to my email promptly and politely with lots of good information so I thought there was no harm in mentioning them.

Anyway the moral of the tale is: you can never be too careful - even the smallest parts can wreak havoc.... In this case two pins in a connector incorrectly positioned. I will post a final update if/when I get the drives running again, following the board swap. Thanks.

 

[cyberjock]

Nice! I've known several people that tried to swap control boards. Assuming that the control board is all that is damaged, it's not a 100% guarantee that the replacement board will work. It's very dependent on the hard drive manufacturer, the firmware of the old and new board, and the model of the hard drive. From the people I've seen that did controller replacements they only had about a 10% chance of working. Mind you, that 10% is including situations where the control board may not have been the only failed component. But neither you nor I have the ability to positively rule out failure of other components at this time anyway.

But, in the bright side, the fact that you got your data back is very lucky. Now hopefully you'll invest in backups. :D