Black Swan? more like evil little kitty event

[meubanks]

HP ProLiant Microserver N40L
1 x AMD Turion II N
16Gb Ram
2 x Seagate Barracuda 3Tb (1 of these since replaced with a WD Blue 3Tb drive)
2 x WD Blue 3Tb
RaidZ-1 (I think) it has been running since 2012 or 2013

Ok so you have heard this kind of problem before but ....
I have recently suffered a ELK event (kind of like a black swan event but caused by my Evil Little Kitty)
and yes that is her real name.
I went away for a week and during that period Elk managed to knock a door over onto my computer table
(picture attached) The NAS is under the door. This caused damage to my NAS Server and destroyed my backup Lacie drive as well, the black thing partially under the door.
The NAS was still powered on when I returned.

The actions I have taken are as follows.

removed all 4 drives, inserted a diagnostic cd to run memory and cpu tests on
the micro server chassis, this passed 2 days of testing.

Ada0 (Seagate) attached drive to secondary system ran SeaTools(DOS), found approx. 110 sector errors corrected.
Ada1 (Seagate) attached drive to secondary system ran SeaTools(DOS), drive not recognized in bios, drive powered up briefly before
making horrible grinding noise and powered down hasn't powered up since.
Ada2 ( WD ) attached drive to secondary system ran SeaTools(DOS), no errors found
Ada3 ( WD ) attached drive to secondary system ran SeaTools(DOS), no errors found

I attempted to boot the NAS but got the same diagnostic as below minus the newly added WD drive in Ada1

and that is basically it, I have tried not to do anything which reduces my chance of recovery since my NAS backup
was literally crushed by the door as well.

I still have my original boot sdc card, although the card reader was smashed the sdc card survived.

[root@ ~]# zpool import

Code:

																								  
   pool: vol1																											  
	 id: 4880409234482636973																										
  state: UNAVAIL																													
 status: One or more devices are missing from the system.																		  
 action: The pool cannot be imported. Attach the missing																			
		devices and try again.																									  
   see: http://illumos.org/msg/ZFS-8000-6X																						  
 config:																															
																																   
		vol1								  UNAVAIL  missing device													  
		  gptid/428ac436-c084-11e6-a318-e839352df633  ONLINE																		
		  gptid/43e80dfd-c084-11e6-a318-e839352df633  ONLINE																		
		  gptid/4a0f9062-c084-11e6-a318-e839352df633  ONLINE																		
																																   
		Additional devices are known to be part of this pool, though their														  
		exact configuration cannot be determined.

[root@ ~]# camcontrol devlist

Code:

<ST3000DM001-9YN166 CC4H>		  at scbus0 target 0 lun 0 (pass0,ada0)															
<WDC WD30EZRZ-00Z5HB0 80.00A80>	at scbus1 target 0 lun 0 (pass1,ada1)															
<WDC WD30EZRZ-00Z5HB0 80.00A80>	at scbus2 target 0 lun 0 (pass2,ada2)															
<WDC WD30EZRZ-00Z5HB0 80.00A80>	at scbus3 target 0 lun 0 (pass3,ada3)															
<Genesys CF  USB3 Reader 1206>	 at scbus7 target 0 lun 0 (pass4,da0)															
<Genesys xD  USB3 Reader 1206>	 at scbus7 target 0 lun 1 (pass5,da1)															
<Genesys SD  USB3 Reader 1206>	 at scbus7 target 0 lun 2 (pass6,da2)

[root@ ~]# gpart show

Code:

																									  
=>		34  5860533101  ada0  GPT  (2.7T)																						
		  34		  94		- free -  (47K)																					
		 128	 4194304	 1  freebsd-swap  (2.0G)																				
	 4194432  5856338696	 2  freebsd-zfs  (2.7T)																				
  5860533128		   7		- free -  (3.5K)																					
																																   
=>		34  5860533101  ada2  GPT  (2.7T)																						
		  34		  94		- free -  (47K)																					
		 128	 4194304	 1  freebsd-swap  (2.0G)																				
	 4194432  5856338696	 2  freebsd-zfs  (2.7T)																				
  5860533128		   7		- free -  (3.5K)																					
																																   
=>		34  5860533101  ada3  GPT  (2.7T)																						
		  34		  94		- free -  (47K)																					
		 128	 4194304	 1  freebsd-swap  (2.0G)																				
	 4194432  5856338696	 2  freebsd-zfs  (2.7T)																				
  5860533128		   7		- free -  (3.5K)																					
																																   
=>	  34  31116221  da0  GPT  (15G)																							  
		34	  1024	1  bios-boot  (512K)																						
	  1058		 6	   - free -  (3.0K)																						
	  1064  31115184	2  freebsd-zfs  (15G)																					  
  31116248		 7	   - free -  (3.5K)

[root@ ~]# glabel status

Code:

																									
									  Name  Status  Components																	  
gptid/49f13e80-c084-11e6-a318-e839352df633	 N/A  ada0p1																		  
gptid/4a0f9062-c084-11e6-a318-e839352df633	 N/A  ada0p2																		  
gptid/427107ff-c084-11e6-a318-e839352df633	 N/A  ada2p1																		  
gptid/428ac436-c084-11e6-a318-e839352df633	 N/A  ada2p2																		  
gptid/43d0e91e-c084-11e6-a318-e839352df633	 N/A  ada3p1																		  
gptid/43e80dfd-c084-11e6-a318-e839352df633	 N/A  ada3p2																		  
gptid/cefb5ac4-b978-11e0-a2d0-e839352df633	 N/A  da0p1  

That is pretty much it I have booted (on a separate sdc card) to 9.3 but other than that
I have been too cautious to do anything else.

Should I post this elsewhere or just give up now? :D

 

[rs225]

You can start with zpool import -f vol1 and if that doesn't go zpool import -f -F vol1

 

[meubanks]

Hi rs225

Thanks for the quick response and your help. I have run those commands unfortunately I do not know how to get the replacement WD drive to be recognized as part of the vol1 pool the pool is showing as having 3 drives but it had 4.

[root@ ~]# zpool import -f vol1

cannot import 'vol1': one or more devices is currently unavailable

[root@ ~]# zpool import -f -F vol1

cannot import 'vol1': one or more devices is currently unavailable

 

[Stux]

Looks like a 4 drive stripe.

How badly do you want your data?

Because I think your only chance is data recovery service on the dead drive and the set.

And your backup... how dead is it? It has an hd inside right?

 

[meubanks]

Heh, it wasn't an ordinary door, it was a fire door so the lacie lost two 1Tb drives (both physical cases bent) The NAS drive is most likely not recoverable, I believe the heads crashed when I powered up for diagnostic. I replaced the controller board and tried as well but that was a no go. I can create a mini clean room and check the platters but that is most likely a no go as well.

 

[rs225]

I would not try and mess with the broken drive. If it needs professional recovery, then that is what it needs.

Post output from zdb -l /dev/ada0

 

[meubanks]

[root@ ~]# zdb -l /dev/ada0

Code:

--------------------------------------------																						
LABEL 0																															
--------------------------------------------																						
failed to unpack label 0																											
--------------------------------------------																						
LABEL 1																															
--------------------------------------------																						
failed to unpack label 1																											
--------------------------------------------																						
LABEL 2																															
--------------------------------------------																						
failed to unpack label 2																											
--------------------------------------------																						
LABEL 3																															
--------------------------------------------																						
failed to unpack label 3

[root@ ~]#

 

[nightshade00013]

Even if the heads crashed it is possible a recovery service can get the majority of the data back. You will likely lose some of what you had but if the backups are dead it's at least something.

If you stripped the data you have to have all of the disks to even attempt to fix the problem. If it was a RaidZ1 you have to have all but one disk and all disks is better if there is one or more with damaged data, anything less than this means ZERO data recovery. I know that rs225 is trying to help but physically damaged drives will not just miraculously work and since you are not sure what you set up to begin with you basically need everything in a working state.

So you have a couple choices pay some money and get something back or take it as a loss and learn multiple lessons.

Problem: stripped data is bad and RaidZ1 is still a big risk
Solution: buy a couple more drives and use RaidZ2 or RaidZ3

Problem: backup device next to the main device is bad
Solution: put the backup device in another room or make sure it is protected in a cabinet or better yet make some offsite backups

Problem: unattached doors in range of servers is bad
Solution: Put the door on it's hinges or take it ELSEWHERE

Problem: the cat hates your computers
Solution: consider a cat carrier or just remove the cat from the equation permanently

 

[meubanks]

Well I wasn't expecting much so I can't say I am disappointed, Sorry if I wasted anyone's time, Thank you all for your help!

 

[rs225]

Sorry, it should have been zdb -l /dev/ada0p2

 

[Stux]

321 backup strategy would've saved the day. Or RaidZ2.

"A 3-2-1 strategy means having at least 3 total copies of your data, 2 of which are local but on different mediums (read: devices), and at least 1 copy offsite."

Why I asked how badly you want the Data. If it's with thousands to you, then you need professional recovery. I'd assume physical case damage on the LaCie is easier to deal with, it probably being a mirror.

 

[warriorcookie]

Heh, it wasn't an ordinary door, it was a fire door so the lacie lost two 1Tb drives (both physical cases bent) The NAS drive is most likely not recoverable, I believe the heads crashed when I powered up for diagnostic. I replaced the controller board and tried as well but that was a no go. I can create a mini clean room and check the platters but that is most likely a no go as well.

Sorry for your loss... But this thread was good food for thought.

As Stux said, if your data is worth it, then pay for professional data recovery.

If you've already come to terms with your loss, but would like to experiment, I recommend checking out this video from Louis Rossmann.

Notice instead of a clean room he uses a laminar flow bench. He pulls the platters from a dead drive and drops them into a good drive with good heads. Seems doable at home if you are careful. But again, if the data is important then just pay for a professional.

 

[Stux]

FWIW, I've heard good things about https://datarecovery.com/

 

[meubanks]

I have done this previously using a very large cardboard box cut a hole for a vacuum cleaner hose (in the top), vacuum should be adjustable type, paint the interior with a white emulsion paint and wrap the top 1/2 in cellophane and voila mini clean room. I however was stupid and feel I deserve my punishment so I bit the bullet and rebuilt on FreeNAS 11. All I can say is fricassee CAT anyone?

 

[nightshade00013]

All I can say is fricassee CAT anyone?

I could use a little target practice, LOL.

 

[Stux]

Well, its a lovely example to point people to about why you want to have your backup pool in a separate machine and why an offsite backup for your most important data is a good idea :)

Nobody expects the spanish inquisition evil little kitty