Server won't boot with drives in after power loss - panic (HELP!!)

[jakessharpe]

Hi,

I'm relatively new to this, this is my third go at a FreeNAS system and this time it was very stable and up for some time. My system is running 9.10 Stable (latest updates) Motherboard is Intel Server Board S3210, Intel Core2Quad Q9650, 8gb Ram (Not ECC I know) and 8x drives (2x WD 1tb, 2x WD 750gb and 4x HGST 500g in a 4 zpool config)

Earlier my system restarted itself, I believe due to power loss (my lights flickered). I heard the system post from across the room, and It booted back up but with an error on when booting

Code:

'KDB: enter: panic
[ thread pid 742 tid 100714 ]
Stopped at			 kdb_enter+0x33e: movq		   $0,kdb_why'

There is more above but I'm not sure how relevant it is, I can take a photo or type out if need be.

I tried booting without the drives in, and this time it booted to the web gui fine, however when I then plugged my drives in every single one was showing 'error', and unless I am missing something I was seemly unable to import them in any way (never done this before though so I could be missing something).

To make it worse I just backed up all my data yesterday to the FreeNAS as I am currently performing a re installation of my main desktop due to a failing hard drive, which is being migrated to an SSD. I backed up all the data to 3 of the 4 zpools, thinking would this be enough redundancy, but I didn't consider the system could go entirely down :( I would have put it on a standard external drive if I had one that wasn't already in FreeNAS, as it was so stable that I put them all in there as network access to these through that system was incredibly useful to me.

Is there any way to retrieve my data at all? Happy to rebuild the system, I just want my data back!

Thanks in advance

 

[BigDave]

I tried booting without the drives in, and this time it booted to the web gui fine, however when I then plugged my drives in every single one was showing 'error', and unless I am missing something I was seemly unable to import them in any way (never done this before though so I could be missing something).

In order to help you, the forum is going to need details about your drive configuration and how those eight drives are hooked to a motherboard with 6 SATA ports! If you provide all the details of your system and how you added the drives over time (I'm guessing you did not start out with these 8 drives from the beginning). Someone will be more likely to jump in and help when the issue you are having is defined in greater detail.

Have you tried following the directions for Importing a Volume in the Manual? http://doc.freenas.org/9.10/storage.html#import-volume

 

[jakessharpe]

Thanks for the reply,

My bad, I did forget to mention two of the SATA ports are provided by a PCI E card in the mobo, a generic card with a Marvell 88SE9235 chip.

The system started with 2 pools and 5 drives. 2x 750gb WD in one mirror used as a Plex Library, and 3x 500GB HGST in Raidz as Time Machine and Backup. I then added a single extra 500gb HGST drive to serve as a iSCSI target for a Steam library.

I then recently added 2x 1tb WD drives and the PCI E card to expand just general storage and the Plex storage

I tried importing the volumes but the drop-down shown in Fig. 8.1.8 in the manual just appears empty. zpool import does nothing either.

I've done a bit more testing and it seems just one of the pools is causing this error. I am able to get the system to boot as normal when only 3 of 4 pools are connected, however the 3x 500gb HGST pool causes that same kernel panic on boot.

I ran a regular scrub and SMART scan and these 3 drives never reported any issues.

Any idea how i can get my data back on this one pool? While not as critical as it first seemed, the old time machine data is crucial to have back.

 

[BigDave]

and 3x 500GB HGST in Raidz as Time Machine and Backup.

So if I understand correctly, this volume/pool that you were using for Time Machine was a three drive STRIPED volume (RAIDz), and this is the volume that is now not showing up or throwing errors?

 

[BigDave]

After re-reading your 2nd reply, I have a suggestion for a bit of trouble shooting.
Remove all of the drives from your machine and hook up the (problematic) three drive pool
directly to your motherboard's SATA ports. Attempt an import of that pool.

If this is successful then I would think the drives making up that pool are good.

My next question would be, were the drives you have errors with, hooked to the
Marvell based card?

That piece of hardware could have been damaged with your fluctuation of electrical current.

 

[jakessharpe]

It was the three drive raidz yes.

Thanks for the advice, I'll have a go at this when I get home from work later and post the results.

They weren't the ones on the card, the two 1tb drives were hooked up via that.


Sent from my iPhone using Tapatalk

 

[jakessharpe]

Okay i stripped the server back and only connected the 3 drives. They now show in zpool import

Code:

[root@JakeFreeNAS ~]# zpool import											 
   pool: Smough																 
	 id: 6105056138951818216													
  state: ONLINE																 
action: The pool can be imported using its name or numeric identifier.		 
config:																		
																				
		Smough										  ONLINE				 
		  raidz1-0									  ONLINE				 
			gptid/12304610-9ae9-11e7-857d-001cc093a225  ONLINE				 
			gptid/12fdd3c0-9ae9-11e7-857d-001cc093a225  ONLINE				 
			gptid/1404952c-9ae9-11e7-857d-001cc093a225  ONLINE				 
[root@JakeFreeNAS ~]# zpool import Smough	 

However when i then import the pool, the server runs a LOT of code on the console screen then crashes. Upon reboot I hit the same issue again, with the kernel panic from my original post appearing again.

 

[rs225]

If it panics again, a photo of the whole screen would be very helpful.

 

[jakessharpe]

Easy enough to reproduce!

 

[rs225]

Try doing zpool import -o readonly=on Smough and see if still panics.

 

[jakessharpe]

Same deal unfortunately, console window shows massive amounts of scrolling code going too fast to read, server turns off and restarts itself then arrives back on same kernel panic again after restart.

 

[rs225]

Hopefully somebody else will have some ideas here. There is also one other thread going with a pool that panics on boot.

 

[jakessharpe]

Thanks anyway. I'll have a look at the other thread. Mainly just want my data back at this point, even if it means trashing the drives and starting over after.

EDIT:

Had another shot at this, quick thread search showed up the command zpool import -o readonly=on -f -R /mnt Smough. Doing this via single user mode with recovery and debug parameters enabled resulted in nothing, just failure to import. Doing this in shell results in the same restart happening again. I'm at a bit of a loss as to what to do now. If anybody knows any way to get the data off the drive that may work please let me know.

 

[BigDave]

Is the SATA add on card still plugged in to the expansion slot? If it is, try removing it
and see if the volume will import once it's out.
Aside from that, perhaps try installing FreeNAS on another machine (with min. 8GB RAM)
and attempting to import the volume to that install.

 

[wblock]

Run memtest on that machine for a while, and look suspiciously at the power supply.