KDB: enter: panic - Unable to boot

[traCk]

Ok guys, so forgive me- Although I have had freenas running for over a year I am still very much a noob.
I was in the GUI today, I had just installed the BTSYNC plugin. I was making a CIFS share for the folder when the GUI became unresponsive. About a minute later I heard my FREENAS system reboot, and it never came back online.

Hooking up a monitor I get the attached, which ends with Stopped at kdb_enter xxxxxx

I have my redundant USB boot drives, I have attempted to reboot with only 1 or the other with no change. I have tried booting from a previous month old boot, no change.

Running a ivy bridge based xeon 4 core, 24 GB EEC DDR3 on a Supermicro Board. Main pool is housed on 3 WD Red 3TB's.

It appears many threads I found with similar issues were due to running low amounts of memory, I feel 24 GB of EEC is sufficient. I run 2 pools, 1 3x3 and 1 1x1 pool, the 1x1 is hardly ever used. The only plugins I am running are Plex, and the recently installed BTSYNC.

What are my options at this point? I have an offsite back up, which admittedly is several months old. I would obviously like to recover the current data.


Thanks!

 

[DrKK]

This is extremely odd.

Your system, as you say, sounds like it has PLENTY of memory for the use case.

Things I would try:

1) Remove *ALL* devices (mice, keyboard, UPS, whatever) from the USB ports. Try again.
2) Make a brand new, fresh, boot device with a virginal FreeNAS on it. Don't worry about the pool import just yet. Can you boot now?

 

[DrKK]

Also can you please be specific about the system. Motherboard model, and HBA's or RAID controllers, etc?

 

[traCk]

Supermicro X9SCM
SATA ports all directly off motherboard to HDD, no RAID or HBA add-in card
Xeon E3-1220 V2
24GB Crucial ECC Memory (8x2 + 4x2)
350W Seasonic 80+ PSU
3x3TB WD RED
1.5TB Samsung Spinpoint (Rarely used)
2x16GB Boot Flash (Redundant)


I will drive to get a fresh drive made here shortly

 

[DrKK]

Well that sounds like a lovely system for FreeNAS. So I doubt that's the problem.

It's possible your pool is horked. Was there any unusual loss of power, unsafe shutdown, busted power supplyu, anything like that?

Can you boot with all your zpool drives disconnected?

 

[traCk]

I haven't had any issues that I can think of lately. The machine is on a UPS which is configured through FreeNAS to properly shutdown if its reserves get low. The last time that happened was 10 months ago, and it properly shut all the way down on UPS power.

The only thing I can say is about 6 months ago I got a message that one of my boot drives was degraded. Shortly after I upgraded to the 9.X fork and have had no errors or issues since. The only reason I didn't swap it out is I didn't take note which drive it was and never had the error pop back up.

I just reinstalled FREENAS iso on a drive, it is installing to a new flash drive as we speak on the system.

 

[traCk]

Got FRENAS installed, got into the GUI, everything seemed good. Imported my main volume, system immediately rebooted. When it came back up I went into the GUI, volume was still not present. Went to import volume again, same result. I can see the console scrolling a ton of characters and then goes blank and system reboots. When I click import volume both of my old volume names show though.

Going to the shell, zpool import shows my pools

zpool import Main (my volume name) says was last used my system XXXXXX , use -f to import anyway.

zpool import -f Main

runs a long script, reboots. Volume still not present in GUI, See Second uploaded picture. This is what displays right before the reboot.

Second pool Archive_Single (Which has basically no useful data on it) recovers fine through GUI.

 

[traCk]

Well, I think I found my culprit.

Although this is ECC Ram I found my system immediately throws errors in memtest.

Removing my 4x2 Kit and leaving the 8X8 kit in resolves these.

Isn't this the whole point of ECC memory not to have this happen?

 

[DrKK]

This does not add up.

Can we have the exact make and model of the RAM sticks you are using? And what source you acquired them from? This particular collection of memory errors just doesn't add up correctly.

 

[traCk]

http://www.kingston.com/dataSheets/KVR1333D3E9SK2_8G.pdf

These modules I purchased along with the Supermicro board, used. My other modules (Crucial 8X2) were purchased new from Amazon.

Crucial kit is https://www.amazon.com/gp/product/B008EMA5VU/?tag=ozlp-20


I have not tried testing just the kingston kit, I still have memtest running from last night on the Crucial kit, so far no errors.

Now I do know this,

The Kingston kit is DDR1333 CAS9 1.5v, the Crucial is DDR1600 CAS11 1.35v. I had emailed Crucial when I originally got these modules, forcing it to run CAS9 1.5v @ 1333 should not be a problem. When they were first installed I ran memtest then without error.

 

[DrKK]

By "4x2" kit, that would typically mean 4 sticks, 2GB. Just FYI. You should have said 2x4, or even 4GBx2, for clarity.

YOu did not say where you bought those from. Was the board and initial set of memory bought from a big retailer (newegg, e.g.,) or from some other kind of reseller or eBay?

And also sir, the exact make and model of *ALL* the RAM. We're not wasting your time asking for it, there is something on my mind.

 

[traCk]

I thought I provided that,
From my previous post:
http://www.kingston.com/dataSheets/KVR1333D3E9SK2_8G.pdf
https://www.amazon.com/gp/product/B008EMA5VU/?tag=ozlp-20
That is the ram that makes up my entire system of 24GB

The board was bought used from a ebay seller,
The board, the 8gb ram kit from kingston and the xeon were purchased off ebay. Everything else was purchased by me new from Newegg or Amazon, including the crucial 16gb kit, drives, psu, etc.

 

[traCk]

Had no luck recovering pool with openindiana. Went ahead and created a new zpool with the same drives, overwriting the data on them. Transferred my latest back over to them, now I guess I wait and see. Got to admit I feel nervous about the whole thing.

 

[SweetAndLow]

Wait so did you get new ram? This is clearly a memory problem and ecc memory can cause a halt in the system if it can't fix the error.

Sent from my Nexus 5X using Tapatalk

 

[traCk]

Wait so did you get new ram? This is clearly a memory problem and ecc memory can cause a halt in the system if it can't fix the error.

Sent from my Nexus 5X using Tapatalk

I was running 2 kits, a 8x2 and a 4x2. Removing the 4x2 fixed my memtest errors. I am running it with just the 8x2 for now, as it is still plenty of ram for my 3x3tb RaidZ pool

 

[traCk]

Just as an update my system has been running fine after removing the 1 kit of memory - I have since taken that kit over to another system and verified it fails there as well. I narrowed down to a specific stick in the kit. I have an RMA started with Kingston but until then I have the 8X2 Crucial Kit running the system.

 

[danb35]

Just as an aside, this is a reason that some people prefer to buy individual sticks of RAM rather than kits--many vendors won't RMA a single stick from a kit, but would want the whole kit back.

 

[Stux]

So, even after removing the dodgy ram the pool was still borked?

 

[traCk]

So, even after removing the dodgy ram the pool was still borked?

Unfortunately I was not able to recover the pool, very disheartening to be honest.

The whole thing has me questioning if Freenas is right for me. Not that Freenas has been anything less than advertised, but with a pool dropping so unexpected it's just a reminder that nothing is reliable enough to not have very constant off system backups. For me, if i'm going to be making daily backups on another system, there is way more versatile and easy to use OS's out there.

 

[traCk]

Just as an aside, this is a reason that some people prefer to buy individual sticks of RAM rather than kits--many vendors won't RMA a single stick from a kit, but would want the whole kit back.

On the other hand, RAM is definitely matched up to be more compatible as a kit. Its a coin toss either way IMO