Fatal Trap 12, after getting "exited on signal 11" emails

[EvanVanVan]

SOLVED!! Not only was I able to boot into FreeNAS by using a new USB drive. I successfully copied my old configuration file and encryption key off of the old USB stick (since I couldn't find my backups).



My FreeNAS system is failing to boot, getting stuck at a Fatal Trap 12 error. Here's how it started, on Aug 1st @ 2 AM I got an email that a zfs scrub was starting. On Aug 2nd @ 3:01 AM I got an email "pid 70360 (smbd), uid 1001: exited on signal 11". At the time I didn't think much of it, and forgot about it. ON Aug 4th @ 3:01 AM I got another email "pid 563 (smbd), uid 1001: exited on signal 11". Again, I did forget about it, until last night when I tried to transfer (CIFS) something over to my server and realized it wasn't responding/connected to the internet.

I hooked up the monitor to the box, and was able to "safely" shutdown by selecting option 11 on the dos menu. After trying to boot back up, i got the Fatal Trap 12 error. I should have taken a picture of the full error, but didn't.

I searched the forums last night and found that Fatal Trap 12 errors can be a few things, "exited on signal 11" usually deals with ram. The first thing I've done is run a memtest. Through a pass and a half it hasn't found any errors. I also saw ZFS Scrubs can cause problems with RAM (but usually when it's NON-ECC, I *have* ECC RAM).

I'm going to let the second memtest pass finish. But a strange thing I think I noticed, when booting with the monitor, one of the first lines that came up was "640/32768K mem" or something along those line. It almost made it look like FreeNAS was only using 640 bytes of memory? Again, I'm not remembering it very well, but if FreeNAS was only accessing that tiny bit of memory it could explain a Fatal Trap 12 error.

Anyother thoughts or suggestions? I saw in another thread someone suggested to try a new usb stick. I really don't want to lose my data so I'm not going to try anything without some guidance.

Thanks

SUPERMICRO MBD-X9SCL-F-O Intel Xeon E3 Server
Intel Xeon E3-1230 V2
Kingston Technology ValueRAM 32GB DDR3 1600MHz PC3 12800 ECC
WD RED (8x) 3GB
IBM ServeRAID M1015 (crossflashed into IT Mode) (All eight of my HDs are plugged into it)

 

[joeschmuck]

Yo can easily tell how much RAM FreeNAS thinks it has, look at the GUI System Information Tab.

And I'll bite... What version of FreeNAS are you running? You never said how the system is working now after rebooting it, can it transfer files, does it crash, what?

 

[EvanVanVan]

Sorry I guess that wasn't clear. When I realized the Nas wasn't responding/connected I couldn't access the GUI by that point.

Since then I haven't been able to boot back into FreeNAS. Maybe I could? After I got the fatal trap, and it stopped loading the dos stuff I never tried to see if it was fully booted and accessible over the lan. I assumed that since I never got back to the dos menu (with those options 1-11) that it wasn't fully booted.

I'm running 9.2.1.3.

Sorry for any typos I'm typing this on my tablet now.

 

[joeschmuck]

Well my first piece of advice would be to take a new/different USB Flash drive and load it with FreeNAS 9.2.1.3 and see if that boots up. If it does and you have your configuration file saved, load your configuration file and hopefully everything will work.

 

[EvanVanVan]

Thanks I'll try that after work tonight.

 

[EvanVanVan]

Haven't had a chance to try a new USB yet, I can't remember if I used FreeNAS 9.2.1 or 9.2.1.3...

Anyway, here a couple of pictures. The first on is where it says "BIOS 640KB/3125028KB available memory." Idk if that's normal or not. (Sorry I didn't have time to turn off the flash for the first one.)

The second picture is my error message. Any thoughts?

 

[ewhac]

Just for laughs: Do you have a keyboard plugged in and, if so, is it a PS/2 type? (psmidentify() refers to the legacy PS/2 keyboard/mouse interface.)

 

[EvanVanVan]

I did have a keyboard, USB though. I probably used it to reboot and never unplugged it. LOL. I hope it's that easy, it did just boot up when I used a new USB stick. I'll try the old stick and no keyboard.

 

[danb35]

I think the 640/3125028 is pretty much normal--low vs. high memory and all. I'm more concerned that it seems to be reporting a total of only about 3 GB.

 

[EvanVanVan]

I used jgreco's post here: http://forums.freenas.org/index.php?threads/recover-freenas-configuration-from-os-drive.15426/ to try and recover my config file. I know I backed it up but can't find it anywhere.

It all worked well until the last cp command it says the file can't be found. For the record I used da9s4, has that part (the daXs4 part) changed?

 

[EvanVanVan]

Uh oh, it might be over. I figured out the right command was actually to copy freenas-v1.db (- not _). I got my settings back, but now my volume1 under Storage says LOCKED.

I tried highlighting the volume and pressing Unlock, it attempts it and fails...Maybe I'm on the wrong version of FreeNAS? It may have been 9.2.1 and not 9.2.1.3.

edit:I'm not going to try anything without guidance like i said..

 

[cyberjock]

1. Your pool is encrypted. You need to unlock it with the key+password or recovery key you saved in a safe place. If you don't have those you are in trouble.
2. All 9.2.1.x releases are vulnerable to a Samba vulnerability. The *only* version anyone on any version of 9.2.1.x should be on is 9.2.1.7. Period.

If your pool is encrypted and you don't know your key+password or recovery key you are already sunk. There is no chance in hell you will see that data again unless you think you're going to hack AES-128 encryption. ;)

 

[EvanVanVan]

The strange thing is, it didn't give me any option for a key+password... It just said "Would you like to unlock?" I choose yes, and it said "attemping..." and then failed.

 

[cyberjock]

was your pool encrypted?

 

[EvanVanVan]

I think so, but I don't remember entirely... I read your posts 16x saying save the config and the encryption keys... I can't figure out where the **** I put them though.

Would trying to load up 9.2.1 hurt and/or help anything?

I guess it couldn't hurt lol

Is the encryption key+pass file kept on the existing flash drive I could copy over?

 

[EvanVanVan]

Found this thread: http://forums.freenas.org/index.php?threads/recover-encryption-key.16593/

Between the last guide I posted and this one, I got the following commands:

mkdir /var/tmp/cfgmount
mount -o ro /dev/da9s4 /var/tmp/cfgmount
cd /data
mkdir geli
cp -av /var/tmp/cfgmount/geli/ /data/geli

SUCCESS!

(Edit: My services were screwed up at first, but I was trying FreeNAS 9.2.1. Went back to 9.2.1.3 that I originally thought I was on and it WORKS!)

 

[EvanVanVan]

Just a bump, because I crushed it. And I backed up my config and encryption key for sure this time and emailed them to myself.

 

[cyberjock]

LOL.. emailed them to yourself. So let me get this straight...

Your config file contains your passwords and such. So thanks for letting your email provider store your info.
Your email also contains your keys.

Any part of this NOT incredibly stupid yet?

So all I have to do is steal your server, which will have your username and password for email and trace that back to get your key and *poof* I have access to your data.

Great idea.

And you chose encryption for what reason again? No.. don't tell me. I don't even want to stick around in this thread any longer. LOL.

 

[EvanVanVan]

Well if you stole my server you'd have access to my data regardless (unless you were planning on removing and leaving me the USB key?). I don't have a passphrase setup, you steal the server you got the data. Don't need to trace back anything if your smart enough to boot it up and look at the config file. (Which of all the people I'm worried of breaking into my house, I'm not concerned they'll be Chinese hackers)

Btw "noobsauce" after stumbling across some of your threads last night, I didn't realize you were such a beginner either. (They just happened to be the same things I was asking.) BTW, you posted in that encryption thread I linked and were informed by Dusan where the encryption key was stored on the USB stick. Yet you're telling me here and in a different thread, another user back in April, "no, if you didn't backup your key you're lost. There's nothing you can do except create a new vpool on the disks and start over." You know/knew that wasn't the case...

Btw my data is not highly sensitive, it's movies and porn.

 

[joeschmuck]

Why encrypt it then? All it does is add a level of difficulty when things go wrong.