FreeNAS Fatal trap 12 reboot loop

[mattsowders1989]

Hello all.
Im not new to using freenas but i have never had to solve any major problems within the OS so im stuck obviously... last night, i had a severe thunderstorm that caused my server to power on and off hard 3 times.. i now get a "fatal trap 12: page fault in kernel mode". the sytem is completely accessible for between 1min 30sec and 2min. once that stop happens, no access. i have tested my hard drives. i have tested my ram. i have tested anything i could think of. once i calmed down some. i got into my GUI long enough to create a backup of my configuration. i removed my config from freenas and saw that it does not crash with my config out. i further explored and saw that my raid 1 status was DEGRADED, along with the status of each disk. (i have two samsung 1 tb drives setup as raid 1) i try to rebuild my raid and i get an error saying that "...is corrupt of invalid." i dont see how it is corrupt or invalid because i can access my files for a brief moment.... like i said. im new to this type of repair and servicing. thank you for your time in advance....

 

[jgreco]

You are very strongly encouraged to yank the USB key, set it aside, and use the same version of FreeNAS, installed on a different USB key, and then upload the configuration to it. This has a more-than-even-odds chance of solving the problem. I've seen USB flash rot once too often, and you get all sorts of stupid problems.

Second most likely cause is insufficient memory. FreeNAS's implementation of ZFS is tuned to work well on moderately sized machines (16, 32, 64GB). A general rule is 8GB or 1GB-per-TB, whichever is larger. It can be made to work with somewhat less memory, but requires some work to do so.

 

[mattsowders1989]

should i use an actual hard drive instead of a usb drive??? should i clone or reinstall?? why is it when my raid isnt active, my server does not fault out??

 

[mattsowders1989]

should i use an actual hard drive instead of a usb drive??? should i clone or reinstall?? why is it when my raid isnt active, my server does not fault out??

i also doubled my RAM just to see if it was that, even though i expected it not to be seeing how the server has been problem free for 2 and a half years

 

[jgreco]

I'd avoid too many changes. Try USB for now.

 

[mattsowders1989]

I'd avoid too many changes. Try USB for now.

i dont want question your knowledge, but why a different usb? why not scan the current to check for problems and if problem free, reinstall using it?? i just want to address this logically

 

[jgreco]

How do you propose to do that? Are you sure you can spot a problem? It sure isn't worth MY time.

 

[mattsowders1989]

I used a different usb key, and got he same results, as i expected....

 

[cyberjock]

If your system was powered off without a proper shutdown you may have corrupted ZFS' metadata. It could result in you being able to use the pool until it gets to the part that is corrupted.

Things I would try:

1. Do a long SMART test if the disks you have left and see if any of them fail. Google and some forum searching should provide you with the appropriate commands if you don't know how. If you are degraded you already have a failed disk. If a second disk is also failing from the power surges you may be in a position where you can't recover any data. Before you ask, there really aren't any ZFS recovery tools. So hopefully you aren't in that position.
2. If you have a few spare disks, try a fresh install of FreeNAS and make a zpool with some spare disks. Throw some data on and off and see if the system has problems. You may have something damaged such as a SATA controller that is responsible for your problems.

It's always recommended that you shutdown servers and unplug them from the wall and network in the event of thunderstorms. I've seen people wipe out every NIC in the house because thunder hit the power lines and it somehow got into the home's Ethernet.

 

[mattsowders1989]

I did a long SMART test on both hard drives. They report to be good. i know i havent lost any data because i can move files on and off the server for about a minute and a half.

 

[cyberjock]

To me that doesn't really prove much. Assuming ZFS' metadata is trashed, you won't have any problems until you hit the bad area. Then things get nasty(which is kind of the scenario you are in). Assuming that is the problem, you do, for certain, have lost data(the corrupted metadata won't point to your file data correctly). How much data is lost will depend on how messed up the metadata is. It's just like a disk drive that starts having problems. You can start copying your data, but if you never try to read or write to the bad areas of the disk you'll never know they are bad or have a problem. When ZFS comes online it does some basic checks and mounts your disks. If it does more stuff in the background after the mounting when the disks are idle, it may be encountering a bad area causing your system to do what it is.

If your long SMART tests are good, then I'm leaning more towards ZFS corruption than anything else. I really don't have any good recommendations at this point except to try some spare disks and see if the same issue occurs. It could be something as simple as some hardware is overheating and causing problems. Time to put on your troubleshooting hat and see how you can narrow down the issue.

Just a quick Google search of your exact error code "fatal trap 12: page fault while in kernel mode" turns up quite a few results. Some of the high post value individuals at the FreeBSD forums said that the error is almost always due to hardware issues.

I would try a fresh config of FreeNAS. Don't import any pools or anything. Just let the system run for 30 minutes and see if anything happens.

 

[mattsowders1989]

ok thanks for your help. i will start testing your theories and see what i come up with. i am assuming that if i were to instal freenas on another box and upload my configuration, it wouldnt work... am i correct?

 

[cyberjock]

It should work. The whole plan with FreeNAS is that if everything failed except your disks you could take your hard drives to another machine and boot it up and be online in minutes(assuming you have the config file backed up). Without the backup you'd have to aut0-import and setup FreeNAS from scratch, but your data would be intact.

 

[mattsowders1989]

i do have a back up. i will do some testing and post with my details

 

[jgreco]

I used a different usb key, and got he same results, as i expected....

Okay, great, then obviously you're smarter than me. I just like to check the easy stuff first, but I'm also fine shutting up and letting you do what you want.

 

[mattsowders1989]

Okay, great, then obviously you're smarter than me. I just like to check the easy stuff first, but I'm also fine shutting up and letting you do what you want.

ok bye

 

[mattsowders1989]

well im almost positive that my ZFS table is corrupt. my up times are only for 2 minutes and i have 900 gig of data that i really dont want to loose.

 

[mattsowders1989]

not ZFS, GPT table.... sorry

 

[gpsguy]

You mentioned doing this "i am assuming that if i were to install freenas on another box and upload my configuration" and cyberjock replied in #13 that it should work. Did you try it?

I too, suffered from fatal trap 12 problems last year. I couldn't even get through the FN boot process, without it crashing. Since I had more $$'s than time, I replaced the hardware.

btw, what version of FN are you running?

 

[mattsowders1989]

You mentioned doing this "i am assuming that if i were to install freenas on another box and upload my configuration" and cyberjock replied in #13 that it should work. Did you try it?

I too, suffered from fatal trap 12 problems last year. I couldn't even get through the FN boot process, without it crashing. Since I had more $$'s than time, I replaced the hardware.

btw, what version of FN are you running?

i did try it and got the same results

im running version 0.7.2