Fatal trap 12: page fault while in kernel mode

[pav]

The error happens at random intervals. Sometimes few times a week, sometimes once a month.

Build: FreeNAS-8.2.0-RELEASE-p1-x64 (r11950)
Platform: Intel(R) Core(TM) i5-3550 CPU @ 3.30GHz
Memory: 3976MB

Screenshot of the error attached:

 

[cyberjock]

First thing I'd try is more memory. Faults of your type are almost always hardware failures or incompatibilities in the hardware with FreeBSD. You could also check to see if your motherboard BIOS is up to date.

Other than that, you're options really are limited except to start troubleshooting things until you find something broken.

 

[Letni]

Fatal trap 12 is a known issue with 8.2.X.. I was seeing it when access shares on NFS.. Upgrade to 8.3.0

 

[TurboSquid]

Fatal trap 12 is a known issue with 8.2.X.. I was seeing it when access shares on NFS.. Upgrade to 8.3.0

I am also experiencing this issue and I am running 8.3.0. It happens completely randomly as the last poster said, sometimes it will happen every day, sometimes it will run for weeks without issue.

I am currently running freenas on:
Build FreeNAS-8.3.0-RELEASE-x64 (r12701M)
Platform AMD E-350 Processor
Memory 8125MB

 

[maglaubig]

I was able to reliably reproduce the behavior from a Win7 Ultimate PC accessing via CIFS. I compared with other PC's running other versions of Win7 (Enterprise) using the exact same CIFS settings, NTLM etc, but couldn't figure it out. Upgrade to 8.3 worked great to fix this.

Take heed of the hardware requirements and BIOS/Firmware. I'm having some difficulty where about after a week on 8.3 I lose access to one of the SATA drives. A reboot always fixes the issue along with a ZFS scrub, but it usually requires power cycling the computer since it hangs on I/O at shutdown and doesn't completely get there.

 

[cyberjock]

I'm having some difficulty where about after a week on 8.3 I lose access to one of the SATA drives. A reboot always fixes the issue along with a ZFS scrub, but it usually requires power cycling the computer since it hangs on I/O at shutdown and doesn't completely get there.

That sounds like more of a hard disk problem.

 

[maglaubig]

I though it was HDD too, although I didn't have the issue on 8.2 and full disk scans didn't yield any results other than the drive was good. Only issue with 8.2 then is that I'd get the crashes at the beginning of this thread. I came across a few other posts in the forums on issues like these specifically with onboard AHCI controllers on 8.3. The SATA controller is not the MegaRAID one but it's similar, don't remember it off hand, but do remember firmware revisions on controllers and motherboards being listed as a big issue in the release notes for 8.3.

Since upgrading the BIOS today it appears the boot time is considerably less. It was taking me upwards of 10 min to come completely online to where I could get at the web GUI. With the BIOS upgrade today I could get to the web GUI in less than 2 min, so I think there's something to it in my case anyway. Fingers are crossed that this gets me stability like I have on other FreeNAS servers I've built.

 

[cyberjock]

It sounds like you did a scrub of your zpool and had no issues so you figured the disks are good.

Have you tried doing a long SMART test of your hard disks? I will grant you that if you do a scrub you are probably dealing with good drives. But a long SMART test is really better designed for finding hard drive defects.

Also, from personal experience I have seen where flaky SATA cables cause a few communication errors here and there. On some SATA chips they seem to drop a hard drive if you accumulate enough during a given power on cycle. If this is your problem you could try replacing the SATA cable since they're cheap. You could also try programming in a reboot via cron every night at like 1am or something. That will "reset" the number of communication errors to zero and may help alleviate the problem. Of course, if that is the problem I'd still replace the SATA cable.

Can you post the model of your motherboard and what bios version you had and upgraded to?

 

[joeschmuck]

@pav,
Your RAM is very low, maybe you need some more tweaking. It looks like you had ran it for almost 16 days before the crash. You should also look into if there was something specific going on during that time when it crashed. You could have a flaky power supply too. I'd say run a RAM test for at least 24 hours, more if you can. Run Prime95 on the CPU, make sure you have a good heatsink on that CPU, but I wouldn't run Prime95 longer than overnight (8 hours). If it doesn't fail in 8 hours I just don't think it will fail at all for that component. If you find something fails and you don't have a spare component, and you have a spare power supply, replace that. A cheap power supply can supply crappy power and cause all kinds of problems. You need to do your homework if you are going to tack down this problem. It could be this NFS bug but if you turn off NFS, then does it all work fine again?

@maglaubig,
You never mentioned if the SATA drive dropping offline was the same drive each time. I would assume it was since you didn't specify. I hope the BIOS update fixed your problems. Also what hard drive model was it? Have you heard of TLER? In a nut shell it's how long a drive can wait to give up some data without throwing an error. NAS drives have this but normal Desktop drives typically do not. So if it takes a little too long to get your data from a hard drive without TLER, the NAS can and will drop your drive offline because it thinks it failed. A desktop drive test may pass just fine. I think I have that correct.

 

[maglaubig]

@pav - I feel like I'm hijacking your thread now. Memtest86+, great memory diagnostic tool (www.memtest.org - NOT .COM). I use it on VMware ESXi servers with upwards of 128GB RAM, no issues. I usually let it run at least 72 hours on new hardware. Boot from a CD/USB and leave it alone, it auto starts. If it doesn't find something, it doesn't exist.

@noobsauce80 - Server board is Intel S3200SH, currently running Nov 29, 2010 BIOS update S3200X38.86B.00.00.0052. SMART tests run daily, both short and long via web GUI, odd and even drive IDs run at separate times. Never had a bad report or errors on POST. Wish I had the old version, this server is remote and I didn't think to have the person helping me through the upgrade write down the previous one. Although SATA cable is possible, I'll say unlikely as it was brand new. Scheduled cron reboots was my next step. Seems like error count might have been to blame along with what @joeschmuck says.

@joeschmuck - you're 100% correct, same drive every time and it took about 1 week to get there. It was always the last ID on the SATA controller. Camcontrol wouldn't report the drive anymore unless with verbose option and then it would leave the adaX number off the end of the line. HDD Model is WDC WD20EARX-32PASB0 51.0AB51 - 2 TB HDD. I'm striping two mirrored vdevs in my config, all the same model drive.

Example of camcontrol on the reportedly failed drive (note passX,adaX are missing from ()'s):

<WDC WD20EARX-32PASB0 51.0AB51> at scbus3 target 0 lun 0 ()

Other drives never reported the issue, although the OS would become unresponsive.

 

[maglaubig]

Just a follow-up to close outstanding questions for me on this thread. Over a week past the BIOS upgrade. No weirdness, system is very responsive. I had some other issues the BIOS upgrade cleared up as well where boot devices would get re-ordered.

 

[Ismael Duarte]

I've this error too, since I've upgraded to 8.3.0.
At this moment I've FreeNAS-8.3.0-RELEASE-p1-x86 (r12825)

Waiting for some fix.
Thank you!

 

[mazdajai]

Had similar problem and waiting for a fix as well -

I'm having some difficulty where about after a week on 8.3 I lose access to one of the SATA drives. A reboot always fixes the issue along with a ZFS scrub, but it usually requires power cycling the computer since it hangs on I/O at shutdown and doesn't completely get there.