panic: general protection fault

[sadokath]

hi!

without big words, here my problem in pictures:

here is my pcivonf -v output for the hardware part.

here my dmesg output

i changed nothing via console. so boot.conf and /etc/ ... are untouched.
no custom cronjobs, one periodic (dayly) zfs snapshot was running (neerly empty volume, changed delta were only some kb).

any ideas... or do i need to provide additional information?

thx

 

[William Grzybowski]

How much RAM do you have? Is it amd64? Did you ran some memory test tool like memtest?

 

[sadokath]

hamster /]$ sysctl vm.kmem_size_min vm.kmem_size_max vm.kmem_size vm.kmem_size_scale
vm.kmem_size_min: 0
vm.kmem_size_max: 329853485875
vm.kmem_size: 4109578240
vm.kmem_size_scale: 1

hamster /]$ sysctl hw.physmem hw.usermem hw.realmem
hw.physmem: 4267454464
hw.usermem: 3900289024
hw.realmem: 5368709120

hamster /]$ sysctl hw.pagesize hw.pagesizes hw.availpages
hw.pagesize: 4096
hw.pagesizes: 4096 2097152 0
hw.availpages: 1041859


hamster /]$ uname -a
FreeBSD hamster.lan 8.2-RELEASE-p1 FreeBSD 8.2-RELEASE-p1 #4: Sat Apr 30 10:39:46 PDT 2011 jpaetzel@servant.iXsystems.com:/usr/home/jpaetzel/freenas/obj.amd64/usr/home/jpaetzel/freenas/FreeBSD/src/sys/FREENAS.amd64 amd64

in short: amd64, 4gb ram

no memtests or such running

 

[sadokath]

new sig with system specs

 

[William Grzybowski]

Sorry I have no idea. 4Gb of RAM is not ideal for ZFS but should not cause panics...

A trap in a python process is very suspicious though... my first guess would be a memory problem..

 

[matthewowen01]

python is running as root so it could easily nuke your system.

there may be an unhanded exception in the python process.

i highly doubt its memory.

 

[William Grzybowski]

python is running as root so it could easily nuke your system.

there may be an unhanded exception in the python process.

i highly doubt its memory.

An unhandled exception in python cause a general protection fault, kernel panic?

Sorry, i don't buy it :)

 

[matthewowen01]

An unhandled exception in python cause a general protection fault, kernel panic?

Sorry, i don't buy it :)

root can do anything it wants, if you tell it to overwrite important things, it will.

 

[sadokath]

over night, it happend again.
uptime was about 9h
no special jobs running. not even files were accessed via lan.

 

[matthewowen01]

well hell, a memtest won't hurt things.

can you run a memory test on your system overnight?

 

[sadokath]

i ran a memtest for 15h now.
no errors found. my ram is ok.

 

[matthewowen01]

good to remove that from the question.

can you configure your server too create a Kernel Crash Dump. before you do change your passwords, they can be included in the Kernel Crash Dump.

instructions are here:
http://www.freebsd.org/doc/en/books/developers-handbook/kerneldebug.html

When you have it let me know and i can make you an account on my SFTP server that you can upload it too. I've never debugged BSD but my last final (CSC445, Theory of Computation 1) is on Tuesday so i have summer to learn. it'll be a nice puzzle to start the summer with.

 

[sadokath]

here my last steps:

* changed passwords
* pluged in 8gb usb stick on an usb port
* created a new volume with that stick (ufs) - was mounted to /mnt/crash

hamster# df -h /dev/ufs/crash
Filesystem Size Used Avail Capacity Mounted on
/dev/ufs/crash 5.3G 4.0K 4.9G 0% /mnt/crash

* Edited /conf/base/etc/rc.conf (dumpdev chaged, dumpdir added)

Result:

hamster# grep crash /conf/base/etc/rc.conf
dumpdev="AUTO" # Device to crashdump to (device name, AUTO, or NO).
dumpdir="/mnt/crash"

* reboot

hamster# grep swap /etc/fstab
/dev/gpt/swap-da0 none swap sw 0 0
/dev/gpt/swap-ada0 none swap sw 0 0
/dev/gpt/swap-ada1 none swap sw 0 0

* hamster# dumpon /dev/gpt/swap-da0
* hamster# dumpon /dev/gpt/swap-ada0
* hamster# dumpon /dev/gpt/swap-ada1


hope thats it! now i wait for the next crash.
or did i miss something?

cheers!

 

[matthewowen01]

honestly i've never done this on BSD before, looks good, let me know when it crashes.

i have finals tomorrow so i may be absent for a bit studying.

 

[ixdwhite]

Could also be a bad CPU/motherboard/etc. memtest86+ is a fairly light exercise from an I/O standpoint.
Also keep an eye on how hot your system is getting, some Atom servers are poorly designed and overheat under load.

 

[matthewowen01]

Could also be a bad CPU/motherboard/etc. memtest86+ is a fairly light exercise from an I/O standpoint.
Also keep an eye on how hot your system is getting, some Atom servers are poorly designed and overheat under load.

When I was restructuring my storage and was pushing 3 gb/s through my LSI SAS 3081E-R, it overheated several times before i figure it out. I had to strap a fan to it.

http://www.lsi.com/storage_home/products_home/host_bus_adapters/sas_hbas/lsisas3081er/index.html

 

[sadokath]

i dont think that its an overheating issue.
there was no load on the system while the two times the panic happend.

and i have 2 heat sensors inside the case: one between my 2 disks (35°C),
one directly attached on the passive cooling block on the cpu (47°C-50°C).

this are no values from the bios or the cpu directly! the sensors are part of the
frontpanel of the case. so the heat sensor is bonded on top of the cpu heatsink.

but if nothing other helpes, i will add some additional fans to keep the system cooler.

 

[gazza]

After some extensive testing on my eee box 202 with freeNAS 8.0 & 8.0.1. I have come to a conclusion that the i386 has some serious issues. I have reverted back to FreeNAS 7.3 and everything working again. If my claims are true, it would be nice to remove the i386 build temporarly until a resolution can be found.
Please note that this ONLY occurs with the i386 build. There are alot of people out there saying that their setup works perfectly with 3-40GB ram. The amd64 works fine guys....

here is the system setup, asus eee box 202 2GB ram (tested).
3 USB hard drives 1TB each running ZFS (yes I know they need alot of ram).

So I have 2GB ram, double of the recommended minimum. I have read alot about kernel tuning and how people are setting their vm.kmem_size and the like to 512M - 1500M +.
Well a word of warning, setting the vm.kmem_size to anything above 680M will most likely cause a panic with a return status of 3.
Setting it to 512M works for x amount of time, until you run some heavy copying stuff to your nas, then things will panic....
I just run a robo copy from windows 7 with a setting of MT:8 to set the multi-threading. This should cause a kernel panic.

I have tried this with a fresh copy of 8.0 and I get the same results...
I have also noticed that the memory graph never goes above 512M.

For now, I am back to 7.3 unless I buy a amd64 instruction set cpu....

 

[sadokath]

I lil bit of times passed by.
Had no time to investigate deeper (moved to another city), but i experience the panic still all 2 to 3 days.

i can exclude the overheating problems.
my server is now located in the basement with constant temperature of < 15°C (59F).
my case also has a 400mm fan thats blowes the cold air directly onto the board/cpu (http://www.zipzoomfly.com/images/Enlarge_image/large/10011089_D.jpg).
temperature sensors signaling good values. so this rly is no problem.

i will try to get a crash dump.

 

[Tekkie]

Actually what I would try is to limit max memory for the kernel and ZFS cache:
vm.kmem_size_max="3584M"
vfs.zfs.arc_max="3072M"

I've got a growing suspicion what on my system random freeze-ups are caused by too little memory being available to the system.