Freenas 8.0 Release Fatal trap 9 Issue

[TwinDaddyKev]

Hi Everyone,

Currently I am testing Freenas 8.0 amd64 Release but getting a Fatal trap 9 issue after 3-6 hours.

Current hardware:

SUPERMICRO MBD-X7SPA-H-O Intel Atom D510 1.66Ghz Server Motherboard
4GB SO-DIMM DDR2
Chenbro case with 4 removable trays
4X1TB WD1001FALS (with TLER enabled)

The error message I am getting is:

Fatal trap 9: general protection fault while in kernel mode
cpuid=0; apic id=00
instruction pointer = 0x200:0xffffffff80631452
stack pointer = 0x28:0xffffff81205eca50
frame pointer = 0x28:0xffffff81205eca80
code segment = base 0x0, limit 0xfffff, type 0x1b
= DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags = interrupt enabled, rersume, IOPL=0
current process = 12 (irq256: em0:rx 0)
trap number = 9
panic : general protection fault
cpuid = 0
Uptime: 6h44m20s

Initial Freenas setup is fine. I was able to get to the Freenas management console. Add the 4 drives as ZFS Raidz single volume. I add some data to the volume. Then I leave it there for a while and when I come back a few hours later, I would get the Fatal trap 9 error.

My other test to see if it was the hardware, I replaced the drives with (4) WD VelociRaptor WD3000HLFS. Create the same ZFS Raidz single volume. Copy some data on there and it ran fine without errors for over 24 hours.

So my question is has anyone successfully run Freenas with WD WD1001FALS drives? Or is this error relate to other hardware issues.

Any suggestions would be appreciated.

Thanks...

TwinDaddyKev

 

[ProtoSD]

We have basically the same hardware except for your disks. I had a bunch of problems like that in the beginning until I bumped up vm.kmem_size & vm.kmem_size_max, in /boot/loader.conf. You'll have to 'mount -uw /' before you edit loader.conf. Let me check what I have mine set to and I'll update this message.

vm.kmem_size="1536M"
vm.kmem_size_max="2048M"

Don't forget to check:

vfs.zfs.prefetch_disable="1"

Think about giving 8.01 beta 4 a try after you get this working. If you upgrade remember that the loader.conf settings don't get saved, so you'll have to edit them again.

 

[TwinDaddyKev]

I actually tried it with 8.0 Release and 8.01 Beta 4 and get the same error message. The weird thing is when I tried the WD3000HLFS VelociRaptor drives, I don't get this issue.
I'll try changing the vm.kem_size setting you suggested.
Thanks for your help.

 

[TwinDaddyKev]

Thanks in advance for your help protosd. You've been a great help and a great contributor to this forum.

As for my issue, I may be having hardware issues. I tried the zfs tuning as you suggested running on FN8 release and FN8.01 Beta 4 but it still crashes. But this time, the web GUI crashes saying it's not available. I can ping the device but can't access it via web GUI or SSH into it. I also tried running it without any hard drives installed except for the 4GB USB drive running FN8.

I am getting a GEOM: ad0s1: geometry does not match label (255h,63s != 16h,255s). And I tried different USB drives and still get this GEOM error. How are you booting on your device and are you getting this GEOM error?

This leads me to believe that the hardware could be an issue. Maybe there is something wrong with the memory modules. Also the BIOS is up to no good. I tried to change the time in the BIOS but it never save. It's always 8 hours ahead for some reason. I flashed it with the latest BIOS firmware R 1.1a but it still didn't fix the time issue.

I am going to run a memory test later today to see if I can find any errors.

 

[ProtoSD]

Thanks TwindaddyKev, I try, but I'm not an expert.

The GEOM error you're getting from your flash drive is normal, at least on every flash drive I've ever booted FreeNAS from I've got that error and FreeNAS still runs fine.

Give the memory test a try and see what that turns up.

 

[TwinDaddyKev]

What BIOS version are you running? I am running R 1.1a. Are you having the BIOS time issue I am having? Mine seems to be 8 hours ahead even when I set it back to PST.

 

[ProtoSD]

I can't say for sure because my system is currently 'headless', but I think when I looked to see if there was a BIOS update, that there was only one version, so I'm pretty sure it's the same as yours.

I haven't noticed anything about the time, I just let NTP handle the time. I do recall that there is a setting in the BIOS that you do need to pay attention to. It's called HPET or High Performance Event Timer, and it needs to be disabled. You'll see, or at least I did, a bunch of errors about the system clock/time or something being adjusted by so many seconds, and I don't recall if it caused crashes or hanging. If you google it you can read more about it, but I think it's a known problem with FreeBSD.

Note: those errors were on the console and in the log file (at least until a reboot), and they didn't happen immediately. Oh, just in case my Signature changes, for future reference, this is for a Supermicro X7SPA-H-O.

UPDATE: Just hooked up a monitor and we have the same BIOS version.

 

[TwinDaddyKev]

Just an update on my issue. It turned out that one of my sodimm is bad. I did a test on it and there were bunch of errors. I also test the other sodimm and it was good. I guess it's time to buy more ram.

I've been up and running with FN8.01 Beta4 with 4 1TB WD1001FALS with TLER enable running ZFS Raidz for over 10 hours without issues. I didn't do any ZFS tuning yet as I am only running with 2GB of ram. Am I supposed to put all the entry at the end of the loader.conf file? Mine looks like this...

#
# Boot loader file for FreeNAS. This relies on a hacked beastie.4th.
#
autoboot_delay="2"
loader_logo="freenas"
#Fix booting from USB device bug
kern.cam.boot_delay=10000

# GEOM support
geom_mirror_load="YES"
geom_stripe_load="YES"
geom_raid3_load="YES"
#geom_raid5_load="YES"
geom_gate_load="YES"
ntfs_load="YES"
smbfs_load="YES"
vm.kmem_size="1536M"
vm.kmem_size_max="2048M"
vfs.zfs.prefetch_disable="1"

hw.hptrr.attach_generic=0

Is that correct?

I've been copying test data and performance has been great averaging around 28-30MB/s via CIFS.

I also disable the High Performance Event Timer as you suggested and the BIOS clock setting is now correct.

Thanks for all your help protosd.

 

[ProtoSD]

Wow, well I guess that's good news, at least you know for sure where the problem is and move on to more more fun stuff.

The loader.conf you just posted looks fine, it looks like it's already been edited.
What other stuff were you talking about adding?
It really doesn't matter where you put it in the file.

Someone did some performance adjustments yesterday that had a big improvment on read performance. I'll have to hunt it down and update this post with a link so you can take advantage of it to.

You're welcome, glad I was able to help!

UPDATE: Here's a link to that thread I just mentioned:
http://forums.freenas.org/showthread.php?1226-Hyper-threading-and-Prefetch

 

[TwinDaddyKev]

I spent the entire weekend troubleshooting this issue and it turns out to be hardware issue. The loader.conf was a c&p from the old configuration. I reinstall FN8.01 Beta4 after I figured out it the bad memory module that crash FN.

I also have the Lian Li PC-Q08A like yours but the CD-ROM cover broke. It was very flimpsy. I had 6 drives in there before but the volume crashed...now I know it was due to memory issue. I'm thinking about buying another MB that support up to 16GB of memory. What do you think about this MB http://***.newegg.com/Product/Product.aspx?Item=N82E16813157247

 

[ProtoSD]

I don't like the fact that it only has 4 SATA ports, but the ability to add up to 16GB of RAM is nice.
There is a whole thread(s) about this board, it's very popular. I haven't read it, but if you're thinking
about this board, do a search for it.

You're right about the case, I wrote a review about it on Newegg. Mine was used and came damaged.
I don't like the fact that the drives are stacked horizontally because the heat just accumulates on
each drive as it goes up the stack. I was going to mod it so the drives were vertically oriented, but
don't have the time right now.

 

[patrickreid]

I was going to try adding these changes, but these parameters are not in my loader.conf. Running 8.3.0-beta-3 could it be in a different location? I am getting panics almost daily under high load