Regular Crash/Kernel Panic - Help

[Moss_Cadu]

We have a freenas system:
FreeBSD 10.3-STABLE #0 r295946+1805185(9.10.2-STABLE):
CPU: Intel(R) Xeon(R) CPU E5-2609 v3 @ 1.90GHz (1900.04-MHz K8-class CPU)
real memory = 36507222016 (34816 MB)
avail memory = 33126821888 (31592 MB)
on a supermicro X10DRH-iT

for a while 23rd feb to 24th feb it has been crashing and restarting several times.
attached are log files from the system but they share a similar message
Fatal trap 12: page fault while in kernel mode
cpuid = 5; apic id = 0a
fault virtual address = 0x20
fault code = supervisor read data, page not present
instruction pointer = 0x20:0xffffffff80395503
stack pointer = 0x28:0xfffffe0a8d14f890
frame pointer = 0x28:0xfffffe0a8d14f8a0
code segment = base 0x0, limit 0xfffff, type 0x1b
= DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags = interrupt enabled, resume, IOPL = 0
current process = 6 (arc_reclaim_thread)

this forum post reports something similar with an arc_reclaim_thread page fault.
https://forums.freenas.org/index.ph...ode-current-process-arc_reclaim_thread.38259/
It's in production so I haven't had the scheduled downtime to do the memory test.

Is there anything else that we could do to diagnose the issue?

 

[dlavigne]

Please also post /var/log/messages.

 

[Moss_Cadu]

okay I have put /var/log/messages on the main post but I believe it has been rotated on the 28th

 

[DrKK]

I'd be interested in knowing what the resolution to this is. Have you verified your install? Should be somewhere in the System->update part of the GUI.

 

[Moss_Cadu]

I've done the verification scan and the UI claims it's the files were verified successfully. (again attached to the main post)

 

[DrKK]

I've done the verification scan and the UI claims it's the files were verified successfully. (again attached to the main post)

Oh I'm sorry, I missed that you already did that.

 

[Moss_Cadu]

Oops, you didn't miss anything it was an edit!

 

[DrKK]

Made a few calls. I am told that if the address reported for the trap 12 changes, then you almost certainly have a hardware issue. If the address is always the same, then, it's likely a bug.

 

[Moss_Cadu]

Well the four crashes I have on file all have the same pointer addresses......

instruction pointer = 0x20:0xffffffff80395503
stack pointer = 0x28:0xfffffe0a8d14f890
frame pointer = 0x28:0xfffffe0a8d14f8a0

Code:

Fatal trap 12: page fault while in kernel mode
cpuid = 2; apic id = 04
fault virtual address   = 0x20
fault code     = supervisor read data, page not present
instruction pointer   = 0x20:0xffffffff80395503
stack pointer    = 0x28:0xfffffe0a8d14f890
frame pointer    = 0x28:0xfffffe0a8d14f8a0
code segment     = base 0x0, limit 0xfffff, type 0x1b
       = DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags   = interrupt enabled, resume, IOPL = 0
current process     = 6 (arc_reclaim_thread)

Fatal trap 12: page fault while in kernel mode
cpuid = 5; apic id = 0a
fault virtual address   = 0x20
fault code     = supervisor read data, page not present
instruction pointer   = 0x20:0xffffffff80395503
stack pointer    = 0x28:0xfffffe0a8d14f890
frame pointer    = 0x28:0xfffffe0a8d14f8a0
code segment     = base 0x0, limit 0xfffff, type 0x1b
       = DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags   = interrupt enabled, resume, IOPL = 0
current process     = 6 (arc_reclaim_thread)

Fatal trap 12: page fault while in kernel mode
cpuid = 3; apic id = 06
fault virtual address   = 0x20
fault code     = supervisor read data, page not present
instruction pointer   = 0x20:0xffffffff80395503
stack pointer    = 0x28:0xfffffe0a8d14f890
frame pointer    = 0x28:0xfffffe0a8d14f8a0
code segment     = base 0x0, limit 0xfffff, type 0x1b
       = DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags   = interrupt enabled, resume, IOPL = 0
current process     = 6 (arc_reclaim_thread)

Fatal trap 12: page fault while in kernel mode
cpuid = 0; apic id = 00
fault virtual address   = 0x20
fault code     = supervisor read data, page not present
instruction pointer   = 0x20:0xffffffff80395503
stack pointer    = 0x28:0xfffffe0a8d14f890
frame pointer    = 0x28:0xfffffe0a8d14f8a0
code segment     = base 0x0, limit 0xfffff, type 0x1b
       = DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags   = interrupt enabled, resume, IOPL = 0
current process     = 6 (arc_reclaim_thread)

Fatal trap 12: page fault while in kernel mode
cpuid = 5; apic id = 0a
fault virtual address   = 0x20
fault code     = supervisor read data, page not present
instruction pointer   = 0x20:0xffffffff80395503
stack pointer    = 0x28:0xfffffe0a8d14f890
frame pointer    = 0x28:0xfffffe0a8d14f8a0
code segment     = base 0x0, limit 0xfffff, type 0x1b
       = DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags   = interrupt enabled, resume, IOPL = 0
current process     = 6 (arc_reclaim_thread)

 

[Moss_Cadu]

I've run a memory test and its come clean and then we had another crash with the same pointer addresses
Dump files attached.

Code:

Fatal trap 12: page fault while in kernel mode
cpuid = 2; apic id = 04
fault virtual address   = 0x20
fault code	 = supervisor read data, page not present
instruction pointer   = 0x20:0xffffffff80395503
stack pointer	= 0x28:0xfffffe0a8d14f890
frame pointer	= 0x28:0xfffffe0a8d14f8a0
code segment	 = base 0x0, limit 0xfffff, type 0x1b
	   = DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags   = interrupt enabled, resume, IOPL = 0
current process	 = 6 (arc_reclaim_thread)

**edit attached /var/log/messages

 

[dlavigne]

If you haven't already, it is worth creating a bug report at bugs.freenas.org. Post the issue number here.

 

[Moss_Cadu]

Done,
https://bugs.freenas.org/issues/21850

 

[Moss_Cadu]

The issue is persisting and I haven't had any more dump files created since the 5th of march. Has anyone got expertise in reading the dmesg log from this box?