FreeNAS 11.2U5 kernel panics

[ziphe]

Hi,

I've been having kernel panics in 11.2U5, but it was also happening on earlier versions. I have the /data/crash dumps, what can I do with this information to help troubleshoot this issue? It happens at around 13-14 days of uptime, and it's a pretty busy box.

More info:

Fatal trap 12: page fault while in kernel mode
cpuid = 4; apic id = 04
fault virtual address = 0x9b
fault code = supervisor read data, page not present
instruction pointer = 0x20:0xffffffff803f3714
stack pointer = 0x28:0xfffffe2016267a60
frame pointer = 0x28:0xfffffe2016267a90
code segment = base 0x0, limit 0xfffff, type 0x1b
= DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags = interrupt enabled, resume, IOPL = 0
current process = 49134 (rclone)


Thanks in advance

 

[Meyers]

I've been experiencing unexpected reboots for months. This post lead me to check out /data/crash and sure enough:

Code:

Fatal trap 12: page fault while in kernel mode
cpuid = 4; apic id = 08
fault virtual address   = 0x3b8
fault code              = supervisor read data, page not present
instruction pointer     = 0x20:0xffffffff80ab391d
stack pointer           = 0x28:0xfffffe08615ad1b0
frame pointer           = 0x28:0xfffffe08615ad250
code segment            = base 0x0, limit 0xfffff, type 0x1b
                        = DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags        = interrupt enabled, resume, IOPL = 0
current process         = 6505 (qbittorrent-nox)

BTW you'll want to post all your relevant hardware and software details if you want any help. See what I've put in my sig for an example.

 

[Meyers]

Here is my ongoing ticket.

 

[ziphe]

I am not 100% certain, but my gut feeling is that high load inside a jail is triggering the panic. I've moved the rclone jobs to a native FreeBSD vm on a different host at this point, so if my theory is correct my kernel panics should go away. I have 5 kernel panics in a row (10-14 days apart) all showing the same trap 12 same supervisor read not present error, and always the running process is rclone. In my opinion this is a smoking gun that this is a software issue and not a hardware problem, which would typically panic the system in a random process, and sometimes even causing different panic types.

CPU: Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz (2200.06-MHz K8-class CPU)
real memory = 137434759168 (131068 MB)
SuperMicro 5049P-E1CTR36L
Chelsio T520

Here are my panics:

Fatal trap 12: page fault while in kernel mode
cpuid = 4; apic id = 04
fault virtual address = 0x9b
fault code = supervisor read data, page not present
instruction pointer = 0x20:0xffffffff803f3714
stack pointer = 0x28:0xfffffe2016267a60
frame pointer = 0x28:0xfffffe2016267a90
code segment = base 0x0, limit 0xfffff, type 0x1b
= DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags = interrupt enabled, resume, IOPL = 0
current process = 49134 (rclone)

Fatal trap 12: page fault while in kernel mode
cpuid = 8; apic id = 08
fault virtual address = 0x9b
fault code = supervisor read data, page not present
instruction pointer = 0x20:0xffffffff803f3674
stack pointer = 0x28:0xfffffe2015de8a60
frame pointer = 0x28:0xfffffe2015de8a90
code segment = base 0x0, limit 0xfffff, type 0x1b
= DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags = interrupt enabled, resume, IOPL = 0
current process = 52098 (rclone)

Fatal trap 12: page fault while in kernel mode
cpuid = 3; apic id = 03
fault virtual address = 0x9b
fault code = supervisor read data, page not present
instruction pointer = 0x20:0xffffffff803f3674
stack pointer = 0x28:0xfffffe201589eac0
frame pointer = 0x28:0xfffffe201589eaf0
code segment = base 0x0, limit 0xfffff, type 0x1b
= DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags = interrupt enabled, resume, IOPL = 0
current process = 15932 (rclone)

Fatal trap 12: page fault while in kernel mode
cpuid = 11; apic id = 11
fault virtual address = 0x90
fault code = supervisor read data, page not present
instruction pointer = 0x20:0xffffffff803f3674
stack pointer = 0x28:0xfffffe2014667ac0
frame pointer = 0x28:0xfffffe2014667af0
code segment = base 0x0, limit 0xfffff, type 0x1b
= DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags = interrupt enabled, resume, IOPL = 0
current process = 86547 (rclone)

Fatal trap 12: page fault while in kernel mode
cpuid = 16; apic id = 16
fault virtual address = 0x90
fault code = supervisor read data, page not present
instruction pointer = 0x20:0xffffffff803f3674
stack pointer = 0x28:0xfffffe2015c7ba60
frame pointer = 0x28:0xfffffe2015c7ba90
code segment = base 0x0, limit 0xfffff, type 0x1b
= DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags = interrupt enabled, resume, IOPL = 0
current process = 29970 (rclone)