Periodic Kernel Panic

[Fingerpuppet]

About June 27th (after a GUI based update) my FreeNAS 9.10 server started behaving strangely. The encrypted ZFS pool would be locked and the uptime graph in the GUI shows huge holes. I didn't think that the server was rebooting, but apparently it was.

Anyway, I have 32 GB of ECC RAM on my E3-1231v3/Supermicro X10SL7-F system and the 16GB flash drive isn't enough for the crash dumps at /var/crash?

What are my steps to diagnose this? Any assistance would be GREATLY appreciated.

 

[Fingerpuppet]

Look what I found!

[root@freenas] /data/crash# ll
total 102
drwxrwxr-x 2 root www 20 Aug 2 03:56 ./
drwxr-xr-x 8 www www 14 Aug 2 03:56 ../
-rw-r--r-- 1 root www 2 Aug 2 03:56 bounds
-rw------- 1 root www 639 Jul 27 03:38 info.0
-rw------- 1 root www 639 Jul 31 04:18 info.1
-rw------- 1 root www 639 Aug 1 03:27 info.2
-rw------- 1 root www 638 Aug 2 03:56 info.3
-rw------- 1 root www 637 Jul 25 09:03 info.4
lrwxr-xr-x 1 root www 6 Aug 2 03:56 info.last@ -> info.3
-rw------- 1 root www 19953 Jul 27 03:38 textdump.tar.0.gz
-rw-r--r-- 1 root www 0 Jun 28 03:43 textdump.tar.0.gz.submitted
-rw------- 1 root www 20304 Jul 31 04:18 textdump.tar.1.gz
-rw-r--r-- 1 root www 0 Jun 29 03:42 textdump.tar.1.gz.submitted
-rw------- 1 root www 21358 Aug 1 03:27 textdump.tar.2.gz
-rw-r--r-- 1 root www 0 Jun 30 03:42 textdump.tar.2.gz.submitted
-rw------- 1 root www 18543 Aug 2 03:56 textdump.tar.3.gz
-rw-r--r-- 1 root www 0 Jul 6 03:41 textdump.tar.3.gz.submitted
-rw------- 1 root www 16652 Jul 25 09:03 textdump.tar.4.gz
-rw-r--r-- 1 root www 0 Jul 25 09:04 textdump.tar.4.gz.submitted
lrwxr-xr-x 1 root www 17 Aug 2 03:56 textdump.tar.last.gz@ -> textdump.tar.3.gz

Now how do I make sense of it?

 

[Fingerpuppet]

[root@freenas] /data/crash# less info.0
Dump header from device /dev/dumpdev
Architecture: amd64
Architecture Version: 1
Dump Length: 581120B (0 MB)
Blocksize: 512
Dumptime: Wed Jul 27 08:35:19 2016
Hostname: freenas.local
Magic: FreeBSD Text Dump
Version String: FreeBSD 10.3-STABLE #0 455d13d(9.10-STABLE): Sun Jun 26 22:47:03 PDT 2016
root@build.ixsystems.com:/tank/home/nightlies/build-freenas9/_BE/objs/tank/home/nightlies/build-freenas9/_BE/true Panic String: I/O to pool 'AlphaPool' appears to be hung on vdev guid 1086804107444168452 at '/dev/gptid/ec140be5-f3a9-11e2-a1b1-94de802cbfa5.eli'.
Dump Parity: 1017242713
Bounds: 0
Dump Status: good
[root@freenas] /data/crash# less info.3
Dump header from device /dev/dumpdev
Architecture: amd64
Architecture Version: 1
Dump Length: 153088B (0 MB)
Blocksize: 512
Dumptime: Tue Aug 2 08:52:50 2016
Hostname: freenas.local
Magic: FreeBSD Text Dump
Version String: FreeBSD 10.3-STABLE #0 455d13d(9.10-STABLE): Sun Jun 26 22:47:03 PDT 2016
root@build.ixsystems.com:/tank/home/nightlies/build-freenas9/_BE/objs/tank/home/nightlies/build-freenas9/_BE/true Panic String: I/O to pool 'AlphaPool' appears to be hung on vdev guid 1086804107444168452 at '/dev/gptid/ec140be5-f3a9-11e2-a1b1-94de802cbfa5.eli'.
Dump Parity: 419747417
Bounds: 3
Dump Status: good
[root@freenas] /data/crash# less info.4
Dump header from device /dev/dumpdev
Architecture: amd64
Architecture Version: 1
Dump Length: 99840B (0 MB)
Blocksize: 512
Dumptime: Mon Jul 25 14:00:16 2016
Hostname: freenas.local
Magic: FreeBSD Text Dump
Version String: FreeBSD 10.3-STABLE #0 455d13d(9.10-STABLE): Sun Jun 26 22:47:03 PDT 2016
root@build.ixsystems.com:/tank/home/nightlies/build-freenas9/_BE/objs/tank/home/nightlies/build-freenas9/_BE/true Panic String: I/O to pool 'AlphaPool' appears to be hung on vdev guid 1086804107444168452 at '/dev/gptid/ec140be5-f3a9-11e2-a1b1-94de802cbfa5.eli'.
Dump Parity: 194044761
Bounds: 4
Dump Status: good
info.4 (END)


I'm using WD Red 3TB drives and they are encrypted.

 

[Fingerpuppet]

SMARTCTL output for the two drives in my mirrored and encrypted zfs pool...

 

[Niklas Ekman]

Hi!

I've just come into the world of FreeNAS. I've just recently bought all the hardware and assembled my server. I've been quite (I hope...) thorough when researching which hardware is best suited for FreeNAS and I hope I've bought the right stuff. This is my setup:

Motherboard: Asrock C236 WSI
RAM: 16 GB Kingston ValueRAM - DDR4 - 8 GB - DIMM 288-pin - 2133 MHz / PC4-17000 - ECC
CPU: Intel Core i3 6100T - 3.2 GHz
Case: Fractal Design Node 304
PSU: Corsair RMx Series RM550x
HDD: WD Red & Samsung drives

I'm periodically getting kernel panics, which annoys me quite a lot. I've tried to search the forums for help but I'm not getting anywhere. I read in this post that I should get something in the /data/crash folder, but it's empty. Also the /var/log/messages file doesn't record anything concerning the crash. I've got the crash both under normal runtime and also during reboot of the server.

I'm a bit at loss now how to continue searching for the error. Can someone help me with instructions on how to enable output to the /data/crash folder so that I at least can see what's going wrong? I'm getting to a prompt which says "db>", maybe I can do something there? Any help would be really helpful.

/Niklas

 

[Fingerpuppet]

I'm still experiencing the kernel panics and it made me start thinking about the silly mantra that it's not a bug, it's a feature. But aside from being flippant, I continued to think about when a kernel panic would be "reasonable and normal" behavior.

I don't think that a kernel panic should ever be normal. Reasonable, maybe. But that comes with some exceptions. Reasonable only if there is a truly unaccountable hardware failure. But I don't see any flaky failures in the smartctl output.

The system runs fine for extended periods of time, but only if the pool remains locked. Unlock the pool and you can use it, but it seems that the clock starts ticking and only a few hours later, it will panic.

Does this qualify as a bug?

 

[dlavigne]

Yes. Make it at bugs.freenas.org, include a debug (made from System -> Advanced -> Save Debug) in the ticket, then post the issue number here.

 

[Fingerpuppet]

Issue #16664 opened with debug .tgz attached. Thank you, Dru.