New FreeNAS 9.1.1 drops and reboots?

[Kleines]

BuildFreeNAS-9.1.0-RELEASE-x64 (dff7d13)
PlatformAMD Opteron(tm) Processor 4238
Memory49107MB
Hardware: Dell Poweredge R515

This is a new FreeNAS build on existing hardware. The server's been in production as another OS for a year before and is known good. I've built it as an iSCSI shared datastore for a vSphere cluster of two new matching PowerEdge R510 boxes.

The issue I'm having is the FreeNAS box is apparently either dropping or rebooting every night. When it does, the cluster of course acts badly and freaks out. To troubleshoot I've set up a separate syslog server and redirected to it.

Here's what I get on both sided of the freeze/reboot:

1. 9/4/2013 3:50 Debug 192.168.10.79 [middleware.notifier:219] Popen()ing: zpool list -H -o health Storage1
   9/4/2013 3:50 Debug 192.168.10.79 [middleware.notifier:219] Popen()ing: zpool list -H -o health Storage1
   9/4/2013 3:50 Debug 192.168.10.79 [middleware.notifier:4106] sysctlbyname: kern.geom.confxml
   9/4/2013 3:51 Info 192.168.10.79 (root) CMD (/usr/local/bin/python /usr/local/www/freenasUI/tools/autosnap.py > /dev/null 2>&1)
   9/4/2013 3:51 Debug 192.168.10.79 [tools.autosnap:107] Checking if process 5156 is still alive
   9/4/2013 3:51 Debug 192.168.10.79 [tools.autosnap:114] Process 5156 gone
   9/4/2013 3:52 Info 192.168.10.79 (root) CMD (/usr/local/bin/python /usr/local/www/freenasUI/tools/autosnap.py > /dev/null 2>&1)
   9/4/2013 3:52 Debug 192.168.10.79 [tools.autosnap:107] Checking if process 5177 is still alive
   9/4/2013 3:52 Debug 192.168.10.79 [tools.autosnap:114] Process 5177 gone
   9/4/2013 3:53 Info 192.168.10.79 (root) CMD (/usr/local/bin/python /usr/local/www/freenasUI/tools/autosnap.py > /dev/null 2>&1)
   9/4/2013 3:53 Debug 192.168.10.79 [tools.autosnap:107] Checking if process 5186 is still alive
   9/4/2013 3:53 Debug 192.168.10.79 [tools.autosnap:114] Process 5186 gone
   9/4/2013 3:54 Info 192.168.10.79 (root) CMD (/usr/local/bin/python /usr/local/www/freenasUI/tools/autosnap.py > /dev/null 2>&1)
   9/4/2013 3:54 Debug 192.168.10.79 [tools.autosnap:107] Checking if process 5195 is still alive
   9/4/2013 3:54 Debug 192.168.10.79 [tools.autosnap:114] Process 5195 gone
   9/4/2013 3:54 Info 192.168.10.79 ABORT_TASK
   9/4/2013 4:00 Info 192.168.10.79 (root) CMD (/usr/local/bin/python /usr/local/www/freenasUI/tools/autosnap.py > /dev/null 2>&1)
   9/4/2013 4:00 Info 192.168.10.79 (root) CMD (/usr/local/bin/python /usr/local/www/freenasUI/tools/alert.py > /dev/null 2>&1)
   9/4/2013 4:00 Info 192.168.10.79 (root) CMD (/usr/local/bin/python /usr/local/bin/mfistatus.py > /dev/null 2>&1)
   9/4/2013 4:00 Info 192.168.10.79 (root) CMD (/bin/sh /root/save_rrds.sh)
   9/4/2013 4:00 Info 192.168.10.79 (root) CMD (newsyslog)
   9/4/2013 4:00 Info 192.168.10.79 (operator) CMD (/usr/libexec/save-entropy)
   9/4/2013 4:00 Info 192.168.10.79 (root) CMD (/usr/libexec/atrun)
   9/4/2013 4:00 Info 192.168.10.79 synchronized to 192.210.137.119
   9/4/2013 4:00 Debug 192.168.10.79 [middleware.notifier:219] Popen()ing: zpool list -H -o health Storage1
   9/4/2013 4:00 Debug 192.168.10.79 [middleware.notifier:219] Popen()ing: zpool list -H -o health Storage1
   9/4/2013 4:00 Debug 192.168.10.79 [middleware.notifier:4106] sysctlbyname: kern.geom.confxml
   9/4/2013 4:00 Info 192.168.10.79 Sep 4 03:59:33 freenas last message repeated 23 times
   9/4/2013 4:01 Info 192.168.10.79 (root) CMD (adjkerntz -a)
   9/4/2013 4:01 Info 192.168.10.79 (root) CMD (/usr/local/bin/python /usr/local/www/freenasUI/tools/autosnap.py > /dev/null 2>&1)​
   How do I go about diagnosing the real issue?​

 

[Kleines]

Update: Found a kernel panic repeating all over the place, so far 54 dumps to go through. I'll update once I'm through them.

Note to self: tell the teddy bear the problem first...

 

[Kleines]

Okay, I've got the most recent crashdump, but I'll be damned if I know how to interpret it. I've posted it at http://pastebin.com/uL4tByBb - any help would be greatly appreciated!

 

[dlavigne]

Please open an issue at bugs.freenas.org. Include a summary of the issue and the pastebin URL to the crashdump.

 

[Kleines]

Please open an issue at bugs.freenas.org. Include a summary of the issue and the pastebin URL to the crashdump.

Thanks, posted to them.

 

[wharfratjoe]

What is the ticket number for this. I am having a similar issue and want to reveiw the ticket before I create one.

 

[Kleines]

What is the ticket number for this. I am having a similar issue and want to reveiw the ticket before I create one.

Ticket# 3116, find it here.