9.2 Random Reboots

Status
Not open for further replies.

BTW

Dabbler
Joined
Feb 1, 2014
Messages
33
Hello,
I have a new system (hardware & OS) that I have configured (attached is the output of hardware).

Problem:
My problem is my system reboots randomly after a syslogd operation. Here is my syslog dump (full log attached). You will notice system reboots at 02:27 & 03:51

Any suggestions?
Thanks

Code:
Line 4417: Feb  3 02:15:01 192.168.8.110 alert.py: [middleware.notifier:213] Popen()ing: zpool list -H -o health agg01
    Line 4422: Feb  3 02:15:01 192.168.8.110 alert.py: [middleware.notifier:213] Popen()ing: zpool list -H -o health agg01
    Line 4423: Feb  3 02:16:23 192.168.8.110 smartd[2734]: Device: /dev/ada0, SMART Prefailure Attribute: 1 Raw_Read_Error_Rate changed from 115 to 119
    Line 4424: Feb  3 02:16:23 192.168.8.110 smartd[2734]: Device: /dev/ada1, SMART Prefailure Attribute: 1 Raw_Read_Error_Rate changed from 112 to 117
    Line 4425: Feb  3 02:16:24 192.168.8.110 smartd[2734]: Device: /dev/ada2, SMART Prefailure Attribute: 1 Raw_Read_Error_Rate changed from 117 to 105
    Line 4426: Feb  3 02:16:24 192.168.8.110 smartd[2734]: Device: /dev/ada2, SMART Prefailure Attribute: 7 Seek_Error_Rate changed from 62 to 63
    Line 4427: Feb  3 02:16:24 192.168.8.110 smartd[2734]: Device: /dev/ada3, SMART Prefailure Attribute: 1 Raw_Read_Error_Rate changed from 117 to 112
    Line 4428: Feb  3 02:16:24 192.168.8.110 smartd[2734]: Device: /dev/ada4, SMART Prefailure Attribute: 1 Raw_Read_Error_Rate changed from 120 to 114
    Line 4430: Feb  3 02:20:00 192.168.8.110 /usr/sbin/cron[4814]: (root) CMD (/usr/local/bin/python /usr/local/www/freenasUI/tools/alert.py > /dev/null 2>&1)
    Line 4431: Feb  3 02:20:00 192.168.8.110 /usr/sbin/cron[4815]: (root) CMD (/usr/libexec/atrun)
    Line 4433: Feb  3 02:20:01 192.168.8.110 alert.py: [middleware.notifier:213] Popen()ing: zpool list -H -o health agg01
    Line 4438: Feb  3 02:20:01 192.168.8.110 alert.py: [middleware.notifier:213] Popen()ing: zpool list -H -o health agg01
    Line 4439: Feb  3 02:22:00 192.168.8.110 /usr/sbin/cron[4851]: (operator) CMD (/usr/libexec/save-entropy)
    Line 4440: Feb  3 02:25:00 192.168.8.110 /usr/sbin/cron[4858]: (root) CMD (/usr/local/bin/python /usr/local/www/freenasUI/tools/alert.py > /dev/null 2>&1)
    Line 4441: Feb  3 02:25:00 192.168.8.110 /usr/sbin/cron[4859]: (root) CMD (/usr/libexec/atrun)
    Line 4443: Feb  3 02:25:02 192.168.8.110 alert.py: [middleware.notifier:213] Popen()ing: zpool list -H -o health agg01
    Line 4448: Feb  3 02:25:02 192.168.8.110 alert.py: [middleware.notifier:213] Popen()ing: zpool list -H -o health agg01
    Line 4449: Feb  3 02:27:33 192.168.8.110 syslogd: restart
    Line 4450: Feb  3 02:27:33 192.168.8.110 syslogd: kernel boot file is /boot/kernel/kernel
    Line 4451: Feb  3 02:27:33 192.168.8.110 kernel: kernel trap 9 with interrupts disabled
    Line 4452: Feb  3 02:27:33 192.168.8.110 kernel:
    Line 4453: Feb  3 02:27:33 192.168.8.110 kernel:
    Line 4454: Feb  3 02:27:33 192.168.8.110 kernel: Fatal trap 9: general protection fault while in kernel mode
    Line 4455: Feb  3 02:27:33 192.168.8.110 kernel: cpuid = 0; apic id = 00
    Line 4456: Feb  3 02:27:33 192.168.8.110 kernel: instruction pointer#011= 0x20:0xffffffff80820b35



Code:
Line 5089: Feb  3 03:40:00 192.168.8.110 /usr/sbin/cron[5673]: (root) CMD (/usr/local/bin/python /usr/local/www/freenasUI/tools/alert.py > /dev/null 2>&1)
    Line 5090: Feb  3 03:40:00 192.168.8.110 /usr/sbin/cron[5674]: (root) CMD (/usr/libexec/atrun)
    Line 5092: Feb  3 03:40:01 192.168.8.110 alert.py: [middleware.notifier:213] Popen()ing: zpool list -H -o health agg01
    Line 5097: Feb  3 03:40:01 192.168.8.110 alert.py: [middleware.notifier:213] Popen()ing: zpool list -H -o health agg01
    Line 5098: Feb  3 03:44:00 192.168.8.110 /usr/sbin/cron[5710]: (operator) CMD (/usr/libexec/save-entropy)
    Line 5099: Feb  3 03:45:00 192.168.8.110 /usr/sbin/cron[5717]: (root) CMD (/usr/local/bin/python /usr/local/www/freenasUI/tools/alert.py > /dev/null 2>&1)
    Line 5100: Feb  3 03:45:00 192.168.8.110 /usr/sbin/cron[5718]: (root) CMD (/usr/libexec/atrun)
    Line 5101: Feb  3 03:45:01 192.168.8.110 alert.py: [middleware.notifier:213] Popen()ing: zpool list -H -o health agg01
    Line 5107: Feb  3 03:45:01 192.168.8.110 alert.py: [middleware.notifier:213] Popen()ing: zpool list -H -o health agg01
    Line 5108: Feb  3 03:50:00 192.168.8.110 /usr/sbin/cron[5755]: (root) CMD (/usr/local/bin/python /usr/local/www/freenasUI/tools/alert.py > /dev/null 2>&1)
    Line 5109: Feb  3 03:50:00 192.168.8.110 /usr/sbin/cron[5756]: (root) CMD (/usr/libexec/atrun)
    Line 5115: Feb  3 03:51:35 192.168.8.110 syslogd: restart
    Line 5116: Feb  3 03:51:35 192.168.8.110 syslogd: kernel boot file is /boot/kernel/kernel
    Line 5117: Feb  3 03:51:35 192.168.8.110 kernel: Copyright (c) 1992-2013 The FreeBSD Project.
    Line 5118: Feb  3 03:51:35 192.168.8.110 kernel: Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
    Line 5119: Feb  3 03:51:35 192.168.8.110 kernel: The Regents of the University of California. All rights reserved.
    Line 5120: Feb  3 03:51:35 192.168.8.110 kernel: FreeBSD is a registered trademark of The FreeBSD Foundation.
    Line 5121: Feb  3 03:51:35 192.168.8.110 kernel: FreeBSD 9.2-RELEASE #0 r+2315ea3: Fri Dec 20 12:48:50 PST 2013
    Line 5122: Feb  3 03:51:35 192.168.8.110 kernel: root@build.ixsystems.com:/tank/home/jkh/checkout/freenas/os-base/amd64/tank/home/jkh/checkout/freenas/FreeBSD/src/sys/FREENAS.amd64 amd64
    Line 5123: Feb  3 03:51:35 192.168.8.110 kernel: gcc version 4.2.1 20070831 patched [FreeBSD]
    Line 5124: Feb  3 03:51:35 192.168.8.110 kernel: CPU: AMD E-350 Processor (1600.00-MHz K8-class CPU)
 

Attachments

  • hardware.txt
    8.7 KB · Views: 357
  • CDANAS001_filtered.txt
    376.7 KB · Views: 390

Hi-Liter

Explorer
Joined
Oct 6, 2011
Messages
83
I was experiencing something similar, I tried disabling USB3 support and deactivating all the auto tunables - which helped, I have yet to figure out which one it was.
 

Yatti420

Wizard
Joined
Aug 12, 2012
Messages
1,437
What specs? I wouldn't use autotune.. Unless you have tons of ram.. USB3 can be an issue.. I'm not sure though if you are running 9.2.0 it should be disabled by default for FreeNAS.. BIOS USB settings could cause issues I guess..
 

Hi-Liter

Explorer
Joined
Oct 6, 2011
Messages
83
USB3 can be an issue.. I'm not sure though if you are running 9.2.0 it should be disabled by default for FreeNAS.
Yes it is, I enabled it because I was using an external USB drive to transfer data to FreeNAS.
 

BTW

Dabbler
Joined
Feb 1, 2014
Messages
33
My FreeNAS version is FreeNAS-9.2.0-RELEASE-x64 (ab098f4). I have not enabled any USB3 support or used any of the Auto-tunes. This is an extremely basic setup at this point cause I want to ensure it is solid before I migrate data to it (case in point).
Thanks
 

BTW

Dabbler
Joined
Feb 1, 2014
Messages
33
No suggestions??? Anyone???

Here is a another question. How do I attach a device (USB preferably, if I have to, a disk) as a dump drive to capture the kernel dumps during a crash?
Thanks
 

Dusan

Guru
Joined
Jan 29, 2013
Messages
1,165
Crash dumps are saved to the system USB stick (/data/crash).
 

DrKK

FreeNAS Generalissimo
Joined
Oct 15, 2013
Messages
3,630
Wow.

What the heck are all those SMART errors on *ALL* the devices, Dusan?

And AMD E-350? That's pretty lightweight (and not supporting ECC RAM).
 

BTW

Dabbler
Joined
Feb 1, 2014
Messages
33
So there is nothing in my (/data/crash) after a system reboot. Is there something I am missing that needs to be configured to capture that or is that set by default?
 

Dusan

Guru
Joined
Jan 29, 2013
Messages
1,165
So there is nothing in my (/data/crash) after a system reboot. Is there something I am missing that needs to be configured to capture that or is that set by default?
It should work automatically. You can check that your /etc/rc.conf file contains a dumpdir="/data/crash" line. But I think you need to have at least one swap device active for the dumps to work.
 

cyberjock

Inactive Account
Joined
Mar 25, 2012
Messages
19,526
Geez.. with read error rates like that something serious is broken! my first guess is bad PSU!
 

BTW

Dabbler
Joined
Feb 1, 2014
Messages
33
I checked my rc.conf and the entry is there. I did run the swapinfo command to verify there was a swap but shows no swaps. Are you referring to a different swap then the 2GB that is created when you create a pool? I have created one pool and presented it as iSCSI (for testing).
Thanks
 

Dusan

Guru
Joined
Jan 29, 2013
Messages
1,165
I checked my rc.conf and the entry is there. I did run the swapinfo command to verify there was a swap but shows no swaps. Are you referring to a different swap then the 2GB that is created when you create a pool?
No, it's the 2GB per device swap. Can you please post output of "cat /etc/fstab" and "gpart show"?
 

BTW

Dabbler
Joined
Feb 1, 2014
Messages
33
Fstab:
Code:
/dev/ufs/FreeNASs1a / ufs ro 1 1
/dev/ufs/FreeNASs3 /cfg ufs rw,noauto 2 2
/dev/ufs/FreeNASs4 /data ufs rw,noatime 2 2


Gpart:
Code:
=>      63  15131573  da0  MBR  (7.2G)
        63  1930257    1  freebsd  [active]  (942M)
  1930320        63      - free -  (31k)
  1930383  1930257    2  freebsd  (942M)
  3860640      3024    3  freebsd  (1.5M)
  3863664    41328    4  freebsd  (20M)
  3904992  11226644      - free -  (5.4G)
 
=>      0  1930257  da0s1  BSD  (942M)
        0      16        - free -  (8.0k)
      16  1930241      1  !0  (942M)
 

BTW

Dabbler
Joined
Feb 1, 2014
Messages
33
I caught the system during a failed reboot after a crash. Any hints?

I think I will run a lower version to see if I get the same results.
c:\users\brandt\desktop\1391607583673.jpg
 

Attachments

  • 1391607583673.jpg
    1391607583673.jpg
    43.3 KB · Views: 337

DrKK

FreeNAS Generalissimo
Joined
Oct 15, 2013
Messages
3,630
That looks hardware-related sir. I am no expert. Maybe someone will chime in.
 

BTW

Dabbler
Joined
Feb 1, 2014
Messages
33
@DrKK
I believe you are correct. I ran some mem tests and it seems my memory is not the exact model on the QVL. I tried the same tests with memory on the QVL and all the tests passed. Seems to be down to a picky mobo.
 

DrKK

FreeNAS Generalissimo
Joined
Oct 15, 2013
Messages
3,630
Excellent. Usually, these things are logical.
 
Status
Not open for further replies.
Top