Random Shutdowns - Help Finding the Cause?

[xCatalystx]

So this has happened twice now in the last week. I've come home to find the FreeNAS powered off. I don't think it is the UPS because I would expect other things to be off aswell. UPS doesn't log any power event either.

What's the best way to debug this software side for issues?
I haven't alot of experience with FreeBSD.

EDIT: I should mention I was running 9.10.2-U1 but just patched it to U5

 

[SweetAndLow]

So this has happened twice now in the last week. I've come home to find the FreeNAS powered off. I don't think it is the UPS because I would expect other things to be off aswell. UPS doesn't log any power event either.

What's the best way to debug this software side for issues?
I haven't alot of experience with FreeBSD.

EDIT: I should mention I was running 9.10.2-U1 but just patched it to U5

Hardware specs? Check /var/log/messages around the time of the outage.

Sent from my Nexus 5X using Tapatalk

 

[xCatalystx]

Heres the output of messages:

Code:

Jun 28 13:37:07 freenas devd: notify_clients: dropping unresponsive client: Broken pipe
Jun 28 13:37:07 freenas kernel: epair0a: link state changed to DOWN
Jun 28 13:37:07 freenas kernel: epair0a: link state changed to DOWN
Jun 28 13:37:07 freenas kernel: epair0b: link state changed to DOWN
Jun 28 13:37:07 freenas kernel: epair0b: link state changed to DOWN
Jun 28 13:37:10 freenas kernel: ifa_del_loopback_route: deletion failed: 48
Jun 28 13:37:10 freenas Freed UMA keg (udp_inpcb) was not empty (240 items).  Lost 24 pages of memory.
Jun 28 13:37:10 freenas Freed UMA keg (udpcb) was not empty (2171 items).  Lost 13 pages of memory.
Jun 28 13:37:10 freenas Freed UMA keg (tcptw) was not empty (630 items).  Lost 14 pages of memory.
Jun 28 13:37:10 freenas Freed UMA keg (tcp_inpcb) was not empty (240 items).  Lost 24 pages of memory.
Jun 28 13:37:10 freenas Freed UMA keg (tcpcb) was not empty (72 items).  Lost 24 pages of memory.
Jun 28 13:37:10 freenas Freed UMA keg (ripcb) was not empty (240 items).  Lost 24 pages of memory.
Jun 28 13:37:10 freenas hhook_vnet_uninit: hhook_head type=1, id=1 cleanup required
Jun 28 13:37:10 freenas hhook_vnet_uninit: hhook_head type=1, id=0 cleanup required
Jun 28 13:37:10 freenas kernel: epair1a: link state changed to DOWN
Jun 28 13:37:10 freenas kernel: epair1a: link state changed to DOWN
Jun 28 13:37:10 freenas kernel: epair1b: link state changed to DOWN
Jun 28 13:37:10 freenas kernel: epair1b: link state changed to DOWN
Jun 28 13:38:36 freenas rc.shutdown: 90 second watchdog timeout expired. Shutdown terminated.
Jun 28 13:38:36 freenas syslog-ng[1521]: syslog-ng shutting down; version='3.6.4'
Jun 28 13:38:36 freenas init: /bin/sh on /etc/rc.shutdown terminated abnormally, going to single user mode
Jun 28 13:38:36 freenas ntpd[2030]: ntpd exiting on signal 15 (Terminated)
Jun 28 13:38:36 freenas upsd[2139]: mainloop: Interrupted system call

Theres nothing for hours before that, and afterwards is me booting the system

Specs: Supermicro X10SL7-F MB, Intel Xeon E3-1231 v3, 32GB ECC Memory, 6x 3tb Raidz2

I happened to be home today when it happened again, i heard windows error and saw my drives were disconnected. Couldn't ping, looked at IPMI and the system state was powered off.

 

[SweetAndLow]

Heres the output of messages:

Code:

Jun 28 13:37:07 freenas devd: notify_clients: dropping unresponsive client: Broken pipe
Jun 28 13:37:07 freenas kernel: epair0a: link state changed to DOWN
Jun 28 13:37:07 freenas kernel: epair0a: link state changed to DOWN
Jun 28 13:37:07 freenas kernel: epair0b: link state changed to DOWN
Jun 28 13:37:07 freenas kernel: epair0b: link state changed to DOWN
Jun 28 13:37:10 freenas kernel: ifa_del_loopback_route: deletion failed: 48
Jun 28 13:37:10 freenas Freed UMA keg (udp_inpcb) was not empty (240 items).  Lost 24 pages of memory.
Jun 28 13:37:10 freenas Freed UMA keg (udpcb) was not empty (2171 items).  Lost 13 pages of memory.
Jun 28 13:37:10 freenas Freed UMA keg (tcptw) was not empty (630 items).  Lost 14 pages of memory.
Jun 28 13:37:10 freenas Freed UMA keg (tcp_inpcb) was not empty (240 items).  Lost 24 pages of memory.
Jun 28 13:37:10 freenas Freed UMA keg (tcpcb) was not empty (72 items).  Lost 24 pages of memory.
Jun 28 13:37:10 freenas Freed UMA keg (ripcb) was not empty (240 items).  Lost 24 pages of memory.
Jun 28 13:37:10 freenas hhook_vnet_uninit: hhook_head type=1, id=1 cleanup required
Jun 28 13:37:10 freenas hhook_vnet_uninit: hhook_head type=1, id=0 cleanup required
Jun 28 13:37:10 freenas kernel: epair1a: link state changed to DOWN
Jun 28 13:37:10 freenas kernel: epair1a: link state changed to DOWN
Jun 28 13:37:10 freenas kernel: epair1b: link state changed to DOWN
Jun 28 13:37:10 freenas kernel: epair1b: link state changed to DOWN
Jun 28 13:38:36 freenas rc.shutdown: 90 second watchdog timeout expired. Shutdown terminated.
Jun 28 13:38:36 freenas syslog-ng[1521]: syslog-ng shutting down; version='3.6.4'
Jun 28 13:38:36 freenas init: /bin/sh on /etc/rc.shutdown terminated abnormally, going to single user mode
Jun 28 13:38:36 freenas ntpd[2030]: ntpd exiting on signal 15 (Terminated)
Jun 28 13:38:36 freenas upsd[2139]: mainloop: Interrupted system call

Theres nothing for hours before that, and afterwards is me booting the system

Specs: Supermicro X10SL7-F MB, Intel Xeon E3-1231 v3, 32GB ECC Memory, 6x 3tb Raidz2

I happened to be home today when it happened again, i heard windows error and saw my drives were disconnected. Couldn't ping, looked at IPMI and the system state was powered off.

Any messages in the ipmi events? Hardware specs are incomplete. What case, what power supply, what boot device and what memory brand.

Sent from my Nexus 5X using Tapatalk

 

[xCatalystx]

Any messages in the ipmi events? Hardware specs are incomplete. What case, what power supply, what boot device and what memory brand.

Sent from my Nexus 5X using Tapatalk

No Events under IPMI that I can see.

CPU: Intel Xeon E3-1231 v3
Mobo: Supermicro X10SL7-F
RAM: 4x 8GB Cruical (CT102472BD160B)
Case: Fractal Define Mini
PSU: Seasonic (cant remeber the model but its ~500w gold)
HDD: 6x WD Red 3TB
OS: 2x 32GB USB's sandisks.

Hardware is almost 3 years old. It quite cold lately so I doubt its a heat issue.

EDIT: zpool reports the last scrub on boot media was almost 2 weeks ago. found no errors.
EDIT2: Uptime before the last reboot was 3months.

 

[SweetAndLow]

No Events under IPMI that I can see.

CPU: Intel Xeon E3-1231 v3
Mobo: Supermicro X10SL7-F
RAM: 4x 8GB Cruical (CT102472BD160B)
Case: Fractal Define Mini
PSU: Seasonic (cant remeber the model but its ~500w gold)
HDD: 6x WD Red 3TB
OS: 2x 32GB USB's sandisks.

Hardware is almost 3 years old. It quite cold lately so I doubt its a heat issue.

EDIT: zpool reports the last scrub on boot media was almost 2 weeks ago. found no errors.
EDIT2: Uptime before the last reboot was 3months.

My guess would be a wife, child or roommate accidently turned it off.

Everything seems ok so far.

Sent from my Nexus 5X using Tapatalk

 

[diedrichg]

I would start with hardware. Reseat the RAM. Check all power cables are secure and that you have the 8-pin CPU power cable attached. Check all the SATA cables are secure. See how that goes.

If you still get shutdowns then swap out the power supply. See how that goes.

If it happens again, take off the CPU cooler, reseat the CPU, clean it with isopropyl alcohol and reapply thermal paste before putting the cooler back on. See how that goes.

 

[xCatalystx]

I doubt it because no one was hope except me today. Other devices on the same PDU also weren't powered off so I'm thinking it's isolated to this box.

any lines I should look for during the last boot under messages?