SOLVED How to diagnose frequent rebooting?

[EvilLeprechaun]

Last night, my FreeNAS-11.3-U1 system "had an unscheduled system reboot." The system came back up and emailed me, so I figured all was fine. Maybe a power hiccup or something.

This morning, it has been rebooting every 20 minutes, give or take a few minutes. I'm not much of a sysadmin, and I'm trying to figure out where I can find out what might be causing the reboots.

Things I have checked:

• Disk temperature (all disks reporting <= 33 degrees C)
• Memory (dashboard reports approximate 6 of my 16 GB is free)
• Disk space (approximately half of my 7TB storage is free)
• Logs (I haven't found anything interesting in the logs, but I might not know what I'm looking for -- everything looks routine when compared to days it's NOT crashing)
• Disk health (`smartctl -t short` runs successfully against all 6 disks, 0 errors found)
• Updates available (none)
• Possible UPS power issues (I have tried it both connected to and unconnected from my UPS)
• Some kind of plugin/VM management interaction (turned off all plugins and VMs, still occurs)
• Cron job (I don't have any)

When the system is running, everything seems to be functioning normally -- web UI loads, all the plugins work correctly... it just keeps rebooting every 20-23 minutes.

What should I be looking at to help me diagnose this constant rebooting?

 

[MikeyG]

System specs?

 

[EvilLeprechaun]

Apologies. Added to my signature.

 

[MikeyG]

You've tried removing each memory stick in case one is bad or trying different memory slots? What about disconnecting all drives from the motherboard (assuming no HBA)? Do you happen to have a spare power supply to test with? Have you tested stability outside FreeNAS with another OS? I assume CPU is at default clocks and not overheating? You've searched for any common issues with that board/CPU going bad or looked into any BIOS updates?

 

[Redcoat]

What does IPMI tell you in the logs?

 

[EvilLeprechaun]

Thanks for the ideas! I checked IPMI, and I didn't see anything in the logs, but I did notice NTP never got set up, so I changed that. The firmware and BIOS were also from 2014, so I updated both of those, which required me to reset the CMOS before the computer would boot again.

Some combination of NTP, firmware update, BIOS update, and flashing the CMOS seems to have done it, because the machine has been up for two hours now. Chalk another one up for "random, confusing hardware issues" I guess.