Odd FreeNAS Randomly rebooting Issue

[statictjo]

Hello I have a question regarding an issue I am currently having. Just wondering what steps to take to diagnose this issue. I have a Freenas 9.10 box running from a bootable usb drive. I have 6 drives I use for storage of a couple jails I currently use for Plex etc. Yesterday this odd occurrence where freenas box would only maintain an up time around an hour to 40 minutes then it will reboot. Trying to diagnose this issue I pulled error log for the times I wrote down these occurrences only to see no record of these time stamps in the logs. It's hinting towards a hardware issue. Anyone have experience with this issue and if there is some tests I can do to determine if its a software issue or hardware and proceed to remedy the problem.

Thanks.

 

[Ericloewe]

Start by following the forum rules and tell us what hardware you're using.

 

[SweetAndLow]

Start by providing your hardware and any errors you see in the logs. I can be your memory, usb stick, power supply or overheating issues.

 

[statictjo]

Hello again,

Specs of server:

- 3x Seagate NAS HDD ST2000VN000 2tb drives
- 3x Western Digital Red Nas 2tb drives

Raid 1

- SuperMicro MBD-A1SAi-2750f-O
Imbedded Intel Atom c2750 2.4hz cpu

- 2x 8gb ram Kingston 8GB 204-Pin DDR3 ECC unbufferred

- Kingston DataTraveler SE9 16gb Usb 2.0
Used as the bootable usb

- EVGA 220-Gs-0550-v1 80 Plus gold 550 w
power supply

Update: I went to sleep woke up with the system still cycling every hour or so, I noticed the system would not post past bios so I pulled one of the memory sticks and it booted successfully, gonna see the up time on the server with it only on one stick of ram could be a bad ram stick. I've also attached a copy of the hardware dump file.

 

[Ericloewe]

Check the IPMI log for RAM errors.

 

[statictjo]

Here is the logs from the IPMI dump file to be honest I am not exactly sure what to look for in here in terms of error codes in this file

 

[anodos]

The CPU temp at the time ran the debug script was 60C. That's pretty high, but as Eric mentioned, log into your BMC and review error logs.

 

[Ericloewe]

The watchdog is triggering the reboots. There are also many unidentified errors - you'll need to check the GUI.

If it was a case of bad RAM, I'd expect many more of those,

 

[statictjo]

I'll monitor the system now that I pulled out one of the ram sticks. I may have sniffed it as a ram issue and got lucky. I've dealt with hardware issues before on PC's I'm just a bit new to Freenas just wanted to single out anything related to freenas since I have little experience with it.

I'll report back in the next couple hours to see how the system behaves.

 

[statictjo]

Update: I pulled out one of the sticks and observed the system was not acting up. Just to cover my basis I reinserted the memory stick and noticed again that it did not crash/reboot like it was before. To confirm the ram is good I performed 2 passes of memtest for both sticks and had Zero errors. I will re-observe the system again and report back. Some useful additional information I first noticed this issue occurred on a previous version of Freenas 9.10.1-U2. I noticed an update and decided to upgrade to U2 thinking it would fix the issue. After noticing the problem continued to do all this ram investigation. I am curious now if there problem lies in my bootable usb drive?

 

[Sakuru]

It sounds like that stick of RAM may not have been seated properly. Sometimes all it takes is a tiny misalignment to throw everything out of whack.

 

[statictjo]

Update: I think I may have solved the issue, I was running memtest86 and all of a sudden my motherboard led turns red following a continuous beep with no crash or anything continued up until it stopped after I reset the machine. I looked in the manual only to pointing to a possible PSU failure. I looked at the beep codes for my motherboard and there isn't anything for a loud continuous loop beep.

 

[statictjo]

Problem solved looks like a heating issue and power supply. Just wanted to say thanks for those who followed this thread and took the time with me to help with my issue.

 

[statictjo]

Everything seemed to be running smooth for a day. I get an update for U4 and now i'm getting crashes again I managed to screenshot the fault. Does anyone else get this issue I attached the screenshot

 

[Stux]

Hardware/Ram failure?

You knew that when your machine failed after 2 years ;)

https://lists.freebsd.org/pipermail/freebsd-questions/2008-March/171715.html

Memtest86 is great and all, but it only picks up the easy stuff, not stray random corruptions which happen once in a blue moon. Unless you're testing on said blue moon.