SOLVED Help Troubleshooting Existing Server

[hunter]

I have had FreeNAS running flawlessly for 4 years on the following hardware:
Supermicro X9SCM-F-O Lga 1155 Motherboard
Intel E3-1230v2 Xenon Quad Core CPU
Kingston 2x8Gb ECC server Memory.

After running for like 232 days without a reboot, I rebooted it this week. Since then, it either turns itself off after up to an hour of running (it brings up FreeNAS 9.11 fine when that happens), or if I reboot it after that, it reboots after maybe 15 seconds. I can hear beep codes as it boots, but they sound like the same ones I have always heard when it was working normally. I do not know how to see error messages from the bios on this headless server.

So far, I have tried: bypassing the UPS, replacing the power supply, and replacing the CMOS battery. I feel like I am just guessing at this point.
Can anyone tell me how to troubleshoot this kind of problem on my headless server?

 

[Chris Moore]

I feel like I am just guessing at this point.

Everything you tried sounds like you were just guessing. The problem you describe sounds like one of your fans probably failed and the system is overheating.

Do you know how to look at the temperature in the IPMI web page? Should look like this:

 

[Jailer]

Open up a console and see what's going on. You've got IPMI on that motherboard, use it.

 

[hunter]

Thank you both. I never learned how to use the IPMI, and I forgot the board had a vga port that would let me see POST messages. I will get out the old vga cord and read up on how to configure and use the IPMI.

 

[Jailer]

IPMI is a network management interface. It's accessed via your network. No need to hook up a monitor keyboard or mouse to use it.

 

[Chris Moore]

One of the pages in the IPMI web interface will look similar to this:

and on that page you can see a preview of what the console would look like.

Just click on that to launch the Java iKVM Viewer which will give you interactive control, from you desktop.
Looks like this:

This is one of the main reasons we buy a server board to begin with. No need to ever connect Monitor, Keyboard, Mouse or CD-ROM because they can all be connected remotely using this utility.

 

[hunter]

IPMI must not need any configuration, that's really slick! Thank you very much for posting the printscreens. I now have IPMI viewer installed on my Windows PC to try what you said. But when I boot, all the fans I can see, are spinning up, but exactly 2 seconds later, before it makes it to POST, makes beeps, or shows me anything via VGA, the fans stop and it seems to turn the power off, then it restarts and continues to cycle that way. I don't think I can reach it through IPMI either (I tried).

I have so little to go on. I tried removing one of the USB thumb drives it boots from, and I tried removing one of the DIMMs and swapping just the 1 DIMM, to no avail. You are right, I continue to guess but I am running out of ideas.

If it was your board, would you remove the motherboard and try it outside the case (shorting the jumpers to start it with a scredriver), and just replace it the motherboard if it does the same thing when it's sitting on a cardboard box?

 

[MrToddsFriends]

If it was your board, would you remove the motherboard and try it outside the case (shorting the jumpers to start it with a scredriver), and just replace it the motherboard if it does the same thing when it's sitting on a cardboard box?

My next step would be to swap either the motherboard or the CPU, whichever is easier (or cheaper) to achieve.

 

[Chris Moore]

If it was your board, would you remove the motherboard and try it outside the case (shorting the jumpers to start it with a scredriver), and just replace it the motherboard if it does the same thing when it's sitting on a cardboard box?

Before I remove it from the chassis, I would reseat all the connections, everything, to see if that makes any difference.
Do you have any accessories attached to the board such as an expansion (PCIe) card?
I have only seen a CPU go bad a few (very few) times since I started working as a service technician back in 1996, and most of those were in the early days due to bad voltage to the chip or, in one case, running it with no cooler. I think it is unlikely that your CPU failed.
I have seen a couple instances where the CPU needed to be reseated, which is a pain, but when I say reseat everything, I mean it.
You said that you already replaced the power supply, after that, if you have reseated everything and you are still having the problem that it will cause the system to spontaneously reboot, my next step would likely be to replace the system board. I am anxious to see where you are at with this after reseating all the connectors and if you have anything plugged into the board that you can disconnect (just for testing) that might be a good idea.
I have had a SAS controller go bad that caused the system to reboot repeatedly until I removed the SAS controller.
It might be a good idea for you to give us a full rundown on all the components in the system. Details could make the difference.

 

[hunter]

Thank you both. I removed the motherboard from the case before seeing your suggestion to reseat all the the connections. It has never had accessory boards attached, just 2 SATA drives (6 Tb Hitachi), and 2 thumb drives attached via USB. Kind of a minimalist setup for FreeNAS. I tried removing all 4 of those items before taking the mother board out to no avail.

I am getting the same rebooting results with the board running out of the case on the cardboard box it came in and with no DIMM memory attached. Even that way, I don't get as far as any beep codes, it just keeps rebooting every 2 seconds. I will try reseating the CPU tomorrow before thinking about a replacement board.

 

[Chris Moore]

I am getting the same rebooting results with the board running out of the case on the cardboard box it came in and with no DIMM memory attached. Even that way, I don't get as far as any beep codes, it just keeps rebooting every 2 seconds. I will try reseating the CPU tomorrow before thinking about a replacement board.

The system can't post unless there is at least one memory module inserted. If I recall correctly, it needs to be in the slot closest to the CPU.

 

[hunter]

The system can't post unless there is at least one memory module inserted. If I recall correctly, it needs to be in the slot closest to the CPU.

Thank you, I will try it with one DIMM and try reseating the CPU today.

Sent from my Nexus 7 using Tapatalk

 

[hunter]

Thanks for the suggestion to try reseating and using one DIMM instead of two. You were right, for one bank the manual says to use 2A which is closer to the CPU than the 2nd bank I was using. I just finished trying it, first with one DIMM and then with another, after reseating the CPU and I get the same results. I'll have to try another board.

Can anyone tell me if I want the same board again? I have Intel E3-1230v2 Xenon Quad Core CPU, it has 1155 form factor. I was happy with the Supermicro X9SCM-F-O Lga 1155. Has something better for the CPU and memory I already have, come along during the last 4 years? (The latest recommended hardware guide for FreeNAS doesn't seem to have anything for 1155 I don't think...)

 

[Chris Moore]

Can anyone tell me if I want the same board again? I have Intel E3-1230v2 Xenon Quad Core CPU, it has 1155 form factor. I was happy with the Supermicro X9SCM-F-O Lga 1155. Has something better for the CPU and memory I already have, come along during the last 4 years? (The latest recommended hardware guide for FreeNAS doesn't seem to have anything for 1155 I don't think...)

If you want to keep the same CPU and RAM, you will just need to keep the same board. They have changed the socket and memory type in such a way that if you go one generation newer you can use the same RAM, but need a new CPU. If you go two generations newer, you need System board, Processor and RAM.
In your place, I would pickup an inexpensive used board from eBay, unless you need something more powerful.
This one for example:
https://www.ebay.com/itm/TESTED-Sup...w-E97379-001-Heatsink-I-O-Shield/232430611431
It is only $60...

 

[pschatz100]

That is a great motherboard - it's what I use:)

Have you tried booting off a new boot device? Create a new install on a new flash drive or SSD. Flash drives are flaky and you see the problems most often when booting or just after boot.

Also, download the manual for IPMI. It's a great tool, but you need to learn how to use it. A simple search on the Supermicro website should do the trick.

 

[hunter]

...
Have you tried booting off a new boot device? Create a new install on a new flash drive or SSD. Flash drives are flaky and you see the problems most often when booting or just after boot.

Also, download the manual for IPMI...

Thanks. Yes, I tried removing the thumb drives used to boot the device and experienced the same boot loop type of behavior. I spent time with the supermicro guide to ipmi, installed it and even got it to work with my board but it was only able to trigger reboots and shutdown. The board never got far enough along to show POST messages or anything from sensors.

I will use ipmi next time now that I see how easy it is and realize there is no configuration needed in the BIOS to enable it.

Sent from my Nexus 7 using Tapatalk

 

[Chris Moore]

I will use ipmi next time now that I see how easy it is and realize there is no configuration needed in the BIOS to enable it.

Depending on the replacement board, and how the previous owner may have configured it, it might need some degree of configuration. The boards I received used had the IP Address configured manually instead of by DHCP, so I had to manually set the IP address in the BIOS which did require me to connect a monitor and keyboard to the system for the first boot. After I had the IPMI interface on the network, I was able to remote in and configure everything else from there.
Your mileage may vary.

 

[Mirfster]

Out of curiosity (and since I am an old fart); can you tell us the actual beep code(s) you are getting (even if it was always the same ones you were used to)?

Like if it is: 3 short beeps, 1 pause, 1 long beep...

 

[hunter]

Out of curiosity (and since I am an old fart); can you tell us the actual beep code(s) you are getting (even if it was always the same ones you were used to)?

Like if it is: 3 short beeps, 1 pause, 1 long beep...

Sure, that's why diagnosing was a challenge: after precisely 2 seconds, the CPU fan spun down, it was quiet for 2 seconds then the fan spun up again and it kept doing that before it had time to POST or produce any beep codes.

 

[Mirfster]

Gotcha, I just re-read all the posts in this thread and since it is not even making it to beeps then I tend to think it has to be CPU, MB, RAM or PSU. Assuming all cables are properly connected and no jumpers have changed...

Usually failure in any of those parts will simply cause the machine to not boot and not provide beep codes. Seems like you are already looking at replacing the MB; which is a very viable option.

Best of luck