Recently upgraded to 9.2.1.8 and now restarting a bunch

[electricd7]

Hello,

I have had a FreeNAS server setup for a very long time (5 years-ish). I tend to update it every year or so. I recently updated to 9.2.1.8 from 9.2.x. Since doing so I have noticed that my freenas server is rebooting every so often...like at least once a day. I can tell this because when I login, the uptime is usually less than 24 hours. I am not sure how to begin to troubleshoot this? Is the log cleared everytime it reboots? Where can I look to see what may have caused it? I don't do much administration on the system, so I am not that familiar with anything other than the Web interface. Please help. Thanks!

 

[dlavigne]

Anything related in /var/log/messages? If not, what time is the system rebooting (from that log)?

Also, post your hardware specs.

 

[electricd7]

Sorry this took so long for me to reply, but here is the output from my log. My machine is a super micro motherboard with 16GB memory and 6x2TB disks i believe. The machine rebooted 15 minutes ago so it should be applicable.

http://homes.joink.com/users/chad/freenas.txt

 

[electricd7]

Anyone? I just can't figure out what is going on with it?

 

[dlavigne]

Any luck with this? Random reboots are often a hardware issue. Is the fan running hot? Bad RAM or disk?

 

[solarisguy]

Let me guess, non-ECC RAM judging by the machine age ?

If non-ECC RAM, can you try testing your RAM?

If ECC RAM, could you try running some CPU stress tests? May be Prime95? AIDA64 can be used for 30 days in trial mode. Etc.

 

[jkh]

Yeah, no smoking guns in the log. Definitely a hardware error of some sort. I would start swapping components, starting with the memory.

 

[solarisguy]

Any luck with this? Random reboots are often a hardware issue. Is the fan running hot? Bad RAM or disk?

Yes, with such an old system, temperature built-up is a definitive possibility.

Please vacuum your system and after it appears to be clean, with vacuum still sucking, dust off all the radiators and fans. (Not all dusters are approved for electronic equipment...).

Observe ESD precautions!

 

[Norleif]

I noticed there is a LSI SAS 2008 in there...
I made my own FreeNAS reboot a few times because I loaded the wrong (too new) firmware on the LSI (reflashed M1015).
The problem went away after I loaded the correct (P16 ??) firmware.

 

[Ericloewe]

P16 is the correct LSI SAS2 firmware for FreeNAS 9.2 and 9.3.

 

[electricd7]

Thanks for all the great ideas! I am going to do the memtest and CPU tests this weekend on the system. Just found it odd that it has been rock-solid for years until I upgraded and wondered if there was some incompatibility. It was only a minor upgrade. I also know that sometime when you turn off things that spin, they don't operate the same way again. I will test all these things out. I do have the IBM M1015 HBA card..how do I check which firmware version is on this card? Thanks again! I will report back.

 

[solarisguy]

Still, consider cleaning...

 

[electricd7]

OK update. I did an overnight memory test (no errors) and 12-hour CPU stress test (no errors, or restarts.) I have since taken the cover off and cleaned all the fans and filters (Lian-LI case with fan filters). It is currently running, so we will see how this goes. Thanks for the advice and I will follow up with results.

 

[electricd7]

Well that didn't take long...Within 2 hours it rebooted again. Nothing in the log. It has to be something with this version and my hardware, I would think. I was able to run the memtest and CPU stresstest much longer without any reboots. Is it possible to downgrade FreeNAS easily? I wish I could remember what version I was on previously..

 

[electricd7]

OK, so I thought I would just say forget it and install 9.3 and see if anything was different. So I just pulled my 9.2.1.8 USB device and put in a 64GB new one. I installed 9.3 and then restored my backed up config. Everything seems OK and working. I noticed, however, that I get the alert: "Firmware version 14 does not match driver version 16 for /dev/mps0" at the top which I haven't seen before. Could this be causing my issued in 9.2.1.8? Did they change the version of the driver recently? I have the IBM M1015 HBA flashed to JBOD mode in my filer.

 

[Ericloewe]

OK, so I thought I would just say forget it and install 9.3 and see if anything was different. So I just pulled my 9.2.1.8 USB device and put in a 64GB new one. I installed 9.3 and then restored my backed up config. Everything seems OK and working. I noticed, however, that I get the alert: "Firmware version 14 does not match driver version 16 for /dev/mps0" at the top which I haven't seen before. Could this be causing my issued in 9.2.1.8? Did they change the version of the driver recently? I have the IBM M1015 HBA flashed to JBOD mode in my filer.

Well, that's likely to cause trouble. FreeNAS 9.2 and 9.3 use the P16 mps driver for LSI SAS 2008 and 2308 controllers (including the M1015). The controller should have the same version firmware flashed, otherwise nasty things may happen.

 

[electricd7]

Yea I wondered. Been awhile since I flashed the card...its already in IT mode, so I think all I need to do is find the P16 firmware files and sas2flash the 2118it.bin file, right? No need to empty out the card as it already thinks its an LSI card at P14?

 

[electricd7]

Wait...I found the 2118IT.bin, but do I need the 2008it.bin? I don't want to brick my card...how do I make sure I get this right?

 

[Ericloewe]

Wait...I found the 2118IT.bin, but do I need the 2008it.bin? I don't want to brick my card...how do I make sure I get this right?

You almost certainly need 2008it.bin. It should flash without annoyances.

 

[electricd7]

Everything I am seeing only references the 2118 firmware for the M1015.