Random reboot

[eroji]

I am having some trouble tracking down the cause of these random reboots after the recent update. I haven't changed any configuration. Reverting back to an older kernel seems to alleviate the problem, but I still had 1 or 2 reboot after 2~3 weeks uptime. Looking at /var/log/messages shows nothing between the last bootup and the sudden reboot. Nothing in /data/crash as well.

I did notice this. Not sure if this is of any relevance.

Code:

Nov 29 15:25:51 freenas syslog-ng[14127]: syslog-ng starting up; version='3.5.6'
Nov 29 15:25:51 freenas kernel: pid 2735 (syslog-ng), uid 0: exited on signal 6 (core dumped)

 

[anodos]

Posting a debug file or at least hardware specs, relevant logs, lists of services used, jails and freenas versions, etc would be great. Without that info, I can only blaim it on chemtrails.

 

[eroji]

Here are the specs for the server:
2x E5 2660
Supermicro X9DRi-F
128GB RAM
Intel X520-DA
Intel 750 400GB for ZIL
4x4TB HGST 7200RPM
8x5TB Toshiba 7200RPM

Running FreeNAS-9.3-STABLE-201511040813

Services on:
CIFS
iSCSI
NFS
S.M.A.R.T.
SNMP
SSH

Running jails are:
transmission
plexmediaserver
owncloud
2 standard jails (SVN and Percona MySQL)

Where else can I look for debug log entries?

 

[anodos]

Here are the specs for the server:
2x E5 2660
Supermicro X9DRi-F
128GB RAM
Intel X520-DA
Intel 750 400GB for ZIL
4x4TB HGST 7200RPM
8x5TB Toshiba 7200RPM

Running FreeNAS-9.3-STABLE-201511040813

Services on:
CIFS
iSCSI
NFS
S.M.A.R.T.
SNMP
SSH

Running jails are:
transmission
plexmediaserver
owncloud
2 standard jails (SVN and Percona MySQL)

Where else can I look for debug log entries?

Click 'system' -> 'advanced' -> 'save debug'

 

[eroji]

Here it is.

 

[DrKK]

I am having some trouble tracking down the cause of these random reboots after the recent update. I haven't changed any configuration. Reverting back to an older kernel seems to alleviate the problem, but I still had 1 or 2 reboot after 2~3 weeks uptime. Looking at /var/log/messages shows nothing between the last bootup and the sudden reboot. Nothing in /data/crash as well.

I did notice this. Not sure if this is of any relevance.

Code:

Nov 29 15:25:51 freenas syslog-ng[14127]: syslog-ng starting up; version='3.5.6'
Nov 29 15:25:51 freenas kernel: pid 2735 (syslog-ng), uid 0: exited on signal 6 (core dumped)

That syslog-ng thing is harmless. Several users get it, and as far as I know, it hasn't created a problem.

Almost certainly that's unrelated.

Sounds like you have decent hardware. I have heard of this happening when one of the USB devices plugged in (e.g., keyboard, mouse) is bad. Might as well rule that out while we're discussing it.

 

[eroji]

Nothing plugged in USB-wise. My UPS has USB but I haven't had that plugged in for months. Long before the reboots started.

 

[DrKK]

Is it possible your watchdogd service is causing the reboot?

What watchdog is, is your BIOS has a "watchdog" in it (which you can turn off, if you want, I believe), and if that is on, it expects to receive a watchdog signal every so often from whatever is running on the box. If it does not, it assumes that the box has frozen up, and reboots it for you. This is useful, of course, for people whose servers are remotely situated, and where a lock-up would require a huge hassle to restart the box.

If your BIOS did not (for whatever reason) receive the watchdog signal when it was expected, the BIOS itself would effectuate a reboot.

At least, that's how I understand it.

But then the question would become, why would your BIOS not be receiving the watchdog signal?

 

[eroji]

Looks like Watch Dog function is disabled in the BIOS.

 

[DrKK]

Looks like Watch Dog function is disabled in the BIOS.

OK, well, then we may have an arduous journey of tracking down the source of your problem; almost certainly it is a piece of hardware.

 

[eroji]

Hmm... Sounds unpleasant.

 

[DrKK]

Interestingly, we are at this very moment, tracking down a similar problem in IRC for @jab416171 , and on HIS box, he went into the IPMI logs, and sure enough, he has several watchdog reboots, even though he thought he had disabled watchdog in the BIOS.

 

[depasseg]

Hmm, I might be experiencing the same watchdog reboot issue (not sure what I have set in BIOS though - I'll check when I can reboot). Seems to be prevalent the past month. This is on my freenas1 (supermicro) server.

 

[eroji]

I am so glad you guys mentioned this. I had went and looked in the IPMI last night and found those identical entries. (Apologies for the lack of screenshot)

Is there a problem ticket already made for the problem?

 

[eroji]

Any additional info on this? Seems like quite a few ppl are having the same problem. (Supermicro motherboard the common factor?) It would be nice to know if there is a fix/workaround available or if a bug ticket was created. I tried looking up the bugs page but couldn't find anything recent. I can create one but don't want to duplicate things if there is already one made.

 

[DrKK]

Wait wait wait. You're assuming this is a FreeNAS/FreeBSD problem. If this were a FreeNAS/FreeBSD problem, we'd be hearing about this 24/7. We're only getting occasional reports. The most likely culprits here, at least as far as I know, is a hardware problem in your boxes.

If anyone has any additional information here, we would all certainly like to know... @cyberjock...et al.

 

[eroji]

I am not ruling out hardware being the culprit. Just trying to figure out what would the best way to troubleshoot the issue. I will try a BIOS update on my end though I did not see any information given by Supermicro on what the latest update fixes. Other than that I don't have any other ideas.

 

[DrKK]

BIOS changelog, at least for whatever board series the X10SLM+F is in.

https://forums.freenas.org/index.php?threads/new-x10-bioses-v3-at-supermicro-changelog.39215/

 

[tobiasbp]

I have the same problem of random reboots. I'm using a Supermicro X7DBE motherboard.

I have just upgraded the BIOS from 2.1a to 2.1c hoping it fixes the problem.

The board does not have IPMI. Watch dog is disabled.

 

[eroji]

I had submitted a bug report and am being told that my motherboard will be blacklisted for the software Watchdog (probably next release?). Seems like that may be a functional workaround for the issue.