Server down (log?)

[BERKUT]

Hello,
Where I can found logs, to check why server down?
In /vat/log/messages
i found only this

Nov 24 00:00:00 NAS06 syslog-ng[12515]: Configuration reload request received, reloading configuration;
Nov 24 02:22:17 NAS06 update_check.py: [freenasOS.Configuration:567] Unable to load http://update.freenas.org/FreeNAS/trains.txt: HTTP Error 404: Not Found
Nov 25 00:00:00 NAS06 syslog-ng[12515]: Configuration reload request received, reloading configuration;
Nov 25 02:22:08 NAS06 update_check.py: [freenasOS.Configuration:567] Unable to load http://update.freenas.org/FreeNAS/trains.txt: HTTP Error 404: Not Found
Nov 25 02:22:11 NAS06 update_check.py: [freenasOS.Configuration:567] Unable to load http://update.freenas.org/FreeNAS/trains.txt: HTTP Error 404: Not Found
Nov 25 02:22:12 NAS06 update_check.py: [freenasOS.Configuration:567] Unable to load http://update.freenas.org/FreeNAS/FreeNAS-9.3-STABLE/ChangeLog.txt: HTTP Error 404: Not Found
Nov 25 10:35:46 NAS06 syslog-ng[2341]: syslog-ng starting up; version='3.5.6'
Nov 25 10:35:46 NAS06 Copyright (c) 1992-2014 The FreeBSD Project.
Nov 25 10:35:46 NAS06 Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
Nov 25 10:35:46 NAS06 The Regents of the University of California. All rights reserved.
Nov 25 10:35:46 NAS06 FreeBSD is a registered trademark of The FreeBSD Foundation.
Nov 25 10:35:46 NAS06 FreeBSD 9.3-RELEASE-p26 #0 r281084+93c5885: Mon Sep 28 13:25:20 PDT 2015
Nov 25 10:35:46 NAS06 root@build3.ixsystems.com:/tank/home/stable-builds/FN/objs/os-base/amd64/tank/home/stable-builds/FN/FreeBSD/src/sys/FREENAS.amd64 amd64

The NAS worked to 10:3*, and after down. Why? Uptime has been ~ 2 mounths.

 

[cyberjock]

That looks like someone hit the reset button, power loss, or a crash. Are any files in /data/crash?

 

[SweetAndLow]

messages would be the place to start but you need to look at older logs since ask of those are probably from the reboot.

 

[BERKUT]

That looks like someone hit the reset button, power loss, or a crash. Are any files in /data/crash?

/data/crash is empty.
and this is can't be reset button.

messages would be the place to start but you need to look at older logs since ask of those are probably from the reboot.

I look in dmesg.today, dmesg.yesterday, messages (in message monthly data)

No any info about system down...

 

[BERKUT]

Any help? Again the servers down, Build 201512121950 & 201511040813 (They are at different locations)
Crash logs is empty... Where search a problem?

 

[Mirfster]

Are these on a UPS?

 

[BERKUT]

yep, of course!

 

[BERKUT]

No one can help? The FreeNas does not have a logging in the time of the down?

 

[Mirfster]

Maybe dlavigne might have some insight on this... /Paging @dlavigne ... :D

 

[dlavigne]

Hard to tell from just the log. Post a debug (you can make it using System -> Advanced -> Save Debug).

 

[BERKUT]

Hard to tell from just the log. Post a debug (you can make it using System -> Advanced -> Save Debug).

NAS06 - build 201511040813
SE2NAS - build 201512121950

debug-NAS06-20160126223311 - down at 2016.01.26 ~ 14:50
debug-NAS06-20160204222217 - down at 2016.02.04 ~ 11:35
debug-SE2NAS-20160202012019 - down at 2016.02.01 ~ 21:10

Thank you, for any help.

 

[dlavigne]

I noticed this in /var/log/messages:

Jan 12 11:03:36 NAS06 smartd[3580]: Device: /dev/ada0, Temperature 41 Celsius reached critical limit of 41 Celsius (Min/Max 22/43)

When is the last time you ran a SMART test?

 

[BERKUT]

SMART tests run every weeks short and 2 times in month full test. In SMART disks no problems.

/dev/ada0 it's SLOG and SMART table is good.

 

[dlavigne]

A heat issue would definitely explain unexpected reboots...

 

[BERKUT]

A heat issue would definitely explain unexpected reboots...

If this heat issue, why has not logged? BIOS is too not have any logs.
~40 Temperature on ada0 it's normal working temperature http://www.hyperxgaming.com/datasheets/SHPM2280P2_us.pdf we just set critical 41 in FreeNas settings. But after 12 january we lowered the temperature on ada0 to 33 celsius.
All other hardware сomponents have max 27-30 Celsius.

 

[pirateghost]

If this heat issue, why has not logged? BIOS is too not have any logs.
~40 Temperature on ada0 it's normal working temperature http://www.hyperxgaming.com/datasheets/SHPM2280P2_us.pdf we just set critical 41 in FreeNas settings. But after 12 january we lowered the temperature on ada0 to 33 celsius.
All other hardware сomponents have max 27-30 Celsius.

Why would BIOS log a temperature issue from a pci-express add-on card?

 

[BERKUT]

Why would BIOS log a temperature issue from a pci-express add-on card?

In BIOS logs has been set - report error from PCI-E. However, problem not in ada0.

 

[pirateghost]

In BIOS logs has been set - report error from PCI-E. However, problem not in ada0.

But how does the bios know what is on that pci-express card?

I don't think you understand my logic....

 

[BERKUT]

But how does the bios know what is on that pci-express card?

I don't think you understand my logic....

Do you think the problem in ada0?

 

[Mirfster]

Just a quick look at two of the debugs... Wondering how come in the the ipmi dumps nothing show for CPU temps? Not sure what 0x0 is for a temp...

Excerpts from your log:

Code:

+--------------------------------------------------------------------------------+
+  ipmitool sdr list | grep Temp  +
+--------------------------------------------------------------------------------+
CPU1 Temp  | 0x00  | ok
CPU2 Temp  | no reading  | ns

+--------------------------------------------------------------------------------+
+  ipmitool sensor  +
+--------------------------------------------------------------------------------+
CPU1 Temp  | 0x0  | discrete  | 0x0000| na  | na  | na  | na  | na  | na   
CPU2 Temp  | na  | discrete  | na  | na  | na  | na  | na  | na  | na 

When I run ipmitool sdr list | grep Temp I get:

Code:

[root@ASC-FREENAS01] ~# ipmitool sdr list | grep Temp
MB Temp          | 40 degrees C      | ok
FP Temp          | 25 degrees C      | ok
BP Temp          | 34 degrees C      | ok
CPU0 Temp        | 69 degrees C      | ok
CPU1 Temp        | disabled          | ns
DIMM Temp        | 43 degrees C      | ok
IOH Temp         | 76 degrees C      | ok