FreeNAS troubleshooting

[Whuckaba]

Good Afternoon-

I need some help troubleshooting. I have been running FreeNas for several months now with no issues. This is my first freebsd (or other unix-like variant) os that I've personally owned... my webserver is linux, but rarely have to do anything with it at the os level...

On Wednesday 23rd I applied an update to FreeNas:

< freenas-boot/ROOT/FreeNAS-9.3-STABLE-201509160044 / zfs rw,noatime,nfsv4acls 0 0
> freenas-boot/ROOT/FreeNAS-9.3-STABLE-201509220011 / zfs rw,noatime,nfsv4acls 0 0
koakuma.local kernel log messages:
> FreeBSD 9.3-RELEASE-p26 #1 r281084+59f7d05: Mon Sep 21 11:47:33 PDT

Over the next several days I found the system turning itself off for no reason.Saturday twice in the span of a few hours.

IPMI doesnt show any abnormal events (temperature, fan speeds, etc). IPMI recording doesn't show any text appearing. I've tried my best to look for logs on the freenas box... but so far I'm not finding anything... logs I find dont seem to report any issues. Not getting any emails from freenas stating any problems (emails do work... getting scrubs and mount changes etc).

So to try to see if it was that update... I rolled back to the previous boot...

< freenas-boot/ROOT/FreeNAS-9.3-STABLE-201509220011 / zfs rw,noatime,nfsv4acls 0 0
> freenas-boot/ROOT/FreeNAS-9.3-STABLE-201509160044 / zfs rw,noatime,nfsv4acls 0 0
koakuma.local kernel log messages:
> FreeBSD 9.3-RELEASE-p25 #0 r281084+d3a5bf7: Tue Sep 15 17:52:04 PDT

and its now been stable for 24+ hours again. the only console message I've not seen before was
Sep 27 00:00:00 koakuma syslog-ng[4773]: Configuration reload request received, reloading configuration;

ASRock C2750D4I: Intel Avoton C2750
4x8gb Crucial ECC ram (I couldnt find 16gb at the time of build... but graphs always show a good headroom)
intel 120gb SSD boot
10x 4TB Western Digital Re (as many on intel as possible, but obviously some on marvell) in z2
corsair hx750i (total over-kill I know)
everything new for this build.


Thanks for any thoughts or insights to look at.

 

[SweetAndLow]

is your pool health? The logs that are interesting will be in /var/log/messages. Finding out the reason for the reboot might be hard, it's usually a hardware failure of some kind.

 

[Whuckaba]

I guess it is hardware... since I came back in from the eclipse to find it off again. Dissapointing since I used stress linux and memtest .. and spent more than 2 weeks checking every drive, the cpu, and memory.

Yeah that is the file I've looked at.. nothing there


Sep 28 00:00:00 koakuma syslog-ng[1997]: Configuration reload request received,
Sep 28 00:59:20 koakuma syslog-ng[1997]: syslog-ng starting up; version='3.5.6'
Sep 28 00:59:20 koakuma Copyright (c) 1992-2014 The FreeBSD Project.

Crash was sometime after midnight... and I restarted it just before 1 am. Although my reporting tab in the web interface stops just before midnight.

according to the reporting, was using ~18G of memory at the time of crash... arch size 12.56G ...
nothing seems out of normal on the other graphs

the only change recently was pointing a seeding torrent to files on the server... no writing, just reading... although as always, the disks all show around 40kB/s write on the disk all the time (never understood why, but has been that way since day 1)... the reading for the torrenting seem to be pulling around 570kB/s

all smartctl show everything has passed.

*shrugs*


edit:

I just am curious what to start at... I would have figured that if it was the marvel controllers, it would just cause the pool to drop or come back and have errors related to that. Or could that really cause it to spontaneously explode and turn off. I'm not experiencing any slow downs I've read about.. getting near about 100 mb/s across gig network, which seems about right to me all things considered.

I'm torn now between buying new/more ram, or just buying a new board like the E3C224D4I-14S .. since it seems more recommended. edit edit: or maybe I'll just buy a M1015 and see if that does the trick...

I just want something that I can just set and forget. I thought I had found that with this, as I had it sitting next to my desk for a 50-something day uptime... finally put it in its new home and within a month now having the issues.