Random reboots with nothing in the logs - cifs, iscsi

[rschwarz_jr]

I have a Dell 2950 II that I have made into a freenas server that is rebooting randomly. I've looked through the system logs and configurations and read the form for things to try and after a week and a half of crashes I'm stumped.

The duration between reboots can be as short as 3 hours or as long as 3 days. There are no core dumps and nothing is happening of note in the logs prior to the reboot. It is used as a iSCSI target for a few ESXI virtual machines and has a couple of CIFS shares. It is in an AD environment.

I don't think its the hardware itself as the server was used for a year + as an ESXI server using local storage with the same two drives and never gave any problems prior to its re-tasking. It's for a very small office so it isn't worked very hard and the crashes are often in the middle of the night when there is little load.

I'd be happy to supply any supplemental information or logs just let me know what would be of use.

Thank you in advance as I am out of ideas.

Specs:
FreeNAS-9.2.1.6-RELEASE-x64 (ddd1e39)
4 X Quad-core Intel(R) Xeon(R) CPU E5345 @ 2.33GHz (8 cores total)
12GB ECC RAM
2 X 1TB WD Red currently in ZFS mirror
Two network cards used. One to my main network for CIFS access and the other is a direct connection to the ESXI server for iSCSI.

dmesg output http://pastebin.com/1ATyvrHy

 

[cyberjock]

Can you post the debug file for your server? You'll have to host it on dropbox or google drive yourself since you can't upload .tgz files to the forum. I'm working on trying to fix that.

 

[rschwarz_jr]

I don't know if it's related to my core problem but I get this message when I go to Settings -> Advanced -> Save Debug

Request Method:GET
Request URL:http://blackhole/system/debug/
Software Version:FreeNAS-9.2.1.6-RELEASE-x64 (ddd1e39)
Exception Type:IOError
Exception Value:
[Errno 2] No such file or directory: '/mnt/MainStorage1/.system/ixdiagnose/ixdiagnose.tgz'
Exception Location:/usr/local/www/freenasUI/../freenasUI/system/views.py in debug, line 557
Server time:Fri, 25 Jul 2014 15:02:24 -0400
Traceback

Request information
GET
No GET data

POST
No POST data

FILES
No FILES data

COOKIES
VariableValue
csrftoken'Fxu2zSh05Gu5qi1CrQGufBZjQpu0jjk6'
sessionid'4ait3qpk5w3m9kcf4lh5nggsi5ov6jjz'
fntreeSaveStateCookie'root%2Croot%2F125%2F163%2F165%2Croot%2F1%2F5%2Croot%2F8%2F25%2Croot%2F125%2Croot%2F108%2Croot%2F34%2Croot%2F34%2F36%2Croot%2F122%2Croot%2F105%2Croot%2F105%2F112%2Croot%2F105%2F109%2Croot%2F51%2Croot%2F51%2F58%2F59%2Croot%2F122%2F126%2Croot%2F123%2Croot%2F53%2F60%2F61%2Croot%2F124%2F162%2F164%2Croot%2F36%2Croot%2F53%2Croot%2F53%2F60%2F61%2F66%2Croot%2F124%2Croot%2F124%2F136%2F140%2Croot%2F137%2F149%2Croot%2F137%2F149%2F150%2Croot%2F137%2F149%2F153%2Croot%2F137%2F149%2F157%2Croot%2F137%2F149%2F161%2Croot%2F137%2F149%2F166%2Croot%2F62%2Croot%2F8%2Croot%2F137'
META
VariableValue
wsgi.multiprocessFalse
HTTP_REFERER'http://blackhole/'
REDIRECT_STATUS'200'
SERVER_SOFTWARE'nginx/1.4.4'
SCRIPT_NAMEu''
REQUEST_METHOD'GET'
PATH_INFOu'/system/debug/'
SERVER_PROTOCOL'HTTP/1.1'
QUERY_STRING''
CONTENT_LENGTH''
HTTP_USER_AGENT'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.153 Safari/537.36'
HTTP_CONNECTION'keep-alive'
HTTP_COOKIE'sessionid=4ait3qpk5w3m9kcf4lh5nggsi5ov6jjz; fntreeSaveStateCookie=root%2Croot%2F125%2F163%2F165%2Croot%2F1%2F5%2Croot%2F8%2F25%2Croot%2F125%2Croot%2F108%2Croot%2F34%2Croot%2F34%2F36%2Croot%2F122%2Croot%2F105%2Croot%2F105%2F112%2Croot%2F105%2F109%2Croot%2F51%2Croot%2F51%2F58%2F59%2Croot%2F122%2F126%2Croot%2F123%2Croot%2F53%2F60%2F61%2Croot%2F124%2F162%2F164%2Croot%2F36%2Croot%2F53%2Croot%2F53%2F60%2F61%2F66%2Croot%2F124%2Croot%2F124%2F136%2F140%2Croot%2F137%2F149%2Croot%2F137%2F149%2F150%2Croot%2F137%2F149%2F153%2Croot%2F137%2F149%2F157%2Croot%2F137%2F149%2F161%2Croot%2F137%2F149%2F166%2Croot%2F62%2Croot%2F8%2Croot%2F137; csrftoken=Fxu2zSh05Gu5qi1CrQGufBZjQpu0jjk6'
SERVER_NAME'localhost'
REMOTE_PORT'39156'
wsgi.url_scheme'http'
SERVER_PORT'80'
SERVER_ADDR'192.168.1.21'
DOCUMENT_ROOT'/usr/local/etc/nginx/html'
DOCUMENT_URI'/system/debug/'
wsgi.input<flup.server.fcgi_base.InputStream object at 0x80f1e3dd0>
HTTP_DNT'1'
HTTP_HOST'blackhole'
wsgi.multithreadTrue
REQUEST_URI'/system/debug/'
HTTP_ACCEPT'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8'
wsgi.version(1, 0)
GATEWAY_INTERFACE'CGI/1.1'
wsgi.run_onceFalse
wsgi.errors<flup.server.fcgi_base.TeeOutputStream object at 0x80f1e3e50>
REMOTE_ADDR'192.168.1.101'
HTTP_ACCEPT_LANGUAGE'en-US,en;q=0.8'
CONTENT_TYPE''
CSRF_COOKIEu'Fxu2zSh05Gu5qi1CrQGufBZjQpu0jjk6'
HTTP_ACCEPT_ENCODING'gzip,deflate,sdch'

 

[rschwarz_jr]

When I run

Code:

/usr/local/bin/ixdiagnose -d /mnt/MainStorage1/.system/ixdiagnose -s -F

manually it hangs on this:

Code:

+ /sbin/camcontrol smpphylist /dev/ses0
camcontrol: error sending command

 

[rschwarz_jr]

Any thoughts?

 

[rschwarz_jr]

I modified the script so it doesn't execute the failing portion. Here is the remaining debug file.
https://dl.dropboxusercontent.com/u/29756995/debug-blackhole-20140729151757.tgz