11 Completely Bugged out!

[Visseroth]

I've been running FreeNAS since version 8 and have never had the problems or as many problems as I'm having now.
I'm running a SuperMicro X8DTH-i
48GB of RAM
Intel(R) Xeon(R) CPU X5690 @ 3.47GHz
RAIDz2 2X6 HGST 4TB Drives
RAIDz2 3X4 Samsung HD204UI's (old used for backups)

I was Previously was running 9.10.2-U6

I've been experiencing random reboots from which I have no idea why it's rebooting. I've looked at the logs and there's nothing to indicate why.
I've looked through previous forums, one guy suggested a log server. Great and novel idea. I'd love to have one up and going but I haven't been able to get one running the way I'd like and if I did it would be on this server which reboots. So no help there!
I don't understand why the logs aren't on the pool(s) so they can be looked through and kept for a set period of time?!?

After the reboot the system seriously hangs with the error "run_interrupt_driven_hooks; still waiting after 60 seconds for xpt_config"


When I decrypt the volume I get an error...

Code:

nvironment: Software Version: FreeNAS-11.0-U3 (c5dcf4416) Request Method: POST Request URL: http://XXX.XXX.XXX.XXX/storage/volume/3/unlock/?X-Progress-ID=(REMOVED) Traceback: File "/usr/local/lib/python3.6/site-packages/django/core/handlers/exception.py" in inner 39. response = get_response(request) File "/usr/local/lib/python3.6/site-packages/django/core/handlers/base.py" in _legacy_get_response 249. response = self._get_response(request) File "/usr/local/lib/python3.6/site-packages/django/core/handlers/base.py" in _get_response 178. response = middleware_method(request, callback, callback_args, callback_kwargs) File "./freenasUI/freeadmin/middleware.py" in process_view 162. return login_required(view_func)(request, *view_args, **view_kwargs) File "/usr/local/lib/python3.6/site-packages/django/contrib/auth/decorators.py" in _wrapped_view 23. return view_func(request, *args, **kwargs) File "./freenasUI/storage/views.py" in volume_unlock 1190. form.done(volume=volume) File "./freenasUI/storage/forms.py" in done 2627. _notifier.reload("disk") File "./freenasUI/middleware/notifier.py" in reload 281. return c.call('service.reload', what, {'onetime': onetime}, **kwargs) File "./freenasUI/middleware/notifier.py" in reload 281. return c.call('service.reload', what, {'onetime': onetime}, **kwargs) File "/usr/local/lib/python3.6/site-packages/middlewared/client/client.py" in call 233. raise CallTimeout("Call timeout") Exception Type: CallTimeout at /storage/volume/3/unlock/ Exception Value: Call timeout 

After the volume is decrypted the jails do not start nor do the jail's storage mount.

So either 11 is seriously buggy or I have a bad install.

Anyone have any thoughts or suggestions?
After this post I'm going to blow away my 11 install, roll back to 9.10.2-U6 and re-running the 11 update to see if I can get some of these errors to go away!

 

[Ericloewe]

At first glance, I suspect a seriously broken install. Try your plan and report back.

What are you using as a boot device?

 

[Visseroth]

32GB SuperMicro SATADOM SD-DM032-PHI

Re-Install is in progress. I had a broken install with the 11-U3 update on a friend's server. Had to roll back to U2, delete U3 and try again. Second time worked but his was in a reboot loop.

 

[Visseroth]

run_interrupt_driven_hooks: still waiting didn't go away. Came got the same error with 9.10.2. Must have something to do with the 2TB drives.
Only way to get it to go away is to power off and power back on.

After rolling back and re-updating all the problems returned.

I also get a error when I start Plex which states...

Code:

cxgbe0:tso4 disabled due to -txcsum.
cxgbe0: tso6 disabled due to -txcsum6.
cxgbe0: enable txsum first.
bridge0: error setting capabilities on cxgbe0: 35

I'm all ears if anyone has any ideas on how to fix these problems

 

[danb35]

I don't understand why the logs aren't on the pool(s)

The logs are wherever you told the system to put the .system dataset. By default, that would be on the data pool once you've created one.

 

[dlavigne]

Which HBA? It might be a driver issue. Some have been fixed for 11.1 and some might not be reported yet.

 

[Visseroth]

The logs are wherever you told the system to put the .system dataset. By default, that would be on the data pool once you've created one.

Well then the logs either are not catching why the reboot took place or are not being persistent.

Which HBA? It might be a driver issue. Some have been fixed for 11.1 and some might not be reported yet.

I'm running LSI cards. If I recall correctly the firmware is version 20 and the driver is 21. I haven't had any notifications for driver updates and these problems didn't start until the upgrade to 11. My server a few times had more than 60 days on it between reboots, now I can't get more than 2 days due to random reboots on top of the other errors.

 

[dlavigne]

We stopped adding alerts a few updates ago for LSI drivers as it was confusing users due to the mismatched driver version naming schema and where to actually download the driver for their specific HBA. Please double-check that you do have the latest driver for your HBA.

 

[Visseroth]

You mean the latest firmware?
To my knowledge the drivers are kept up to date by the FreeNAS team. If I have to start troubleshooting drivers and trying to get them installed and working I'll find another NAS.
And my cards are LSI 9211-8i and if I remember correctly are in IT mode (Bridge)
Network interface is a Chelsio T420-CR 10GB fiber with one module (Intel)

Edit: According to the website, the latest firmware is the P20, which is what I'm running...
https://www.broadcom.com/products/storage/host-bus-adapters/sas-9210-8i#downloads

 

[Visseroth]

Anyone have any ideas?

I also forgot to mention, as I forgot until just now when I logged into my Windows machine that I'm now unable to access my SMB shares. FreeNAS logs says authentication failure but I know my user name and password, it works everywhere else except in Windows since the 11 upgrade.

 

[dlavigne]

Assuming you're not using NTLM auth (search the forum for that one), it's hard to tell without a system debug. If you decide to create a report at bugs.freenas.org, include the debug and post the issue number here.

 

[Visseroth]

I may likely create a bug report. I rolled back to the latest 9.10.2-U6 and have not had a problem with random reboots.
As for NTLM auth, no, I'm not using it. I had to actually google search it to find out what it is. I figured I wasn't using it since I didn't know, but I wanted to be sure.