FreeNAS-11.0-U2 (e417d8aa5) - Constant Tracebacks

[HeloJunkie]

I am running FreeNAS-11.0-U2 (e417d8aa5) and I am getting constant tracebacks when attempting to do stuff via the GUI. What I am attempting to do works (ie, update smart cron, add vol, etc) but it still throws a traceback. I am looking for any ideas as to where to start with the troubleshooting...

My server config is:
EMC Isilon SuperMicro X8DT6-A Motherboard
2 x Intel Xeon E5603 4 Core Processors
96GB (12 x 8GB) DDR3 PC10600 (1333) REG ECC Memory
2 x SanDisk 8GB SATADOM Boot Drives (Mirrored)
LSI SAS3081E-R w/Expander
36 x 2TB 7200RPM Hitachi HDS72303 Hard Drives
4 x 9 Drive RAIDZ2 VDEVs
Dual 1200 Watt Gold Power Supplies
APC Smart-UPS RT 3000
FreeNAS-11.0-U2 (e417d8aa5)




Here is the traceback:

Code:

Request Method: POST
Request URL: http://freenas-2/admin/tasks/smarttest/edit/2/
Software Version: FreeNAS-11.0-U2 (e417d8aa5)
Exception Type: CallTimeout
Exception Value:
Call timeout
Exception Location: /usr/local/lib/python3.6/site-packages/middlewared/client/client.py in call, line 233
Server time: Thu, 17 Aug 2017 15:01:52 -0700


Environment:

Software Version: FreeNAS-11.0-U2 (e417d8aa5)
Request Method: POST
Request URL: http://ase-freenas-2/admin/tasks/smarttest/edit/2/


Traceback:
File "/usr/local/lib/python3.6/site-packages/django/core/handlers/exception.py" in inner
  39.			 response = get_response(request)
File "/usr/local/lib/python3.6/site-packages/django/core/handlers/base.py" in _legacy_get_response
  249.			 response = self._get_response(request)
File "/usr/local/lib/python3.6/site-packages/django/core/handlers/base.py" in _get_response
  178.			 response = middleware_method(request, callback, callback_args, callback_kwargs)
File "./freenasUI/freeadmin/middleware.py" in process_view
  162.		 return login_required(view_func)(request, *view_args, **view_kwargs)
File "/usr/local/lib/python3.6/site-packages/django/contrib/auth/decorators.py" in _wrapped_view
  23.				 return view_func(request, *args, **kwargs)
File "./freenasUI/freeadmin/options.py" in wrapper
  208.				 return self._admin.admin_view(view)(*args, **kwargs)
File "/usr/local/lib/python3.6/site-packages/django/utils/decorators.py" in _wrapped_view
  149.					 response = view_func(request, *args, **kwargs)
File "/usr/local/lib/python3.6/site-packages/django/views/decorators/cache.py" in _wrapped_view_func
  57.		 response = view_func(request, *args, **kwargs)
File "./freenasUI/freeadmin/site.py" in inner
  142.			 return view(request, *args, **kwargs)
File "./freenasUI/freeadmin/options.py" in edit
  566.					 mf.save()
File "./freenasUI/tasks/forms.py" in save
  453.		 notifier().restart("smartd")
File "./freenasUI/middleware/notifier.py" in restart
  273.			 return c.call('service.restart', what, {'onetime': onetime}, **kwargs)
File "./freenasUI/middleware/notifier.py" in restart
  273.			 return c.call('service.restart', what, {'onetime': onetime}, **kwargs)
File "/usr/local/lib/python3.6/site-packages/middlewared/client/client.py" in call
  233.			 raise CallTimeout("Call timeout")

Exception Type: CallTimeout at /admin/tasks/smarttest/edit/2/
Exception Value: Call timeout

Request information
GET
No GET data

POST
Variable Value
__all__ ''
smarttest_disks '{serial_lunid}MK015xxxxxxxERA_5000cca223c2d62a'
smarttest_type 'S'
smarttest_desc ''
smarttest_hour '01'
smarttest_daymonth '*/2'
smarttest_month '12'
smarttest_dayweek '7'
__form_id 'dialogForm_smarttest'
FILES
No FILES data

COOKIES
Variable Value
sessionid '2lythgyiy594kdefdn0rta0tpad97k60'
fntreeSaveStateCookie 'root%2Croot%2F35%2F36%2Crootdgdfgdgdhgfhdfg%2F1%2Croot%2F162%2Croot%2F35'
csrftoken '4YjvtR3UfghUbkjy9gfghpLaZ7hO7ZH3S0HV'
META
Variable Value

 

[Chris Moore]

My server config is:
EMC Isilon SuperMicro X8DT6-A Motherboard
2 x Intel Xeon E5603 4 Core Processors
96GB (12 x 8GB) DDR3 PC10600 (1333) REG ECC Memory
2 x SanDisk 8GB SATADOM Boot Drives (Mirrored)
LSI SAS3081E-R w/Expander
36 x 2TB 7200RPM Hitachi HDS72303 Hard Drives
4 x 9 Drive RAIDZ2 VDEVs
Dual 1200 Watt Gold Power Supplies
APC Smart-UPS RT 3000
FreeNAS-11.0-U2 (e417d8aa5)

Nice setup. You might want to post this in the "Report a Bug" section so a developer can look at it.

 

[HeloJunkie]

Nice setup. You might want to post this in the "Report a Bug" section so a developer can look at it.

Thanks, Chris - I assume you mean Development -> Bug Reporting Discussion, correct? I did not find a "Report a Bug" Forum here. Should I just delete this and post it there or post in both places or does this forum software allow cross-posting? I didn't find it if it does.


Thanks Again

 

[Ericloewe]

The traceback looks related to storage, more specifically, SMART tests.

Is the HBA and backplane firmware up to date?

 

[HeloJunkie]

Hi @Ericloewe

That is just this particular traceback. It happens all the time. This time it just happened that I was swapping out hard drives on one of my vdevs. Another time it was when I added a new jail (with a different traceback). What I am trying to do always ends up working (ie - the jail was actually there after the traceback) but I am still getting the traces.

 

[Chris Moore]

Thanks, Chris - I assume you mean Development -> Bug Reporting Discussion, correct? I did not find a "Report a Bug" Forum here. Should I just delete this and post it there or post in both places or does this forum software allow cross-posting? I didn't find it if it does.


Thanks Again :)

https://bugs.freenas.org/projects/freenas

 

[Ericloewe]

Have you verified the install? What's the boot pool scrub schedule?

 

[HeloJunkie]

Have you verified the install? What's the boot pool scrub schedule?

Yes sir -

I do a scrub about once a month on the boot drive (the default I think).

I am running version 16 of the firmware and 21 of the drivers for the LSI card according to the dmesg:

Code:

mps0: <Avago Technologies (LSI) SAS2008> port 0xe000-0xe0ff mem 0xfad3c000-0xfad3ffff,0xfad40000-0xfad7ffff irq 32 at device 0.0 on pci5
mps0: Firmware: 16.00.01.00, Driver: 21.01.00.00-fbsd
mps0: IOCCapabilities: 1285c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,EventReplay,HostDisc>

 

[Ericloewe]

The firmware should be P20.00.07, but that doesn't explain non-storage problems...

Can you show us a different traceback?

 

[HeloJunkie]

@Ericloewe

OK, so I went back and tried to get another traceback and I think you are right, it appears that it is only when I try and do something with smart settings. There are no alerts, the pools appear fine, the changes I am trying to make get made (or at least show up in the GUI after the traceback). So I upgraded the firmware to 20.00.06. I could not find 20.00.07 on Broadcom's website).

Code:

root@freenas-2:~ # sas2flash -list
LSI Corporation SAS2 Flash Utility
Version 16.00.00.00 (2013.03.01)
Copyright (c) 2008-2013 LSI Corporation. All rights reserved

		Adapter Selected is a LSI SAS: SAS2008(B2) 

		Controller Number			  : 0
		Controller					 : SAS2008(B2) 
		PCI Address					: 00:05:00:00
		SAS Address					: 5003048-0-1408-1e00
		NVDATA Version (Default)	   : 14.01.00.08
		NVDATA Version (Persistent)	: 14.01.00.08
		Firmware Product ID			: 0x2213 (IT)
		Firmware Version			   : 20.00.06.00
		NVDATA Vendor				  : LSI
		NVDATA Product ID			  : SAS9211-8i
		BIOS Version				   : 07.31.00.00
		UEFI BSD Version			   : N/A
		FCODE Version				  : N/A
		Board Name					 : SAS9211-8i
		Board Assembly				 : N/A
		Board Tracer Number			: N/A

		Finished Processing Commands Successfully.
		Exiting SAS2Flash.

However, upgrading the flash did not resolve the problem. I still get the traceback when trying to alter settings, specifically every single time I try to alter S.M.A.R.T. settings. I am able to add a new pool, destroy a pool, add datasets, etc all with no tracebacks, but anything at all with the smart tests causes the traceback.

 

[Ericloewe]

What about the backplane? Can you try attaching some drives directly to the SAS controller?

 

[HeloJunkie]

OK, I was able to find the P20.00.07 firmware and updated the card:

Code:

root@freenas-2:~ # sas2flash -list
LSI Corporation SAS2 Flash Utility
Version 16.00.00.00 (2013.03.01)
Copyright (c) 2008-2013 LSI Corporation. All rights reserved

		Adapter Selected is a LSI SAS: SAS2008(B2) 

		Controller Number			  : 0
		Controller					 : SAS2008(B2) 
		PCI Address					: 00:05:00:00
		SAS Address					: 5003048-0-1408-1e00
		NVDATA Version (Default)	   : 14.01.00.06
		NVDATA Version (Persistent)	: 14.01.00.08
		Firmware Product ID			: 0x2213 (IT)
		Firmware Version			   : 20.00.07.00
		NVDATA Vendor				  : LSI
		NVDATA Product ID			  : SAS9210-8i
		BIOS Version				   : 07.31.00.00
		UEFI BSD Version			   : N/A
		FCODE Version				  : N/A
		Board Name					 : SAS9211-8i
		Board Assembly				 : N/A
		Board Tracer Number			: N/A

		Finished Processing Commands Successfully.
		Exiting SAS2Flash.

Still same issue. This has an integrated backplane, I don't have the cables necessary to connect directly to the drives from the controller but I do have a completely separate identical unit I think I will throw FreeNAS on it and see if I get the same results. Maybe it is something with the card or backplane. I can also try a different version of FreeNAS and see if it is something with 11. I'll report back. Thanks for the direction @Ericloewe

 

[HeloJunkie]

OK, I did a fresh install of FreeNAS-9.10.2-U3 (e1497f269) and no problems or trace-backs occur. I am doing an upgrade from U3 to 11 right now. Will report back once the upgrade has completed.

 

[HeloJunkie]

OK, so a fresh install of 9.10.2 produces no tracebacks at all. After installing 9.10.2 I upgraded to 11.0-U2 and sure enough, the tracebacks are back. I then did a complete fresh install directly to 11.0-U2 and got the same tracebacks. I have three of these systems so I had a spare and fired it up and flashed the 2008 to 20.00.07 and tried the same test. Same exact outcome. Works great on 9.10.2 but upgrading to 11.0-U2 or installing 11.0-U2 from scratch leads right back to the tracebacks.

Any thoughts @Ericloewe ?

 

[Ericloewe]

The backplane firmware may need updating, too.

 

[HeloJunkie]

Hummm...even though it works on 9 just fine?

 

[Ericloewe]

It's worth a try.

 

[HeloJunkie]

OK, so the backplane P/N is 415-0049-02 and there is no firmware update for those backplanes. According to my vendor, they are running the latest firmware. All three systems work fine on 9.10.2, all three throw the tracebacks on 11.0-U2.

Any other ideas?

 

[Ericloewe]

I'd try filing a bug report. Maybe iX's contact at Broadcom can help.

 

[HeloJunkie]

OK, So I decided to do some additional testing. As I said, I have three of these units, one is a production backup of a primary NAS so I left it alone. The other two are identical systems with the exception of the amount of RAM that has been installed in the systems.

I started with fresh installs of 9.10 as before and went back up to 11. But this time I pulled all of the drives and started adding them back into the chassis a few at a time to determine when they would fault. I got the same exact results on both systems. The tracebacks are always when I manipulate the pool or add/remove SMART tests.

Here is the overview of the tests
With no pools, there are no errors. I can add/remove/change SMART tests all day long, no tracebacks
With 1 pool, 2 Drives = No Problems
With 1 pool, 3 Drive or more = tracebacks

This is the same behavior on both test systems.