Call timeout managing extents

[Thibaut]

Hello,

We're running FreeNAS 11.2-RELEASE-U1 on the following server:

Supermicro X9DRD-7LN4F-JBOD
2 x Xeon E-2620 v2 @ 2.10GHz (24 cores)
192 GB ECC RAM
36 x 3TB HDs

This machine is mainly aimed at making a lot of iSCSI extents of type "Device" (> 300) available to a pool of 15 workstations, each acting as an iSCSI target. Each link is created as a "Target / Extent" under "Associated Targets". A maximum of 8 simultaneous extents per target has been enforced. So far so good. Unfortunately, at a certain point when the server is under standard load, using the FreeNAS GUI to create new extents becomes impossible.

Using the new web interface doesn't show any information, simply not displaying anything under the "Device" list that should pop up and turning the grey line to red once it has been clicked on.
Using the "legacy" interface continuously returns a "Call timeout" error, showing the traceback copied here below, thus not letting the opportunity to effectively create the new extent...

Sometimes, when the server load is lighter, both interfaces work as expected.
We suspect that the origin of the problem is that the uwsgi configuration for the nginx/Django driven FreeNAS interface is somehow setup with a maximum wait time that is exceeded while waiting for a response, generating the "call timeout" error reported in the legacy interface, but we have no clue where to modify this value and mostly whether it would be possible, or even advisable to do so !

The only workaround that we found up to now is to directly edit the /data/freenas-v1.db file, adding new entries in the "services_iscsitargetextent" table by hand, which is obviously not an ideal solution

We'd appreciate any advise or indication of whether our suspicion is correct and that the "call timeout" error is well produced by the script taking too long to execute.
If so, is it possible to change the maximum wait time for the FreeNAS interface's scripts and then where should we look to adapt this ?
Last, would such a modification be damageable to the general behavior/stability of the system ?

Thank you very much,

Thibaut

ERROR / TRACEBACK CODE returned in FreeNAS legacy interface
----------------------------

Code:

Request Method:     GET
Request URL:     http://172.16.0.211:65080/legacy/admin/services/iscsitargetextent/add/
Software Version:     FreeNAS-11.2-RELEASE-U1 (31f889bbf)
Exception Type:     CallTimeout
Exception Value:     

Call timeout

Exception Location:     /usr/local/lib/python3.6/site-packages/middlewared/client/client.py in call, line 447
Server time:     Tue, 19 Feb 2019 01:00:01 +0100

Traceback
...................
Environment:

Software Version: FreeNAS-11.2-RELEASE-U1 (31f889bbf)
Request Method: GET
Request URL: http://172.16.0.211:65080/legacy/admin/services/iscsitargetextent/add/


Traceback:
File "/usr/local/lib/python3.6/site-packages/django/core/handlers/exception.py" in inner
  42.             response = get_response(request)
File "/usr/local/lib/python3.6/site-packages/django/core/handlers/base.py" in _legacy_get_response
  249.             response = self._get_response(request)
File "/usr/local/lib/python3.6/site-packages/django/core/handlers/base.py" in _get_response
  178.             response = middleware_method(request, callback, callback_args, callback_kwargs)
File "./freenasUI/freeadmin/middleware.py" in process_view
  163.         return login_required(view_func)(request, *view_args, **view_kwargs)
File "/usr/local/lib/python3.6/site-packages/django/contrib/auth/decorators.py" in _wrapped_view
  23.                 return view_func(request, *args, **kwargs)
File "./freenasUI/freeadmin/options.py" in wrapper
  216.                 return self._admin.admin_view(view)(*args, **kwargs)
File "/usr/local/lib/python3.6/site-packages/django/utils/decorators.py" in _wrapped_view
  149.                     response = view_func(request, *args, **kwargs)
File "/usr/local/lib/python3.6/site-packages/django/views/decorators/cache.py" in _wrapped_view_func
  57.         response = view_func(request, *args, **kwargs)
File "./freenasUI/freeadmin/site.py" in inner
  143.             return view(request, *args, **kwargs)
File "./freenasUI/freeadmin/options.py" in add
  403.             mf = mf()
File "./freenasUI/services/forms.py" in __init__
  831.                     'iscsi.extent.disk_choices').items())
File "./freenasUI/services/forms.py" in __init__
  831.                     'iscsi.extent.disk_choices').items())
File "/usr/local/lib/python3.6/site-packages/middlewared/client/client.py" in call
  447.             raise CallTimeout("Call timeout")

Exception Type: CallTimeout at /legacy/admin/services/iscsitargetextent/add/
Exception Value: Call timeout
...................

Request information
...................
GET

No GET data
POST

No POST data
FILES

No FILES data
COOKIES
Variable     Value
fntreeSaveStateCookie     'root'
csrftoken     '********'
sessionid     'zunpmoozy001ai4bh38ryh8mg50fibon'
META
Variable     Value

 

[Chris Moore]

I think this might be something for you to open a bug report on. If there is a problem with the GUI, the developers need to know so they can fix it. Please post the number to the ticket here.
If your FreeNAS GUI is not working, you should be able to go here: https://redmine.ixsystems.com/projects/freenas/

 

[Thibaut]

Thank you Chris !

I've filed this in the bug report system as issue #75774

In the meantime any suggestion on how to handle this problem, what to do or what NOT to do is still welcome !

Best regards,
Thibaut

 

[jakubjb]

Hi Thibaut, Chris!
I have the same issue with 11.2-U6 right now. I checked bug report and saw:
"call timeouts are often an indication of a failing boot device. Does a fresh install to a new USB stick resolve the issue?"
I have installed system on two new USB sticks (mirror) last month. Freenas-boot pool is healthy, scrub successful, the same with System->Update->Verify install.
Is fresh install still the only answer to this?

 

[jakubjb]

Read speed felt to ground, so all looks fine, while looking at statuses, but simple file copy (read) shows the problem. I definitelly end USB sticks usage. It's not the first time, it's happening more frequently with newer sticks, sadly.