Jails OSError [Errno 6] Device not configured

[gmhafiz]

Last week my syncthing jail began to fail. Its UI at port 8080 was showing the default nginx page. Inaccessible.

Today, the syncthing UI is totally inaccessible. On top of that my plex jail failed too, getting a connection refused

Code:

curl -vv http://192.168.1.106:8080/
*   Trying 192.168.1.106:8080...
* TCP_NODELAY set
* connect to 192.168.1.106 port 8080 failed: Connection refused
* Failed to connect to 192.168.1.106 port 8080: Connection refused
* Closing connection 0
curl: (7) Failed to connect to 192.168.1.106 port 8080: Connection refused

Code:

curl -vv http://192.168.1.104:32400/web/index.html
*   Trying 192.168.1.104:32400...
* TCP_NODELAY set
* connect to 192.168.1.104 port 32400 failed: Connection refused
* Failed to connect to 192.168.1.104 port 32400: Connection refused
* Closing connection 0
curl: (7) Failed to connect to 192.168.1.104 port 32400: Connection refused

I can confirm all my files are still present by accessing by sftp.

When trying to access the Jail menu, I get OSError [Errno 6] Device not configured

Code:

Error: Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/iocage_lib/ioc_json.py", line 1266, in check_default_config
    default_props = json.load(default_json)
  File "/usr/local/lib/python3.9/json/__init__.py", line 293, in load
    return loads(fp.read(),
OSError: [Errno 6] Device not configured

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/middlewared/main.py", line 137, in call_method
    result = await self.middleware._call(message['method'], serviceobj, methodobj, params, app=self,
  File "/usr/local/lib/python3.9/site-packages/middlewared/main.py", line 1209, in _call
    return await self.run_in_executor(prepared_call.executor, methodobj, *prepared_call.args)
  File "/usr/local/lib/python3.9/site-packages/middlewared/main.py", line 1113, in run_in_executor
    return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs))
  File "/usr/local/lib/python3.9/concurrent/futures/thread.py", line 52, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/local/lib/python3.9/site-packages/middlewared/schema.py", line 977, in nf
    return f(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/middlewared/plugins/jail_freebsd.py", line 1054, in get_activated_pool
    pool = ioc.IOCage(skip_jails=True, reset_cache=True).get('', pool=True)
  File "/usr/local/lib/python3.9/site-packages/iocage_lib/iocage.py", line 95, in __init__
    self.generic_iocjson = ioc_json.IOCJson()
  File "/usr/local/lib/python3.9/site-packages/iocage_lib/ioc_json.py", line 1372, in __init__
    super().__init__(location, checking_datasets, silent, callback)
  File "/usr/local/lib/python3.9/site-packages/iocage_lib/ioc_json.py", line 432, in __init__
    self.default_config = self.check_default_config()
  File "/usr/local/lib/python3.9/site-packages/iocage_lib/ioc_json.py", line 1301, in check_default_config
    self.backup_iocage_jail_conf(default_json_location)
  File "/usr/local/lib/python3.9/site-packages/iocage_lib/ioc_json.py", line 919, in backup_iocage_jail_conf
    shutil.copy(
  File "/usr/local/lib/python3.9/shutil.py", line 418, in copy
    copyfile(src, dst, follow_symlinks=follow_symlinks)
  File "/usr/local/lib/python3.9/shutil.py", line 285, in copyfile
    copyfileobj(fsrc, fdst)
  File "/usr/local/lib/python3.9/shutil.py", line 205, in copyfileobj
    buf = fsrc_read(length)
OSError: [Errno 6] Device not configured

In alert, I get

Code:

Failed to check for alert HasUpdate: Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/middlewared/plugins/alert.py", line 740, in 
__run_source alerts = (await alert_source.check()) or [] File "/usr/local/lib/python3.9/site-packages/middlewared/alert/base.py", line 210, in check return await 
self.middleware.run_in_thread(self.check_sync) File "/usr/local/lib/python3.9/site-packages/middlewared/utils/run_in_thread.py", line 10, in run_in_thread return await 
self.loop.run_in_executor(self.run_in_thread_executor, functools.partial(method, *args, **kwargs)) File "/usr/local/lib/python3.9/site-packages/middlewared/utils
/io_thread_pool_executor.py", line 25, in run result = self.fn(*self.args, **self.kwargs) File "/usr/local/lib/python3.9/site-packages/middlewared/alert/source/update.py", 
line 67, in check_sync path = self.middleware.call_sync("update.get_update_location") File "/usr/local/lib/python3.9/site-packages/middlewared/main.py", line 1257, in 
call_sync return self.run_coroutine(methodobj(*prepared_call.args)) File "/usr/local/lib/python3.9/site-packages/middlewared/main.py", line 1297, in run_coroutine 
return fut.result() File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 438, in result return self.__get_result() File "/usr/local/lib/python3.9/concurrent/futures
/_base.py", line 390, in __get_result raise self._exception File "/usr/local/lib/python3.9/site-packages/middlewared/plugins/update.py", line 412, in get_update_location 
os.chmod(path, 0o755) OSError: [Errno 6] Device not configured: '/var/db/system/update' 

/var/db/system/update gas some files in it

 

[gmhafiz]

It looks like it may be related to https://jira.ixsystems.com/browse/NAS-110671

I have updated truenas to the latest version and that seems to fix this problem.

My jails, except syncthing are up. CPU and HDD utilization are also down .