Force Switch Jail Tank

[Stealthbird]

Hello Everyone,
I am in a spot of bother and need some assistance.
I am running Freenas on a tower server with internal storage and also have an external disk array connected via fibre channel. Broken into two tanks depending on whether or not it's internal (lo_tank) or external (ext_tank)
Unfortunately, I've lost a drive in my external disk array and as I am not physically able to replace the drive (I am currently working in the USA until January, and my server is located in my home in the UK) I decided it was best to just shut down the disk array.

My jails and some plugins were running on this ext_tank, but I would now like to spin up some jails in the lo_tank.
I am having issues switching the jails system from ext_tank to lo_tank. I know I can't access the old jails, I just want to make new ones, but the system will not let me activate the lo_tank for jails or plugins and throws errors.

When I open the jail page in the gui.
ZFSException
cannot open 'ext_tank': pool I/O is currently suspended

Code:

Error: Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/iocage_lib/zfs.py", line 20, in run
    cp.check_returncode()
  File "/usr/local/lib/python3.7/subprocess.py", line 444, in check_returncode
    self.stderr)
subprocess.CalledProcessError: Command '['zfs', 'get', '-H', '-o', 'property,value', 'all', 'ext_tank']' returned non-zero exit status 1.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/middlewared/main.py", line 130, in call_method
    io_thread=False)
  File "/usr/local/lib/python3.7/site-packages/middlewared/main.py", line 1095, in _call
    return await run_method(methodobj, *args)
  File "/usr/local/lib/python3.7/site-packages/middlewared/main.py", line 1019, in _run_in_conn_threadpool
    return await self.run_in_executor(self.__ws_threadpool, method, *args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/middlewared/main.py", line 1007, in run_in_executor
    return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs))
  File "/usr/local/lib/python3.7/concurrent/futures/thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/local/lib/python3.7/site-packages/middlewared/schema.py", line 965, in nf
    return f(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/middlewared/plugins/jail.py", line 914, in get_activated_pool
    pool = ioc.IOCage(skip_jails=True, reset_cache=True).get('', pool=True)
  File "/usr/local/lib/python3.7/site-packages/iocage_lib/iocage.py", line 95, in __init__
    self.pool = PoolAndDataset().get_pool()
  File "/usr/local/lib/python3.7/site-packages/iocage_lib/iocage.py", line 66, in get_pool
    return ioc_json.IOCJson().json_get_value("pool")
  File "/usr/local/lib/python3.7/site-packages/iocage_lib/ioc_json.py", line 1321, in __init__
    super().__init__(location, checking_datasets, silent, callback)
  File "/usr/local/lib/python3.7/site-packages/iocage_lib/ioc_json.py", line 428, in __init__
    self.pool, self.iocroot = self.get_pool_and_iocroot()
  File "/usr/local/lib/python3.7/site-packages/iocage_lib/ioc_json.py", line 552, in get_pool_and_iocroot
    pool = get_pool()
  File "/usr/local/lib/python3.7/site-packages/iocage_lib/ioc_json.py", line 453, in get_pool
    if pool.active:
  File "/usr/local/lib/python3.7/site-packages/iocage_lib/pools.py", line 27, in active
    return Dataset(self.name, cache=self.cache).properties.get(
  File "/usr/local/lib/python3.7/site-packages/iocage_lib/dataset.py", line 38, in __init__
    self._properties = deepcopy(cache.datasets.get(self.resource_name))
  File "/usr/local/lib/python3.7/site-packages/iocage_lib/cache.py", line 22, in datasets
    ioc_pool = iocage_activated_pool()
  File "/usr/local/lib/python3.7/site-packages/iocage_lib/zfs.py", line 97, in iocage_activated_pool
    if dataset_properties(pool).get('org.freebsd.ioc:active') == 'yes':
  File "/usr/local/lib/python3.7/site-packages/iocage_lib/zfs.py", line 88, in dataset_properties
    return properties(dataset, 'zfs')
  File "/usr/local/lib/python3.7/site-packages/iocage_lib/zfs.py", line 54, in properties
    resource_type, 'get', '-H', '-o', 'property,value', 'all', dataset
  File "/usr/local/lib/python3.7/site-packages/iocage_lib/zfs.py", line 22, in run
    raise ZFSException(cp.returncode, cp.stderr)
iocage_lib.zfs.ZFSException: cannot open 'ext_tank': pool I/O is currently suspended

I clear this message and go to "choose pool for Plugin and Jail Storage", and then get this

CallError
[EFAULT] Failed to activate lo_tank: cannot open 'ext_tank': pool I/O is currently suspended

Code:

Error: Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/iocage_lib/zfs.py", line 20, in run
    cp.check_returncode()
  File "/usr/local/lib/python3.7/subprocess.py", line 444, in check_returncode
    self.stderr)
subprocess.CalledProcessError: Command '['zfs', 'get', '-H', '-o', 'property,value', 'all', 'ext_tank']' returned non-zero exit status 1.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/middlewared/plugins/jail.py", line 1316, in activate
    iocage.activate(pool['name'])
  File "/usr/local/lib/python3.7/site-packages/iocage_lib/iocage.py", line 361, in activate
    pool.deactivate_pool()
  File "/usr/local/lib/python3.7/site-packages/iocage_lib/pools.py", line 53, in deactivate_pool
    Dataset(self.name, cache=self.cache).set_property(
  File "/usr/local/lib/python3.7/site-packages/iocage_lib/dataset.py", line 38, in __init__
    self._properties = deepcopy(cache.datasets.get(self.resource_name))
  File "/usr/local/lib/python3.7/site-packages/iocage_lib/cache.py", line 22, in datasets
    ioc_pool = iocage_activated_pool()
  File "/usr/local/lib/python3.7/site-packages/iocage_lib/zfs.py", line 97, in iocage_activated_pool
    if dataset_properties(pool).get('org.freebsd.ioc:active') == 'yes':
  File "/usr/local/lib/python3.7/site-packages/iocage_lib/zfs.py", line 88, in dataset_properties
    return properties(dataset, 'zfs')
  File "/usr/local/lib/python3.7/site-packages/iocage_lib/zfs.py", line 54, in properties
    resource_type, 'get', '-H', '-o', 'property,value', 'all', dataset
  File "/usr/local/lib/python3.7/site-packages/iocage_lib/zfs.py", line 22, in run
    raise ZFSException(cp.returncode, cp.stderr)
iocage_lib.zfs.ZFSException: cannot open 'ext_tank': pool I/O is currently suspended


During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/middlewared/main.py", line 130, in call_method
    io_thread=False)
  File "/usr/local/lib/python3.7/site-packages/middlewared/main.py", line 1095, in _call
    return await run_method(methodobj, *args)
  File "/usr/local/lib/python3.7/site-packages/middlewared/main.py", line 1019, in _run_in_conn_threadpool
    return await self.run_in_executor(self.__ws_threadpool, method, *args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/middlewared/main.py", line 1007, in run_in_executor
    return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs))
  File "/usr/local/lib/python3.7/concurrent/futures/thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/local/lib/python3.7/site-packages/middlewared/schema.py", line 965, in nf
    return f(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/middlewared/plugins/jail.py", line 1318, in activate
    raise CallError(f'Failed to activate {pool["name"]}: {e}')
middlewared.service_exception.CallError: [EFAULT] Failed to activate lo_tank: cannot open 'ext_tank': pool I/O is currently suspended

How can I get around this. I don't see why the tank that the jails are currently, and the fact that it is inaccessible, prevents me changing to a new jail.


System Config
FreeBSD 11.3-RELEASE-p7 (FreeNAS.amd64)

Hopefully someone can help me with this.

Kind Regards

 

[Samuel Tai]

According to /usr/local/lib/python3.7/site-packages/middlewared/plugins/jail.py, lines 1312-1322, there can be only one zpool activated for jails/plugins. When you switch zpools for jails, you nuke all the old jails and plugins. When you shut down zpool ext_tank, the GUI's no longer able to reach that zpool to clear all the existing jails and deactivate it in preparation for activating zpool lo_tank to host jails.

 

[Stealthbird]

Is there any way I can get around this?

 

[Samuel Tai]

The only way I can think of is to reinstall, but that's dangerous to do if you're not physically at the console.

 

[Samuel Tai]

I have no idea how this will interact with the FreeNAS GUI. The following is completely untested. Looking at the source code for iocage/activate.py, activation merely adds the ZFS property org.freebsd.ioc:active=yes to the zpool. You could try zfs set org.freebsd.ioc:active=yes lo_tank and zfs set org.freebsd.ioc:active=no ext_tank. (The 2nd may or may not work, as ext_tank is offline.) Whether that will get you through the GUI, I can't say.

 

[Stealthbird]

Yes, I tried to change the zfs property. Unfortunately, like you said the second one did not work for the same reason as before, and it would not change the jail pool.
I think I might just get my parents to start the disk array back up and hopefully I I can get it changed, otherwise, I guess I'll have to wait until I get back home. :S