Unable to access Jails and Plugins from web GUI after upgrade from 11.2 to 11.3

[EricSten]

When I try to access either the Jails or Plugins from the web GUI the progress indicator spins for a while and finally pops up a message saying it cannot unmount the most recently mounted jail with the following error "cannot unmount '/mnt/iocage/jails/webserver/root': Device busy". Full error below. I can shutdown the jail that the error is generated on and I get the same error on the next jail.

All the jails are working but I can't manage them from the GUI or look at the available plugins.

Thoughts or suggestions?

Code:

Error: Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/iocage_lib/zfs.py", line 20, in run
    cp.check_returncode()
  File "/usr/local/lib/python3.7/subprocess.py", line 444, in check_returncode
    self.stderr)
subprocess.CalledProcessError: Command '['zfs', 'inherit', 'mountpoint', 'Data/iocage']' returned non-zero exit status 1.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/middlewared/main.py", line 130, in call_method
    io_thread=False)
  File "/usr/local/lib/python3.7/site-packages/middlewared/main.py", line 1091, in _call
    return await run_method(methodobj, *args)
  File "/usr/local/lib/python3.7/site-packages/middlewared/main.py", line 1015, in _run_in_conn_threadpool
    return await self.run_in_executor(self.__ws_threadpool, method, *args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/middlewared/main.py", line 1003, in run_in_executor
    return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs))
  File "/usr/local/lib/python3.7/concurrent/futures/thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/local/lib/python3.7/site-packages/middlewared/schema.py", line 964, in nf
    return f(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/middlewared/plugins/jail.py", line 563, in query
    self.check_dataset_existence()
  File "/usr/local/lib/python3.7/site-packages/middlewared/plugins/jail.py", line 876, in check_dataset_existence
    IOCCheck(migrate=True, reset_cache=True)
  File "/usr/local/lib/python3.7/site-packages/iocage_lib/ioc_check.py", line 62, in __init__
    self.__check_migrations__()
  File "/usr/local/lib/python3.7/site-packages/iocage_lib/ioc_check.py", line 76, in __check_migrations__
    self.iocage_dataset.inherit_property('mountpoint')
  File "/usr/local/lib/python3.7/site-packages/iocage_lib/resource.py", line 33, in inherit_property
    inherit_property(self.resource_name, prop)
  File "/usr/local/lib/python3.7/site-packages/iocage_lib/zfs.py", line 233, in inherit_property
    return run(['zfs', 'inherit', ds_property, dataset]).returncode == 0
  File "/usr/local/lib/python3.7/site-packages/iocage_lib/zfs.py", line 22, in run
    raise ZFSException(cp.returncode, cp.stderr)
iocage_lib.zfs.ZFSException: cannot unmount '/mnt/iocage/jails/webserver/root': Device busy

 

[tobtschik]

Have you tried stopping all the jails, activating iocage on another pool (a new pool on a USB Thumb Drive will do) and reactivating on your main pool? Maybe iocage can self heal or repair whats wrong. At least you would know if managing Jails from the GUI would start to work again.

 

[ghostwolf59]

All my manually created jails (not plugins) *iocage upgraded to 11.2.-U7 that worked perfect under 11.2-U7 * disappeared with 11.3 *cant even get it back to 11.2-U7 using saved config - Not happy Jane!!!!

 

[tobtschik]

Take a look into the jail's datasets. There is a file called config.json. After the upgrade to 11.3 I had two files in there: config.json and config_backup.json. Had to mv config_backup.json config.json for every jail and mv defaults_backup.json defaults.json in iocage dataset to bring all jails back to life after downgrading/rebooting to 11.2-U7.

I need my Docker VM and don't have the time to rebuild another one, so I did not give it another try to upgrade to 11.3 yet.

 

[tobtschik]

What I found out: 11.3 seems to upgrade the conf.json Files for the new NAT functionality. 11.2-U7 iocage cannot read these configs. Why it broke during the upgrade? That I don't know.

 

[EricSten]

Have you tried stopping all the jails, activating iocage on another pool (a new pool on a USB Thumb Drive will do) and reactivating on your main pool? Maybe iocage can self heal or repair whats wrong. At least you would know if managing Jails from the GUI would start to work again.

So I did try stopping and restarting the jails with no effect. I was hesitant with activating iocage on another pool but got forced into it. One of my jails got corrupted and I tried to restore from a backup (using iocage export weekly, using iocage import to restore) but due to the amount of data in the jail it froze up during the import and crashed the OS. Upon reboot the system took it upon itself to "reset" the pool where my iocage was activated. Before iocage was activated on my Data pool with the dataset name of iocage. So the jails were accessed via /mnt/iocage/jail/... After the crash the jails are now accessed via /mnt/Data/iocage with iocage being a dataset under a dataset named Data on the Data pool (yes, I could have done better with my naming). Now that they are running from /mnt/Data/iocage/jail path and all seems to be working fine. I can open the Jails and Plugin GUI options without issue.

As to the import issue, apparently this is a known/open issue. From what I understand when you import a jail it decompresses the image into memory before sending it to ZFS. If you don't have enough memory to hold the image... boom. This article explains this and how to work around it: https://github.com/iocage/iocage/issues/1086

Thanks for the help!

Eric

 

[KevDog]

@EricSten -- I don't know about the memory issue. But znapsend is the workaround?? Weird. I use that utility for backing up my ZFS created VM's to a FreeNAS dataset.

 

[EricSten]

KevDog. I didn't use znapsend for my issue. I did the following, changing for my particular path and jail:

as a work around you can
you may need todo zfs destroy -r zroot/iocage/jails/plex to clean up a failed import
adjust the paths for your setup

1. unzip the exported jail
2. zfs recv -F zroot/iocage/jails/plex < plex_2019-10-30
   zfs recv -F zroot/iocage/jails/plex/root < plex_2019-10-30_root

 

[JohnDigital]

Im having these issues too. I noticed it about the time I went to 11.3-RC2 and forgot as everything works as it should. But its still doing this after upgrade to STABLE and got all jails running 11.3.

This is the only thread I can really find on this as relating to FreeNAS and 11.3. How widespread is this? Should I just forget for awhile again and its going to be fixed, or I need to fix it?

 

[ghostwolf59]

Im having these issues too. I noticed it about the time I went to 11.3-RC2 and forgot as everything works as it should. But its still doing this after upgrade to STABLE and got all jails running 11.3.

This is the only thread I can really find on this as relating to FreeNAS and 11.3. How widespread is this? Should I just forget for awhile again and its going to be fixed, or I need to fix it?

I have no idea whats going on with the upgrade. I ended up installing 11.2-U7, applied my saved config - accepted that none of my jails worked as they were tagged CORRUPT. Applied the 11.3 upgrade.
Were then forced to blow away all my jails (entire iocage pool) and then recreated each jail from scratch.
All works now.

btw. none of the jails could be created using the gui - Had to login as root and do the creation from the command line
i.e iocage create -n <jailname> -r 11.3-RELEASE ip4_addr="vnet0|nnn.nnn.n.nn/24" defaultrouter="nnn.nnn.n.n" vnet="on" allow_raw_sockets="1" boot="on"

 

[JohnDigital]

Management of the jails is done as normal via ssh. Its only when the gui jails/plugins page is opened. Its not effecting anything else as far as I see. Im not sure if it bothers me enough to blow it all away. I did to one of my less used jails and then the error switched to the next jail.

 

[ByteNick]

Still no solution for this?
I know it works via SSH, but...

 

[JohnDigital]

I went so far as to blow away a few less used jails and redo them only to have the error afterwards too. I still get unable to unmount, device busy errors.

 

[John Regan]

Also having this issue. Was hoping someone would do the heavy lifting and figure out a fix. I may just have to abandon my jails I have and use a VM instead.

 

[JohnDigital]

Ok I did the leg work and figured out how to fix this.

Login to the freenas host via ssh. Then stop each jail but first list all jails if you need to. Thiese were my steps.

Code:

iocage list
iocage stop JAILNAME

#------> Repeat this command for each jail

#------>Then forcibly zfs unmount each jail.  I cannot know what your actual command should be, but mine are this.

zfs unmount -f /mnt/iocage/jails/JAILNAME

#------>Repeat this command for each jail.

Now just remount the jails

zfs mount -a

Now you can access the plugin and jail GUI page without error. Then from the GUI restart each jail. I have not tested this any further. Not sure if it works thru reboots and such.

 

[ByteNick]

@John Digital : Thank you so much!
It worked like a charm!

 

[JohnDigital]

Your welcome

 

[John Regan]

Ok I did the leg work and figured out how to fix this.

Login to the freenas host via ssh. Then stop each jail but first list all jails if you need to. Thiese were my steps.

Code:

iocage list
iocage stop JAILNAME

#------> Repeat this command for each jail

#------>Then forcibly zfs unmount each jail.  I cannot know what your actual command should be, but mine are this.

zfs unmount -f /mnt/iocage/jails/JAILNAME

#------>Repeat this command for each jail.

Now just remount the jails

zfs mount -a

Now you can access the plugin and jail GUI page without error. Then from the GUI restart each jail. I have not tested this any further. Not sure if it works thru reboots and such.

Thanks worked for me too. Appreciate your help.

 

[JohnDigital]

My pleasure, glad to help