Jails no longer show up and receiving ('Dataset with path not found',) in multiple locations.

[Poetart]

Was attempting to update to fix some weird virtualization issues I was having. 11.3 did not fix the issue and jails disappeared for me. Reverted back to a VMware snapshot and jails still did not work. Reverted back to a Freenas snapshot and that did not resolve the issue. Have a settings backup from before I ran any updates so I did a fresh install on another drive and loaded the configs. Same thing. Checked the location of the raw files of the Jails and nothing seems to be missing from what I could tell. Changing the directories of the jail manager does not do anything either.

Main issues:

No jails show up in the Jails tab

Can not create a new jail from plugins or jails:
Error: ('Dataset with path not found',)
No more info on error


Error from going to plugins --> Installed

Code:

('Dataset with path not found',)
Error: concurrent.futures.process._RemoteTraceback:
"""
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/concurrent/futures/process.py", line 175, in _process_worker
    r = call_item.fn(*call_item.args, **call_item.kwargs)
  File "/usr/local/lib/python3.6/site-packages/middlewared/worker.py", line 133, in main_worker
    res = loop.run_until_complete(coro)
  File "/usr/local/lib/python3.6/asyncio/base_events.py", line 484, in run_until_complete
    return future.result()
  File "/usr/local/lib/python3.6/site-packages/middlewared/worker.py", line 93, in _run
    return await self._call(name, serviceobj, methodobj, params=args, job=job)
  File "/usr/local/lib/python3.6/site-packages/middlewared/worker.py", line 85, in _call
    return methodobj(*params)
  File "/usr/local/lib/python3.6/site-packages/middlewared/worker.py", line 85, in _call
    return methodobj(*params)
  File "/usr/local/lib/python3.6/site-packages/middlewared/schema.py", line 668, in nf
    return f(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/middlewared/plugins/jail.py", line 546, in list_resource
    resource_list = iocage.list("all", plugin=True)
  File "/usr/local/lib/python3.6/site-packages/iocage_lib/iocage.py", line 1169, in list
    quick=quick
  File "/usr/local/lib/python3.6/site-packages/iocage_lib/ioc_list.py", line 64, in list_datasets
    iocage_lib.ioc_json.IOCJson().json_check_default_config()
  File "/usr/local/lib/python3.6/site-packages/iocage_lib/ioc_json.py", line 1924, in json_check_default_config
    self.json_write(default_props, default_json_location)
  File "/usr/local/lib/python3.6/site-packages/iocage_lib/ioc_json.py", line 431, in json_write
    if template else None
  File "libzfs.pyx", line 645, in libzfs.ZFS.get_dataset_by_path
libzfs.ZFSException: Dataset with path  not found
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/middlewared/main.py", line 165, in call_method
    result = await self.middleware.call_method(self, message)
  File "/usr/local/lib/python3.6/site-packages/middlewared/main.py", line 1107, in call_method
    return await self._call(message['method'], serviceobj, methodobj, params, app=app, io_thread=False)
  File "/usr/local/lib/python3.6/site-packages/middlewared/main.py", line 1052, in _call
    return await self._call_worker(name, *args)
  File "/usr/local/lib/python3.6/site-packages/middlewared/main.py", line 1072, in _call_worker
    return await self.run_in_proc(main_worker, name, args, job)
  File "/usr/local/lib/python3.6/site-packages/middlewared/main.py", line 995, in run_in_proc
    return await self.run_in_executor(self.__procpool, method, *args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/middlewared/main.py", line 971, in run_in_executor
    return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs))
libzfs.ZFSException: ('Dataset with path  not found',)

zpool list -v:

Code:

NAME                                     SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH  ALTROOT
Datastore                                109T  63.8T  44.9T        -         -    11%    58%  1.00x  ONLINE  /mnt
  raidz2                                36.2T  21.3T  15.0T        -         -    12%    58%
    gptid/a59f84da-0466-11e9-b843-000c297fa509      -      -      -        -         -      -      -
    gptid/a65990d7-0466-11e9-b843-000c297fa509      -      -      -        -         -      -      -
    gptid/a71b521b-0466-11e9-b843-000c297fa509      -      -      -        -         -      -      -
    gptid/a7d5fc78-0466-11e9-b843-000c297fa509      -      -      -        -         -      -      -
    gptid/a896c65a-0466-11e9-b843-000c297fa509      -      -      -        -         -      -      -
  raidz2                                36.2T  21.2T  15.0T        -         -    10%    58%
    gptid/a97bb127-0466-11e9-b843-000c297fa509      -      -      -        -         -      -      -
    gptid/aa385c8a-0466-11e9-b843-000c297fa509      -      -      -        -         -      -      -
    gptid/aaff6a9c-0466-11e9-b843-000c297fa509      -      -      -        -         -      -      -
    gptid/abd1c168-0466-11e9-b843-000c297fa509      -      -      -        -         -      -      -
    gptid/aca49950-0466-11e9-b843-000c297fa509      -      -      -        -         -      -      -
  raidz2                                36.2T  21.4T  14.9T        -         -    12%    58%
    gptid/ad7c885a-0466-11e9-b843-000c297fa509      -      -      -        -         -      -      -
    gptid/ae62ba03-0466-11e9-b843-000c297fa509      -      -      -        -         -      -      -
    gptid/af364090-0466-11e9-b843-000c297fa509      -      -      -        -         -      -      -
    gptid/b011a411-0466-11e9-b843-000c297fa509      -      -      -        -         -      -      -
    gptid/b0e8b8f3-0466-11e9-b843-000c297fa509      -      -      -        -         -      -      -
SSD                                      460G   227G   233G        -         -    34%    49%  1.00x  ONLINE  /mnt
  gptid/03d03205-00c7-11e9-896e-000c297fa509   460G   227G   233G        -         -    34%    49%
freenas-boot                             238G   769M   237G        -         -      -     0%  1.00x  ONLINE  -
  da17p2                                 238G   769M   237G        -         -      -     0%

Code:

root@freenas:~ # zfs list
NAME                                                          USED  AVAIL  REFER  MOUNTPOINT
Datastore                                                    37.8T  24.6T  37.8T  /mnt/Datastore
Datastore/.bhyve_containers                                  70.5M  24.6T  70.5M  /mnt/Datastore/.bhyve_containers
Datastore/iocage                                             4.66M  24.6T  3.74M  /mnt/Datastore/iocage
Datastore/iocage/download                                     156K  24.6T   156K  /mnt/Datastore/iocage/download
Datastore/iocage/images                                       156K  24.6T   156K  /mnt/Datastore/iocage/images
Datastore/iocage/jails                                        156K  24.6T   156K  /mnt/Datastore/iocage/jails
Datastore/iocage/log                                          156K  24.6T   156K  /mnt/Datastore/iocage/log
Datastore/iocage/releases                                     156K  24.6T   156K  /mnt/Datastore/iocage/releases
Datastore/iocage/templates                                    156K  24.6T   156K  /mnt/Datastore/iocage/templates
SSD                                                           227G   218G    88K  /mnt/SSD
SSD/.system                                                   714M   218G    96K  legacy
SSD/.system/configs-3bc724bbb18d493585697320a07060e7          371M   218G   371M  legacy
SSD/.system/cores                                            9.97M   218G  9.97M  legacy
SSD/.system/rrd-3bc724bbb18d493585697320a07060e7              326M   218G   326M  legacy
SSD/.system/samba4                                           5.62M   218G  5.62M  legacy
SSD/.system/syslog-3bc724bbb18d493585697320a07060e7          1.14M   218G  1.14M  legacy
SSD/.system/webui                                              88K   218G    88K  legacy
SSD/iocage                                                    221G   218G  8.74M  /mnt/SSD/iocage
SSD/iocage/download                                           409M   218G    88K  /mnt/SSD/iocage/download
SSD/iocage/download/11.2-RELEASE                              272M   218G   272M  /mnt/SSD/iocage/download/11.2-RELEASE
SSD/iocage/download/11.3-RELEASE                              137M   218G   137M  /mnt/SSD/iocage/download/11.3-RELEASE
SSD/iocage/images                                              88K   218G    88K  /mnt/SSD/iocage/images
SSD/iocage/jails                                              219G   218G    96K  /mnt/SSD/iocage/jails
SSD/iocage/jails/Jackett                                      504M   218G    96K  /mnt/SSD/iocage/jails/Jackett
SSD/iocage/jails/Jackett/root                                 504M   218G  1.44G  /mnt/SSD/iocage/jails/Jackett/root
SSD/iocage/jails/Tautulli 2.0                                 521M   218G    92K  /mnt/SSD/iocage/jails/Tautulli 2.0
SSD/iocage/jails/Tautulli 2.0/root                            521M   218G  1.46G  /mnt/SSD/iocage/jails/Tautulli 2.0/root
SSD/iocage/jails/deluge                                       427M   218G   268K  /mnt/SSD/iocage/jails/deluge
SSD/iocage/jails/deluge/root                                  427M   218G  1.05G  /mnt/SSD/iocage/jails/deluge/root
SSD/iocage/jails/plex                                         216G   218G   184K  /mnt/SSD/iocage/jails/plex
SSD/iocage/jails/plex/root                                    216G   218G   217G  /mnt/SSD/iocage/jails/plex/root
SSD/iocage/jails/sonarr                                      1.58G   218G   276K  /mnt/SSD/iocage/jails/sonarr
SSD/iocage/jails/sonarr/root                                 1.58G   218G  2.23G  /mnt/SSD/iocage/jails/sonarr/root
SSD/iocage/log                                                120K   218G   120K  /mnt/SSD/iocage/log
SSD/iocage/releases                                          1.34G   218G    88K  /mnt/SSD/iocage/releases
SSD/iocage/releases/11.2-RELEASE                             1011M   218G    88K  /mnt/SSD/iocage/releases/11.2-RELEASE
SSD/iocage/releases/11.2-RELEASE/root                        1011M   218G   998M  /mnt/SSD/iocage/releases/11.2-RELEASE/root
SSD/iocage/releases/11.3-RELEASE                              358M   218G    88K  /mnt/SSD/iocage/releases/11.3-RELEASE
SSD/iocage/releases/11.3-RELEASE/root                         358M   218G   358M  /mnt/SSD/iocage/releases/11.3-RELEASE/root
SSD/iocage/templates                                           88K   218G    88K  /mnt/SSD/iocage/templates
SSD/jails                                                    5.87G   218G   104K  /mnt/SSD/jails
SSD/jails/.warden-template-pluginjail-11.0-x64                539M   218G   539M  /mnt/SMB-Storage/jails/.warden-template-pluginjail-11.0-x64
SSD/jails/.warden-template-standard-11.0-x64                 1.78G   218G  1.78G  /mnt/SMB-Storage/jails/.warden-template-standard-11.0-x64
SSD/jails/.warden-template-standard-11.0-x64-20180528214233  1.78G   218G  1.78G  /mnt/SMB-Storage/jails/.warden-template-standard-11.0-x64-20180528214233
SSD/jails/.warden-template-standard-11.0-x64-20180815111841  1.79G   218G  1.79G  /mnt/SMB-Storage/jails/.warden-template-standard-11.0-x64-20180815111841
freenas-boot                                                  768M   230G    64K  none
freenas-boot/ROOT                                             767M   230G    29K  none
freenas-boot/ROOT/Initial-Install                               1K   230G   758M  legacy
freenas-boot/ROOT/default                                     767M   230G   760M  legacy

Freenas ver currently : 11.2 u7

 

[jag131990]

How did you fix this? I am having the exact same problem!

 

[Poetart]

How did you fix this? I am having the exact same problem!

It has to do with jail names. If you look at the ones I have, there is a jail I had called
SSD/iocage/jails/Tautulli 2.0
If I remember right, the "." was the issue that crashed the whole thing. Might have been the space too but I changed deleted the whole jail as even when I got it working it said it was "corrupt". I assume that was from me changing the root jail name and it was easier for me to just import the whole thing to a new jail then to make the old one work. After that I upgraded back to 11.3 and everything came back up.
I cant remember the commands off the top of my head but if you have any more issues let me know and I can try to help.

 

[Clinderw]

I'm having this exact issue after having to roll back to 11.2 (booting from USB was causing me issues, oddly, with 11.3). I don't have any jails with spaces or periods. Anything else involved in the solve here?

 

[okynnor]

I'm having the exact same issue where I upgraded to 11.3U1. When the FTP feature didn't work, I rolled back to 11.2U8. The Jails are not installing now. I get "('Dataset with path not found',)" but the pool is configured and activated for Jails. Another bugs? have to wait?

Error: Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/middlewared/main.py", line 166, in call_method
result = await self.middleware.call_method(self, message)
File "/usr/local/lib/python3.6/site-packages/middlewared/main.py", line 1093, in call_method
return await self._call(message['method'], serviceobj, methodobj, params, app=app, io_thread=False)
File "/usr/local/lib/python3.6/site-packages/middlewared/main.py", line 1037, in _call
return await self._call_worker(name, *args)
File "/usr/local/lib/python3.6/site-packages/middlewared/main.py", line 1058, in _call_worker
return await self.run_in_proc(main_worker, name, args, job)
File "/usr/local/lib/python3.6/site-packages/middlewared/main.py", line 990, in run_in_proc
return await async_run_in_executor(loop, executor, method, *args, **kwargs)
File "/usr/local/lib/python3.6/site-packages/middlewared/utils/asyncio_.py", line 41, in async_run_in_executor
raise result
libzfs.ZFSException: ('Dataset with path not found',)

 

[Clinderw]

I couldn’t get this to work so I upgraded again and so far has been fine. Weird path and pattern. Admittedly frustrating. The boot size was really small on the roll back too...it’s like something with the jails fell out.

After going forward again to 11.3 it’s been working.

Chris

 

[okynnor]

I couldn’t get this to work so I upgraded again and so far has been fine. Weird path and pattern. Admittedly frustrating. The boot size was really small on the roll back too...it’s like something with the jails fell out.

After going forward again to 11.3 it’s been working.

Chris

I can't go forward to 11.3 until the FTP server feature is fixed though.

 

[okynnor]

I wanted to write my resulting course of action given the amount of issues which I've described in this thread. Thanks to clindenw, I have followed his advise and to reinstall FreeNAS. The most current version as of this writing is 11.3U1. I didn't move my data to the Cloud first. I simply reinstalled FreeNAS, then I didn't restore the configuration. I chose to redo all my Services from scratch by reattaching the Pool with all the data intact. It worked! The following services were fully restored:

1. iSCSI
2. FTP
3. SMB
4. NFS

I was successful in installing SyncThing too. I notice that how Jails are handled have changed when moving from 11.2 U8 to 11.3U1.

All the services are working

A word of caution though. Make sure that you're really, really sure before upgrading. Once you do, and things don't work as they should, rolling back the boot environment doesn't fix things and can make things more complicated.