Truenas Core 12.0-u1/u3 Unable to Export/Disconnect Pools/Datasets

[Emile.Belcourt]

Hello all,

I'm having a bit of an issue at the moment, I've been unable to export/disconnect any pools nor can I delete any datasets, I get either:
-[EBUSY] unmount failed when Export/Disconnect Pool (full error in spoiler below)

Show : Error for ZFS Export/Disconnect

Error: concurrent.futures.process._RemoteTraceback:
"""
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/middlewared/plugins/zfs.py", line 139, in do_delete
zfs.destroy(name, force=options['force'])
File "libzfs.pyx", line 391, in libzfs.ZFS.__exit__
File "/usr/local/lib/python3.8/site-packages/middlewared/plugins/zfs.py", line 139, in do_delete
zfs.destroy(name, force=options['force'])
File "libzfs.pyx", line 1330, in libzfs.ZFS.destroy
libzfs.ZFSException: unmount failed

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/local/lib/python3.8/concurrent/futures/process.py", line 239, in _process_worker
r = call_item.fn(*call_item.args, **call_item.kwargs)
File "/usr/local/lib/python3.8/site-packages/middlewared/worker.py", line 94, in main_worker
res = MIDDLEWARE._run(*call_args)
File "/usr/local/lib/python3.8/site-packages/middlewared/worker.py", line 45, in _run
return self._call(name, serviceobj, methodobj, args, job=job)
File "/usr/local/lib/python3.8/site-packages/middlewared/worker.py", line 39, in _call
return methodobj(*params)
File "/usr/local/lib/python3.8/site-packages/middlewared/worker.py", line 39, in _call
return methodobj(*params)
File "/usr/local/lib/python3.8/site-packages/middlewared/schema.py", line 977, in nf
return f(*args, **kwargs)
File "/usr/local/lib/python3.8/site-packages/middlewared/plugins/zfs.py", line 144, in do_delete
raise CallError(str(e), errno_)
middlewared.service_exception.CallError: [EBUSY] unmount failed
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/middlewared/job.py", line 367, in run
await self.future
File "/usr/local/lib/python3.8/site-packages/middlewared/job.py", line 403, in __run_body
rv = await self.method(*([self] + args))
File "/usr/local/lib/python3.8/site-packages/middlewared/schema.py", line 973, in nf
return await f(*args, **kwargs)
File "/usr/local/lib/python3.8/site-packages/middlewared/plugins/pool.py", line 1579, in export
await self.middleware.call('zfs.pool.delete', pool['name'])
File "/usr/local/lib/python3.8/site-packages/middlewared/main.py", line 1238, in call
return await self._call(
File "/usr/local/lib/python3.8/site-packages/middlewared/main.py", line 1195, in _call
return await methodobj(*prepared_call.args)
File "/usr/local/lib/python3.8/site-packages/middlewared/service.py", line 487, in delete
rv = await self.middleware._call(
File "/usr/local/lib/python3.8/site-packages/middlewared/main.py", line 1203, in _call
return await self._call_worker(name, *prepared_call.args)
File "/usr/local/lib/python3.8/site-packages/middlewared/main.py", line 1209, in _call_worker
return await self.run_in_proc(main_worker, name, args, job)
File "/usr/local/lib/python3.8/site-packages/middlewared/main.py", line 1136, in run_in_proc
return await self.run_in_executor(self.__procpool, method, *args, **kwargs)
File "/usr/local/lib/python3.8/site-packages/middlewared/main.py", line 1110, in run_in_executor
return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs))
middlewared.service_exception.CallError: [EBUSY] unmount failed

-Error deleting dataset <i>Dataset Name</i> when deleting dataset

Show : Error for Dataset Delete

[EFAULT] Failed to delete dataset: cannot unmount '/mnt/Hot_NVMe_Pool/Active_Data_Share': unmount failed

I have made sure there is no data in the dataset, it's not being used by NFS, SMB/CIFs or any other service I can think of (I may be unaware of some but I have done the dutiful). In middlewared.log when attempting to delete a dataset, I get the following:

Show : middlewared.log exerpt for dataset removal

[2021/04/22 15:51:12] (ERROR) middlewared.job.run():379 - Job <bound method accepts.<locals>.wrap.<locals>.nf of <middlewared.plugins.pool.PoolService object at 0x81cfa1df0>> failed
concurrent.futures.process._RemoteTraceback:
"""
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/middlewared/plugins/zfs.py", line 139, in do_delete
zfs.destroy(name, force=options['force'])
File "libzfs.pyx", line 391, in libzfs.ZFS.__exit__
File "/usr/local/lib/python3.8/site-packages/middlewared/plugins/zfs.py", line 139, in do_delete
zfs.destroy(name, force=options['force'])
File "libzfs.pyx", line 1330, in libzfs.ZFS.destroy
libzfs.ZFSException: unmount failed

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/local/lib/python3.8/concurrent/futures/process.py", line 239, in _process_worker
r = call_item.fn(*call_item.args, **call_item.kwargs)
File "/usr/local/lib/python3.8/site-packages/middlewared/worker.py", line 94, in main_worker
res = MIDDLEWARE._run(*call_args)
File "/usr/local/lib/python3.8/site-packages/middlewared/worker.py", line 45, in _run
return self._call(name, serviceobj, methodobj, args, job=job)
File "/usr/local/lib/python3.8/site-packages/middlewared/worker.py", line 39, in _call
return methodobj(*params)
File "/usr/local/lib/python3.8/site-packages/middlewared/worker.py", line 39, in _call
return methodobj(*params)
File "/usr/local/lib/python3.8/site-packages/middlewared/schema.py", line 977, in nf
return f(*args, **kwargs)
File "/usr/local/lib/python3.8/site-packages/middlewared/plugins/zfs.py", line 144, in do_delete
raise CallError(str(e), errno_)
middlewared.service_exception.CallError: [EBUSY] unmount failed
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/middlewared/job.py", line 367, in run
await self.future
File "/usr/local/lib/python3.8/site-packages/middlewared/job.py", line 403, in __run_body
rv = await self.method(*([self] + args))
File "/usr/local/lib/python3.8/site-packages/middlewared/schema.py", line 973, in nf
return await f(*args, **kwargs)
File "/usr/local/lib/python3.8/site-packages/middlewared/plugins/pool.py", line 1579, in export
await self.middleware.call('zfs.pool.delete', pool['name'])
File "/usr/local/lib/python3.8/site-packages/middlewared/main.py", line 1238, in call
return await self._call(
File "/usr/local/lib/python3.8/site-packages/middlewared/main.py", line 1195, in _call
return await methodobj(*prepared_call.args)
File "/usr/local/lib/python3.8/site-packages/middlewared/service.py", line 487, in delete
rv = await self.middleware._call(
File "/usr/local/lib/python3.8/site-packages/middlewared/main.py", line 1203, in _call
return await self._call_worker(name, *prepared_call.args)
File "/usr/local/lib/python3.8/site-packages/middlewared/main.py", line 1209, in _call_worker
return await self.run_in_proc(main_worker, name, args, job)
File "/usr/local/lib/python3.8/site-packages/middlewared/main.py", line 1136, in run_in_proc
return await self.run_in_executor(self.__procpool, method, *args, **kwargs)
File "/usr/local/lib/python3.8/site-packages/middlewared/main.py", line 1110, in run_in_executor
return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs))
middlewared.service_exception.CallError: [EBUSY] unmount failed
[2021/04/22 15:52:31] (ERROR) ZFSDatasetService.do_delete():860 - Failed to delete dataset
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/middlewared/plugins/zfs.py", line 853, in do_delete
subprocess.run(
File "/usr/local/lib/python3.8/subprocess.py", line 512, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['zfs', 'destroy', '-r', 'Hot_NVMe_Pool/Active_Data_Share']' returned non-zero exit status 1.

I cannot unmount or remove the dataset using either: zfs destroy Hot_NVMe_Pool/Active_Data_Share or sudo umount -f /mnt/Hot_NVMe_Pool/Active_Data_Share. Below is the output of each:

root@med-stor-01[/var/log]# zfs destroy Hot_NVMe_Pool/Active_Data_Share
cannot unmount '/mnt/Hot_NVMe_Pool/Active_Data_Share': unmount failed
root@med-stor-01[/var/log]# sudo umount -f /mnt/Hot_NVMe_Pool/Active_Data_Share
umount: unmount of /mnt/Hot_NVMe_Pool/Active_Data_Share failed: Device busy

I also cannot destroy the pool with: zpool destroy Hot_NVMe_Pool, I get the error:

root@med-stor-01[/var/log]# zpool destroy Hot_NVMe_Pool
cannot unmount '/mnt/Hot_NVMe_Pool/Active_Data_Share': unmount failed

Below is the zfs list for the pool:

NAME USED AVAIL REFER MOUNTPOINT
Hot_NVMe_Pool 50.6M 2.15T 153K /mnt/Hot_NVMe_Pool
Hot_NVMe_Pool/Active_Data_Share 192K 2.15T 192K /mnt/Hot_NVMe_Pool/Active_Data_Share

This was in the message log at the same time:

Show : Message log

Apr 22 15:51:10 med-stor-01 GEOM_ELI: Device mirror/swap4.eli destroyed.
Apr 22 15:51:11 med-stor-01 GEOM_MIRROR: Device swap4: provider destroyed.
Apr 22 15:51:11 med-stor-01 GEOM_MIRROR: Device swap4 destroyed.
Apr 22 15:51:11 med-stor-01 GEOM_ELI: Device mirror/swap3.eli destroyed.
Apr 22 15:51:11 med-stor-01 GEOM_MIRROR: Device swap3: provider destroyed.
Apr 22 15:51:11 med-stor-01 GEOM_MIRROR: Device swap3 destroyed.
Apr 22 15:51:11 med-stor-01 GEOM_ELI: Device mirror/swap2.eli destroyed.
Apr 22 15:51:12 med-stor-01 GEOM_MIRROR: Device swap2: provider destroyed.
Apr 22 15:51:12 med-stor-01 GEOM_MIRROR: Device swap2 destroyed.
Apr 22 15:51:12 med-stor-01 GEOM_ELI: Device mirror/swap1.eli destroyed.
Apr 22 15:51:12 med-stor-01 GEOM_MIRROR: Device swap1: provider destroyed.
Apr 22 15:51:12 med-stor-01 GEOM_MIRROR: Device swap1 destroyed.
Apr 22 15:51:12 med-stor-01 GEOM_ELI: Device mirror/swap0.eli destroyed.
Apr 22 15:51:12 med-stor-01 GEOM_MIRROR: Device swap0: provider destroyed.
Apr 22 15:51:12 med-stor-01 GEOM_MIRROR: Device swap0 destroyed.

There is nothing in lsof when I grep for the pool name either.

I'm a little bit stumped, I thought it might have just been the NVMe pool but I also cannot delete datasets from the main HDD pool. I have also tried turning all services off and deleting, still no effect. Lastly, I updated from 12.0-u1 to 12.0-u3 and still no effect.

Interestingly as well, my Zabbix SNMP monitor flags a swap error when I try to delete the Hot_NVMe_Pool, could be relevant?

Any help would be appreciated!

Many thanks,
Emile

 

[Emile.Belcourt]

Thought I'd report back that the case is closed.

I forgot about Jails! I had the two datasets I was trying to delete mounted as mount points!

Lesson learned :)