Cannot create dataset, operation not permitted

[hybris]

Hello!

I have a rather annoying problem on Truenas Scale (TrueNAS-SCALE-22.12.4.2).

When I try to create a new dataset (called test123 in this example) it fails, logs shown below.

Things I have also tried:
1) I have also tried manually but maybe it is not supposed to work? (cd /mnt/hdd_pool/; mkdir test123) It also fails with operation not permitted and I am the root user.
2) Rebooted the server.
3) Export / import the pool.
4) It is possible to create a new dataset inside an existing dataset.

Do you have any suggestions?

Thank you.


[EFAULT] Failed to mount dataset: cannot mount '/mnt/hdd_pool/test123': failed to create mountpoint: Operation not permitted Error: concurrent.futures.process._RemoteTraceback: """ Traceback (most recent call last): File "/usr/lib/python3/dist-packages/middlewared/plugins/zfs.py", line 1114, in mount dataset.mount() File "libzfs.pyx", line 465, in libzfs.ZFS.__exit__ File "/usr/lib/python3/dist-packages/middlewared/plugins/zfs.py", line 1114, in mount dataset.mount() File "libzfs.pyx", line 3972, in libzfs.ZFSDataset.mount libzfs.ZFSException: cannot mount '/mnt/hdd_pool/test123': failed to create mountpoint: Operation not permitted During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/lib/python3.9/concurrent/futures/process.py", line 243, in _process_worker r = call_item.fn(*call_item.args, **call_item.kwargs) File "/usr/lib/python3/dist-packages/middlewared/worker.py", line 115, in main_worker res = MIDDLEWARE._run(*call_args) File "/usr/lib/python3/dist-packages/middlewared/worker.py", line 46, in _run return self._call(name, serviceobj, methodobj, args, job=job) File "/usr/lib/python3/dist-packages/middlewared/worker.py", line 40, in _call return methodobj(*params) File "/usr/lib/python3/dist-packages/middlewared/worker.py", line 40, in _call return methodobj(*params) File "/usr/lib/python3/dist-packages/middlewared/schema.py", line 1383, in nf return func(*args, **kwargs) File "/usr/lib/python3/dist-packages/middlewared/plugins/zfs.py", line 1117, in mount raise CallError(f'Failed to mount dataset: {e}') middlewared.service_exception.CallError: [EFAULT] Failed to mount dataset: cannot mount '/mnt/hdd_pool/test123': failed to create mountpoint: Operation not permitted """ The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/middlewared/main.py", line 184, in call_method result = await self.middleware._call(message['method'], serviceobj, methodobj, params, app=self) File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1317, in _call return await methodobj(*prepared_call.args) File "/usr/lib/python3/dist-packages/middlewared/service.py", line 940, in create rv = await self.middleware._call( File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1317, in _call return await methodobj(*prepared_call.args) File "/usr/lib/python3/dist-packages/middlewared/schema.py", line 1247, in nf res = await f(*args, **kwargs) File "/usr/lib/python3/dist-packages/middlewared/schema.py", line 1379, in nf return await func(*args, **kwargs) File "/usr/lib/python3/dist-packages/middlewared/plugins/pool.py", line 3470, in do_create await self.middleware.call('zfs.dataset.mount', data['name']) File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1368, in call return await self._call( File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1325, in _call return await self._call_worker(name, *prepared_call.args) File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1331, in _call_worker return await self.run_in_proc(main_worker, name, args, job) File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1246, in run_in_proc return await self.run_in_executor(self.__procpool, method, *args, **kwargs) File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1231, in run_in_executor return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs)) middlewared.service_exception.CallError: [EFAULT] Failed to mount dataset: cannot mount '/mnt/hdd_pool/test123': failed to create mountpoint: Operation not permitted

 

[anodos]

This sounds like for some reason your root-level dataset is readonly.

 

[hybris]

Thanks for helping me!

I have checked using the shell:

# zfs get readonly hdd_pool
NAME PROPERTY VALUE SOURCE
hdd_pool readonly off local

I have also toggled readonly on / off in the GUI but it does not fix the problem.

Was this the correct way to check?

Do you have any other ideas on what I can do?

 

[hybris]

I upgraded to Cobia 23.10 and the problem went away.