Missing dataset in GUI in Cobia beta

[liraelia]

Hello,
I have upgraded to TrueNAS-SCALE-23.10-BETA.1, did the pool upgrade and now my datasets aren't showing in GUI.

I get the following error from /var/log/middlewared.log when I click on datasets in GUI

Code:

[2023/08/18 12:14:42] (WARNING) application.call_method():249 - Exception while calling pool.dataset.details(*[])

Traceback (most recent call last):

  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 213, in call_method

    result = await self.middleware._call(message['method'], serviceobj, methodobj, params, app=self)

             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1372, in _call

    return await self.run_in_executor(prepared_call.executor, methodobj, *prepared_call.args)

           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1271, in run_in_executor

    return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs))

           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/usr/lib/python3.11/concurrent/futures/thread.py", line 58, in run

    result = self.fn(*self.args, **self.kwargs)

             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/usr/lib/python3/dist-packages/middlewared/schema/processor.py", line 181, in nf

    return func(*args, **kwargs)

           ^^^^^^^^^^^^^^^^^^^^^

  File "/usr/lib/python3/dist-packages/middlewared/schema/processor.py", line 50, in nf

    res = f(*args, **kwargs)

          ^^^^^^^^^^^^^^^^^^

  File "/usr/lib/python3/dist-packages/middlewared/plugins/pool_/dataset_details.py", line 209, in details

    self.collapse_datasets(dataset, info, mnt_info)

  File "/usr/lib/python3/dist-packages/middlewared/plugins/pool_/dataset_details.py", line 235, in collapse_datasets

    self.collapse_datasets(child, info, mnt_info)

  File "/usr/lib/python3/dist-packages/middlewared/plugins/pool_/dataset_details.py", line 235, in collapse_datasets

    self.collapse_datasets(child, info, mnt_info)

  File "/usr/lib/python3/dist-packages/middlewared/plugins/pool_/dataset_details.py", line 233, in collapse_datasets

    self.normalize_dataset(dataset, info, mnt_info)

  File "/usr/lib/python3/dist-packages/middlewared/plugins/pool_/dataset_details.py", line 229, in normalize_dataset

    dataset['rsync_tasks_count'] = self.get_rsync_tasks_count(dataset, info['rsync'])

                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/usr/lib/python3/dist-packages/middlewared/plugins/pool_/dataset_details.py", line 418, in get_rsync_tasks_count

    if pathlib.Path(ds['mountpoint']).is_relative_to(i['path']):

       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/usr/lib/python3.11/pathlib.py", line 872, in __new__

    self = cls._from_parts(args)

           ^^^^^^^^^^^^^^^^^^^^^

  File "/usr/lib/python3.11/pathlib.py", line 510, in _from_parts

    drv, root, parts = self._parse_args(args)

                       ^^^^^^^^^^^^^^^^^^^^^^

  File "/usr/lib/python3.11/pathlib.py", line 494, in _parse_args

    a = os.fspath(a)

        ^^^^^^^^^^^^

TypeError: expected str, bytes or os.PathLike object, not NoneType

and my datasets tab looks like this

Is there anything that can be done?
Thank you

 

[ABain]

@liraelia another user submitted a bug reported on this and a fix is expected in RC.1.
NAS-123560

 

[morganL]

@liraelia another user submitted a bug reported on this and a fix is expected in RC.1.
NAS-123560

@liraelia

RC.1 is expected at end of September.

If data is not critical, then the Nightly images could be used.

If anyone else sees the issue, please log it here. Its not clear what % of users get this problem and whether its related to cloudsync or rsync tasks that have been scheduled.

 

[hege]

I saw this issue as well, but didn't capture any logs. For what it's worth, I have a cloudsync task, so it's plausibly the same issue as NAS-123560.

For now I'm back on 22.12 due to broken UPS and SNMP, both issues seem to be fixed already. Any chance a BETA.2 is cut before end of September? Looks like all my blocker bugs have fixes, but I'm not sure I'd trust a nightly build with my data.

 

[liraelia]

I don't have any cloud sync tasks. I do have one rsync task, I have disabled it and the rsync service before upgrading.
I tried to delete it now and my datasets are visible in gui again.

 

[jace92]

Confirming I have the same missing Datasets issue on the Cobia Beta

 

[Saberwolf]

i can confirm just remove your cloud sync in your data protection tab and data set return reboot server.

 

[jace92]

Oh, that's fun. I can also confirm that removing your cloud sync task shows the datasets again. Now I'm re-adding it to see if that breaks it again. I hope not...

Do we know why that's the issue/fix? (Other than because it's a bug on a beta product)

 

[Saberwolf]

Oh, that's fun. I can also confirm that removing your cloud sync task shows the datasets again. Now I'm re-adding it to see if that breaks it again. I hope not...

Do we know why that's the issue/fix? (Other than because it's a bug on a beta product)

i can confirm it has been fixed for next release but they are collecting and working on ups and few other bugs it is slated to be fixed in RC.1

 

[jace92]

i can confirm it has been fixed for next release but they are collecting and working on ups and few other bugs it is slated to be fixed in RC.1

That's good, thanks for that update!

 

[jace92]

Well... re-adding the task broke it again. Darn

 

[NickF]

Did you guys make a Jira ticket yet? Please report. Sounds like an unintended interaction may have caused a bug.

TrueNAS - Issues - iXsystems TrueNAS Jira

ixsystems.atlassian.net

 

[ABain]

Did you guys make a Jira ticket yet? Please report. Sounds like an unintended interaction may have caused a bug.

TrueNAS - Issues - iXsystems TrueNAS Jira

ixsystems.atlassian.net

See above in the thread, there is already a bug and it will be available in RC.1

 

[ULTIoNT]

Sorry I didn't read other responses carefully. Indeed this is a problem with the cloud sync tasks. When they are removed, the datasets shows up again.
---
Also encountered this problem. I just moved my system dataset from boot-pool to another SSD data pool and then datasets are gone in the WebUI while everything else seems to be intact, including apps, VMs, SMB, pointing to these datasets when modifying configs in the WebUI, etc. The errors I see are the same as liraelia has pointed out:

Code:

[2023/08/26 19:56:09] (WARNING) application.call_method():249 - Exception while calling pool.dataset.details(*[])
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 213, in call_method
    result = await self.middleware._call(message['method'], serviceobj, methodobj, params, app=self)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1372, in _call
    return await self.run_in_executor(prepared_call.executor, methodobj, *prepared_call.args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1271, in run_in_executor
    return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/schema/processor.py", line 181, in nf
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/schema/processor.py", line 50, in nf
    res = f(*args, **kwargs)
          ^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/plugins/pool_/dataset_details.py", line 209, in details
    self.collapse_datasets(dataset, info, mnt_info)
  File "/usr/lib/python3/dist-packages/middlewared/plugins/pool_/dataset_details.py", line 235, in collapse_datasets
    self.collapse_datasets(child, info, mnt_info)
  File "/usr/lib/python3/dist-packages/middlewared/plugins/pool_/dataset_details.py", line 235, in collapse_datasets
    self.collapse_datasets(child, info, mnt_info)
  File "/usr/lib/python3/dist-packages/middlewared/plugins/pool_/dataset_details.py", line 233, in collapse_datasets
    self.normalize_dataset(dataset, info, mnt_info)
  File "/usr/lib/python3/dist-packages/middlewared/plugins/pool_/dataset_details.py", line 228, in normalize_dataset
    dataset['cloudsync_tasks_count'] = self.get_cloudsync_tasks_count(dataset, info['cloud'])
                                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/plugins/pool_/dataset_details.py", line 408, in get_cloudsync_tasks_count
    if pathlib.Path(ds['mountpoint']).is_relative_to(i['path']):
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/pathlib.py", line 872, in __new__
    self = cls._from_parts(args)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/pathlib.py", line 510, in _from_parts
    drv, root, parts = self._parse_args(args)
                       ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/pathlib.py", line 494, in _parse_args
    a = os.fspath(a)
        ^^^^^^^^^^^^
TypeError: expected str, bytes or os.PathLike object, not NoneType

It seems like there is some `None` mountpoints in the response to `pool.dataset`. I took a look at the response to `/pool/dataset` (I am not familiar with websocket) and it seems that there is going to be `"mountpoint": null` for zvols inside a dataset and maybe that's what confuses middleware. In my case, they are `SSD/virtual-machines/OpenWRT`and `SSD/virtual-machines/Ubuntu` themselves and wherever they are considered children.

 

[Saberwolf]

Sorry I didn't read other responses carefully. Indeed this is a problem with the cloud sync tasks. When they are removed, the datasets shows up again.
---
Also encountered this problem. I just moved my system dataset from boot-pool to another SSD data pool and then datasets are gone in the WebUI while everything else seems to be intact, including apps, VMs, SMB, pointing to these datasets when modifying configs in the WebUI, etc. The errors I see are the same as liraelia has pointed out:

Code:

[2023/08/26 19:56:09] (WARNING) application.call_method():249 - Exception while calling pool.dataset.details(*[])
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 213, in call_method
    result = await self.middleware._call(message['method'], serviceobj, methodobj, params, app=self)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1372, in _call
    return await self.run_in_executor(prepared_call.executor, methodobj, *prepared_call.args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1271, in run_in_executor
    return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/schema/processor.py", line 181, in nf
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/schema/processor.py", line 50, in nf
    res = f(*args, **kwargs)
          ^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/plugins/pool_/dataset_details.py", line 209, in details
    self.collapse_datasets(dataset, info, mnt_info)
  File "/usr/lib/python3/dist-packages/middlewared/plugins/pool_/dataset_details.py", line 235, in collapse_datasets
    self.collapse_datasets(child, info, mnt_info)
  File "/usr/lib/python3/dist-packages/middlewared/plugins/pool_/dataset_details.py", line 235, in collapse_datasets
    self.collapse_datasets(child, info, mnt_info)
  File "/usr/lib/python3/dist-packages/middlewared/plugins/pool_/dataset_details.py", line 233, in collapse_datasets
    self.normalize_dataset(dataset, info, mnt_info)
  File "/usr/lib/python3/dist-packages/middlewared/plugins/pool_/dataset_details.py", line 228, in normalize_dataset
    dataset['cloudsync_tasks_count'] = self.get_cloudsync_tasks_count(dataset, info['cloud'])
                                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/plugins/pool_/dataset_details.py", line 408, in get_cloudsync_tasks_count
    if pathlib.Path(ds['mountpoint']).is_relative_to(i['path']):
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/pathlib.py", line 872, in __new__
    self = cls._from_parts(args)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/pathlib.py", line 510, in _from_parts
    drv, root, parts = self._parse_args(args)
                       ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/pathlib.py", line 494, in _parse_args
    a = os.fspath(a)
        ^^^^^^^^^^^^
TypeError: expected str, bytes or os.PathLike object, not NoneType

It seems like there is some `None` mountpoints in the response to `pool.dataset`. I took a look at the response to `/pool/dataset` (I am not familiar with websocket) and it seems that there is going to be `"mountpoint": null` for zvols inside a dataset and maybe that's what confuses middleware. In my case, they are `SSD/virtual-machines/OpenWRT`and `SSD/virtual-machines/Ubuntu` themselves and wherever they are considered children.

Thanks for the enthusiasm but if you look at the above Jira ticket they have found the solution it has to deal with cloud replication task ease workaround is to disable them / remove until the next release.

 

[cfmedicus]

Just happened to me right now - I was shocked, but all my machines are running. in the CLI all Datasets are there. Last task before "loosing" the Datasets on the ui was to add a rsync task for one (sub)Dataset.

 

[jace92]

I updated to the RC for 23.10 yesterday and I still show my datasets after re-configuring the Cloud Sync.