TrueNAS 13.0 - Cannot disconnect OFFLINE pool: Error: 'dict' object has no attribute 'endswith'

[Force Majeure]

I need some of your collective wisdom...
I have an archive pool that went bad. Degraded. Stuck at 47% resilvering for 5 days. Many faulted drives. I have a backup of the data, so I want to scrap this pool and start from scratch.
Problem is that the pool would not disconnect. TrueNAS would say it was busy.

So I pulled all the drives from the shelf and did a hard reset on the server. Now the pool shows as OFFLINE rather than DEGRADED.

But it still won't disconnect when told to do so from the GUI.

Here is the error readout:

Error: Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/middlewared/job.py", line 355, in run
await self.future
File "/usr/local/lib/python3.9/site-packages/middlewared/job.py", line 391, in __run_body
rv = await self.method(*([self] + args))
File "/usr/local/lib/python3.9/site-packages/middlewared/schema.py", line 975, in nf
return await f(*args, **kwargs)
File "/usr/local/lib/python3.9/site-packages/middlewared/plugins/pool.py", line 1598, in export
disks = [i async for i in await self.middleware.call('pool.get_disks', oid)]
File "/usr/local/lib/python3.9/site-packages/middlewared/plugins/pool.py", line 1598, in <listcomp>
disks = [i async for i in await self.middleware.call('pool.get_disks', oid)]
File "/usr/local/lib/python3.9/site-packages/middlewared/plugins/pool.py", line 1047, in get_disks
if disk_name := await self.middleware.call('disk.label_to_disk', prov, False, cache):
File "/usr/local/lib/python3.9/site-packages/middlewared/main.py", line 1279, in call
return await self._call(
File "/usr/local/lib/python3.9/site-packages/middlewared/main.py", line 1247, in _call
return await self.run_in_executor(prepared_call.executor, methodobj, *prepared_call.args)
File "/usr/local/lib/python3.9/site-packages/middlewared/main.py", line 1152, in run_in_executor
return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs))
File "/usr/local/lib/python3.9/concurrent/futures/thread.py", line 58, in run
result = self.fn(*self.args, **self.kwargs)
File "/usr/local/lib/python3.9/site-packages/middlewared/plugins/disk_/disk_info_freebsd.py", line 123, in label_to_disk
dev = self.label_to_dev(label, geom_scan, cache) or label
File "/usr/local/lib/python3.9/site-packages/middlewared/plugins/disk_/disk_info_freebsd.py", line 104, in label_to_dev
if label.endswith('.nop'):
AttributeError: 'dict' object has no attribute 'endswith'

Zpool clear does nothing. Says the pool is not there to clear (which is true).

I'm out of ideas at this point. Do I need to re-install TrueNAS? Because I don't see another way out of this. Or is there a CLI command that can nuke a misbehaving pool like this?

Thanks

 

[Force Majeure]

BTW: this is a SuperMicro server. The archives is on a NetApp DS4246 shelf connected via a LSI SAS9200-8e card. In case that's relevant.

 

[Samuel Tai]

Just a wild-ass guess here. Have you tried nuking the zpool cache /data/zfs/zpool.cache?

 

[Force Majeure]

I have not. Are you suggesting that I should? And if so, how would I go about doing that? And is that not dangerous for my other pools?

 

[Samuel Tai]

You didn't mention any other pools. If that's the case, your safest option is to reinstall.

 

[Force Majeure]

I feared so. Thanks for the advice anyways.

 

[Force Majeure]

I saved a config file. I did a reinstall. Then I loaded my config file... And the bug is still there. It must be something in the configuration, because it carried over in the config file.

 

[winnielinnie]

In the command-line, does the zpool command reveal anything?

Code:

zpool status -v

 

[Force Majeure]

Ok. So I rolled back to a previous backup config file from April 1st and that solved the issues.

 

[Force Majeure]

UPDATE: months later, when I decide to pull a different pool from the same server, the bug returns. The pool is shown as "OFFLINE" in the TrueNAS Gui, but I can't disconnect it. I get the same error message.
When I do "zpool status -v" or "zool list", the faulty pool is not present.
What's up with this bug? Is it just a me problem, or is anybody else having this issue?

 

[Chopper_Rob]

I have the same issue disconnection an offline pool, the pool had not more shares and was running with legacy encryption.

 

[Chopper_Rob]

I was still runninng 13.0U4, upgrading to 13.0U5.3 fixed the issue. After upgrading i was able to disconnect the pool.