High CPU, high load avg

[VulcanRidr]

I have an issue with high load and the kernel process spiking between 400-900% CPU, load average is between 55 and 200, and I rebooted it, and it spiked as soon as it came up. My hardware is Intel(R) Xeon(R) CPU E5-2620 v4 @ 2.10GHz, 64GB RAM, 60 drive pool with 6 10 drive RAID-Z2 VDEVS arranged in a RAID-Z2. I am running 12.0u8.

I got a complaint from a data science team saying that their queries against the NFS mount from one of the datasets were extremely slow. I got a request to reboot from mgmt, and after reboot (system came up at 17:01 and as I started digging into it, I have seen the following python errors in middlewared.log:

Code:

[2022/03/21 17:09:17] (WARNING) middlewared._loop_monitor_thread():1490 - Task seems blocked:
  File "/usr/local/lib/python3.9/site-packages/middlewared/main.py", line 1213, in _call return await methodobj(*prepared_call.args)
  File "/usr/local/lib/python3.9/site-packages/middlewared/schema.py", line 975, in nf return await f(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/middlewared/plugins/datastore/read.py", line 92, in query aliases = self._get_queryset_joins(table)
  File "/usr/local/lib/python3.9/site-packages/middlewared/plugins/datastore/read.py", line 200, in _get_queryset_joins result.update(self._get_queryset_joins(alias))
  File "/usr/local/lib/python3.9/site-packages/middlewared/plugins/datastore/read.py", line 190, in _get_queryset_joins for column in table.c:
  File "/usr/local/lib/python3.9/site-packages/sqlalchemy/util/langhelpers.py",line 883, in __get__obj.__dict__[self.__name__] = result = self.fget(obj)
  File "/usr/local/lib/python3.9/site-packages/sqlalchemy/sql/selectable.py", line 631, in columns self._populate_column_collection()
  File "/usr/local/lib/python3.9/site-packages/sqlalchemy/sql/selectable.py", line 1354, in _populate_column_collection col._make_proxy(self)
  File "/usr/local/lib/python3.9/site-packages/sqlalchemy/sql/schema.py", line 1608, in _make_proxy c = self._constructor(
  File "/usr/local/lib/python3.9/site-packages/middlewared/sqlalchemy.py", line29, in __init__super().__init__(*args, **kwargs)

Has anyone seen behavior out of their NAS similar to this? Is it actually related to middlewared and the loop monitor thread? What other information can I provide?

Thanks,
--vr