TrueNAS CORE webUI non-responsive, /usr/local/sbin/hactl error on SSH

[brettw.10]

The webUI of my TrueNAS CORE 12.0-U7 system running on an Intel NUC DC32171YE with 16GB RAM and 128GB SSD keeps becoming unresponsive after a period of time after a reboot. Initially, the webUI is responsive after a reboot and I can SSH into the box without issue. After a period of time (undetermined), the webUI becomes unresponsive, and SSHing into the box gives me the dreaded /usr/local/sbin/hactl error:

Traceback (most recent call last):
File "/usr/local/sbin/hactl", line 171, in <module>
main(args.command, args.q)
File "/usr/local/sbin/hactl", line 17, in main
client = Client()
File "/usr/local/lib/python3.9/site-packages/middlewared/client/client.py", line 283, in __init__
self._ws.connect()
File "/usr/local/lib/python3.9/site-packages/middlewared/client/client.py", line 124, in connect
rv = super(WSClient, self).connect()
File "/usr/local/lib/python3.9/site-packages/ws4py/client/__init__.py", line 223, in connect
bytes = self.sock.recv(128)
socket.timeout: timed out

I have tried restarting middlewared, but it stalls waiting for a process to terminate/respond:

root@thevault:~ # service middlewared restart
Stopping middlewared.
Waiting for PIDS: 305

I tried reinstalling (upgrade installation), but this hasn't helped either.
Running dmesg -a, I also see a lot of these errors:

Dec 23 12:14:07 thevault 1 2021-12-23T12:14:07.417422+11:00 thevault.netzwerg.solutions collectd 1554 - - Traceback (most recent call last):
File "/usr/local/lib/collectd_pyplugins/disktemp.py", line 62, in read
with Client() as c:
File "/usr/local/lib/python3.9/site-packages/middlewared/client/client.py", line 283, in __init__
self._ws.connect()
File "/usr/local/lib/python3.9/site-packages/middlewared/client/client.py", line 124, in connect
rv = super(WSClient, self).connect()
File "/usr/local/lib/python3.9/site-packages/ws4py/client/__init__.py", line 223, in connect
bytes = self.sock.recv(128)
socket.timeout: timed out

As well as a bunch of these:

sonewconn: pcb 0xfffff8007a80fb70: Listen queue overflow: 193 already in queue awaiting acceptance (10 occurrences)
sonewconn: pcb 0xfffff8007a80fb70: Listen queue overflow: 193 already in queue awaiting acceptance (7 occurrences)
sonewconn: pcb 0xfffff8007a80fb70: Listen queue overflow: 193 already in queue awaiting acceptance (7 occurrences)

Any assistance that y'all can provide would be greatly appreciated.

 

[marahin]

Have you managed to find a solution?

 

[quada]

also getting these same errors after upgrading from anglefish to bluefin. UI stops after about 30 minutes and also brings down the SMB shares. Can only connect through ssh to get these errors.