Upgrade to 13.0-U3.1 - Lost GUI after starting jail

[Paul Crook]

OS: TrueNAS-13.0-U3.1 (3153285d7d)
Motherboard: SuperMicro X10SRH-CLN4F-O
CPU: Intel Xeon E5-2650L v4
RAM: 128 GB ECC
Boot: SuperMicro SSD-DM032 SATA DOM
HBA: Embedded SAS3008
Storage: WD Gold RAIDZ2

I just updated from 11.2 to 11.3 to 12.0 to 13.0-U3.1 over the past two days.
Today I created my first jail and started it. The GUI froze at "Starting job ...".
I couldn't reach the GUI from another browser tab.
The system continued to serve files on the network with no issues.

/var/log/messages shows the following error exactly every 5 minutes to the second.

Code:

Nov 17 17:09:34 freenas 1 2022-11-17T17:09:34.245613-07:00 freenas.local collectd 3469 - - Traceback (most recent call last):
  File "/usr/local/lib/collectd_pyplugins/disktemp.py", line 62, in read
    with Client() as c:
  File "/usr/local/lib/python3.9/site-packages/middlewared/client/client.py", line 286, in __init__
    self._ws.connect()
  File "/usr/local/lib/python3.9/site-packages/middlewared/client/client.py", line 124, in connect
    rv = super(WSClient, self).connect()
  File "/usr/local/lib/python3.9/site-packages/ws4py/client/__init__.py", line 223, in connect
    bytes = self.sock.recv(128)
socket.timeout: timed out

I found this in /var/log/middleware.log:

Code:

[2022/11/17 16:02:07] (ERROR) iocage.callback():67 - Please convert back to a jail before trying to start primary_services
[2022/11/17 16:06:07] (WARNING) middlewared.process():47 - EventSource 'reporting.realtime' run() failed
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/middlewared/event.py", line 45, in process
    self.run()
  File "/usr/local/lib/python3.9/site-packages/middlewared/plugins/reporting/events.py", line 166, in run
    last_interface_speeds.update({'time': time.monotonic(), 'speeds': self.get_interface_speeds()})
  File "/usr/local/lib/python3.9/site-packages/middlewared/plugins/reporting/events.py", line 74, in get_interface_speeds
    interfaces = self.middleware.call_sync('interface.query')
  File "/usr/local/lib/python3.9/site-packages/middlewared/main.py", line 1302, in call_sync
    return self.run_coroutine(self.run_in_executor(prepared_call.executor, methodobj, *prepared_call.args))
  File "/usr/local/lib/python3.9/site-packages/middlewared/main.py", line 1333, in run_coroutine
    raise RuntimeError('Middleware is terminating')
RuntimeError: Middleware is terminating

I don't know what it means to convert back to a jail before trying to start primary_services(the name of my jail). I only created it and tried to start it.

This is similar to the issue described in the resource Python 3.9 crashes - A TrueCommand perspective which suggests service middlewared restartbut that hangs forever Waiting for PIDS: 2039.

It is also similar to NAS-109709 - many users are reporting python38 core dumping, but that supposedly only affected 12.0 versions and is closed.

I have not yet forced a middlewared restart nor rebooted in case more data needs to be collected while the system's in this state. There are no core files in /var/db/system/cores.

What should I do next?

 

[Paul Crook]

Bump

 

[Samuel Tai]

By any chance did you give the same IP to both the jail and your TrueNAS server?

 

[Paul Crook]

By any chance did you give the same IP to both the jail and your TrueNAS server?

Thanks for responding.

I chose DHCP rather than a hard address, but I hadn’t configured the DHCP server to give the jail an IP address yet. I didn’t know what the jail’s MAC address was going to be. I thought I‘d wait until I could see the MAC in the DHCP logs and then add the jail’s MAC-based configuration. I was surprised to see the whole TrueNAS GUI go down Instead.

I had set the jail to auto-start and thought if I rebooted the NAS, the GUI would go down when the jail started up, but it didn’t. So I deleted the jail and am back at square 1. I had used the advanced jail configuration and maybe something I chose there triggered this problem, but I did not record my choices.

I’d still like to use a jail to run a few services, but I could use a VM instead if this problem persists. I think the more important issue is that the GUI went down so easily.