FreeNAS-11.3-U2.1 middleware blocks system

Jost Wittmann

Dabbler
Joined
Feb 23, 2014
Messages
34
Hi!
I am running FreeNAS-11.3-U2.1 on a Lenovo ThinkServer TS140, with Intel(R) Xeon(R) CPU E3-1240 V2 @ 3.40GHz and 16GB RAM EEC.

The middleware seems to cause troubles:

Code:
[2020/07/21 08:47:21] (WARNING) middlewared._loop_monitor_thread():1285 - Task seems blocked:
  File "/usr/local/bin/middlewared", line 10, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.7/site-packages/middlewared/main.py", line 1448, in main
    startup_seq_path=startup_seq_path,
  File "/usr/local/lib/python3.7/site-packages/middlewared/main.py", line 1302, in run
    self.loop.run_forever()
  File "/usr/local/lib/python3.7/asyncio/base_events.py", line 534, in run_forever
    self._run_once()
  File "/usr/local/lib/python3.7/asyncio/base_events.py", line 1771, in _run_once
    handle._run()
  File "/usr/local/lib/python3.7/asyncio/events.py", line 88, in _run
    self._context.run(self._callback, *self._args)
  File "/usr/local/lib/python3.7/site-packages/middlewared/main.py", line 1081, in _call
    return await methodobj(*args)
  File "/usr/local/lib/python3.7/site-packages/middlewared/plugins/account.py", line 91, in user_extend
    user['sshpubkey'] = f.read()



As a result, the system is mostly frozen: the GUI is inaccessible, and many processes freeze. Fortunately SSH works, and a panic reset does the trick:
Code:
sysctl debug.kdb.panic=1


Voilà: Good as new.

This happened several times during the last days. Do you have any recommendations for how to deal with this?
Thank you very much in andvance
 

Jost Wittmann

Dabbler
Joined
Feb 23, 2014
Messages
34
I am still struggeling with this issue.

The server sometimes shows the following screen:
Middleware.jpeg


If I hit any key on the keyboard, I get back to the menu where I can configure the network, and reboot a.s.o.. However, the web-GUI is still inaccessible and most of the services seem to be down.

Seems to be Python related. Seems also to be related to this Bug: https://redmine.ixsystems.com/issues/27013

Any help very much appreciated!
 

Tony-1971

Contributor
Joined
Oct 1, 2016
Messages
147
Hi,

are there reasons not to upgrade to the lastest 11.3 release?
Maybe the 11.3-U4.1 resolve the problem...

Best Regards,
Antonio
 

Jost Wittmann

Dabbler
Joined
Feb 23, 2014
Messages
34
Forgot to mention that I switched to
Version:
TrueNAS-12.0-BETA

Would you recommend to change to the latest FreeNAS version?
 

Tony-1971

Contributor
Joined
Oct 1, 2016
Messages
147
The lastest Freenas version is 11.3-U4.1
If you want to try TrueNAS Core then install RC1

Best Regards,
Antonio
 

Jost Wittmann

Dabbler
Joined
Feb 23, 2014
Messages
34
I'm now running TrueNAS-12.0-RC1. Let's see if this fixes things...
 

Jost Wittmann

Dabbler
Joined
Feb 23, 2014
Messages
34
Nope, switching to TrueNAS-12.0-RC1 didn't fix things :-(

Server just froze again. By now I understand that it helps to hit any key on the keyoard that is connected to the server.

When I connect using ssh, I see the first few lines of the welcome screen. But after the "Welcome to freenas" - line, there is no reaction on behalf of the server. The line "Traceback..." appears only once I hit any key on the local keyboard connected to the server:
1601844129217.png


I'd be very grateful for help! While writing this post, the server actually froze a second time (so only minutes after the "I fixed it by a key-stroke"...).
 
Last edited:

Tony-1971

Contributor
Joined
Oct 1, 2016
Messages
147
The script is used for High Availability (when there are more than one node).
Code:
root@freenas-sm[~]# /usr/local/sbin/hactl status
Node status: Not an HA node

If I try to run it by hand the response is immediate so I think that in your case there is a network problem (maybe DNS).

Best Regards,
Antonio

P.S.: I don't think that HA is available in TrueNAS Core, I can't find anything about it in web interface
 

Jost Wittmann

Dabbler
Joined
Feb 23, 2014
Messages
34
Not sure if this helps, but when I choose to configure DNS, the problem actually occurs reliably:
signal-2020-12-03-212504.jpeg


That seems to confirm your guess, Tony.
 

Jost Wittmann

Dabbler
Joined
Feb 23, 2014
Messages
34
I am currently trying to avoid the problem by
  1. saved the config of TrueNAS
  2. Install Proxmox
  3. Install TrueNAS as VM
  4. reload the config
Let's see what happens....
 

StevoFNF

Cadet
Joined
May 7, 2021
Messages
8
I am currently trying to avoid the problem by
  1. saved the config of TrueNAS
  2. Install Proxmox
  3. Install TrueNAS as VM
  4. reload the config
Let's see what happens....

Would you mind updating on your situation?

I just found this thread after days of digging through my setup because I have nearly the exact same issue as you. My rig was working reliably for years and in the last few weeks randomly started losing all NICs and requiring a full reboot in order to restore connectivity. I've replaced the drives, the boot partition, updated to latest TrueNAS, deleted and allowed autotune to redo the tuneables, nothing seemed to remedy the issue.

I am now highly suspicious that the NIC is dying. I removed my PCIE NIC earlier today so I can't fully recreate your experience but when I try to do the configure DNS from the shell I do not encounter the issue as you said you could recreate reliably.

If you did resolve your problem, was it a Network card switch? Any information would be appreciated. Thanks.
 

Jost Wittmann

Dabbler
Joined
Feb 23, 2014
Messages
34
Hi Stevo,
I am sorry for all the trouble you went through. I know exactly how you feel...

Unfortunately, I do not have much to report that could be of any help:
Since I installed TrueNAS as a VM in Proxmox, the problem did not reoccur.

I shut down a lot of the services that I used to have running onn TrueNAS earlier. So this is potentially also a reason that the middleware-problem doesn't occur any longer.
All the best!
 
Top