Register for the iXsystems Community to get an ad-free experience and exclusive discounts in our eBay Store.

Python 3.9 crashes - A TrueCommand perspective

Western Digital Drives - The Preferred Drives of FreeNAS and TrueNAS CORE

Basil Hendroff

Neophyte Sage
Joined
Jan 4, 2014
Messages
1,491
This is not a TC issue, but it does highlight the value of TC in identifying patterns. Here's a TC dashboard view of a number of servers.

tc76.jpg


One of the systems is offline. I'm also not able to connect to the server UI.

tn04.jpg


This is symptomatic of the python crash that been reported in many guises on the forum. The way to temporarily resolve this is to SSH into the server and restart the middleware. A server restart isn't necessary, but the issue reappears every couple of days, which is annoying.
Code:
root@truenas-l2[/]# service middlewared restart
Stopping middlewared.
root@truenas-l2[/]#

Magically, the server UI is now accessible and the server is no longer offline in TC. Sure enough, the python issue is logged on the TC Alerts card for the server.

tc58.jpg


Patterns have emerged. Returning to the first image of the TC dashboard:
  1. I've only ever seen python crashes on the two servers with Intel processors and never on the servers with AMD processors.
  2. The middleware crashes more frequently on the Intel server running 12.0-U4 and less on the Intel server running 12.0-U3.1.
There is already a ticket out for the python crash issue NAS-109709. The ticket refers to an image that supposedly fixes the core dump issue, but as the Intel servers are production servers, I'm unable to test the fix and will instead await the release of 12.0-U5. I'll add this thread to the ticket though.
 

revengineer

Member
Joined
Oct 27, 2019
Messages
154
I only have a statistic of 1 server running on Intel. on U3.1 I had frequent middleware crashes every few days. On U4, issues are gone, running 18 days now without middleware crash. I am running regular U4, not the latest modded version posted by Caleb in the epic ticket. So I cannot backup your second point.
 

Basil Hendroff

Neophyte Sage
Joined
Jan 4, 2014
Messages
1,491
I only have a statistic of 1 server running on Intel. on U3.1 I had frequent middleware crashes every few days. On U4, issues are gone, running 18 days now without middleware crash. I am running regular U4, not the latest modded version posted by Caleb in the epic ticket. So I cannot backup your second point.
Thanks for the feedback. It's curious I have the reverse behaviour on point 2. What makes this more curious is that both servers are on identical h/w apart from slightly different Intel CPUs. What I might try is to switch down to a 12-U3.1 boot environment on the server currently running 12.0-U4 and see if the behaviour changes.
 

revengineer

Member
Joined
Oct 27, 2019
Messages
154
@Basil Hendroff Have you tried the modified U4 posted in the epic ticket? This supposedly fixed residual issues with these core dumps. If this does not work for you then we may received in August a U5 that is still not fully working. So testing this now would be good.
 

Basil Hendroff

Neophyte Sage
Joined
Jan 4, 2014
Messages
1,491
Have you tried the modified U4 posted in the epic ticket?
Good thought, but as I indicated in the OP, these are production servers so I not prepared to use these to test the modified U4 on them.
 

revengineer

Member
Joined
Oct 27, 2019
Messages
154
Good thought, but as I indicated in the OP, these are production servers so I not prepared to use these to test the modified U4 on them.
Got it, understood. I am puzzled why the memory leak associated with the third-party python library, which caused the core dumps, would be processor specific. Then we will keep our fingers crossed that U5 is the fix all have been waiting for.
 

Basil Hendroff

Neophyte Sage
Joined
Jan 4, 2014
Messages
1,491
Well, I rolled back down to U3.1 on the Intel server (HP Gen 8 microserver with an Intel Xeon E3-1220L V2) running U4 last Wed and I've not had a middleware crash since. It was crashing every couple of days with U4. Hoping U4.1 addresses the issue.
 

revengineer

Member
Joined
Oct 27, 2019
Messages
154
I do not see any middleware crashes addressed in the U4.1 fix list. This seems to be mainly for enclosure related issues, and they threw in the dashboard CPU widget fix. So it seems they are holding further python fixes until U5.
 

Basil Hendroff

Neophyte Sage
Joined
Jan 4, 2014
Messages
1,491
Damn it! I spoke to soon and jinxed myself. The middleware crashed on that server overnight o_O
 

Basil Hendroff

Neophyte Sage
Joined
Jan 4, 2014
Messages
1,491
Since upgrading to U4.1, my middleware crashes seem to have ceased.
 
Top