Tried TrueNAS 12 and rolled back to 11.3

[alexr]

I've been trying [unsuccessfully] to get dotnet working in a jail and there was a possibly relevant VM change in May that got picked up by TN 12, so I figured I'd try upgrading.

TL;DR: I rolled back to 11.3 for several reasons and that change didn't fix dotnet.

iX-built dual socket Xeon Gold, 24 cores, 192GB. The upgrade went smoothly. I started up my one bhyve VM (Ubuntu 20.04 to use dotnet and VS Code) fine. Then I proceeded to start up my ten jails one at a time. The first few were fine, but then they started to take progressively longer to start up, taking up to 10m each. Obvious CPU hogs were SNMP and MySQL in librenms, but there were plenty of idle cores left.

About 5m after getting it all started, I got a notification that my IMAP and SMTP servers had stopped responding again. The logs had lots of messages from dovecot about how locks were taking 100s, etc.

Eventually the web UI stopped responding and "sudo reboot" hung, so I had to pull the power and revert to 11.3.

 

[sretalla]

Obvious CPU hogs were SNMP

This is a problem for many of us... if you can live without SNMP, disable the service and the server will run well.

 

[alexr]

This is a problem for many of us... if you can live without SNMP, disable the service and the server will run well.

How can a single runaway process that isn’t spawning lots of threads seize up an entire machine? I was running htop and the machine and a load of 3.

 

[sretalla]

How can a single runaway process that isn’t spawning lots of threads seize up an entire machine? I was running htop and the machine and a load of 3.

It seems that it blocks the middleware from doing its work, so things like jails and VMs continue to run with little impact even though the GUI and some other system features are blocked.

 

[alexr]

It seems that it blocks the middleware from doing its work, so things like jails and VMs continue to run with little impact even though the GUI and some other system features are blocked.

I'm not seeing how middleware locking up would slow other processes badly enough to stall things in a jail or seize up a reboot.

 

[sretalla]

I'm not seeing how middleware locking up would slow other processes badly enough to stall things in a jail or seize up a reboot.

I don't pretend to deeply understand the root cause of it either, but I can confirm what I saw, which is that with the Python3.8 process from SNMP no longer running, the system is fine and has been up for 3 days now without needing a reboot after initially needing a reboot in less than 24 hours with it running.

 

[Jared Potter]

I guess this is the issue I'm having as the UI and SSH and console via impi all become unresponsive after less than 24 hours on 12-RELEASE. If I catch it while it's just slow I can send a restart otherwise the only solution is to hit the reset on the server. The problem is that snmp is turned off in service and I've never used it... Or at least in any active way... How do I turn it off? Otherwise it is "off" and my issue is similar but not the same. I have two servers doing the same exact thing with different hardware configurations.

 

[sretalla]

I have seen another issue on my VM system where the middleware daemon freezes, which stops the GUI from presenting the login screen. I have been able to connect via SSH though and could restart it (service middlewared restart) and all came back to normal with the GUI