VERY HIGH CPU after upgrade from FreeNAS 11.3

[prophoto]

My machine ran for 4 days fine, then I had to shut down to install some hardware. Been fine another 3 days. Very odd.

 

[mrMuppet]

I have this same sysctl -a issue as well. Quickly after boot of a TrueNAS-12.0-RELEASE system, the system gets bogged down with what appear to be tuck sysctl -a processes. I do not have SNMP turned on. This appears to be gathering stats for reporting, but the calls are either backing up or hanging in some fashion.

Code:

/bin/sh -c /sbin/sysctl -a 2>/dev/null | grep -E "hw.realmem|hw.physmem|vm.stats.vm.v_page_count|vm.stats.vm.v_wire_count|vm.stats.vm.v_active_count|vm.stats.vm.v_inactive_count|vm.stats.vm.v_cache_count|vm.stats.vm.v_free_count|vm.stats.vm.v_page_size"

Can get the system back into shape with a

Code:

killall -9 sysctl

I tried the killall and yes, there was no 100% sysctl task this morning!
But! my iocage were not responding either! Seems to be another problem. Still had to reboot the system to get everything responding again.
On the screenshot you see my System this morning before restart.

 

[mrMuppet]

Uuh. And as I See now: it's swaping! That shouldn't be.. M

 

[AirborneTrooper]

I’m still surprised no official update has been put out yet. TrueNAS was released weeks ago and they are aware of these issues

 

[mrMuppet]

I saw my drive starting to swap again some minutes ago. And all the problems I have seem to start always at 23:00 to 02:00... The only Services that start in the night on my system are some plex cleaning up tasks. So I stopped plex for this night. In this second the swapping was gone. Let's See if it's gone till tomorrow morning.

BTW my sysctl-Tasks are no longer listet as having high processors loads. I tried it with "killall -9 sysctl" and sysctl was no longer listet in "top" but ui and jails were still not responding in the morning.
Then I tried setting up a new bootdrive (last time I updated 11.3 to 12) and imported my settings. After ging this way the sysctl didn't appear in the night (even without killall) but System was still not responding in the morning.

Edit: my perception that the drive Was already swpping this night was wrong. Perhaps it starts later in the night. I hope not.

 

[scoger]

I have this same sysctl -a issue as well. Quickly after boot of a TrueNAS-12.0-RELEASE system, the system gets bogged down with what appear to be tuck sysctl -a processes. I do not have SNMP turned on. This appears to be gathering stats for reporting, but the calls are either backing up or hanging in some fashion.

Code:

/bin/sh -c /sbin/sysctl -a 2>/dev/null | grep -E "hw.realmem|hw.physmem|vm.stats.vm.v_page_count|vm.stats.vm.v_wire_count|vm.stats.vm.v_active_count|vm.stats.vm.v_inactive_count|vm.stats.vm.v_cache_count|vm.stats.vm.v_free_count|vm.stats.vm.v_page_size"

Can get the system back into shape with a

Code:

killall -9 sysctl

OK, for those following along at home. Two things have gotten my TrueNAS back into shape.

1. Code:

   sysctl -a

   this thing was actually a process getting spawned from a jail running homebridge-config-ui-x. That has been fixed in the test version, details here https://github.com/oznu/homebridge-config-ui-x/issues/956
2. On the recommendation of much reading this weekend, I turned off AFP and made SMB shares for my TimeMachine backups, which works WAY better.

The system is no longer swamped by a jail trying to monitor CPU/Memory and my TimeMachine backups work way better.

Special shout out to htop tree view (becasue I forgot about it) to help me track this down and oznu@githir for the quick turn around

 

[AirborneTrooper]

OK, for those following along at home. Two things have gotten my TrueNAS back into shape.

1. Code:

   sysctl -a

   this thing was actually a process getting spawned from a jail running homebridge-config-ui-x. That has been fixed in the test version, details here https://github.com/oznu/homebridge-config-ui-x/issues/956

I've had Homebridge turned off for weeks now. Glad to see that there was a fix. I just enabled it and upgraded.

 

[AirborneTrooper]

I am still being middlewared even on the latest version of the homebridge ui x with the configset.

 

[scoger]

Have you tried narrowing this down with htop in tree view to see what's taking all the CPU time? Also, what other TrueNAS services do you have turned on?

 

[aakelley]

I think I have the Homebridge related issue as well (high cpu usage that stopped when I stopped that Jail). I wasn't clear on when the fix would be in the main release channel?

 

[scoger]

I think I have the Homebridge related issue as well (high cpu usage that stopped when I stopped that Jail). I wasn't clear on when the fix would be in the main release channel?

Looks like the config to turn off monitoring was added to release in 4.33 if I'm reading github correctly. https://github.com/oznu/homebridge-config-ui-x/commit/ab5c399ba73fb2624f3f71c3c9f6e399f5742d74

Don't forget to add the config option to turn off monitoring per the docs.

 

[AirborneTrooper]

I think I have the Homebridge related issue as well (high cpu usage that stopped when I stopped that Jail). I wasn't clear on when the fix would be in the main release channel?

Make sure you're using at least 4.33 of the UI-X. In your config file, make sure disableServerMetricsMonitoring is added and set to true. Once it's saved do a restart. Here's an example of my portion for the config

Code:

    "platforms": [
        {
            "name": "Config",
            "port": 8080,
            "auth": "form",
            "theme": "auto",
            "restart": "pm2 start homebridge -- -D -I",
            "tempUnits": "f",
            "lang": "auto",
            "disableServerMetricsMonitoring": "true",
            "log": {
                "method": "custom",
                "path": "/var/log/homebridge.log",
                "command": "tail -f /root/.pm2/logs/homebridge-out.log /root/.pm2/logs/homebridge-error.log"
            },
            "platform": "config"

 

[AirborneTrooper]

Mine is still causing issues so I've elected to destroy the jail and I'll be moving it to a raspberry pi

 

[sretalla]

Mine is still causing issues so I've elected to destroy the jail and I'll be moving it to a raspberry pi

After U1 comes out, you may be able to try it again, but in general it seems there may be some issues with the way it's doing those system calls for it to work in the long term.

 

[aakelley]

Yeah, mine too. Updated UI-X to 4.35, added parameter to config, restarted Homebridge and CPU still gets slammed.

 

[AirborneTrooper]

After U1 comes out, you may be able to try it again, but in general it seems there may be some issues with the way it's doing those system calls for it to work in the long term.

Not sure when the planned release date is but I have a Pi only running Pi-hole so plenty enough room and processing to also run HomeBridge. A month and nothing but issues so it was best to remove it from my TrueNAS box. Just don't have the time to deal with rebooting and what not when it's primary use is for Plex.

 

[sretalla]

Not sure when the planned release date is but I have a Pi only running Pi-hole so plenty enough room and processing to also run HomeBridge. A month and nothing but issues so it was best to remove it from my TrueNAS box. Just don't have the time to deal with rebooting and what not when it's primary use is for Plex.

100% fair assessment.

Release date should be in a week or so.

 

[aakelley]

I just upgraded to 12U1 and can confirm that I am able to run the home bridge jail without slamming my CPU. All is right with the world again ;-)