Massive System Slowdown

[jsylvia007]

EDIT2: Now at over 24 hours since a reboot (for the first time since the upgrade to TN 12.0 Core)... after killing the python3.8 process all still running well and GUI/CLI responsive.

Does this mean that even with the process disabled in the services panel, after a reboot you need to remember to kill the process because it starts on its own?

 

[sretalla]

Does this mean that even with the process disabled in the services panel, after a reboot you need to remember to kill the process because it starts on its own?

No reboot yet.
The service was stopped (and disabled on startup), then later, the process was still seen running, so killed.

I don't expect that SNMP will start again, but will report back if I eventually need to reboot and see the python3.8 process again.

 

[jsylvia007]

No reboot yet.
The service was stopped (and disabled on startup), then later, the process was still seen running, so killed.

I don't expect that SNMP will start again, but will report back if I eventually need to reboot and see the python3.8 process again.

Thanks for the clarification!

 

[Prophet4NO1]

Is there anything else that uses python3.8? I know that is one of the things that pops up in the errors. But with out knowing what is calling to run it, I am not sure killing will do much. Guess I need to dive in deeper and see what is using it.

 

[FrankNAS]

Is there anything else that uses python3.8? I know that is one of the things that pops up in the errors. But with out knowing what is calling to run it, I am not sure killing will do much. Guess I need to dive in deeper and see what is using it.

other things do use python but if you want to make sure its the SNMP one you can

Code:

ps auxw | grep snmp

and kill the PID of the SNMP python item
also disable the SNMP service

doing both has resolved the management slowdown/timeout issues I was having

 

[ornias]

Disktemp.py crashing is a known issue and I already traced the bug on the bug report issue.
It has nothing to do with python, it's a bug in the disktemp.py stack.

 

[sretalla]

kill the PID of the SNMP python item
also disable the SNMP service

doing both has resolved the management slowdown/timeout issues I was having

And for me also. All good now for more than 24 hours.

 

[AirborneTrooper]

I restarted last night and checked SNMP which wasn't toggled on at all. This morning my GUI is working again. I know my issue is the middleware and not anything with the drive temps. I just ran

Code:

 ps auxw | grep snmp 

and it's just sitting there in ssh for 5 mins so far. This is beyond frustrating.

 

[AirborneTrooper]

Had to reboot again and I only get one result right now for the command.

Code:

root@truenas[~]# ps auxw | grep snmp
root       6806    0.0  0.0   11540    2972  0  S+   08:19     0:00.00 grep snmp

 

[sretalla]

what about ps auxw | grep python3.8 ?

 

[AirborneTrooper]

Code:

root@truenas[~]# ps auxw | grep pytho3.8
root       8012    0.0  0.0    4888    2552  0  R+   09:19     0:00.00 grep pytho3.8

 

[AirborneTrooper]

GUI always runs great after reboot but I suspect later this evening or by tomorrow morning it'll slow down again. In the GUI, I often see one of my threads shoot up to 83-100% usage. I have a Dell R710 with 2 Intel(R) Xeon(R) CPU X5675 @ 3.07GH and 64GB or RAM. Never seen it spike like that on 11.3 and I share plex with friends and family and often have to transcode.

 

[sretalla]

I suspect that although your symptoms are the same, you don't have the SNMP variant of the problem... I'm not sure if anyone has seen other confirmed causes... can you see anything in top that points you to something? once it's running, press j to expose the jail IDs column to see if the heavy process is in a jail or on the host (JID 0). You can find the JID list in iocage list or jls.

 

[AirborneTrooper]

The only error I've seen is the same python3.8 middleware that you have seen. The only services I run are S.M.A.R.T., SMB, SSH, and UPS though. Next time it freezes up I'll try to run that python command

 

[jsylvia007]

The only error I've seen is the same python3.8 middleware that you have seen. The only services I run are S.M.A.R.T., SMB, SSH, and UPS though. Next time it freezes up I'll try to run that python command

Just to note, in your previous post you spelled python incorrectly, so that might be why nothing appeared. At least for me though, my processes run as just "python" not "python3.8". I would just grep for python alone.

This is my output from the grep (note that the snmp-agent process isn't running because I've already killed it):

Code:

nas01# ps -auxwwww | grep python
root        1213   0.1  2.5 1902844 1644360  -  S    21Oct20   323:45.91 python3.8: middlewared (python3.8)
root        1216   0.0  0.0   21532   11936  -  S    21Oct20     0:07.32 /usr/local/bin/python3.8 -c from multiprocessing.resource_tracker import main;main(11)
root        1258   0.0  0.3  206632  167296  -  I    21Oct20    10:23.97 python3.8: middlewared (worker) (python3.8)
root        1259   0.0  0.3  209540  167648  -  I    21Oct20    10:26.97 python3.8: middlewared (worker) (python3.8)
root        1260   0.0  0.3  218264  168788  -  I    21Oct20    10:23.82 python3.8: middlewared (worker) (python3.8)
root        1261   0.0  0.3  224312  168900  -  I    21Oct20    10:25.18 python3.8: middlewared (worker) (python3.8)
root        1262   0.0  0.3  208844  167368  -  I    21Oct20    10:32.61 python3.8: middlewared (worker) (python3.8)
root        1620   0.0  0.1   59524   47656  -  I    21Oct20     0:00.95 python3.8: /usr/local/bin/python3.8 -c from multiprocessing.spawn import spawn_main; spawn_main(tracker_fd=12, pipe_handle=47) --multiprocessing-fork (python3.8)
daemon      2720   0.0  0.1   71864   44440  -  I    21Oct20     0:52.28 python3 /usr/local/bin/wsdd.py (python3.8)
root        3365   0.0  0.1   49244   35984 u0  Is+  21Oct20     0:00.71 python3 /etc/netcli (python3.8)
root        3357   0.0  0.1   49232   35980 v0  Is+  21Oct20     0:00.70 python3 /etc/netcli (python3.8)
root       86190   0.0  0.0   11508    2968  0  S+   11:22       0:00.00 grep python

I can also say that the GUI is certainly sluggish, but it doesn't time-out and isn't causing issues that I can see. The command line is back to being nice and snappy, no lag there at all any longer for me.

 

[sretalla]

At least for me though, my processes run as just "python"

If I used the GUI shell, I saw it was truncated, but in a separate terminal, I saw it as python3.8... at least for the one I needed to kill to get things back.

 

[AirborneTrooper]

I’ll have to run it again when I’m back at home. If I knew I’d have these headaches I wouldn’t have upgraded and waited. I upgraded my zpools tho

 

[AirborneTrooper]

Here's python3.8 but GUI is still working. I do see threads jumping up to 82% usage right now though. Will have to compare when GUI becomes unresponsive again.

Code:

root@truenas[~]# ps auxw | grep python3.8
root         462    0.0  1.8 1797808 1233156  -  S    08:39       3:49.35 python3.8: middlewared (python3.8)
root         465    0.0  0.0   21532   11936  -  I    08:39       0:00.05 /usr/local/bin/python3.8 -c from multiprocessing.resource
root         507    0.0  0.2  206768  164728  -  I    08:39       0:16.80 python3.8: middlewared (worker) (python3.8)
root         508    0.0  0.2  204608  164692  -  I    08:39       0:17.23 python3.8: middlewared (worker) (python3.8)
root         509    0.0  0.2  239216  166020  -  I    08:39       0:17.72 python3.8: middlewared (worker) (python3.8)
root         510    0.0  0.2  210020  163468  -  I    08:39       0:16.80 python3.8: middlewared (worker) (python3.8)
root         511    0.0  0.3  210608  170780  -  I    08:39       0:18.67 python3.8: middlewared (worker) (python3.8)
root         865    0.0  0.1   59524   47632  -  I    08:39       0:00.55 python3.8: /usr/local/bin/python3.8 -c from multiprocessi
daemon      1638    0.0  0.1   72336   44660  -  I    08:39       0:00.25 python3 /usr/local/bin/wsdd.py (python3.8)
root        1901    0.0  0.1   49232   35716 v0  Is+  08:39       0:00.31 python3 /etc/netcli (python3.8)
root       25407    0.0  0.0   11564    2980  0  S+   15:45       0:00.00 grep python3.8

 

[mrMuppet]

I've had the same problems in the last days. I think i fixed it for me by stopping all iocages one by one - by doing this i found out that one old couchpotato-jail did the 100% Processor load. By stopping that jail it jumped to 10 to 30 % . I hope that helps you.

 

[AirborneTrooper]

Interesting. I have all my jails on 12.1-RELEASE and the apps are updated too.