Hello,
Today I noticed sluggish response times from a CLI ssh session, and found the load was consistently above 14 (using top and uptime). There were multiple python3.6 processes that seemed to be causing most of the CPU load. After a reboot the system load dropped, but was still running way to high (4-8). Not sure how long this has been going on, but the sluggish response times from the CLI ssh session was first noticed today, and the ssh sessions are usually open for weeks, with pretty consistent usage; if not daily, at least every other day. Below is a current top output, and attached is a snapshot of the system hardware from the GUI. Reviewed a few threads about high load, but most were complaints about loads averages below 1, and it was not clear what if anything cleared the issue. Note there is one zfs pool that is at 90% utilization, but its been above 85% for over a year now, and the CPU loads were not an issue until recently. I've been thinking of moving some of the data in that pool for a while now to drop the utilization, and will begin the migration later this week. Also thinking about upgrading to 11.2, but before doing so wanted to see if there were any suggestions on steps to try and figure out what is going on before upgrading.
root@titanium:/mnt/media_NAS/movies # zpool list
NAME SIZE ALLOC FREE EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
freenas-boot 28.8G 901M 27.9G - - 3% 1.00x ONLINE -
media_NAS 43.5T 39.5T 3.95T - 34% 90% 1.00x ONLINE /mnt
media_NAS3 32.5T 18.3T 14.2T - 1% 56% 1.00x ONLINE /mnt
Top snap:
last pid: 7279; load averages: 5.30, 6.10, 6.08 up 0+00:52:22 12:59:00
49 processes: 1 running, 48 sleeping
CPU: 1.6% user, 0.0% nice, 49.6% system, 0.6% interrupt, 48.2% idle
Mem: 621M Active, 113M Inact, 5000M Wired, 26G Free
ARC: 1571M Total, 1145M MFU, 389M MRU, 1276K Anon, 10M Header, 26M Other
1355M Compressed, 4469M Uncompressed, 3.30:1 Ratio
Swap: 6144M Total, 6144M Free
PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND
220 root 21 21 0 196M 137M kqread 6 1:03 9.42% python3.6
3100 root 1 24 0 106M 87140K select 6 0:38 5.10% python3.6
3242 root 12 20 0 97820K 13888K nanslp 6 0:24 3.35% collectd
3133 root 16 46 0 40532K 21884K uwait 1 0:12 0.49% consul
6348 root 1 20 0 7948K 3572K CPU0 0 0:02 0.23% top
1789 root 2 20 0 18968K 6600K kqread 0 0:00 0.05% syslog-ng
3054 nobody 1 20 0 7144K 3140K select 6 0:01 0.03% mdnsd
4796 root 1 20 0 13216K 7368K select 1 0:00 0.01% sshd
2198 root 1 20 0 12512K 12620K select 3 0:00 0.01% ntpd
4112 root 1 20 0 6496K 2532K nanslp 4 0:00 0.01% cron
2388 root 1 20 0 37092K 13204K select 3 0:00 0.01% nmbd
3111 root 1 20 0 152M 117M kqread 4 0:24 0.00% uwsgi
2403 root 1 20 0 128M 97988K select 6 0:00 0.00% smbd
4343 root 16 20 0 220M 142M umtxn 7 0:15 0.00% uwsgi
4316 root 1 52 0 79644K 67368K ttyin 7 0:09 0.00% python3.6
2392 root 1 20 0 171M 138M select 6 0:02 0.00% smbd
1917 root 1 -52 r0 3520K 3584K nanslp 1 0:01 0.00% watchdogd
3126 root 15 20 0 45784K 11748K uwait 6 0:01 0.00% consul-alerts
3047 www 1 20 0 12920K 8040K kqread 2 0:00 0.00% nginx
7212 root 1 26 0 174M 140M select 7 0:00 0.00% smbd
3461 root 9 20 0 31724K 12688K uwait 1 0:00 0.00% consul
Thanks,
Today I noticed sluggish response times from a CLI ssh session, and found the load was consistently above 14 (using top and uptime). There were multiple python3.6 processes that seemed to be causing most of the CPU load. After a reboot the system load dropped, but was still running way to high (4-8). Not sure how long this has been going on, but the sluggish response times from the CLI ssh session was first noticed today, and the ssh sessions are usually open for weeks, with pretty consistent usage; if not daily, at least every other day. Below is a current top output, and attached is a snapshot of the system hardware from the GUI. Reviewed a few threads about high load, but most were complaints about loads averages below 1, and it was not clear what if anything cleared the issue. Note there is one zfs pool that is at 90% utilization, but its been above 85% for over a year now, and the CPU loads were not an issue until recently. I've been thinking of moving some of the data in that pool for a while now to drop the utilization, and will begin the migration later this week. Also thinking about upgrading to 11.2, but before doing so wanted to see if there were any suggestions on steps to try and figure out what is going on before upgrading.
root@titanium:/mnt/media_NAS/movies # zpool list
NAME SIZE ALLOC FREE EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
freenas-boot 28.8G 901M 27.9G - - 3% 1.00x ONLINE -
media_NAS 43.5T 39.5T 3.95T - 34% 90% 1.00x ONLINE /mnt
media_NAS3 32.5T 18.3T 14.2T - 1% 56% 1.00x ONLINE /mnt
Top snap:
last pid: 7279; load averages: 5.30, 6.10, 6.08 up 0+00:52:22 12:59:00
49 processes: 1 running, 48 sleeping
CPU: 1.6% user, 0.0% nice, 49.6% system, 0.6% interrupt, 48.2% idle
Mem: 621M Active, 113M Inact, 5000M Wired, 26G Free
ARC: 1571M Total, 1145M MFU, 389M MRU, 1276K Anon, 10M Header, 26M Other
1355M Compressed, 4469M Uncompressed, 3.30:1 Ratio
Swap: 6144M Total, 6144M Free
PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND
220 root 21 21 0 196M 137M kqread 6 1:03 9.42% python3.6
3100 root 1 24 0 106M 87140K select 6 0:38 5.10% python3.6
3242 root 12 20 0 97820K 13888K nanslp 6 0:24 3.35% collectd
3133 root 16 46 0 40532K 21884K uwait 1 0:12 0.49% consul
6348 root 1 20 0 7948K 3572K CPU0 0 0:02 0.23% top
1789 root 2 20 0 18968K 6600K kqread 0 0:00 0.05% syslog-ng
3054 nobody 1 20 0 7144K 3140K select 6 0:01 0.03% mdnsd
4796 root 1 20 0 13216K 7368K select 1 0:00 0.01% sshd
2198 root 1 20 0 12512K 12620K select 3 0:00 0.01% ntpd
4112 root 1 20 0 6496K 2532K nanslp 4 0:00 0.01% cron
2388 root 1 20 0 37092K 13204K select 3 0:00 0.01% nmbd
3111 root 1 20 0 152M 117M kqread 4 0:24 0.00% uwsgi
2403 root 1 20 0 128M 97988K select 6 0:00 0.00% smbd
4343 root 16 20 0 220M 142M umtxn 7 0:15 0.00% uwsgi
4316 root 1 52 0 79644K 67368K ttyin 7 0:09 0.00% python3.6
2392 root 1 20 0 171M 138M select 6 0:02 0.00% smbd
1917 root 1 -52 r0 3520K 3584K nanslp 1 0:01 0.00% watchdogd
3126 root 15 20 0 45784K 11748K uwait 6 0:01 0.00% consul-alerts
3047 www 1 20 0 12920K 8040K kqread 2 0:00 0.00% nginx
7212 root 1 26 0 174M 140M select 7 0:00 0.00% smbd
3461 root 9 20 0 31724K 12688K uwait 1 0:00 0.00% consul
Thanks,
