Massive System Slowdown

Prophet4NO1

Dabbler
Joined
Sep 11, 2016
Messages
20
So, I have been having issues with my FreeNAS/TrueNAS server. I get the following errors in the log that come back after rebooting. Usually 6-12 hours later. Nothing else is in the /var/log/messages file before or after these types of entry.


Oct 22 11:10:42 freenas 1 2020-10-22T11:10:42.551320-05:00 freenas.local collectd 1887 - - Traceback (most recent call last):
File “/usr/local/lib/collectd_pyplugins/disktemp.py”, line 63, in read
temperatures = c.call(‘disk.temperatures’, self.disks, self.powermode)
File “/usr/local/lib/collectd_pyplugins/disktemp.py”, line 63, in read
temperatures = c.call(‘disk.temperatures’, self.disks, self.powermode)
File “/usr/local/lib/python3.8/site-packages/middlewared/client/client.py”, line 421, in call
raise CallTimeout(“Call timeout”)
middlewared.client.client.CallTimeout: Call timeout


When these messages start to pile up, the system grinds down to a near halt. I do not think the messages are the problem themselves, but a symptom of another issue. This also effects performace in jails as well. Plex for example can not even manage to stream video with no transcoding. There are no errors in SMART, so I do not think it is a drive issue, but I could be wrong. One thing I did notice on the dashboard is that the "services" uses about double the amount of RAM it normally does. In four years or so this server has been running, it usually stayed around 5-6GB for services. Now, once the errors happen and the system slows, usage for services doubles to nearly 12GB. There is a total of 32GB in the system. I rarely get to let the system run more than a day before having to reboot because of this, so RAM is mostly empty aside from services use.

I googled and looked around here, can not find much. To try and solve the issue I have checked firewall logs for outside communication being blocked (per one thread I found with similar errors), nothing. I changed USB flash drives, nothing. Fresh install of FreeNAS on the new drive, nothing.

At this point, I am assuming some sort of hardware issue, but I am not sure. Been messing with this for months. I am currently on the latest TrueNAS release. Just updated last night. I would love any ideas on where it look to try and smoke this out. The machine runs great after a reboot. But at some point just falls on it's face. Thanks in advance.
 

sretalla

Powered by Neutrality
Moderator
Joined
Jan 1, 2016
Messages
9,702
I am also seeing middleware timeouts on my Primary NAS (the only BareMetal instance I have).

That then puts the system into non-responsive mode on the GUI and I guess other things counting on middleware are also broken. Jails continue to work fine, but controlling them is slow/impossible (middleware again, I suppose).

TrueCommand also (incorrectly) reports the system as unavailable (but this is no suporise if middleware isn't responding).

The other 3 instances I have don't produce this error. (All VMs, 1 on ESXi, 2 on Proxmox).

I notice at around the time TrueCommand starts reporting the outage, there are a bunch of messages about UPS polling (although those are present both before and after that for a long time also) I have reviewed the UPS settings and decided to leave the service stopped for a while to see if it's related.

A few points of suspicion...

The UPS settings had the e-mail option turned on.

My email settings were pointing at a server that was no longer functioning, so all email would have been failing.


I have now corrected those points, so together with the UPS service being disabled (I think it hasn't been working properly anyway due to the server hosting the UPS having changed in some way) and the mail settings correct, I hope the middleware will manage to stand up for the rest of the day (and longer).
 

sretalla

Powered by Neutrality
Moderator
Joined
Jan 1, 2016
Messages
9,702
It seems that the problem was not those things... TrueCommand reports it offline again after just a few hours.

The last thing I can see in the logs (middleware, since messages has nothing for a few hours before):
Code:
[2020/10/23 12:15:46] (WARNING) application.call_method():173 - Exception while calling sharing.webdav.query(*[])
concurrent.futures.process._RemoteTraceback:
"""
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/concurrent/futures/process.py", line 239, in _process_worker
    r = call_item.fn(*call_item.args, **call_item.kwargs)
  File "/usr/local/lib/python3.8/site-packages/middlewared/worker.py", line 91, in main_worker
    res = MIDDLEWARE._run(*call_args)
  File "/usr/local/lib/python3.8/site-packages/middlewared/worker.py", line 45, in _run
    return self._call(name, serviceobj, methodobj, args, job=job)
  File "/usr/local/lib/python3.8/site-packages/middlewared/worker.py", line 33, in _call
    with Client('ws+unix:///var/run/middlewared-internal.sock', py_exceptions=True) as c:
  File "/usr/local/lib/python3.8/site-packages/middlewared/client/client.py", line 281, in __init__
    self._ws.connect()
  File "/usr/local/lib/python3.8/site-packages/middlewared/client/client.py", line 124, in connect
    rv = super(WSClient, self).connect()
  File "/usr/local/lib/python3.8/site-packages/ws4py/client/__init__.py", line 223, in connect
    bytes = self.sock.recv(128)
socket.timeout: timed out
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/middlewared/main.py", line 137, in call_method
    result = await self.middleware._call(message['method'], serviceobj, methodobj, params, app=self,
  File "/usr/local/lib/python3.8/site-packages/middlewared/main.py", line 1191, in _call
    return await methodobj(*prepared_call.args)
  File "/usr/local/lib/python3.8/site-packages/middlewared/schema.py", line 973, in nf
    return await f(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/middlewared/service.py", line 442, in query
    result = await self.middleware.call(
  File "/usr/local/lib/python3.8/site-packages/middlewared/main.py", line 1233, in call
    return await self._call(
  File "/usr/local/lib/python3.8/site-packages/middlewared/main.py", line 1191, in _call
    return await methodobj(*prepared_call.args)
  File "/usr/local/lib/python3.8/site-packages/middlewared/schema.py", line 973, in nf
    return await f(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/middlewared/plugins/datastore/read.py", line 163, in query
    result = await self._queryset_serialize(
  File "/usr/local/lib/python3.8/site-packages/middlewared/plugins/datastore/read.py", line 207, in _queryset_serialize
    extend_context_value = await self.middleware.call(extend_context, extra_options)
  File "/usr/local/lib/python3.8/site-packages/middlewared/main.py", line 1233, in call
    return await self._call(
  File "/usr/local/lib/python3.8/site-packages/middlewared/main.py", line 1191, in _call
    return await methodobj(*prepared_call.args)
  File "/usr/local/lib/python3.8/site-packages/middlewared/service.py", line 577, in sharing_task_extend_context
    'locked_datasets': await self.middleware.call('zfs.dataset.locked_datasets'),
  File "/usr/local/lib/python3.8/site-packages/middlewared/main.py", line 1233, in call
    return await self._call(
  File "/usr/local/lib/python3.8/site-packages/middlewared/main.py", line 1199, in _call
    return await self._call_worker(name, *prepared_call.args)
  File "/usr/local/lib/python3.8/site-packages/middlewared/main.py", line 1205, in _call_worker
    return await self.run_in_proc(main_worker, name, args, job)
  File "/usr/local/lib/python3.8/site-packages/middlewared/main.py", line 1132, in run_in_proc
    return await self.run_in_executor(self.__procpool, method, *args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/middlewared/main.py", line 1106, in run_in_executor
    return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs))
socket.timeout: timed out
[2020/10/23 12:15:46] (DEBUG) application.__crash_reporting():181 - [Crash Reporting] is disabled using sentinel file.
[2020/10/23 12:15:46] (DEBUG) application.__crash_reporting():181 - [Crash Reporting] is disabled using sentinel file.
[2020/10/23 12:15:46] (WARNING) asyncio.write():901 - socket.send() raised exception.
[2020/10/23 12:15:46] (WARNING) asyncio.write():901 - socket.send() raised exception.
[2020/10/23 12:15:46] (WARNING) asyncio.write():901 - socket.send() raised exception.
[2020/10/23 12:15:46] (WARNING) asyncio.write():901 - socket.send() raised exception.
[2020/10/23 12:15:46] (WARNING) asyncio.write():901 - socket.send() raised exception.
[2020/10/23 12:15:46] (WARNING) asyncio.write():901 - socket.send() raised exception.
[2020/10/23 12:15:46] (WARNING) asyncio.write():901 - socket.send() raised exception.
[2020/10/23 12:15:46] (WARNING) asyncio.write():901 - socket.send() raised exception.
[2020/10/23 12:15:46] (WARNING) asyncio.write():901 - socket.send() raised exception.
[2020/10/23 12:15:46] (WARNING) asyncio.write():901 - socket.send() raised exception.
[2020/10/23 12:15:47] (WARNING) asyncio.write():901 - socket.send() raised exception.
[2020/10/23 12:15:47] (WARNING) asyncio.write():901 - socket.send() raised exception.
[2020/10/23 12:15:47] (WARNING) asyncio.write():901 - socket.send() raised exception.
[2020/10/23 12:15:47] (WARNING) asyncio.write():901 - socket.send() raised exception.
[2020/10/23 12:15:47] (WARNING) asyncio.write():901 - socket.send() raised exception.
[2020/10/23 12:15:48] (DEBUG) urllib3.connectionpool._new_conn():955 - Starting new HTTPS connection (1): update-master.ixsystems.com:443
[2020/10/23 12:15:50] (DEBUG) urllib3.connectionpool._make_request():428 - https://update-master.ixsystems.com:443 "GET /TrueNAS/trains_redir.json HTTP/1.1" 200 233
[2020/10/23 12:15:50] (DEBUG) freenasOS.Configuration.TryGetNetworkFile():606 - TryGetNetworkFile(['https://update.ixsystems.com/TrueNAS/trains.txt', 'https://update-master.ixsystems.com/TrueNAS/trains.txt'])
[2020/10/23 12:15:50] (DEBUG) urllib3.connectionpool._new_conn():955 - Starting new HTTPS connection (1): update.ixsystems.com:443
[2020/10/23 12:15:50] (WARNING) asyncio.write():901 - socket.send() raised exception.
[2020/10/23 12:15:51] (DEBUG) urllib3.connectionpool._make_request():428 - https://update.ixsystems.com:443 "GET /TrueNAS/trains.txt HTTP/1.1" 301 169
[2020/10/23 12:15:51] (DEBUG) urllib3.connectionpool._new_conn():955 - Starting new HTTPS connection (1): update.freenas.org:443
[2020/10/23 12:15:52] (DEBUG) urllib3.connectionpool._make_request():428 - https://update.freenas.org:443 "GET /TrueNAS/trains.txt HTTP/1.1" 200 377
[2020/10/23 12:15:52] (DEBUG) freenasOS.Configuration.TryGetNetworkFile():745 - TryGetNetworkFile(['https://update.ixsystems.com/TrueNAS/trains.txt', 'https://update-master.ixsystems.com/TrueNAS/trains.txt']):  Read 377 bytes total
[2020/10/23 12:15:52] (DEBUG) freenasOS.Configuration.TryGetNetworkFile():606 - TryGetNetworkFile(['https://update-master.ixsystems.com/TrueNAS/TrueNAS-12.0-STABLE/LATEST'])
[2020/10/23 12:15:52] (DEBUG) urllib3.connectionpool._new_conn():955 - Starting new HTTPS connection (1): update-master.ixsystems.com:443
[2020/10/23 12:15:52] (DEBUG) urllib3.connectionpool._make_request():428 - https://update-master.ixsystems.com:443 "GET /TrueNAS/TrueNAS-12.0-STABLE/LATEST HTTP/1.1" 200 1622
[2020/10/23 12:15:52] (DEBUG) freenasOS.Configuration.TryGetNetworkFile():745 - TryGetNetworkFile(['https://update-master.ixsystems.com/TrueNAS/TrueNAS-12.0-STABLE/LATEST']):  Read 1622 bytes total
[2020/10/23 12:15:52] (DEBUG) freenasOS.Configuration.TryGetNetworkFile():606 - TryGetNetworkFile(['https://update-master.ixsystems.com/updates/ix_crl.pem'])
[2020/10/23 12:15:52] (DEBUG) urllib3.connectionpool._new_conn():955 - Starting new HTTPS connection (1): update-master.ixsystems.com:443
[2020/10/23 12:15:53] (DEBUG) urllib3.connectionpool._make_request():428 - https://update-master.ixsystems.com:443 "GET /updates/ix_crl.pem HTTP/1.1" 200 1028
[2020/10/23 12:15:53] (DEBUG) freenasOS.Configuration.CheckFreeSpace():77 - CheckFreeSpace(path=/tmp/tmprrfcp133.pem, pool=None, required=1028)
[2020/10/23 12:15:53] (DEBUG) freenasOS.Configuration.TryGetNetworkFile():745 - TryGetNetworkFile(['https://update-master.ixsystems.com/updates/ix_crl.pem']):  Read 1028 bytes total
[2020/10/23 12:15:53] (DEBUG) freenasOS.Configuration.TryGetNetworkFile():606 - TryGetNetworkFile(['https://update.ixsystems.com/TrueNAS/Validators/ValidateUpdate-mg3ef525.txt', 'https://update-master.ixsystems.com/TrueNAS/Validators/ValidateUpdate-mg3ef525.txt'])
[2020/10/23 12:15:53] (DEBUG) urllib3.connectionpool._new_conn():955 - Starting new HTTPS connection (1): update.ixsystems.com:443
[2020/10/23 12:15:54] (DEBUG) urllib3.connectionpool._make_request():428 - https://update.ixsystems.com:443 "GET /TrueNAS/Validators/ValidateUpdate-mg3ef525.txt HTTP/1.1" 301 169
[2020/10/23 12:15:54] (DEBUG) urllib3.connectionpool._new_conn():955 - Starting new HTTPS connection (1): update.freenas.org:443
[2020/10/23 12:15:54] (DEBUG) urllib3.connectionpool._make_request():428 - https://update.freenas.org:443 "GET /TrueNAS/Validators/ValidateUpdate-mg3ef525.txt HTTP/1.1" 200 185
[2020/10/23 12:15:54] (DEBUG) freenasOS.Configuration.CheckFreeSpace():77 - CheckFreeSpace(path=/tmp/tmpxf13q4zj, pool=None, required=185)
[2020/10/23 12:15:54] (DEBUG) freenasOS.Configuration.TryGetNetworkFile():745 - TryGetNetworkFile(['https://update.ixsystems.com/TrueNAS/Validators/ValidateUpdate-mg3ef525.txt', 'https://update-master.ixsystems.com/TrueNAS/Validators/ValidateUpdate-mg3ef525.txt']):  Read 185 bytes total
[2020/10/23 12:17:25] (WARNING) middlewared._loop_monitor_thread():1404 - Task seems blocked:
  File "/usr/local/lib/python3.8/site-packages/middlewared/main.py", line 1191, in _call
    return await methodobj(*prepared_call.args)
  File "/usr/local/lib/python3.8/site-packages/middlewared/schema.py", line 973, in nf
    return await f(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/middlewared/plugins/system.py", line 549, in info
    product_version = (await(await Popen(
  File "/usr/local/lib/python3.8/asyncio/subprocess.py", line 236, in create_subprocess_exec
    transport, protocol = await loop.subprocess_exec(
  File "/usr/local/lib/python3.8/asyncio/base_events.py", line 1630, in subprocess_exec
    transport = await self._make_subprocess_transport(
  File "/usr/local/lib/python3.8/asyncio/unix_events.py", line 197, in _make_subprocess_transport
    transp = _UnixSubprocessTransport(self, protocol, args, shell,
  File "/usr/local/lib/python3.8/asyncio/base_subprocess.py", line 36, in __init__
    self._start(args=args, shell=shell, stdin=stdin, stdout=stdout,
  File "/usr/local/lib/python3.8/asyncio/unix_events.py", line 789, in _start
    self._proc = subprocess.Popen(
  File "/usr/local/lib/python3.8/subprocess.py", line 854, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "/usr/local/lib/python3.8/subprocess.py", line 1664, in _execute_child
    os.close(errpipe_read)

[2020/10/23 12:17:51] (WARNING) middlewared._loop_monitor_thread():1404 - Task seems blocked:
  File "/usr/local/lib/python3.8/site-packages/middlewared/main.py", line 1191, in _call
    return await methodobj(*prepared_call.args)
  File "/usr/local/lib/python3.8/site-packages/middlewared/schema.py", line 973, in nf
    return await f(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/middlewared/plugins/system.py", line 554, in info
    manufacturer = (await(await Popen(
  File "/usr/local/lib/python3.8/asyncio/subprocess.py", line 236, in create_subprocess_exec
    transport, protocol = await loop.subprocess_exec(
  File "/usr/local/lib/python3.8/asyncio/base_events.py", line 1630, in subprocess_exec
    transport = await self._make_subprocess_transport(
  File "/usr/local/lib/python3.8/asyncio/unix_events.py", line 197, in _make_subprocess_transport
    transp = _UnixSubprocessTransport(self, protocol, args, shell,
  File "/usr/local/lib/python3.8/asyncio/base_subprocess.py", line 36, in __init__
    self._start(args=args, shell=shell, stdin=stdin, stdout=stdout,
  File "/usr/local/lib/python3.8/asyncio/unix_events.py", line 789, in _start
    self._proc = subprocess.Popen(
  File "/usr/local/lib/python3.8/subprocess.py", line 854, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "/usr/local/lib/python3.8/subprocess.py", line 1637, in _execute_child
    self.pid = _posixsubprocess.fork_exec(

[2020/10/23 12:22:28] (WARNING) middlewared._loop_monitor_thread():1404 - Task seems blocked:
  File "/usr/local/lib/python3.8/site-packages/middlewared/plugins/service_/services/base.py", line 72, in get_state
    return await self._get_state_freebsd()
  File "/usr/local/lib/python3.8/site-packages/middlewared/plugins/service_/services/base_freebsd.py", line 27, in _get_state_freebsd
    proc = await run(*cmd, check=False, encoding="utf-8")
  File "/usr/local/lib/python3.8/site-packages/middlewared/utils/__init__.py", line 53, in run
    proc = await asyncio.create_subprocess_exec(*args, **kwargs)
  File "/usr/local/lib/python3.8/asyncio/subprocess.py", line 236, in create_subprocess_exec
    transport, protocol = await loop.subprocess_exec(
  File "/usr/local/lib/python3.8/asyncio/base_events.py", line 1630, in subprocess_exec
    transport = await self._make_subprocess_transport(
  File "/usr/local/lib/python3.8/asyncio/unix_events.py", line 202, in _make_subprocess_transport
    watcher.add_child_handler(transp.get_pid(),
  File "/usr/local/lib/python3.8/asyncio/unix_events.py", line 1299, in add_child_handler
    thread.start()
  File "/usr/local/lib/python3.8/threading.py", line 857, in start
    self._started.wait()
  File "/usr/local/lib/python3.8/threading.py", line 558, in wait
    signaled = self._cond.wait(timeout)
  File "/usr/local/lib/python3.8/threading.py", line 302, in wait
    waiter.acquire()

[2020/10/23 12:24:51] (WARNING) middlewared._loop_monitor_thread():1404 - Task seems blocked:
  File "/usr/local/lib/python3.8/site-packages/middlewared/plugins/service_/services/base.py", line 72, in get_state
    return await self._get_state_freebsd()
  File "/usr/local/lib/python3.8/site-packages/middlewared/plugins/service_/services/base_freebsd.py", line 27, in _get_state_freebsd
    proc = await run(*cmd, check=False, encoding="utf-8")
  File "/usr/local/lib/python3.8/site-packages/middlewared/utils/__init__.py", line 53, in run
    proc = await asyncio.create_subprocess_exec(*args, **kwargs)
  File "/usr/local/lib/python3.8/asyncio/subprocess.py", line 236, in create_subprocess_exec
    transport, protocol = await loop.subprocess_exec(
  File "/usr/local/lib/python3.8/asyncio/base_events.py", line 1630, in subprocess_exec
    transport = await self._make_subprocess_transport(
  File "/usr/local/lib/python3.8/asyncio/unix_events.py", line 202, in _make_subprocess_transport
    watcher.add_child_handler(transp.get_pid(),
  File "/usr/local/lib/python3.8/asyncio/unix_events.py", line 1299, in add_child_handler
    thread.start()
  File "/usr/local/lib/python3.8/threading.py", line 857, in start
    self._started.wait()
  File "/usr/local/lib/python3.8/threading.py", line 558, in wait
    signaled = self._cond.wait(timeout)
  File "/usr/local/lib/python3.8/threading.py", line 302, in wait
    waiter.acquire()



A little before that I seem to have a lot of these:

Code:
[2020/10/23 12:03:14] (WARNING) middlewared._loop_monitor_thread():1404 - Task seems blocked:
  File "/usr/local/lib/python3.8/site-packages/middlewared/main.py", line 1191, in _call
    return await methodobj(*prepared_call.args)
  File "/usr/local/lib/python3.8/site-packages/middlewared/schema.py", line 973, in nf
    return await f(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/middlewared/plugins/system.py", line 544, in info
    product = (await(await Popen(
  File "/usr/local/lib/python3.8/asyncio/subprocess.py", line 236, in create_subprocess_exec
    transport, protocol = await loop.subprocess_exec(
  File "/usr/local/lib/python3.8/asyncio/base_events.py", line 1630, in subprocess_exec
    transport = await self._make_subprocess_transport(
  File "/usr/local/lib/python3.8/asyncio/unix_events.py", line 197, in _make_subprocess_transport
    transp = _UnixSubprocessTransport(self, protocol, args, shell,
  File "/usr/local/lib/python3.8/asyncio/base_subprocess.py", line 36, in __init__
    self._start(args=args, shell=shell, stdin=stdin, stdout=stdout,
  File "/usr/local/lib/python3.8/asyncio/unix_events.py", line 789, in _start
    self._proc = subprocess.Popen(
  File "/usr/local/lib/python3.8/subprocess.py", line 854, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "/usr/local/lib/python3.8/subprocess.py", line 1658, in _execute_child
    part = os.read(errpipe_read, 50000)

[2020/10/23 12:06:59] (WARNING) middlewared._loop_monitor_thread():1404 - Task seems blocked:
  File "/usr/local/lib/python3.8/site-packages/middlewared/schema.py", line 973, in nf
    return await f(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/middlewared/plugins/network.py", line 2233, in query
    resolvconf = (await run('resolvconf', '-l')).stdout.decode()
  File "/usr/local/lib/python3.8/site-packages/middlewared/utils/__init__.py", line 53, in run
    proc = await asyncio.create_subprocess_exec(*args, **kwargs)
  File "/usr/local/lib/python3.8/asyncio/subprocess.py", line 236, in create_subprocess_exec
    transport, protocol = await loop.subprocess_exec(
  File "/usr/local/lib/python3.8/asyncio/base_events.py", line 1630, in subprocess_exec
    transport = await self._make_subprocess_transport(
  File "/usr/local/lib/python3.8/asyncio/unix_events.py", line 197, in _make_subprocess_transport
    transp = _UnixSubprocessTransport(self, protocol, args, shell,
  File "/usr/local/lib/python3.8/asyncio/base_subprocess.py", line 36, in __init__
    self._start(args=args, shell=shell, stdin=stdin, stdout=stdout,
  File "/usr/local/lib/python3.8/asyncio/unix_events.py", line 789, in _start
    self._proc = subprocess.Popen(
  File "/usr/local/lib/python3.8/subprocess.py", line 854, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "/usr/local/lib/python3.8/subprocess.py", line 1658, in _execute_child
    part = os.read(errpipe_read, 50000)

[2020/10/23 12:09:06] (WARNING) middlewared._loop_monitor_thread():1404 - Task seems blocked:
  File "/usr/local/lib/python3.8/site-packages/middlewared/schema.py", line 973, in nf
    return await f(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/middlewared/plugins/system.py", line 567, in info
    ecc_memory = bool(RE_ECC_MEMORY.findall((await run(['dmidecode', '-t', '16'])).stdout.decode()))
  File "/usr/local/lib/python3.8/site-packages/middlewared/utils/__init__.py", line 53, in run
    proc = await asyncio.create_subprocess_exec(*args, **kwargs)
  File "/usr/local/lib/python3.8/asyncio/subprocess.py", line 236, in create_subprocess_exec
    transport, protocol = await loop.subprocess_exec(
  File "/usr/local/lib/python3.8/asyncio/base_events.py", line 1630, in subprocess_exec
    transport = await self._make_subprocess_transport(
  File "/usr/local/lib/python3.8/asyncio/unix_events.py", line 197, in _make_subprocess_transport
    transp = _UnixSubprocessTransport(self, protocol, args, shell,
  File "/usr/local/lib/python3.8/asyncio/base_subprocess.py", line 36, in __init__
    self._start(args=args, shell=shell, stdin=stdin, stdout=stdout,
  File "/usr/local/lib/python3.8/asyncio/unix_events.py", line 789, in _start
    self._proc = subprocess.Popen(
  File "/usr/local/lib/python3.8/subprocess.py", line 854, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "/usr/local/lib/python3.8/subprocess.py", line 1658, in _execute_child
    part = os.read(errpipe_read, 50000)

[2020/10/23 12:09:41] (WARNING) application.call_method():173 - Exception while calling sharing.webdav.query(*[])
concurrent.futures.process._RemoteTraceback:
"""
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/concurrent/futures/process.py", line 239, in _process_worker
    r = call_item.fn(*call_item.args, **call_item.kwargs)
  File "/usr/local/lib/python3.8/site-packages/middlewared/worker.py", line 91, in main_worker
    res = MIDDLEWARE._run(*call_args)
  File "/usr/local/lib/python3.8/site-packages/middlewared/worker.py", line 45, in _run
    return self._call(name, serviceobj, methodobj, args, job=job)
  File "/usr/local/lib/python3.8/site-packages/middlewared/worker.py", line 33, in _call
    with Client('ws+unix:///var/run/middlewared-internal.sock', py_exceptions=True) as c:
  File "/usr/local/lib/python3.8/site-packages/middlewared/client/client.py", line 281, in __init__
    self._ws.connect()
  File "/usr/local/lib/python3.8/site-packages/middlewared/client/client.py", line 124, in connect
    rv = super(WSClient, self).connect()
  File "/usr/local/lib/python3.8/site-packages/ws4py/client/__init__.py", line 223, in connect
    bytes = self.sock.recv(128)
socket.timeout: timed out
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/middlewared/main.py", line 137, in call_method
    result = await self.middleware._call(message['method'], serviceobj, methodobj, params, app=self,
  File "/usr/local/lib/python3.8/site-packages/middlewared/main.py", line 1191, in _call
    return await methodobj(*prepared_call.args)
  File "/usr/local/lib/python3.8/site-packages/middlewared/schema.py", line 973, in nf
    return await f(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/middlewared/service.py", line 442, in query
    result = await self.middleware.call(
  File "/usr/local/lib/python3.8/site-packages/middlewared/main.py", line 1233, in call
    return await self._call(
  File "/usr/local/lib/python3.8/site-packages/middlewared/main.py", line 1191, in _call
    return await methodobj(*prepared_call.args)
  File "/usr/local/lib/python3.8/site-packages/middlewared/schema.py", line 973, in nf
    return await f(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/middlewared/plugins/datastore/read.py", line 163, in query
    result = await self._queryset_serialize(
  File "/usr/local/lib/python3.8/site-packages/middlewared/plugins/datastore/read.py", line 207, in _queryset_serialize
    extend_context_value = await self.middleware.call(extend_context, extra_options)
  File "/usr/local/lib/python3.8/site-packages/middlewared/main.py", line 1233, in call
    return await self._call(
  File "/usr/local/lib/python3.8/site-packages/middlewared/main.py", line 1191, in _call
    return await methodobj(*prepared_call.args)
  File "/usr/local/lib/python3.8/site-packages/middlewared/service.py", line 577, in sharing_task_extend_context
    'locked_datasets': await self.middleware.call('zfs.dataset.locked_datasets'),
  File "/usr/local/lib/python3.8/site-packages/middlewared/main.py", line 1233, in call
    return await self._call(
  File "/usr/local/lib/python3.8/site-packages/middlewared/main.py", line 1199, in _call
    return await self._call_worker(name, *prepared_call.args)
  File "/usr/local/lib/python3.8/site-packages/middlewared/main.py", line 1205, in _call_worker
    return await self.run_in_proc(main_worker, name, args, job)
  File "/usr/local/lib/python3.8/site-packages/middlewared/main.py", line 1132, in run_in_proc
    return await self.run_in_executor(self.__procpool, method, *args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/middlewared/main.py", line 1106, in run_in_executor
    return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs))
socket.timeout: timed out
[2020/10/23 12:09:41] (DEBUG) application.__crash_reporting():181 - [Crash Reporting] is disabled using sentinel file.
[2020/10/23 12:13:59] (WARNING) middlewared._loop_monitor_thread():1404 - Task seems blocked:
  File "/usr/local/bin/middlewared", line 10, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.8/site-packages/middlewared/main.py", line 1577, in main
    Middleware(
  File "/usr/local/lib/python3.8/site-packages/middlewared/main.py", line 1421, in run
    self.loop.run_forever()
  File "/usr/local/lib/python3.8/asyncio/base_events.py", line 570, in run_forever
    self._run_once()
  File "/usr/local/lib/python3.8/asyncio/base_events.py", line 1859, in _run_once
    handle._run()
  File "/usr/local/lib/python3.8/asyncio/events.py", line 81, in _run
    self._context.run(self._callback, *self._args)
  File "/usr/local/lib/python3.8/site-packages/aiohttp/web_ws.py", line 281, in send_str
    await self._writer.send(data, binary=False, compress=compress)
  File "/usr/local/lib/python3.8/site-packages/aiohttp/http_websocket.py", line 649, in send
    await self._send_frame(message, WSMsgType.TEXT, compress)
  File "/usr/local/lib/python3.8/site-packages/aiohttp/http_websocket.py", line 620, in _send_frame
    self.transport.write(header + message)
  File "/usr/local/lib/python3.8/asyncio/selector_events.py", line 908, in write
    n = self._sock.send(data)

[2020/10/23 12:15:45] (WARNING) middlewared._loop_monitor_thread():1404 - Task seems blocked:
  File "/usr/local/lib/python3.8/asyncio/base_events.py", line 570, in run_forever
    self._run_once()
  File "/usr/local/lib/python3.8/asyncio/base_events.py", line 1859, in _run_once
    handle._run()
  File "/usr/local/lib/python3.8/asyncio/events.py", line 81, in _run
    self._context.run(self._callback, *self._args)
  File "/usr/local/lib/python3.8/site-packages/aiohttp/web_ws.py", line 281, in send_str
    await self._writer.send(data, binary=False, compress=compress)
  File "/usr/local/lib/python3.8/site-packages/aiohttp/http_websocket.py", line 649, in send
    await self._send_frame(message, WSMsgType.TEXT, compress)
  File "/usr/local/lib/python3.8/site-packages/aiohttp/http_websocket.py", line 618, in _send_frame
    self.transport.write(message)
  File "/usr/local/lib/python3.8/asyncio/selector_events.py", line 921, in write
    self._loop._add_writer(self._sock_fd, self._write_ready)
  File "/usr/local/lib/python3.8/asyncio/selector_events.py", line 301, in _add_writer
    self._selector.modify(fd, mask | selectors.EVENT_WRITE,
  File "/usr/local/lib/python3.8/selectors.py", line 261, in modify
    key = self.register(fileobj, events, data)
  File "/usr/local/lib/python3.8/selectors.py", line 526, in register
    self._selector.control([kev], 0, 0)


The GUI is currently working (without a refresh), but is noticeably slow to respond.

EDIT: I noticed that the TrueCommand GUI wasn't responding, so I redeployed that container and it came back... not sure if it's a two-way thing.
 

Prophet4NO1

Dabbler
Joined
Sep 11, 2016
Messages
20
The slowdown for me is machine wide. SSH is slow. I have a monitor and keyboard in the rack, so I connected them up and tried the console, also slow. Even local on the console a simple command takes 10+ seconds to get a response. I am stumped at this point.
 

sretalla

Powered by Neutrality
Moderator
Joined
Jan 1, 2016
Messages
9,702
I noticed a bunch of messages about swapspace before my system froze almost completely and needed a hard reset yesterday, so I adjusted the vfs.zfs.arc_max (which had been set very close to my actual RAM size and had worked fine for years, but perhaps FreeBSD 12 is now taking it more literally and allocating more ARC than is healthy for the system and causing swapping.

Since doing that yesterday, everything ran fine... until this morning again slow and unresponsive system, but this time no swapspace errors.

Instability introduced for just one of my 3 TrueNAS 12.0 Core systems with the upgrade, but unfortunately this is the system I actually count on the most, so this is really hurting.

I get this pattern repeating itself a lot before, during and after the slowness starts:
Code:
Oct 25 09:42:27 medianas 1 2020-10-25T09:42:27.818845+01:00 medianas.ntau.chau.ch smartd 4149 - - Device: /dev/nvme1, Temperature 67 Celsius reached critical limit of 42 Celsius (Min/Max 63/68)
Oct 25 09:42:27 medianas 1 2020-10-25T09:42:27.820280+01:00 medianas.ntau.chau.ch smartd 4149 - - Device: /dev/nvme1, Temperature 67 Celsius reached critical limit of 42 Celsius (Min/Max 63/68)
Oct 25 09:42:27 medianas 1 2020-10-25T09:42:27.821073+01:00 medianas.ntau.chau.ch smartd 4149 - - Device: /dev/nvme0, Temperature 51 Celsius reached critical limit of 42 Celsius (Min/Max 49/54)
Oct 25 09:42:27 medianas 1 2020-10-25T09:42:27.822036+01:00 medianas.ntau.chau.ch smartd 4149 - - Device: /dev/nvme0, Temperature 51 Celsius reached critical limit of 42 Celsius (Min/Max 49/54)
Oct 25 09:43:24 medianas 1 2020-10-25T09:43:24.228248+01:00 medianas.ntau.chau.ch collectd 26588 - - Traceback (most recent call last):
  File "/usr/local/lib/collectd_pyplugins/disktemp.py", line 62, in read
    with Client() as c:
  File "/usr/local/lib/python3.8/site-packages/middlewared/client/client.py", line 281, in __init__
    self._ws.connect()
  File "/usr/local/lib/python3.8/site-packages/middlewared/client/client.py", line 124, in connect
    rv = super(WSClient, self).connect()
  File "/usr/local/lib/python3.8/site-packages/ws4py/client/__init__.py", line 223, in connect
    bytes = self.sock.recv(128)
socket.timeout: timed out
Oct 25 09:48:24 medianas 1 2020-10-25T09:48:24.228191+01:00 medianas.ntau.chau.ch collectd 26588 - - Traceback (most recent call last):
  File "/usr/local/lib/collectd_pyplugins/disktemp.py", line 62, in read
    with Client() as c:
  File "/usr/local/lib/python3.8/site-packages/middlewared/client/client.py", line 281, in __init__
    self._ws.connect()
  File "/usr/local/lib/python3.8/site-packages/middlewared/client/client.py", line 124, in connect
    rv = super(WSClient, self).connect()
  File "/usr/local/lib/python3.8/site-packages/ws4py/client/__init__.py", line 223, in connect
    bytes = self.sock.recv(128)
socket.timeout: timed out
Oct 25 09:53:24 medianas 1 2020-10-25T09:53:24.248498+01:00 medianas.ntau.chau.ch collectd 26588 - - Traceback (most recent call last):
  File "/usr/local/lib/collectd_pyplugins/disktemp.py", line 62, in read
    with Client() as c:
  File "/usr/local/lib/python3.8/site-packages/middlewared/client/client.py", line 281, in __init__
    self._ws.connect()
  File "/usr/local/lib/python3.8/site-packages/middlewared/client/client.py", line 124, in connect
    rv = super(WSClient, self).connect()
  File "/usr/local/lib/python3.8/site-packages/ws4py/client/__init__.py", line 223, in connect
    bytes = self.sock.recv(128)
socket.timeout: timed out
Oct 25 09:58:24 medianas 1 2020-10-25T09:58:24.224643+01:00 medianas.ntau.chau.ch collectd 26588 - - Traceback (most recent call last):
  File "/usr/local/lib/collectd_pyplugins/disktemp.py", line 62, in read
    with Client() as c:
  File "/usr/local/lib/python3.8/site-packages/middlewared/client/client.py", line 281, in __init__
    self._ws.connect()
  File "/usr/local/lib/python3.8/site-packages/middlewared/client/client.py", line 124, in connect
    rv = super(WSClient, self).connect()
  File "/usr/local/lib/python3.8/site-packages/ws4py/client/__init__.py", line 223, in connect
    bytes = self.sock.recv(128)
socket.timeout: timed out
Oct 25 10:03:24 medianas 1 2020-10-25T10:03:24.267265+01:00 medianas.ntau.chau.ch collectd 26588 - - Traceback (most recent call last):
  File "/usr/local/lib/collectd_pyplugins/disktemp.py", line 62, in read
    with Client() as c:
  File "/usr/local/lib/python3.8/site-packages/middlewared/client/client.py", line 281, in __init__
    self._ws.connect()
  File "/usr/local/lib/python3.8/site-packages/middlewared/client/client.py", line 124, in connect
    rv = super(WSClient, self).connect()
  File "/usr/local/lib/python3.8/site-packages/ws4py/client/__init__.py", line 223, in connect
    bytes = self.sock.recv(128)
socket.timeout: timed out
Oct 25 10:08:24 medianas 1 2020-10-25T10:08:24.224157+01:00 medianas.ntau.chau.ch collectd 26588 - - Traceback (most recent call last):
  File "/usr/local/lib/collectd_pyplugins/disktemp.py", line 62, in read
    with Client() as c:
  File "/usr/local/lib/python3.8/site-packages/middlewared/client/client.py", line 281, in __init__
    self._ws.connect()
  File "/usr/local/lib/python3.8/site-packages/middlewared/client/client.py", line 124, in connect
    rv = super(WSClient, self).connect()
  File "/usr/local/lib/python3.8/site-packages/ws4py/client/__init__.py", line 223, in connect
    bytes = self.sock.recv(128)
socket.timeout: timed out
Oct 25 10:12:28 medianas 1 2020-10-25T10:12:28.223269+01:00 medianas.ntau.chau.ch smartd 4149 - - Device: /dev/nvme1, Temperature 67 Celsius reached critical limit of 42 Celsius (Min/Max 63/68)
Oct 25 10:12:28 medianas 1 2020-10-25T10:12:28.224666+01:00 medianas.ntau.chau.ch smartd 4149 - - Device: /dev/nvme1, Temperature 67 Celsius reached critical limit of 42 Celsius (Min/Max 63/68)
Oct 25 10:12:28 medianas 1 2020-10-25T10:12:28.225180+01:00 medianas.ntau.chau.ch smartd 4149 - - Device: /dev/nvme0, Temperature 52 Celsius reached critical limit of 42 Celsius (Min/Max 49/54)
Oct 25 10:12:28 medianas 1 2020-10-25T10:12:28.225637+01:00 medianas.ntau.chau.ch smartd 4149 - - Device: /dev/nvme0, Temperature 52 Celsius reached critical limit of 42 Celsius (Min/Max 49/54)
Oct 25 10:13:24 medianas 1 2020-10-25T10:13:24.227535+01:00 medianas.ntau.chau.ch collectd 26588 - - Traceback (most recent call last):
  File "/usr/local/lib/collectd_pyplugins/disktemp.py", line 62, in read
    with Client() as c:
  File "/usr/local/lib/python3.8/site-packages/middlewared/client/client.py", line 281, in __init__
    self._ws.connect()
  File "/usr/local/lib/python3.8/site-packages/middlewared/client/client.py", line 124, in connect
    rv = super(WSClient, self).connect()
  File "/usr/local/lib/python3.8/site-packages/ws4py/client/__init__.py", line 223, in connect
    bytes = self.sock.recv(128)
socket.timeout: timed out
Oct 25 10:18:33 medianas 1 2020-10-25T10:18:33.790352+01:00 medianas.ntau.chau.ch collectd 26588 - - Traceback (most recent call last):
  File "/usr/local/lib/collectd_pyplugins/disktemp.py", line 62, in read
    with Client() as c:
  File "/usr/local/lib/python3.8/site-packages/middlewared/client/client.py", line 281, in __init__
    self._ws.connect()
  File "/usr/local/lib/python3.8/site-packages/middlewared/client/client.py", line 124, in connect
    rv = super(WSClient, self).connect()
  File "/usr/local/lib/python3.8/site-packages/ws4py/client/__init__.py", line 223, in connect
    bytes = self.sock.recv(128)
socket.timeout: timed out


I have now removed it from TrueCommand to see if it's something triggered by that.

Nothing special in the middleware log, nor the daemon log to indicate why the system would be struggling other than this in middleware:
Code:
[2020/10/25 02:55:45] (WARNING) middlewared._loop_monitor_thread():1404 - Task seems blocked:
  File "/usr/local/lib/python3.8/asyncio/events.py", line 81, in _run
    self._context.run(self._callback, *self._args)
  File "/usr/local/lib/python3.8/site-packages/middlewared/schema.py", line 973, in nf
    return await f(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/middlewared/plugins/pool.py", line 356, in is_upgraded
    proc = await Popen(
  File "/usr/local/lib/python3.8/asyncio/subprocess.py", line 216, in create_subprocess_shell
    transport, protocol = await loop.subprocess_shell(
  File "/usr/local/lib/python3.8/asyncio/base_events.py", line 1597, in subprocess_shell
    transport = await self._make_subprocess_transport(
  File "/usr/local/lib/python3.8/asyncio/unix_events.py", line 197, in _make_subprocess_transport
    transp = _UnixSubprocessTransport(self, protocol, args, shell,
  File "/usr/local/lib/python3.8/asyncio/base_subprocess.py", line 36, in __init__
    self._start(args=args, shell=shell, stdin=stdin, stdout=stdout,
  File "/usr/local/lib/python3.8/asyncio/unix_events.py", line 789, in _start
    self._proc = subprocess.Popen(
  File "/usr/local/lib/python3.8/subprocess.py", line 854, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "/usr/local/lib/python3.8/subprocess.py", line 1658, in _execute_child
    part = os.read(errpipe_read, 50000)

[2020/10/25 09:12:16] (WARNING) middlewared._loop_monitor_thread():1404 - Task seems blocked:
  File "/usr/local/lib/python3.8/asyncio/events.py", line 81, in _run
    self._context.run(self._callback, *self._args)
  File "/usr/local/lib/python3.8/site-packages/middlewared/schema.py", line 973, in nf
    return await f(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/middlewared/plugins/pool.py", line 356, in is_upgraded
    proc = await Popen(
  File "/usr/local/lib/python3.8/asyncio/subprocess.py", line 216, in create_subprocess_shell
    transport, protocol = await loop.subprocess_shell(
  File "/usr/local/lib/python3.8/asyncio/base_events.py", line 1597, in subprocess_shell
    transport = await self._make_subprocess_transport(
  File "/usr/local/lib/python3.8/asyncio/unix_events.py", line 197, in _make_subprocess_transport
    transp = _UnixSubprocessTransport(self, protocol, args, shell,
  File "/usr/local/lib/python3.8/asyncio/base_subprocess.py", line 36, in __init__
    self._start(args=args, shell=shell, stdin=stdin, stdout=stdout,
  File "/usr/local/lib/python3.8/asyncio/unix_events.py", line 789, in _start
    self._proc = subprocess.Popen(
  File "/usr/local/lib/python3.8/subprocess.py", line 854, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "/usr/local/lib/python3.8/subprocess.py", line 1658, in _execute_child
    part = os.read(errpipe_read, 50000)
 

pschatz100

Guru
Joined
Mar 30, 2014
Messages
1,184
@Prophet4NO1 How about providing your system configuration and FreeNAS version. it is really not possible to comment without knowing the details.
 

sretalla

Powered by Neutrality
Moderator
Joined
Jan 1, 2016
Messages
9,702
Maybe worth noting is that my system uses the igb driver for the one NIC I have connected since I see that there are some timeouts of things that may be linked to network connectivity.
 

Prophet4NO1

Dabbler
Joined
Sep 11, 2016
Messages
20
mobo - Supermicro X10SLH-F-0

CPU - E3-1241V3

RAM - Crucial 32GB (4x8GB) 18KSF1G72AZ-1G6E1 1600mhz ECC

HDD - 6x3TB, one pool, two VDEV. Second VDEV was an expansion to the original. RAIDZ1 There are two Tosiba drives and the rest are HGST. Same specs and the slowdown happened well after adding the second VDEV.

The current version of TruNAS, 12.0 Release. But the issue has been ongoing for months going back to much older versions. Started back around January or so. I even tried rolling back to older versions before getting a new flash drive and doing a fresh install. At this point I am thinking this is hardware related.

All temps system wide are fine. Most in the 30-40C range. Heavy load does not effect it. Like transcoding Plex 4K down to 1080p. Everything runs fine. Up untill something makes the system angry. The only consistancy in it is that it takes usually 6-12 hours for the issue to crop up, regardless of system load.

Could it be drive related maybe? The two Toshiba drives are orriginal with the build. One died a year ago and I replaced it with an HGST. Maybe the other two are crapping out but not tripping SMART?
 

Prophet4NO1

Dabbler
Joined
Sep 11, 2016
Messages
20
So, something I had not thought about, Jail OS upgrades. I looked at my jails. All are on 11.3 u7. So, I pulled the newest version I could, 12.1, and upgraded them. The jury is still out, but the system has been up for several hours so far with no problems. I will post another update if the issues persist or if things seem to be running smoothly now.
 

sretalla

Powered by Neutrality
Moderator
Joined
Jan 1, 2016
Messages
9,702
I have upgraded 3 of my 10 jails.

I have just had another freeze... can't seem to make it 24 hours without a problem, but at least this rules out TrueCommand.

I will see if I can get the jails upgraded too... maybe a quicker test would just be to stop as many as possible and upgrade the essentials.
 

sretalla

Powered by Neutrality
Moderator
Joined
Jan 1, 2016
Messages
9,702
I notice that I can't delete my 11.1 U5 boot environment... If it weren't such a pain to reach the back of that system, I'd do a rebuild and restore the config on a fresh boot environment.
 

mpkaye

Cadet
Joined
Jun 26, 2019
Messages
4
I just experienced this and noticed that there was a python 3.8 process pinning a hyperthread at 100%. Upon investigation it was the SNMP service causing the problem, so I have stopped that service for now. Issue has gone away.
 

jsylvia007

Explorer
Joined
Oct 4, 2011
Messages
84

Prophet4NO1

Dabbler
Joined
Sep 11, 2016
Messages
20

jsylvia007

Explorer
Joined
Oct 4, 2011
Messages
84
I just read it. Seems similar. I am not getting the CPU pinning though. CPU is barley being used based on the desktop screen and htop. Strange. I will check SNMP though and see if that has any effect. Not sure if it is on off hand.
So it did fix the problem for me. My GUI is still extremely slow, BUT, the terminal is back to being responsive and I haven't gotten anymore errors on the console.
 

Prophet4NO1

Dabbler
Joined
Sep 11, 2016
Messages
20
Already off on mine. So, that is not my issue. I have SMART, SMB, SSH, and UPS services running, that is it right now. Maybe the UPS one could be an issue since it involves monitoring?
 

jsylvia007

Explorer
Joined
Oct 4, 2011
Messages
84
Already off on mine. So, that is not my issue. I have SMART, SMB, SSH, and UPS services running, that is it right now. Maybe the UPS one could be an issue since it involves monitoring?
Well that's a shame... Sorry to hear that. I also have NFS, SMART, SMB, & SSH. Like I said, the GUI is still... not great... But the command line is completely back and much more responsive. I don't use UPS, I have another solution with a Raspberry Pi for my whole home-lab.

That being said, Did you check to see if the process was running from the command line? Perhaps it got kicked off, seperate from the GUI? Never know, and worth a check if you didn't originally. The process was: python3 /usr/local/bin/snmp-agent.py.
 

Prophet4NO1

Dabbler
Joined
Sep 11, 2016
Messages
20
I checked with htop. Nothing stood out. No heavy CPU usage. The one thing that seemed a tad high was my syncthing jail, nearly a gig of RAM used. But nearly no CPU.
 

sretalla

Powered by Neutrality
Moderator
Joined
Jan 1, 2016
Messages
9,702
I have indeed seen heavy python3.8 threads pinned at 100% and was running the SNMP service (for no reason now, as I had enabled it 5 years ago before TrueCommand existed and never really got around to putting it to work).

I have stopped the service at the CLI and disabled the service via GUI (both with difficulty as it's running slowly already now).

The slowness has not abated after stopping the service, but jails continue to respond for now.

I'll report back potentially after I need to reboot and will see with SNMP disabled how it goes.

EDIT: after finally getting top to run, I found that the python3.8 process was still there even with SNMP stopped, so I killed it and things seem OK for now. I'll continue to give feedback over time.

EDIT2: Now at over 24 hours since a reboot (for the first time since the upgrade to TN 12.0 Core)... after killing the python3.8 process all still running well and GUI/CLI responsive.
 
Last edited:
Top