Hello,
I been trying to troubleshoot this server for a while now. I had it running FreeNas for a couple years and then it started to crash. It would reboot every couple of days.
I took it offline to diagnosis. Didn't see anything so I thought I would update it and remove the encryption.
Updated it to truenas scale. It would hang and never recover until reboot.
I checked ram, it passed MemTest86
I replaced Power Supply. No change.
I updated BIOS from 4.2 to 8.02 and it no longer hangs, but I get constant errors after 5-12 hours. Then every hour at least, sometimes every 10 minutes, until reboot.
Specs
TrueNAS-SCALE-22.12.3.3
AMD Ryzen 7 3700X 8-Core Processor
Asrock B450 PRO 4
L8.02 Bios
64 GB ECC ~ 2600 MHZ
2 Pool
1 Pool Zfs raidz3 - 10 Drives 14-TB ea - No encryption
1 Pool just a 2TB ssd
2 x expansion cards
1 x intel modem expansion card for network
GPU Nvida P2000
Main OS on an nvme ~250 gb I think
I am thinking next step is replace the processor, but I don't know.
Both logs indicate some sort of timeout error,
[err -110]: b'Connection timed out'
The attached jpg is what I got when I tried to restart through GUI.
I have a similar system AMD Ryzen 5 PRO 4650G running fine.
If anyone has any ideas I would love to hear them.
I been trying to troubleshoot this server for a while now. I had it running FreeNas for a couple years and then it started to crash. It would reboot every couple of days.
I took it offline to diagnosis. Didn't see anything so I thought I would update it and remove the encryption.
Updated it to truenas scale. It would hang and never recover until reboot.
I checked ram, it passed MemTest86
I replaced Power Supply. No change.
I updated BIOS from 4.2 to 8.02 and it no longer hangs, but I get constant errors after 5-12 hours. Then every hour at least, sometimes every 10 minutes, until reboot.
Specs
TrueNAS-SCALE-22.12.3.3
AMD Ryzen 7 3700X 8-Core Processor
Asrock B450 PRO 4
L8.02 Bios
64 GB ECC ~ 2600 MHZ
2 Pool
1 Pool Zfs raidz3 - 10 Drives 14-TB ea - No encryption
1 Pool just a 2TB ssd
2 x expansion cards
1 x intel modem expansion card for network
GPU Nvida P2000
Main OS on an nvme ~250 gb I think
I am thinking next step is replace the processor, but I don't know.
Code:
Error: Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/middlewared/main.py", line 204, in call_method
result = await self.middleware._call(message['method'], serviceobj, methodobj, params, app=self)
File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1344, in _call
return await methodobj(*prepared_call.args)
File "/usr/lib/python3/dist-packages/middlewared/schema.py", line 1246, in nf
res = await f(*args, **kwargs)
File "/usr/lib/python3/dist-packages/middlewared/schema.py", line 1378, in nf
return await func(*args, **kwargs)
File "/usr/lib/python3/dist-packages/middlewared/plugins/docker_linux/images.py", line 68, in query
for image in await docker.images.list():
File "/usr/lib/python3/dist-packages/aiodocker/images.py", line 31, in list
response = await self.docker._query_json("images/json", "GET", params=params)
File "/usr/lib/python3/dist-packages/aiodocker/docker.py", line 302, in _query_json
async with self._query(
File "/usr/lib/python3/dist-packages/aiodocker/utils.py", line 309, in __aenter__
resp = await self._coro
File "/usr/lib/python3/dist-packages/aiodocker/docker.py", line 240, in _do_query
await self._check_version()
File "/usr/lib/python3/dist-packages/aiodocker/docker.py", line 192, in _check_version
ver = await self._query_json("version", versioned_api=False)
File "/usr/lib/python3/dist-packages/aiodocker/docker.py", line 302, in _query_json
async with self._query(
File "/usr/lib/python3/dist-packages/aiodocker/utils.py", line 309, in __aenter__
resp = await self._coro
File "/usr/lib/python3/dist-packages/aiodocker/docker.py", line 250, in _do_query
response = await self.session.request(
File "/usr/lib/python3/dist-packages/aiohttp/client.py", line 544, in _request
await resp.start(conn)
File "/usr/lib/python3/dist-packages/aiohttp/client_reqrep.py", line 905, in start
self._continue = None
File "/usr/lib/python3/dist-packages/aiohttp/helpers.py", line 656, in __exit__
raise asyncio.TimeoutError from None
asyncio.exceptions.TimeoutErrorCode:
New alerts:
Failed to check for alert Smartd: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/middlewared/plugins/alert.py", line 784, in __run_source alerts = (await alert_source.check()) or [] File "/usr/lib/python3/dist-packages/middlewared/alert/base.py", line 223, in check return await self.middleware.run_in_thread(self.check_sync) File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1261, in run_in_thread return await self.run_in_executor(self.thread_pool_executor, method, *args, **kwargs) File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1258, in run_in_executor return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs)) File "/usr/lib/python3.9/concurrent/futures/thread.py", line 52, in run result = self.fn(*self.args, **self.kwargs) File "/usr/lib/python3/dist-packages/middlewared/alert/source/smartd.py", line 22, in check_sync if not self.middleware.call_sync("service.started", "smartd"): File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1414, in call_sync return self.run_coroutine(methodobj(*prepared_call.args)) File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1454, in run_coroutine return fut.result() File "/usr/lib/python3.9/concurrent/futures/_base.py", line 433, in result return self.__get_result() File "/usr/lib/python3.9/concurrent/futures/_base.py", line 389, in __get_result raise self._exception File "/usr/lib/python3/dist-packages/middlewared/schema.py", line 1378, in nf return await func(*args, **kwargs) File "/usr/lib/python3/dist-packages/middlewared/schema.py", line 1246, in nf res = await f(*args, **kwargs) File "/usr/lib/python3/dist-packages/middlewared/plugins/service.py", line 201, in started state = await service_object.get_state() File "/usr/lib/python3/dist-packages/middlewared/plugins/service_/services/base.py", line 37, in get_state return await self.middleware.run_in_thread(self._get_state_sync) File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1261, in run_in_thread return await self.run_in_executor(self.thread_pool_executor, method, *args, **kwargs) File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1258, in run_in_executor return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs)) File "/usr/lib/python3.9/concurrent/futures/thread.py", line 52, in run result = self.fn(*self.args, **self.kwargs) File "/usr/lib/python3/dist-packages/middlewared/plugins/service_/services/base.py", line 42, in _get_state_sync state = unit.Unit.ActiveState File "/usr/lib/python3/dist-packages/pystemd/base.py", line 191, in _call return func(self, name, *args) File "/usr/lib/python3/dist-packages/pystemd/base.py", line 127, in _get_property return bus.get_property( File "pystemd/dbuslib.pyx", line 478, in pystemd.dbuslib.DBus.get_property pystemd.dbusexc.DBusTimeoutError: [err -110]: b'Connection timed out'
Both logs indicate some sort of timeout error,
[err -110]: b'Connection timed out'
The attached jpg is what I got when I tried to restart through GUI.
I have a similar system AMD Ryzen 5 PRO 4650G running fine.
If anyone has any ideas I would love to hear them.
Attachments
Last edited:
