Restoring and troubleshooting a system crash

[hrayrwannis]

Hello everyone,

I am getting the following alert from truenas for a while now - is it anything serious?

my system does crash once a week though not sure if it could be related to this.

thanks for any insight.

H

TrueNAS @ truenas

The following alert has been cleared:

• Failed to check for alert UnencryptedDatasets: concurrent.futures.process._RemoteTraceback: """ Traceback (most recent call last): File "/usr/lib/python3.9/concurrent/futures/process.py", line 243, in _process_worker r = call_item.fn(*call_item.args, **call_item.kwargs) File "/usr/lib/python3/dist-packages/middlewared/worker.py", line 115, in main_worker res = MIDDLEWARE._run(*call_args) File "/usr/lib/python3/dist-packages/middlewared/worker.py", line 46, in _run return self._call(name, serviceobj, methodobj, args, job=job) File "/usr/lib/python3/dist-packages/middlewared/worker.py", line 40, in _call return methodobj(*params) File "/usr/lib/python3/dist-packages/middlewared/worker.py", line 40, in _call return methodobj(*params) File "/usr/lib/python3/dist-packages/middlewared/schema.py", line 1254, in nf validate_return_type(f, res, nf.returns) File "/usr/lib/python3/dist-packages/middlewared/schema.py", line 1227, in validate_return_type clean_and_validate_arg(verrors, schema, res_entry) File "/usr/lib/python3/dist-packages/middlewared/schema.py", line 1233, in clean_and_validate_arg value = attr.clean(arg) File "/usr/lib/python3/dist-packages/middlewared/schema.py", line 1109, in clean tmpval = copy.deepcopy(value) File "/usr/lib/python3.9/copy.py", line 146, in deepcopy y = copier(x, memo) File "/usr/lib/python3.9/copy.py", line 205, in _deepcopy_list append(deepcopy(a, memo)) File "/usr/lib/python3.9/copy.py", line 146, in deepcopy y = copier(x, memo) File "/usr/lib/python3.9/copy.py", line 230, in _deepcopy_dict y[deepcopy(key, memo)] = deepcopy(value, memo) File "/usr/lib/python3.9/copy.py", line 146, in deepcopy y = copier(x, memo) File "/usr/lib/python3.9/copy.py", line 205, in _deepcopy_list append(deepcopy(a, memo)) File "/usr/lib/python3.9/copy.py", line 146, in deepcopy y = copier(x, memo) File "/usr/lib/python3.9/copy.py", line 230, in _deepcopy_dict y[deepcopy(key, memo)] = deepcopy(value, memo) File "/usr/lib/python3.9/copy.py", line 146, in deepcopy y = copier(x, memo) File "/usr/lib/python3.9/copy.py", line 205, in _deepcopy_list append(deepcopy(a, memo)) File "/usr/lib/python3.9/copy.py", line 146, in deepcopy y = copier(x, memo) File "/usr/lib/python3.9/copy.py", line 230, in _deepcopy_dict y[deepcopy(key, memo)] = deepcopy(value, memo) File "/usr/lib/python3.9/copy.py", line 146, in deepcopy y = copier(x, memo) File "/usr/lib/python3.9/copy.py", line 205, in _deepcopy_list append(deepcopy(a, memo)) File "/usr/lib/python3.9/copy.py", line 146, in deepcopy y = copier(x, memo) File "/usr/lib/python3.9/copy.py", line 228, in _deepcopy_dict memo[id(x)] = y UnboundLocalError: local variable 'y' referenced before assignment """ The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/middlewared/plugins/alert.py", line 784, in __run_source alerts = (await alert_source.check()) or [] File "/usr/lib/python3/dist-packages/middlewared/alert/source/datasets.py", line 18, in check for dataset in await self.middleware.call('pool.dataset.query', [['encrypted', '=', True]]): File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1395, in call return await self._call( File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1355, in _call return await self.run_in_executor(prepared_call.executor, methodobj, *prepared_call.args) File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1258, in run_in_executor return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs)) File "/usr/lib/python3.9/concurrent/futures/thread.py", line 52, in run result = self.fn(*self.args, **self.kwargs) File "/usr/lib/python3/dist-packages/middlewared/schema.py", line 1252, in nf res = f(*args, **kwargs) File "/usr/lib/python3/dist-packages/middlewared/schema.py", line 1382, in nf return func(*args, **kwargs) File "/usr/lib/python3/dist-packages/middlewared/plugins/pool.py", line 3074, in query self.__transform(self.middleware.call_sync( File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1418, in call_sync return self.run_coroutine(self._call_worker(name, *prepared_call.args)) File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1454, in run_coroutine return fut.result() File "/usr/lib/python3.9/concurrent/futures/_base.py", line 433, in result return self.__get_result() File "/usr/lib/python3.9/concurrent/futures/_base.py", line 389, in __get_result raise self._exception File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1358, in _call_worker return await self.run_in_proc(main_worker, name, args, job) File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1273, in run_in_proc return await self.run_in_executor(self.__procpool, method, *args, **kwargs) File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1258, in run_in_executor return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs)) UnboundLocalError: local variable 'y' referenced before assignment

 

[hrayrwannis]

So further to this, the server stopped posting and when i rebooted it never posted again.

Once i plugged into a monitor i can see all sorts of failed things that couldn't initialize. At first it was middleware which never started - even when trying to start/restart it as i've read some places...also some weird things like python couldn't read adress and error of CPU 10 socket 20...etc so then:

- I've thought maybe ram - so i put in two sticks of new RAM
- I've tried to see if the boot drive failed (SSD) so i installed a fresh copy on a Nvme
- THe fresh copy on the Nvme was 23.10 version which finally posted when i tried to upload the config file and restarted the same things started to happen
- then i thought maybe it had to be the original version 22.13.3.3 that i had running so I found and imaged and installed that on the NVME
- so the fresh copy posted and then when i tried to get the config file uploaded then the same happend.
- now i see it stalls ad ix-zfs.service start....

so what can i do to retrieve my pool without loosing everything and any idea what might have happened? for example, is it hardware or software??

thanks in advance.

 

[hrayrwannis]

So the saga continues...

I've changed today the hard disk data cables as well - so for the 3 main HDDs - and kept the NVME as the boot drive. So it finally booted and posted the web interface, i had to reconnect the main nic to the bridge (for some reason the config file did not restore this correctly?).

After having to boot couple of times for mounting the pool and trying to restore the apps...It did get stuck again at ix-zfs.service start again. I had to restart again.

SO the main problem now is that all the apps are gone and when i am mounting the pool i getting all sorts of errors that the ix-applications folder is not found but its there. So not sure what the matter.

I am still not sure what caused the crash and still can't get the apps back even when the pool mounts the page no applications running...Any ideas?

thanks,

 

[hrayrwannis]

How can i get all my apps back?? it keeps saying stuff like:

[EFAULT] SCALE version does not contain already applied kubernetes migrations ( 'middlewared.plugins.kubernetes_linux.migrations.0002_k3s_snapshots, middlewared.plugins.kubernetes_linux.migrations.0003_remove_docker' )

"

• Failed to configure kubernetes cluster for Applications: SCALE version does not contain already applied kubernetes migrations ( 'middlewared.plugins.kubernetes_linux.migrations.0002_k3s_snapshots, middlewared.plugins.kubernetes_linux.migrations.0003_remove_docker' )

anyone anything???