SOLVED TrueNAS install broken after unexpected power outage

[CheeryFlame]

Hello, I'm currently living the nightmare of watching my system failing after many months of learning and setup.

Since the power outage it's taking around 30 minutes to boot TrueNAS and the web interface is slower than normal. SMB shares aren't working, I can't access them even though the service is up and running. The apps aren't working either, I'm getting the message Applications are not running.

I tried resetting the bios clock since it wasn't setup right and that was suggested while searching on this forum.

Upon boot it seems like it's reconstructing the whole boot system and this is one thing that makes the boot lasting much longer than normal.

Also I'm seeing

"Failed to start Virtualization daemon"

"Failed to start System Logger Demon".

"Failed to start Docker Application Container Engine"

Also I saw this error and had to fix database.

Since I'm new to TrueNAS and this is my first server, I'm afraid I don't know where to start to troubleshoot this behaviour.

Thank you to anyone that will help me recover this mess!

 

[morganL]

If you can boot the system.. I'd start by disabling the services. Just see if we can get the system healthy.

Which version of SCALE?

It's likely that some piece of hardware is unhealthy and causing issues.... are there any alerts?

How old is the SATADOM??? These do have a habit of wearing out.

 

[CheeryFlame]

Which version of SCALE?

Thank you for your reply! I'm sorry it took so long to reply I had a big day at work.

My version of Scale is TrueNAS-22.12.0, I just noticed there was an update available. I guess I should wait before trying to update.

I found this new error as well in the iDRAC java console

If you can boot the system.. I'd start by disabling the services. Just see if we can get the system healthy.

I've disabled the following services: NFS, Rsync, S.M.A.R.T. and SMB, also unchecked the start automatically. I rebooted and there were many errors and it rebuilt some stuff as well. I made a backup of my configs but I'm afraid they could be corrupted. Still can't access apps and booting is taking a lot of time.

It's likely that some piece of hardware is unhealthy and causing issues.... are there any alerts?

These are the only hardware errors I got on iDRAC.

Drive 12 and 13 are my mirrored SSDs for my applications. Although they're back in iDRAC and I'm able to navigate to it in the Shell and I can see the ix-applications folder. It doesn't look broken and TrueNAS haven't reported any errors regarding those 2 drives.

Drive 16 and 17 are 2x of the 4 drives of a jbod temporary drive that I can access over SMB.

The BP1 SAS A2 cable is properl;y connected and it really seems that all of those errors are related to the unexpected interruption and that everything is back.

How old is the SATADOM??? These do have a habit of wearing out.

Unfortunately I can't tell but it's used from TechMikeNY which usually provides reliable hardware. I've asked for help on Reddit this morning on how to test out a Satadom drive but unfortunately no one replied back.

Also worth mentioning that this seems to be taking more time than it used to before, I may have forgotten though.

What could I do next? Thank you very much!

 

[CheeryFlame]

Here are more screenshots of the boot I think is worth sharing.

 

[morganL]

Not sure the status here.

Can you confirm what happens when the system boots. What seems to work and what does not?

 

[CheeryFlame]

Hello, I had to do a fresh install on another drive. I won't be using Satadom ever again. I've been told Scale doesn't jusst load in the ram but writes a lot and isn't the best candidate for Satadom. I'm not sure this was written in TrueNAS docs but I think it would be great to write it somewhere.

 

[morganL]

Hello, I had to do a fresh install on another drive. I won't be using Satadom ever again. I've been told Scale doesn't jusst load in the ram but writes a lot and isn't the best candidate for Satadom. I'm not sure this was written in TrueNAS docs but I think it would be great to write it somewhere.

It writes some data, but not excessively.

SATADOMs aren't great, but I suspect your SATADOM was failing before SCALE or because of power event.