SOLVED TrueNAS Network services failing after reboot

[kepford]

Version: TrueNAS-12.0-U2
Hardware: TrueNAS Mini E+
* 4-Core 2.2GHz Intel Processor
* 16GB of High-speed ECC Memory
* 4x 1GbE NICs

Note: I'm pretty new to TrueNAS

* While I was working on one of my VMs on TrueNAS I noticed some odd errors when I tried to create a new VM.
* Saw SQL errors in the web GUI
* I tried checking for updates to TrueNAS and that produced similar errors
* Thought I would try just rebooting. That was a mistake.
* Network services never started after reboot
* Had to hook up a monitor and keyboard because the server never came back on the network
* Not able to ping external or internal addresses from TrueNAS console. Have no network connection
* In the console running `ifconfig` shows that none of the interfaces (two of which are plugged into my switch and are lighting up) are connected or have IPs
* NIC lights up and shows up with `ifconfig`. Therefore I don't believe this is a hardware issue.
* I tried running `/etc/netcli` to configure the network interface and see what options I have.
* I get python errors with the netcli commands
* Not sure what is causing all this but it doesn't seem good.
* My best guess is a failed TrueNAS upgrade and I didn't notice the issue until now.

What should I do?
Is there a offline manual update process I can use?
Can I re-install TrueNAS from a thumb drive without losing my existing configuration?

Any pointers or direction appreciated. I'm still learning TrueNAS coming from Linux :)

 

[kepford]

Debug information:

https://gist.github.com/kepford/8cba0a8c4d2cc9235bfb7fe83a486831

 

[NugentS]

Do you have a copy of the config file?
If yes - then just rebuild onto a good USB thumb drive, load the config and see what happens - should help with the debugging. Not a good idea to keep that setup - but it should help debug
If no - then the config should be available on the main pool - just not sure where

 

[kepford]

I was able to copy the freenas-v1.db DB onto a thumb drive and inspect it and check it for database corruption. It seems like it is ok.
I will try installing what you suggest here. Thanks!

 

[kepford]

Booted from the latest TrueNAS ISO on a usb flash drive:
Installed fresh copy of TrueNAS on another flash drive
After the GUI loaded I uploaded my old DB in the GUI
Rebooted
All my settings came back

Jail
VM
Pools
Settings

So far everything looks good. Thanks for your help

 

[NugentS]

Looks like the TrueNAS Mini E+ boots from a SataDom (16GB)
Now you know your pools etc are good - its time to reinstall on the SataDom and then reload the config again. If that works then great - or maybe the SATADom is knackered.
Make sure the system Dataset is on the main pool, and not the USB / SataDom
And if you want to stick with the USB then I suggest a mirror, with two different makes - or replace the SataDom (if needed)

 

[kepford]

The system dataset is on the main pool so at some point today I'll reinstall the OS on the SATADom and re-import the config file. Thanks for you help.

 

[NugentS]

Good luck

 

[kepford]

All good. OS installed. Everything is working as expected.