After a reboot, getting "Connecting to FreeNAS ... Make sure the FreeNAS system is powered on and connected to the network."

[riz94107]

I have a FreeNAS 11 system that I'm hoping to upgrade to TrueNAS core 12 or 13. This machine had an uptime of over a year, so because I have a couple important VMs on it, I wanted to reboot before doing the upgrade to make sure everything came back OK. It did - the VMs are all running fine (I can log into them), and I can log into the FreeNAS host via SSH, but I can't actually get to the web interface. I get this message:

Connecting to FreeNAS ... Make sure the FreeNAS system is powered on and connected to the network.

...one thing I did notice, is that the SSL cert has expired, and I figure maybe that has something to do with it? However, because I can't get to the UI, I can't actually change anything!

The host is remote, but as I said, I can SSH in and I have access to a serial console.... but why is this suddenly happening (I used the UI to reboot an hour ago!) and how can I fix it so I can update my SSL cert and also upgrade to TruNAS core?

Thanks,
+j

 

[riz94107]

How obnoxious - with no input or fiddling of any kind overnight, there's a login screen this morning. Gotta love unpredictable systems.

 

[riz94107]

Spoke too soon - I can get logged in, then as soon as I try to do anything useful, the UI chugs for a minute or so, then gives me the "Connecting to FreeNAS" screen like before. UGH.

 

[Whattteva]

Sounds like you have either stability problem or network connection problem.
Is your connection stable through SSH or is it flaky also? How long did you stay logged in to the SSH doing anything useful?

 

[riz94107]

No on both counts - this hardware has been 100% stable for years (uptime 377 days with lots of activity prior to the reboot, which was manual and just to make sure if I was going to have a problem (as I appear to be having!) it would happen without the additional chaos of an upgrade around it), and the network is solid - I still have an ssh session I opened last night, and I was doing stuff on it actively for over an hour with nothing unusual.

 

[Whattteva]

Seems strange. I don't think expired cert would do anything other than cause your web browser to complain about it and being insecure. Sounds like maybe an unstable nginx process, but then you rebooted and it doesn't help. Maybe backup your config and reinstall fresh and restore config. If it's a simple corrupted installation, that should fix it and installation + config restore should only take 10 mins at most, likely faster.

 

[riz94107]

While I doubt I'm going to do that (the idea of reinstalling a critical remote system while it's unstable makes me pretty nervous) at least until I visit the machine in 6 or 12 months, how would I go about getting a config backup without access to the UI? Or do anything else, for that matter? If there's any docs about how to reset stuff from the command line (say, from the serial console - which I also have access to), I'd love to know about it.

 

[Whattteva]

While I doubt I'm going to do that (the idea of reinstalling a critical remote system while it's unstable makes me pretty nervous) at least until I visit the machine in 6 or 12 months, how would I go about getting a config backup without access to the UI? Or do anything else, for that matter? If there's any docs about how to reset stuff from the command line (say, from the serial console - which I also have access to), I'd love to know about it.

I believe the config file is located under /data/freenas-v1.db. I'm not entirely 100% sure on this, so hopefully someone else will confirm this for me.

 

[Redcoat]

so hopefully someone else will confirm this for me.

Yes!
And if the boot media happens to be corrupt, it's also at /var/db/system/configs<a very large number> from the autosave at 03h45 every morning unless your system files have been moved back to the boot media from the first-created pool.

 

[riz94107]

Just a quick update regarding my situation, in case anyone cares - thanks for all the responses.

So I grabbed the config from `/data/freenas-v1.db` just in case (since my old backup was quite old), and decided to poke around a bit to see what I could recover. After letting things sit a while, I got the UI back again - but this time I paid more attention to how it behaved. Pretty much everything that was retrieving from the database was hanging indefinitely - couldn't get a list of certs, couldn't list the VMs (even though they were definitely running - I could use their services). Clearly, things were not great, but the config _looked_ fine when I poked around a bit with `sqlite`. I decided another reboot was in order before planning for complete meltdown - so, after taking a bit more emergency data, I did a `shutdown -r now` from the command line while watching on serial console.

It did the complete shutdown routine - got as far as:

`Syncing disks, vnodes remaining... 0 0 0 0 0 0 0 0 0 0 done
All buffers synced.`

...and then hung there. For a full 5 minutes. Eventually, I got tired of waiting and did a power cycle - I figured, it *looked* like all the buffers had been flushed. After an uncomfortably long time (normal for this hardware, unfortunately), I got the FreeNAS splash screen. When everything came back, this time it looks a lot better - menu items work, etc. and I've already updated the SSL cert so at least that doesn't weigh on me.

Now to white-knuckle it through updating to 12 and 13 in relatively rapid succession - at least I have all day for this. :)

Thanks again for the help.

 

[Whattteva]

That's good news. Sounds like you're back in business. Hopefully, you won't run into anymore issues (which I feel is unlikely give that you were able to upgrade in relatively rapid succession).