website times out and can't ssh as root

[penguin_brian]

This *was* working and I have not made any changes:

Code:

brian@canidae:~$ curl -v https://minion.pri/
*   Trying 2001:44b8:4112:8a03::c:443...
* Connected to minion.pri (2001:44b8:4112:8a03::c) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
[no response]

Code:

brian@canidae:~$ ssh root@minion.pri
Last login: Wed Jan 26 09:52:16 2022 from 2001:44b8:4112:8a02:e88:d16b:f1f6:c2ed
FreeBSD 12.2-RELEASE-p10 b26f74b5984(HEAD) TRUENAS

    TrueNAS (c) 2009-2021, iXsystems, Inc.
    All rights reserved.
    TrueNAS code is released under the modified BSD license with some
    files copyrighted by (c) iXsystems, Inc.

    For more information, documentation, help or support, go here:
    http://truenas.com
Welcome to TrueNAS




[delay]
Traceback (most recent call last):
  File "/usr/local/sbin/hactl", line 171, in <module>
    main(args.command, args.q)
  File "/usr/local/sbin/hactl", line 17, in main
    client = Client()
  File "/usr/local/lib/python3.9/site-packages/middlewared/client/client.py", line 283, in __init__
[no response, no prompt]

Meanwhile the box is still up and running, I can still ssh as non-root, etc. Although commands like "top" will hang.

Guessing a reboot might help. As I can't find anyway of getting root access, might be the only option in fact.

But wondering what on earth is going on here?

Later: seems like my boot USB device is dead. Not sure why this should cause problems on running system though. My understanding was that the boot device is only used for booting...

Just wondering what is the best way tor recover?

dmesg shows:

Code:

[...]
(da4:umass-sim0:0:0:0): CAM status: CCB request completed with an error
(da4:umass-sim0:0:0:0): Retrying command, 0 more tries remain
(da4:umass-sim0:0:0:0): READ(10). CDB: 28 00 00 6e 1a 8c 00 00 80 00
(da4:umass-sim0:0:0:0): CAM status: CCB request completed with an error
(da4:umass-sim0:0:0:0): Error 5, Retries exhausted
Solaris: WARNING: Pool 'boot-pool' has encountered an uncorrectable I/O failure and has been suspended.
[...]

Can't mirror the boot device because (a) can't access website (b) probably can't mirror SUSPENDED pool anyway.

What is the best way to proceed?

Maybe the "Boot the installer with virtualbox and install onto new USB then swap it with the old stick" method? https://www.truenas.com/community/t...able-usb-for-my-freenas-box.73944/post-513198

 

[penguin_brian]

Maybe the "Boot the installer with virtualbox and install onto new USB then swap it with the old stick" method? https://www.truenas.com/community/t...able-usb-for-my-freenas-box.73944/post-513198

Looks like if I do that, it doesn't pick up my old config. Wonder how to do this?

 

[Samuel Tai]

Save your config under System->General->Download config. On the new USB stick, upload your config under System->General->Upload config.

 

[penguin_brian]

Hmmm. Looks like I cant do that anymore. Even before I rebooted I couldn't get to the website.

Does it matter if the old config is... old? Also what filename should I be looking for? I am sure I downloaded the config, but not sure of the filename.

 

[penguin_brian]

Never mind, I found a backup I kept from September last year, during my upgrade then. Not as old as I thought. Kept it on an encrypted partition.

Fingers crossed it works. Fingers crossed I made this backup after I fixed up the broken but appeared to be working network config.

Filename: minion-TrueNAS-12.0-U4-20210914111102.tar

 

[penguin_brian]

All good. All working. Glad I found that backup. Looks like it had recent network config too.

Wondering: Where would I be without the backup? Would I be stuck reconfiguring stuff from scratch, or is there someway of recovering from an existing system?

 

[Samuel Tai]

Wondering: Where would I be without the backup? Would I be stuck reconfiguring stuff from scratch

Yup, reconfiguring from scratch.

 

[penguin_brian]

OK, good to know. Thanks for you help.

Now I found a backup from October also, but I think the September one is fine.

Only minor thing I have found missing so far is my ssh key authorization. Guessing I created the /root/.ssh/authorized_keys manually instead of via the website, and somewhere during the process my root home directory was recreated. Doesn't matter, but unexpected.

 

[Samuel Tai]

Add your new key back via the UI, and then create another config backup. Search around the forums; other people have created scripts to automatically back up configs on a regular basis.