Web interface not reachable

[Tony-1971]

Hi,

Suddenly I can't reach TrueNas web interface but other services seems working well.
File /var/log/nginx/error.log contains a lot of errors; following the last ones:

Code:

...
Dec 21 18:18:06 freenas-sm nginx: 2020/12/21 18:18:06 [error] 11690#100975: *16 kevent() reported that connect() failed (61: Connection refused) while connecting to upstream, client: 192.168.50.132, server: localhost, request: "GET /websocket HTTP/1.1", upstream: "http://127.0.0.1:6000/websocket", host: "freenas-emx.localdomain:443"
Dec 21 18:18:07 freenas-sm nginx: 2020/12/21 18:18:07 [error] 11690#100975: *18 kevent() reported that connect() failed (61: Connection refused) while connecting to upstream, client: 192.168.50.132, server: localhost, request: "GET /websocket HTTP/1.1", upstream: "http://127.0.0.1:6000/websocket", host: "freenas-emx.localdomain:80"

I restart ngnix it but the problem is yet present.

Ports 80 and 443 seems open:

Code:

$ nmap freenas-emx -p 80,443
Starting Nmap 7.80 ( https://nmap.org ) at 2020-12-21 18:32 CET
Nmap scan report for freenas-emx (192.168.50.3)
Host is up (0.00027s latency).
rDNS record for 192.168.50.3: freenas-emx.localdomain

PORT    STATE SERVICE
80/tcp  open  http
443/tcp open  https

Nmap done: 1 IP address (1 host up) scanned in 0.10 seconds

but when I try to access web interface there is a message that say (transalated from italian): Connect to TrueNAS... Are you sure that TrueNAS system is switched on and connected to the network.

Best Regards,
Antonio

 

[Chris Moore]

The times that I have seen this type of thing, it has been a boot disk failure. Do you just have the one, not a boot mirror?

When the boot drive fails, everything in RAM will keep running fine. The system will not fail until it needs to read something from the boot pool that is not cached in RAM. I had a system that ran for almost a day with a failed boot drive. It just wouldn't do anything that it was not already doing at the time the boot drive failed.

There may be no choice but to try a reboot. I suggest a mirrored pair of SSDs for the boot pool. That is what I use on the systems at work. They don't have to be new or fancy. I have several old 80 or 120 GB drives that have previously been used in laptop computers and such that work fine as FreeNAS boot drives.

 

[Tony-1971]

Yes, I have only one boot disk on SATA DOM. When I login using ssh I see the following error:

Code:

$ ssh freenas-emx
tony@freenas-emx's password:
Last login: Mon Dec 21 09:38:14 2020 from 192.168.50.86
FreeBSD 12.2-RELEASE-p2 663e6b09467(HEAD) TRUENAS

        TrueNAS (c) 2009-2020, iXsystems, Inc.
        All rights reserved.
        TrueNAS code is released under the modified BSD license with some
        files copyrighted by (c) iXsystems, Inc.

        For more information, documentation, help or support, go here:
        http://truenas.com
Welcome to FreeNAS
Could not chdir to home directory /nonexistent: No such file or directory

Warning: settings changed through the CLI are not written to
the configuration database and will be reset on reboot.

% sudo su -
Password:
Sorry, try again.
Password:
Traceback (most recent call last):
  File "/usr/local/sbin/hactl", line 171, in <module>
    main(args.command, args.q)
  File "/usr/local/sbin/hactl", line 17, in main
    client = Client()
  File "/usr/local/lib/python3.8/site-packages/middlewared/client/client.py", line 281, in __init__
    self._ws.connect()
  File "/usr/local/lib/python3.8/site-packages/middlewared/client/client.py", line 124, in connect
    rv = super(WSClient, self).connect()
  File "/usr/local/lib/python3.8/site-packages/ws4py/client/__init__.py", line 216, in connect
    self.sock.connect(self.bind_addr)
ConnectionRefusedError: [Errno 61] Connection refused

Warning: settings changed through the CLI are not written to
the configuration database and will be reset on reboot.

If I check the boot disk status from CLI it seems OK:

Code:

root@freenas-sm[~]# zpool status -v freenas-boot
  pool: freenas-boot
 state: ONLINE
status: Some supported features are not enabled on the pool. The pool can
        still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
        the pool may no longer be accessible by software that does not support
        the features. See zpool-features(5) for details.
  scan: scrub repaired 0B in 00:00:53 with 0 errors on Mon Dec 14 03:45:53 2020
config:

        NAME          STATE     READ WRITE CKSUM
        freenas-boot  ONLINE       0     0     0
          ada0p2      ONLINE       0     0     0

errors: No known data errors
root@freenas-sm[~]# zpool list -v freenas-boot
NAME           SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
freenas-boot    14G  6.08G  7.92G        -         -      -    43%  1.00x    ONLINE  -
  ada0p2        14G  6.08G  7.92G        -         -      -  43.4%      -  ONLINE

I execute another scrub from command line and no problems are detected:

Code:

root@freenas-sm[~]# zpool scrub freenas-boot

root@freenas-sm[~]# zpool status -v freenas-boot
  pool: freenas-boot
 state: ONLINE
status: Some supported features are not enabled on the pool. The pool can
        still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
        the pool may no longer be accessible by software that does not support
        the features. See zpool-features(5) for details.
  scan: scrub repaired 0B in 00:00:54 with 0 errors on Tue Dec 22 10:08:08 2020
config:

        NAME          STATE     READ WRITE CKSUM
        freenas-boot  ONLINE       0     0     0
          ada0p2      ONLINE       0     0     0

errors: No known data errors

There is something else I can check before reboot?
Thank you.

Best Regards,
Antonio

 

[Chris Moore]

There is something else I can check before reboot?

There is a service that can be restarted to effectively reboot the Web UI without restarting the server. I can't remember the name of the service though. It used to be service netif restart if I recall correctly, but I think they changed that when they redesigned the UI a few versions back.
Possibly worth trying. If you find something, please post back. I will keep searing the forum to see if I can find something.

 

[Tony-1971]

Yes the service is present and seems to manage network interface.
But before restart network interfaces I check the status of WebUI and now it works again.
Not sure why because the only action I performed manually was scrub of boot pool.
The last error in /var/log/messages is:

Code:

Dec 22 10:02:48 freenas-sm 1 2020-12-22T10:02:48.753435+01:00 freenas-sm.local collectd 3675 - - Traceback (most recent call last):
  File "/usr/local/lib/collectd_pyplugins/disktemp.py", line 62, in read
    with Client() as c:
  File "/usr/local/lib/python3.8/site-packages/middlewared/client/client.py", line 281, in __init__
    self._ws.connect()
  File "/usr/local/lib/python3.8/site-packages/middlewared/client/client.py", line 124, in connect
    rv = super(WSClient, self).connect()
  File "/usr/local/lib/python3.8/site-packages/ws4py/client/__init__.py", line 216, in connect
    self.sock.connect(self.bind_addr)
ConnectionRefusedError: [Errno 61] Connection refused

Log time is very close to scrub time...
I think that reboot / service restart is not needed now but I will continue to watch the server.

Best Regards,
Antonio