Lost the WebUI

[Heracles]

Hi,

I recently upgraded my 3 NAS servers to TrueNAS Core 12.0 - U1. On my main one, Atlas (described in more details in my signature if needed), the Web interface is not reachable anymore. I receive the first page saying that it is connecting to TrueNAS and asking to be sure that the server is running. Not making it to the login form anymore...

Usually, I reach that Web interface while going through HAProxy. The idea is to require an SSL certificate before granting the access. But here, nothing is working anymore : Not through HAProxy, not with a direct call to the local DNS name and neither with the IP Address. All of them stop at the same first connecting message. I tried with different browsers (Chrome and Firefox) and using both normal and incognito / private mode. No luck either.

After some research, I tried to restart Nginx. It restarted successfully but not progress.

Code:

root@Atlas[~]# service nginx restart
Performing sanity check on nginx configuration:
nginx: the configuration file /usr/local/etc/nginx/nginx.conf syntax is ok
nginx: configuration file /usr/local/etc/nginx/nginx.conf test is successful
Stopping nginx.
Waiting for PIDS: 1881.
Performing sanity check on nginx configuration:
nginx: the configuration file /usr/local/etc/nginx/nginx.conf syntax is ok
nginx: configuration file /usr/local/etc/nginx/nginx.conf test is successful
Starting nginx.
root@Atlas[~]# service nginx status        
nginx is running as pid 38492.

In the past, there was another service to restart, django, but that one seems to be gone in TrueNAS.

Code:

root@Atlas[~]# service django status
django does not exist in /etc/rc.d or the local startup
directories (/etc/ix.rc.d /usr/local/etc/rc.d), or is not executable

In Nginx error log, I see the error about connection refused for URL /websocket when trying to connect it on the loopback (127.0.0.1) and port 6000. First source IP is my Docker host running many services as containers, including TrueCommand. Second one is a management station I used to connect directly to TrueNAS.

Code:

Jan 13 00:07:15 Atlas.jb.lan nginx: 2021/01/13 00:07:15 [error] 38547#101090: *963 kevent() reported that connect() failed (61: Connection refused) while connecting to upstream, client: 172.24.128.46, server: localhost, request: "GET /websocket HTTP/1.1", upstream: "http://127.0.0.1:6000/websocket", host: "atlas.jb.lan:443"
./nginx/error.log:Jan 13 00:30:27 Atlas.jb.lan nginx: 2021/01/13 00:30:27 [error] 38547#101090: *1886 kevent() reported that connect() failed (61: Connection refused) while connecting to upstream, client: 172.24.128.16, server: localhost, request: "GET /websocket HTTP/1.1", upstream: "http://127.0.0.1:6000/websocket", host: "atlas.jb.lan"

A quick netstat -an showed that this port 6000 is not listened by the TCP/IP stack.

Code:

root@Atlas[/var/log/nginx]# netstat -an | grep 6000
fffff800370a6000 seqpac      0      0                0 fffff800370a6100                0                0 /var/run/devd.seqpacket.pipe
fffff800370a6100 seqpac      0      0                0 fffff800370a6000                0                0

To reboot the server is to be avoided. I must completely shutdown my entire environment for that because Atlas is hosting an ESXi datastore over iSCSI for my main server. Rebooting requires that I power off everything, put ESXi in maintenance mode, do the reboot and restart everything manually.

I still have full access and full control over SSH.

Any idea what I should do in TrueNAS for kick starting that WebUI as restarting Nginx is not enough ? What is supposed to be listening that port 6000 and how can I restart it ?

Thanks for your help here,

 

[Heracles]

Ok... I should have keep working my case a few more minutes...

I did a compare between Atlas (Web UI not working) and Hades (Web UI working properly). I noticed that many middlewared processes were missing in Atlas. A search about it returned me this thread. Indeed, a service middlewared restart fixed everything for me too.