[11.2] Web GUI doesn't work behind a reverse proxy

[seedz]

The new web GUI in the 11.2 doesn't work when behind an nginx reverse proxy server :

maybe there's something to configure in either freenas or nginx, but i have no clue what
Web GUI works fine when accessed directly from its IP.

both Nginx and freenas are set up with SSL with the exact same certificate from letsencrypt

this is what my nginx.conf looks like for the freenas part :

Code:

	   # HTTP server
		server {
				listen		  80;
				listen		  [::]:80;
				server_name	 XXXXXXXXXX;

				location /.well-known {
						alias /usr/local/www/freenas_stub/.well_known;
				}

				location / {
						return 301 https://XXXXXXXX;
				}
		}

		# HTTPS server
		server {
				listen		  443 ssl;
				listen		  [::]:443 ssl;
				server_name	 XXXXXXX;

				ssl					 on;
				ssl_certificate		 /usr/local/etc/letsencrypt/live/XXXXXXXXX/fullchain.pem;
				ssl_certificate_key	 /usr/local/etc/letsencrypt/live/XXXXXXXXXX/privkey.pem;
				ssl_session_timeout 1d;
				ssl_session_cache shared:SSL:50m;
				ssl_session_tickets off;

				ssl_protocols TLSv1.2;
				ssl_ciphers 'ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RS
A-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256';
				ssl_prefer_server_ciphers on;

				ssl_stapling on;
				ssl_stapling_verify on;

				location / {
				  proxy_pass					https://192.168.1.40:443;

				  proxy_http_version 1.1;
				  proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
				  proxy_set_header Host $http_host;
				  proxy_set_header X-Forwarded-Proto https;
				  proxy_redirect off;

				 proxy_ssl_certificate		 /usr/local/etc/letsencrypt/live/XXXXXXXX/fullchain.pem;
				 proxy_ssl_certificate_key	 /usr/local/etc/letsencrypt/live/XXXXXXXX/privkey.pem;

				}
		}

 

[m0nkey_]

There is more than just HTTP and HTTPS going on. The back-end requires other ports to be open in order to operate.

If you're attempting to open the UI to the Internet, I strongly advise against it. It's safer to remotely administer FreeNAS via a VPN.

 

[seedz]

this is strange, because this is happening within my local network

I'm attempting to get a cert on a name, the finality being to make nginx answer diffently depending if the address asking is local or remote.
(local would get the freenas page, remote would only get access to the challenge for certbot - i've edited out my filters as my IPv6 range is there)
> this worked flawlessly with the old GUI

and no, don't tell me about the DNS challenge, i'm unable to do one.

 

[seedz]

so, i had the idea of checking the freenas box config files.
turns out it uses nginx as well, so i copied over this whole section into my nginx proxy file (changing the IP along the way), and it worked :

Code:

		location /websocket {
			proxy_pass http://127.0.0.1:6000/websocket;
			proxy_http_version 1.1;
			proxy_set_header X-Real-Remote-Addr $remote_addr;
			proxy_set_header X-Real-Remote-Port $remote_port;
			proxy_set_header Upgrade $http_upgrade;
			proxy_set_header Connection "upgrade";
		}

		location /websocket/shell {
			proxy_pass http://127.0.0.1:6000/_shell;
			proxy_http_version 1.1;
			proxy_set_header X-Real-Remote-Addr $remote_addr;
			proxy_set_header X-Real-Remote-Port $remote_port;
			proxy_set_header Upgrade $http_upgrade;
			proxy_set_header Connection "upgrade";
			proxy_send_timeout 7d;
			proxy_read_timeout 7d;
		}

		location /api/v2.0 {
			proxy_pass http://127.0.0.1:6000/api/v2.0;
			proxy_http_version 1.1;
			proxy_set_header X-Real-Remote-Addr $remote_addr;
			proxy_set_header X-Real-Remote-Port $remote_port;
			proxy_set_header Host $host;
			proxy_set_header X-Forwarded-For $remote_addr;
		}

		location /_download {
			proxy_pass http://127.0.0.1:6000;
			proxy_http_version 1.1;
			proxy_set_header X-Real-Remote-Addr $remote_addr;
			proxy_set_header X-Real-Remote-Port $remote_port;
		}

		location /_upload {
			proxy_pass http://127.0.0.1:6000;
			# make sure nginx does not buffer the upload and pass directly to middlewared
			proxy_request_buffering off;
			proxy_http_version 1.1;
			proxy_set_header X-Real-Remote-Addr $remote_addr;
			proxy_set_header X-Real-Remote-Port $remote_port;
		}

 

[seedz]

for those who are curious, here's how my nginx proxy file around my freenas box section looks like :

Code:

#########################################################
#													   #
#				   FNAS.EXAMPLE.COM					#
#													   #
#########################################################

		# HTTP server
		server {
				listen		  80;
				listen		  [::]:80;
				server_name	 fnas.example.com;

				include		 /usr/local/etc/nginx/letsencrypt.conf;
				location / {
						return 301 https://fnas.example.com;
				}
		}

		# HTTPS server
		server {
				listen		  443 ssl;
				listen		  [::]:443 ssl;
				server_name	 fnas.example.com;

				ssl_certificate		 /usr/local/etc/letsencrypt/live/fnas.example.com/fullchain.pem;
				ssl_certificate_key	 /usr/local/etc/letsencrypt/live/fnas.example.com/privkey.pem;

				include		 /usr/local/etc/nginx/letsencrypt.conf;

				location /websocket {
						proxy_pass http://10.0.0.1:6000/websocket;
						proxy_http_version 1.1;
						proxy_set_header X-Real-Remote-Addr $remote_addr;
						proxy_set_header X-Real-Remote-Port $remote_port;
						proxy_set_header Upgrade $http_upgrade;
						proxy_set_header Connection "upgrade";
				}

				location /websocket/shell {
						proxy_pass http://10.0.0.1:6000/_shell;
						proxy_http_version 1.1;
						proxy_set_header X-Real-Remote-Addr $remote_addr;
						proxy_set_header X-Real-Remote-Port $remote_port;
						proxy_set_header Upgrade $http_upgrade;
						proxy_set_header Connection "upgrade";
						proxy_send_timeout 7d;
						proxy_read_timeout 7d;
				}

				location /api/v2.0 {
						proxy_pass http://10.0.0.1:6000/api/v2.0;
						proxy_http_version 1.1;
						proxy_set_header X-Real-Remote-Addr $remote_addr;
						proxy_set_header X-Real-Remote-Port $remote_port;
						proxy_set_header Host $host;
						proxy_set_header X-Forwarded-For $remote_addr;
				}

				location /_download {
						proxy_pass http://10.0.0.1:6000;
						proxy_http_version 1.1;
						proxy_set_header X-Real-Remote-Addr $remote_addr;
						proxy_set_header X-Real-Remote-Port $remote_port;
				}

				location /_upload {
						proxy_pass http://10.0.0.1:6000;
						# make sure nginx does not buffer the upload and pass directly to middlewared
						proxy_request_buffering off;
						proxy_http_version 1.1;
						proxy_set_header X-Real-Remote-Addr $remote_addr;
						proxy_set_header X-Real-Remote-Port $remote_port;
				}

				location / {
				#	   deny					10.0.0.2;					 #router IP
						allow				   FFFF:FFFF:FFFF:FFFF::/65;	 #ISP assigned IPv6 range
						allow				   10.0.0.0/8;				#your IPv4 range
						deny					all;

						proxy_pass					  https://10.0.0.1:443;

						proxy_http_version			  1.1;
						proxy_set_header				X-Forwarded-For $proxy_add_x_forwarded_for;
						proxy_set_header				Host $http_host;
						proxy_set_header				X-Forwarded-Proto https;
						proxy_redirect				  off;

						proxy_ssl_certificate		   /usr/local/etc/letsencrypt/live/fnas.example.com/fullchain.pem;
						proxy_ssl_certificate_key	   /usr/local/etc/letsencrypt/live/fnas.example.com/privkey.pem;

				}
		}

this is my letsencrypt.conf file :

Code:

location /.well-known {
		root /usr/local/www/letsencrypt/;
}

certbot being configured to use /usr/local/www/letsencrypt folder as the web root for challenges

and here is my "root" ssl.conf file :

Code:

				ssl_certificate		 /usr/local/etc/letsencrypt/live/example.com/fullchain.pem;
				ssl_certificate_key	 /usr/local/etc/letsencrypt/live/example.com/privkey.pem;

				ssl_session_timeout	 1d;
				ssl_session_cache	   shared:SSL:50m;
				ssl_session_tickets	 off;
				ssl_dhparam			 /usr/local/etc/ssl/private/dhparams_4096_pem;

				ssl_protocols		   TLSv1.2;
				ssl_ciphers 'ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:DHE-RSA-AES256-SHA256';
				ssl_prefer_server_ciphers on;

				ssl_stapling			on;
				ssl_stapling_verify	 on;

I've set the whole letsencrypt folder in a non-shared ZFS zvol that i simply mount in the jails i need the certs in.

Code:

/usr/local/etc/letsencrypt jail side
/mnt/POOL/VOL/certs freenas side

As for passing the certs to freenas, i've set up a folder inside which looks like this :

Code:

/usr/local/etc/letsencrypt/FNAS # ls -la
total 18
drwxr-xr-x   2 root  wheel   4 Apr 26 22:34 .
drwxrwxr-x  11 root  wheel  12 May 21 17:55 ..
lrwxr-xr-x   1 root  wheel  36 Apr 26 22:34 fnas_example_com.crt -> ../live/fnas.example.com/fullchain.pem
lrwxr-xr-x   1 root  wheel  34 Apr 26 22:34 fnas_example_com.key -> ../live/fnas.example.com/privkey.pem

and of course, in FreeNas GUI, i've set up this tunable :

Code:

variable : SSLDIR
value : /mnt/POOL/VOL/certs/FNAS
type : rc.conf

 

[Spencer Skinner]

I gave this a try and the issue ive run into is that when i try and connect to the reverse proxy address it just comes up withe a pulsating freenas logo in the corner and then nothing else

 

[meku]

In my experience only the websockets need to be specified in the reverse proxy config. Manually proxying the other endpoints seems redundant to me at best.

Aside from the SSL options the following should be enough to create a reverse SSL proxy of https://fnas-ssl.example.com => http://fnas.example.com

Code:

    server {
        listen       443 ssl;
        server_name ~^(fnas-ssl\.)(?<domain>.+)$;
        
        location / {
            proxy_pass http://fnas.$domain:80;
        }
        location /websocket {
            # this is required for ui
            proxy_pass http://fnas.$domain/websocket;
            proxy_http_version 1.1;
            proxy_set_header Upgrade $http_upgrade;
            proxy_set_header Connection "upgrade";
            proxy_buffering off;
            proxy_read_timeout 86400;
        }
        location /websocket/shell {
            # this is required for VM serial consoles
            proxy_pass http://fnas.$domain/websocket/shell;
            proxy_http_version 1.1;
            proxy_set_header Upgrade $http_upgrade;
            proxy_set_header Connection "upgrade";
            proxy_buffering off;
            proxy_read_timeout 86400;
        }
    }

 

[JustinClift]

and no, don't tell me about the DNS challenge, i'm unable to do one.

Out of curiosity, is that because you don't have write access to your DNS records (eg can't create a TXT entry), or is it something else?

Asking because that's something I hit a while back too, as my DNS provider doesn't have an API which can be scripted by certbot to automatically do the update.

This is an (undocumented?) `manual` workaround for that though, which I use. It basically spits out the TXT record info needed, and I manually add that to the DNS records via webui. Certbot then verifies things + generates the certificate.

 

[jonmayer]

Thank you for this! I had no idea how to forward to websocket connection needed. This worked perfectly.

 

[seedz]

Out of curiosity, is that because you don't have write access to your DNS records (eg can't create a TXT entry), or is it something else?

Asking because that's something I hit a while back too, as my DNS provider doesn't have an API which can be scripted by certbot to automatically do the update.

This is an (undocumented?) `manual` workaround for that though, which I use. It basically spits out the TXT record info needed, and I manually add that to the DNS records via webui. Certbot then verifies things + generates the certificate.

yup, i do not have access to my DNS in any way.
It's a corporate-like DNS, and i'm part of a sub-branch.
So if i want a change made, i have to go through a 2 to 3 weeks delay, which is too much for this kind of thing :p

The configuration above is my own at home, and i could maybe use DNS side verification but I won't, since it enables me to test things before deploying them at work and to switch between DNS / registrars at will without any hassle !
I've used this principle to deploy FreeNAS as a redundant* back up solution for Unitrends and script file-based back ups, but also for NextCloud, BigBlueButton, WordPress, reverse proxy with cache / https / IP filtering, VPN access

*if you're curious, main back up is based on a quad 10Tb drive setup on striped mirrors with an access to FiberChannel storages used by ESX, with a back up off site on a quad 8Tb drive RAIDz1 setup via a slower GB connexion, with a once a week replication between the two.

 

[KevDog]

Just curious -- I don't know much about websockets, however there are ws and wss connections with wss implying secure. Is there a way to proxy_pass to an https connection rather than http connection? I'm not sure this would even work since clearly your proxy would be functioning as a MITM.

 

[glauco]

Thank you, this is a brilliant idea!
The /websocket location was enough to get a working web UI, but to get more stuff working, you need to copy over more locations.

so, i had the idea of checking the freenas box config files.
turns out it uses nginx as well, so i copied over this whole section into my nginx proxy file (changing the IP along the way), and it worked :

 

[raoul]

I have a slightly similar issue with the reports dashboard when I click on "Reporting", form the web dev console i get:

Code:

Mixed Content: The page at 'https://nas.mydomain/reportsdashboard/cpu' was loaded over HTTPS, but requested an insecure XMLHttpRequest endpoint 'http://nas.mydomain/ui/'. This request has been blocked; the content must be served over HTTPS.
(anonymous) @ polyfills.8e04786abe2dac2d705d.js:1
D.a.<computed> @ polyfills.8e04786abe2dac2d705d.js:1
...
Show 53 more frames
main.576b904c8f53bfe85d8a.js:1 ERROR DOMException: Failed to execute 'send' on 'XMLHttpRequest': Failed to load 'https://nas.mydomain/'.
    at https://nas.mydomain/ui/polyfills.8e04786abe2dac2d705d.js:1:43541

I have a reverse proxy with nginx.

Connecting with http using the IP the charts are plotted as expected.

 

[raoul]

I answered myself after searching last night and now. The trick is to enable the redirect directive in the main location:

Code:

location / {
                include snippets/proxy-params.conf;
                proxy_pass http://freenas.local/;
                proxy_redirect http:// https://;
        }