Resource icon

Reverse Proxy using Caddy (with optional automatic TLS)

NasKar

Guru
Joined
Jan 8, 2016
Messages
739
I've been trying to get my reverse proxy working with my nextcloud install. Tried to use HAProxy in pfsense but couldn't get it going so I'm trying to go back to caddy RP in a FN jail. All was working with this setup before my update to 12.2U3. Can't seem to get it working and I think the issue is with my firewall.
Theorectically if my Caddyfile worked before that same file should work with a fresh install of the RP jail.

Setup:
My pfsense has a NAT rules for ports 80 and 443 directed to my caddy RP jail. Those autocreated rules are above my default block rules on my WAN interface.
The caddy.log file show
Code:
"error":"authorization failed: HTTP 400 urn:ietf:params:acme:error:connection - Timeout during connect (likely firewall problem)","order":"https://acme-staging-v02.api.letsencrypt.org/acme/order/19380616/45879732","attempt":2,"max_attempts":3}

So I presume that the jail cannot get out on ports 80 and 443.

Here is my Caddyfile
Code:
{
  acme_ca https://acme-staging-v02.api.letsencrypt.org/directory
  email mydomain@gmail.com
}

mydomain.com {
   root * /usr/local/www/html/
   file_server
   header {
      enable HSTS
      Strict-Transport-Security max-age=31536000;
   }
}
cloud.mydomain.com {

  encode gzip
  reverse_proxy http://192.168.5.81
  header {
     enable HSTS
     Strict-Transport-Security max-age=31536000;
        }
}

collabora.mydomain.com {
   encode gzip

   @collabora {
     path /loleaflet/*          # Loleaflet is the client part of LibreOffice Online
     path /hosting/discovery    # WOPI discovery URL
     path /hosting/capabilities # Show capabilities as json
     path /lool/*               # Main websocket, uploads/downloads, presentations
   }
  reverse_proxy @collabora http://192.168.5.89:9980
}


I'm stumped and any help even to narrow down where to look would be greatly appreciated. Maybe the same issue preventing the HAProxy from working.
 
Joined
Jan 4, 2014
Messages
1,644
Last edited:

NasKar

Guru
Joined
Jan 8, 2016
Messages
739
I think I finally found the issue with the firewall. It was my client VPN. When I turn it off I can access nextcloud from cell phone. It's odd because I'm almost positive that it used to work with the VPN on before I upgraded the jail to 12.2 Release. Maybe the VPN company made a change. Had the VPN setup so I would only work on certain computers and turn off for certain websites. Will have to do more research.

Is this warning something to be concerned about?
Code:
service caddy validate
2021/05/06 01:05:23.823    INFO    using provided configuration    {"config_file": "/usr/local/www/Caddyfile", "config_adapter": "caddyfile"}
2021/05/06 01:05:23.826    WARN    input is not formatted with 'caddy fmt'    {"adapter": "caddyfile", "file": "/usr/local/www/Caddyfile", "line": 2}
2021/05/06 01:05:23.826    INFO    tls.cache.maintenance    started background certificate maintenance    {"cache": "0xc0003aa000"}
2021/05/06 01:05:23.826    INFO    http    server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS    {"server_name": "srv0", "https_port": 443}
2021/05/06 01:05:23.826    INFO    http    enabling automatic HTTP->HTTPS redirects    {"server_name": "srv0"}
2021/05/06 01:05:23.827    INFO    tls.cache.maintenance    stopped background certificate maintenance    {"cache": "0xc0003aa000"}


Also seeing these errors in the caddy.log after it says certificate obtained succesfully ....
Code:
{"level":"error","ts":1620252254.2255273,"logger":"http.handlers.reverse_proxy","msg":"aborting with incomplete response","error":"http2: stream closed"}
{"level":"error","ts":1620252254.22561,"logger":"http.handlers.reverse_proxy","msg":"aborting with incomplete response","error":"http2: stream closed"}
{"level":"error","ts":1620252254.2256765,"logger":"http.handlers.reverse_proxy","msg":"aborting with incomplete response","error":"http2: stream closed"}
{"level":"error","ts":1620252254.2256844,"logger":"http.handlers.reverse_proxy","msg":"aborting with incomplete response","error":"http2: stream closed"}
{"level":"error","ts":1620252254.544693,"logger":"http.handlers.reverse_proxy","msg":"aborting with incomplete response","error":"context canceled"}
{"level":"error","ts":1620252255.4778025,"logger":"http.handlers.reverse_proxy","msg":"aborting with incomplete response","error":"context canceled"}
{"level":"error","ts":1620252257.3815832,"logger":"http.handlers.reverse_proxy","msg":"aborting with incomplete response","error":"http2: stream closed"}
{"level":"error","ts":1620252257.3816237,"logger":"http.handlers.reverse_proxy","msg":"aborting with incomplete response","error":"http2: stream closed"}
{"level":"error","ts":1620263094.5911913,"logger":"http.handlers.reverse_proxy","msg":"aborting with incomplete response","error":"client disconnected"}
 
Joined
Jan 4, 2014
Messages
1,644
Is this warning something to be concerned about?
Nope. https://caddy.community/t/caddy-2-4-0-beta-1-is-now-available/11519/29

Also seeing these errors in the caddy.log after it says certificate obtained succesfully ....
Not sure, but...
Code:
   header {
      enable HSTS
      Strict-Transport-Security max-age=31536000;
   }

I can understand the application of this for cloud.mydomain.com (Nextcloud is expecting this), but not domain.com. Try commenting it out there and see what happens when you restart caddy.

Failing that, maybe submit a support ticket on the Caddy forum.
 

NasKar

Guru
Joined
Jan 8, 2016
Messages
739
Nope. https://caddy.community/t/caddy-2-4-0-beta-1-is-now-available/11519/29


Not sure, but...
Code:
   header {
      enable HSTS
      Strict-Transport-Security max-age=31536000;
   }

I can understand the application of this for cloud.mydomain.com (Nextcloud is expecting this), but not domain.com. Try commenting it out there and see what happens when you restart caddy.

Failing that, maybe submit a support ticket on the Caddy forum.
Commenting the STS fixed the errors and caddy fmt —overwrite fixed the warning.
 

Electr0

Dabbler
Joined
Dec 18, 2020
Messages
47
Why is nothing ever simple?

I tried to run this script but first got this error:
Code:
Stopped caddy due to VNET failure
Failed to create jail

I then tried to create the jail manually and run through the commands in the script.

Next, the go install github.com/caddyserver/xcaddy/cmd/xcaddy@latest command failed with /bin/sh: go: not found - so I installed GO.

I've run the command again, but now I get the error:
Code:
go install github.com/caddyserver/xcaddy/cmd/xcaddy@latest:
module github.com/caddyserver/xcaddy/cmd/xcaddy:
Get "https://proxy.golang.org/github.com/caddyserver/xcaddy/cmd/xcaddy/@v/list":
x509: certificate signed by unknown authority

Now I'm stuck...
What does this mean, and how do I fix it?

Do I somehow need to copy the list of CAs from the main TrueNAS install into the Caddy Jail, so that GO can read the Cert in the GitHub repo?

------

Okay, so I managed to fix the certificate error by running pkg install ca_root_nss.

Now I've reached the end of the script and have everything setup, including the following line in the Caddyfile:
Code:
*:80 {
    root * /usr/local/www/html
    file_server
    reverse_proxy /sonarr* 192.168.0.200:8989
    reverse_proxy /radarr* 192.168.0.200:9898
    reverse_proxy /nzbget* 192.168.0.200:6789
}


However anything I append after my TrueNAS IP (/sonarr, /radarr, /nzbet) just continues to resolve to the TrueNAS Web UI.

When navigating to 192.168.0.199:2020, it resolves to the "Hello, world!" page, so Caddy appears to be working correctly.
 
Last edited:

sretalla

Powered by Neutrality
Moderator
Joined
Jan 1, 2016
Messages
9,702
You probably need to install ca_root_nss for that.

I see that the script wants to install "nano","bash","go","git", so you should probably make sure all of them are there too.
 

Jailer

Not strong, but bad
Joined
Sep 12, 2014
Messages
4,974
However anything I append after my TrueNAS IP (/sonarr, /radarr, /nzbet) just continues to resolve to the TrueNAS Web UI.
This doesn't make any sense. Unless you installed them on the base system (which you shouldn't) you need to point them to the jails IP where they are installed and not the TrueNAS web UI IP.
 

danb35

Hall of Famer
Joined
Aug 16, 2011
Messages
15,458
However anything I append after my TrueNAS IP (/sonarr, /radarr, /nzbet) just continues to resolve to the TrueNAS Web UI.
That's to be expected--the Caddy jail should be running on its own IP.
When navigating to 192.168.0.199:2020, it resolves to the "Hello, world!" page, so Caddy appears to be working correctly.
So either it isn't using the Caddyfile you've prepared (service caddy reload should do that) or there's some kind of IP conflict (like you're somehow using the same IP for the jail as you're using for your TrueNAS box).
 

Electr0

Dabbler
Joined
Dec 18, 2020
Messages
47
That's to be expected--the Caddy jail should be running on its own IP.
Okay, I'm confused. I thought the whole point of running a reverse proxy was so I could point my browser at [TrueNAS_IP]/sonarr and it would resolve to my Sonarr Web UI, exactly as if I'd typed [TrueNAS_IP]:8989?

I am currently using the following IPs:
TrueNAS : 192.168.0.200
Caddy Jail : 192.168.0.199

If I try to access 192.168.0.199/sonarr I'm just met with a blank white page.

Caddy is using the correct file, because if I comment out, or uncomment the following lines:
Code:
:2020 {                                                        
        respond "Hello, world!"                                
} 

And then try to access 192.168.0.199:2020, "Hello, world!" is either not displayed or displayed, respectively.
 
Last edited:

danb35

Hall of Famer
Joined
Aug 16, 2011
Messages
15,458
I thought the whole point of running a reverse proxy was so I could point my browser at [TrueNAS_IP]/sonarr
No, you missed the point. It can't do that on the TrueNAS IP, as TrueNAS already has its own web server listening on that IP. It instead would do that using whatever IP (or better yet, hostname) belongs to the Caddy server.
If I try to access 192.168.0.199/sonarr I'm just met with a blank white page.
I guess the first thing to do is make sure it's actually using the Caddyfile you've entered. In the jail, if you create /usr/local/www/html/test.txt, can you browse to it at 192.168.0.199/test.txt?
 

sretalla

Powered by Neutrality
Moderator
Joined
Jan 1, 2016
Messages
9,702
If I try to access 192.168.0.199/sonarr I'm just met with a blank white page.
Seems to me that your caddy config is also pointing to the host and not the IP of the jails for sonarr, etc.
 

danb35

Hall of Famer
Joined
Aug 16, 2011
Messages
15,458
Seems to me that your caddy config is also pointing to the host and not the IP of the jails for sonarr, etc.
I expect that's because TrueNAS defaults to using NAT for plugins. Why anyone thought that was a good idea is a mystery to me.
 

Electr0

Dabbler
Joined
Dec 18, 2020
Messages
47
No, you missed the point. It can't do that on the TrueNAS IP, as TrueNAS already has its own web server listening on that IP. It instead would do that using whatever IP (or better yet, hostname) belongs to the Caddy server.
Okay, understood.

I guess the first thing to do is make sure it's actually using the Caddyfile you've entered. In the jail, if you create /usr/local/www/html/test.txt, can you browse to it at 192.168.0.199/test.txt?
Hmmm... I edited my post and it seems it's using the correct Caddy file. But the directory "html" doesn't exist in "www".
I created it and put a test.txt file in there, but i can't access it from 192.168.0.199/text.txt

Seems to me that your caddy config is also pointing to the host and not the IP of the jails for sonarr, etc.
Ohh... My Jails don't have external IPs, only internal IP that are in the form of 172.16.0.2/30.
Hence why I can access them via their port number, appended to my TrueNAS IP.

So in order for all this to work properly, each Jail (Sonarr, Radarr, NZBget) should have it's own (TrueNAS) external IP address, like 192.168.0.xxx?
 

danb35

Hall of Famer
Joined
Aug 16, 2011
Messages
15,458
I created it and put a test.txt file in there, but i can't access it from 192.168.0.199/text.txt
What does "can't access" mean? What happens when you try?
So in order for all this to work properly, each Jail (Sonarr, Radarr, NZBget) should have it's own (TrueNAS) external IP address, like 192.168.0.xxx?
Uncertain at this point; I'm not aware that it's been tried. I don't know of any reason that it shouldn't work the way you're using it, but (again) I'm not aware that anyone's tried as yet.
 

Electr0

Dabbler
Joined
Dec 18, 2020
Messages
47
What does "can't access" mean? What happens when you try?
Sorry, I should have clarified: It just "loads" a blank white page, even though I entered text in the test.txt file

Could it be a permissions issue?

I just had a look in the caddy file from the includes directory that's supposed to go into /usr/local/etc/rc.d/:
Code:
# Defaults                                                      
: ${caddy_enable:=NO}                                          
: ${caddy_adapter:=caddyfile}                                  
: ${caddy_config:=/usr/local/etc/caddy/Caddyfile}              
: ${caddy_directory:=/var/db/caddy}                            
: ${caddy_extra_flags:=""}                                      
: ${caddy_logdir:="/var/log/${name}"}                          
: ${caddy_logfile:="${caddy_logdir}/${name}.log"}              
: ${caddy_user:="root"}                                        
: ${caddy_group:="wheel"}                                      
                                                               
# Config and base directories                                  
: ${XDG_CONFIG_HOME:="${caddy_directory}/config"}
: ${XDG_DATA_HOME:="${caddy_directory}/data"}
export XDG_CONFIG_HOME XDG_DATA_HOME


It seems like : ${caddy_directory:=/var/db/caddy} is pointing to the wrong place?
Shouldn't that be /usr/local/www
 
Last edited:

sretalla

Powered by Neutrality
Moderator
Joined
Jan 1, 2016
Messages
9,702
Ohh... My Jails don't have external IPs, only internal IP that are in the form of 172.16.0.2/30.
Hence why I can access them via their port number, appended to my TrueNAS IP.

So in order for all this to work properly, each Jail (Sonarr, Radarr, NZBget) should have it's own (TrueNAS) external IP address, like 192.168.0.xxx?
Just like with docker containers, if you want to work between jails, you need to use the 172.16.0.0/30 addresses, not the host IP.

The biggest problem with that is that they can't really be made static, so resolving it with some kind of nameserver is prefereble, but I don't see much material about that out there.

The ultimate best option is to have the jails use VNET and use their 192.168... addresses so everything works like you would expect it to.
 

danb35

Hall of Famer
Joined
Aug 16, 2011
Messages
15,458
So in order for all this to work properly, each Jail (Sonarr, Radarr, NZBget) should have it's own (TrueNAS) external IP address, like 192.168.0.xxx?
OK, I've now been able to confirm this is not required (VMs are awfully handy for testing)--here's my Caddyfile:
Code:
:80 {
    root * /usr/local/www/html
    file_server
    reverse_proxy /sonarr* 192.168.1.121:8989
}

(The TrueNAS IP is 192.168.1.121; Sonarr installed using the plugin with NAT), and here's Sonarr through the Caddy reverse proxy:
1633620859206.png

...and here's the setting in Sonarr you were missing in order to make it work:
1633620894848.png

(it's under Settings -> General). The other applications should have a similar setting.
 

danb35

Hall of Famer
Joined
Aug 16, 2011
Messages
15,458
It seems like : ${caddy_directory:=/var/db/caddy} is pointing to the wrong place?
Shouldn't that be /usr/local/www
No, because (1) caddy_directory doesn't control where Caddy looks for the Caddyfile, and (2) the script runs sysrc caddy_config="/usr/local/www/Caddyfile" to tell it to look there.
Just like with docker containers, if you want to work between jails, you need to use the 172.16.0.0/30 addresses, not the host IP.
This does not appear to be correct, as shown above.
 

Electr0

Dabbler
Joined
Dec 18, 2020
Messages
47
Okay, I changed that setting in Sonarr that you posted but there must be something wrong with my setup, somehow.

TrueNAS IP: 192.168.0.200
Caddy IP: 192.168.0.199

http://192.168.0.200:8989 | Loads Sonarr Web UI (But now redirects to http://192.168.0.200:8989/sonarr)
http://192.168.0.200:8989/sonarr | Loads Sonarr Web UI (Works as it should - default setup)
http://192.168.0.200/sonarr | Redirects to TrueNAS Web UI

http://192.168.0.199/sonarr | Blank white page

http://192.168.0.199:2020/ | "Hello, world!"

http://192.168.0.199/test.txt | Blank white page (Should download a .txt file or show it's conents)
 
Top