SOLVED Hosting subdomains from different jails

toxikat

Dabbler
Joined
Nov 3, 2022
Messages
27
Hello, I have a high-level web hosting question which I am wondering.

I currently have a domain, call it "b.c" from google domains. I have my nextcloud plugin running in its jail, and I have the dynamic DNS set to "my.b.c" as an A record. This is working fine.

In a separate jail, I have jellyfin running. I can access it locally via "http://192.168.1.101:8096/", but I'd like to be able to access it remotely via "watch.b.c". I used nginx for reverse proxying before (correct me if the terminology is wrong) when going through the motions for nextcloud by referencing this tutorial by sysadmin102.

I tried to do the same thing again but realized that my server can only have 1 dynamic dns, which makes sense to me.

I'm wondering how to accomplish my goal now. I was thinking that I would need to do something along the lines of changing my dynamic DNS to be "b.c" and then making CNAME or A records (not sure which) for my NC and jellyfin? So maybe I'd need to make a CNAME record "my.b.c" for NC and "watch.b.c" for jellyfin, but I don't know how to proceed from there since they'll be in different jails.

Thanks in advance to the community.
 

danb35

Hall of Famer
Joined
Aug 16, 2011
Messages
15,504
You have two distinct issues:
  • How to get both names to resolve to the correct IP address, and
  • How to send traffic for each name to the right place
The first is a DNS issue, and your instincts are pretty close. You'd set an A record for something (doesn't much matter which--the base domain, my.domain, watch.domain, whatever), and have that automatically updated using dynamic DNS. You'd then set CNAME records for the other names pointing to that one.

The second issue is addressed using a reverse proxy--where to run it and which software to use are both up to you; my answer is to run it on my OPNsense router and to use Caddy as the software. If your router does or can support running as a reverse proxy, I'd favor doing that. Otherwise, you can run it in a jail under TrueNAS, in a VM under TrueNAS or elsewhere, or even on separate dedicated hardware. For software, again, I like Caddy (here's how to install and use that in a jail: https://www.truenas.com/community/r...-using-caddy-with-optional-automatic-tls.114/), largely because it's so simple to configure and handles all the TLS stuff automatically. Other options would include Traefik, Nginx (optionally with Nginx Proxy Manager, if running under Linux), Apache, and HAProxy.

If you're running the reverse proxy somewhere other than on your router, you'd need to forward ports 80 and 443 to it. Then configure it appropriately and you should be good to go.
 

toxikat

Dabbler
Joined
Nov 3, 2022
Messages
27
Hey @danb35, thanks for the answer. I am researching routers to buy (my current one is provided by the ISP. It's got all the basic functionality and can hit the advertised gigabit internet speeds, but some annoying things about it include not supporting loopback resolution and, well, I guess I can't run a reverse proxy on it).

In the meantime, I'd like to get it installed on my nas (in a jail as you mentioned).

The first is a DNS issue, and your instincts are pretty close. You'd set an A record for something (doesn't much matter which--the base domain, my.domain, watch.domain, whatever), and have that automatically updated using dynamic DNS. You'd then set CNAME records for the other names pointing to that one.

To clarify, I can do something like this?
- "b.c" DDNS
- "my.b.c" CNAME
- "watch.b.c" CNAME.

I'm currently using nginx for my nextcloud jail. Is it possible to mix and match nginx and caddy, or is it better to just use one?

Anyways, I'm still a bit confused about how to actually configure nginx in a jail to talk to the other jails, per se. I thought they were isolated from each other. Could I get an example or some reference?
 

danb35

Hall of Famer
Joined
Aug 16, 2011
Messages
15,504
I am researching routers to buy
I'm a big fan of mini-PCs like these:
...as you can then run whatever software you want on it. I favor OPNsense; pfSense is another big name. If you like a CLI, VyOS is a possibility. Mikrotik's RouterOS runs on x86 too. Or there are integrated router/server solutions like Nethserver (which I also use, but not as a router). The hardware's relatively future-proof, and you have plenty of options for software, lots of them free.

To clarify, I can do something like this?
- "b.c" DDNS
- "my.b.c" CNAME
- "watch.b.c" CNAME.
Correct. The CNAME in each case would be to b.c.
I'm currently using nginx for my nextcloud jail. Is it possible to mix and match nginx and caddy, or is it better to just use one?
There's no inherent reason it would be a problem to put a Nginx server behind a Caddy proxy, or vice versa.
Anyways, I'm still a bit confused about how to actually configure nginx in a jail to talk to the other jails, per se. I thought they were isolated from each other.
Jails are mostly isolated from each other, but they do ordinarily share a common network, and it's over that network that they'd communicate. Setting Caddy up for this is very straightforward--to proxy to the two hosts you mention, you'd set up its config file (or "Caddyfile") like this:
Code:
my.b.c {
    reverse_proxy 10.0.0.10 # Internal IP of your Nextcloud jail
}

watch.b.c {
    reverse_proxy 10.0.0.20:8096 # Internal IP and port of your Jellyfin jail
}

When a client from the public Internet wants to go to watch.b.c, it queries DNS and gets your public IP address. When that client then hits your router, the router forwards ports 80/443 to the Caddy jail. When the connection hits your Caddy installation, Caddy sees that the client wants to go to watch.b.c, and sends the request to that jail.
 

toxikat

Dabbler
Joined
Nov 3, 2022
Messages
27
It's been about a week and I've had a chance to give this another shot.

I made some progress after some initial failed experimentation where I've got the "end user experience" to the same as before:
`my.b.c` resolves to nextcloud correctly.
`watch.b.c` fails with the following error:
1668386708512.png



My current setup:
I've been following a very helpful guide I found on this forums about Caddy and Nextcloud setup scripts. I installed caddy with the cloudflare DNS plugin.

After reading the README on the github, I figured that it's best to move to use CloudFlare as the DNS provider, since it was the best supported one. I am currently on Google domains which I previously used for nextcloud and Dynamic DNS. I was able move to use Cloudflare in front of the domain. Here is my cloudflare config now:
1668387079604.png


Below is my caddy Caddyfile
Code:
{
        acme_ca https://acme-staging-v02.api.letsencrypt.org/directory
        email myemail@gmail.com
}

:2020 {
        respond "hi"
}

my.b.c {
        #tls {
        #       dns cloudflare tbh not sure what token to use here? When I visit this site on external network, I see that https lock is used so I guess I omit this ....
        #}
        reverse_proxy 192.168.1.70 # nextcloud
}

nya.b.c {
        respond "test"
}

watch.b.c {
        reverse_proxy 192.168.1.101:8096 # jellyfin
}



I'm not sure what I need to look into to get further subdomains working. I tried looking into others with the same 525 error on the caddy forums many seemed to be docker related, which I don't think applies here. Some further guidance would be appreciated, thanks a ton!
 

danb35

Hall of Famer
Joined
Aug 16, 2011
Messages
15,504
I'm not sure what I need to look into to get further subdomains working.
If you want to be able to access watch. (or anything else) from outside your LAN, you'll need to forward ports 80 and 443 to your Caddy jail--and if you do that, there really isn't any reason to be using Cloudflare for your DNS, as you'd be able to get certs without DNS validation.
 

toxikat

Dabbler
Joined
Nov 3, 2022
Messages
27
I might be getting it a bit confused in my head then. I can forward ports 80 and 443 to my caddy jail (192.168.1.199). I must've gotten confused and carried away getting cloudflare set up when following the caddy and nextcloud scripts.

Then in that case, would it be reasonable to do something like:
Google domains:
b.c DDNS
my.b.cCNAME
watch.b.cCNAME
nya.b.cCNAME

In Truenas, set the DDNS to be b.c.

Would I need to make modifications with the Caddyfile? I'm having some issues with too many redirects.
 
Last edited:

Jailer

Not strong, but bad
Joined
Sep 12, 2014
Messages
4,977
All my subdomains are set up as A records with cloudflare with the content column being my public IP address. I have a static public IP so I don't know how you would set that up with a dynamic one.

The only CNAME MX and TXT records I have are for mailgun for the email service I use for a forum.
 

toxikat

Dabbler
Joined
Nov 3, 2022
Messages
27
I couldn't get it working properly with just Google domains. I think I'm probably missing a key piece of info.

I switched back to Cloudflare as previously described. I got my jellyfin (watch.b.c) working and also my nya.b.c working on an external network (phone on data, so not using my router).

However, on an external network, nextcloud doesn't work. I get a 502 bad gateway from cloudflare indicating my.b.c has a host error.
 

toxikat

Dabbler
Joined
Nov 3, 2022
Messages
27
I'm having another look at this issue this weekend. I realized I left out a bit of information. As it turns out, my nextcloud scripted installation (mentioned above) autoconfigured a Caddyfile in that nextcloud jail. Here it is:


Code:
root@nextcloud:/usr/local/www # cat Caddyfile
{
        # debug
        #acme_ca https://acme-staging-v02.api.letsencrypt.org/directory
        email email@gmail.com
        # default_sni my.b.c
}

my.b.c {
        root * /usr/local/www/nextcloud
        file_server
        log {
                output file /var/log/my.b.c.log
        }

        php_fastcgi 127.0.0.1:9000 {
                env front_controller_active true
        }

        tls {
                dns cloudflare ksZTxdpJ2howALeeSii8tL9_wKmTyuOdK2JGErLc5
        }

        header {
                # enable HSTS
                Strict-Transport-Security max-age=31536000;
        }

        # client support (e.g. os x calendar / contacts)
        redir /.well-known/carddav /remote.php/dav 301
        redir /.well-known/caldav /remote.php/dav 301
        redir /.well-known/webfinger /index.php/.well-known/webfinger 301
        redir /.well-known/nodeinfo /index.php/.well-known/nodeinfo 301

        # .htaccess / data / config / ... shouldn't be accessible from outside
        @forbidden {
                path /.htaccess
                path /data/*
                path /config/*
                path /db_structure
                path /.xml
                path /README
                path /3rdparty/*
                path /lib/*
                path /templates/*
                path /occ
                path /console.php
        }

        respond @forbidden 404
}


other.b.c {
#       tls {
#               dns cloudflare
#       }

        respond "please work"
}


I have verified with service caddy status that it is running as well.

Still no progress on this issue unfortunately. I'm wondering if this "inner" caddyfile is maybe conflicting with the outer one in the Caddy jail?
 

toxikat

Dabbler
Joined
Nov 3, 2022
Messages
27
Bumping this since I found some more information after digging in my caddy jail. I was able to run my caddy with `caddy run debug` and saw this error from my 502:

Code:
2022/11/26 06:01:28.658 ERROR   http.log.error.log0     x509: cannot validate certificate for 192.168.1.70 because it doesn't contain any IP SANs       {"request": {"remote_ip": "external.ip.redacted", "remote_port": "32258", "proto": "HTTP/2.0", "method": "GET", "host": "my.b.c", "uri": "/favicon.ico", "headers": {"Sec-Fetch-Site": ["same-origin"], "Sec-Fetch-Mode": ["no-cors"], "Accept-Language": ["en-US,en;q=0.9"], "Cf-Connecting-Ip": ["205.251.233.53"], "Dnt": ["1"], "Sec-Ch-Ua-Mobile": ["?0"], "Cf-Ray": ["770068e20ef7ec58-SEA"], "Sec-Ch-Ua": ["\"Chromium\";v=\"106\", \"Google Chrome\";v=\"106\", \"Not;A=Brand\";v=\"99\""], "User-Agent": ["Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/106.0.0.0 Safari/537.36"], "Sec-Fetch-Dest": ["image"], "Cookie": [], "Accept-Encoding": ["gzip"], "X-Forwarded-For": ["205.251.233.53"], "X-Forwarded-Proto": ["https"], "Accept": ["image/avif,image/webp,image/apng,image/svg+xml,image/*,*/*;q=0.8"], "Cf-Visitor": ["{\"scheme\":\"https\"}"], "Sec-Ch-Ua-Platform": ["\"macOS\""], "Referer": ["https://my.b.c/"], "Cf-Ipcountry": ["US"], "Cdn-Loop": ["cloudflare"]}, "tls": {"resumed": false, "version": 772, "cipher_suite": 4865, "proto": "h2", "server_name": "my.b.c"}}, "duration": 0.003838388, "status": 502, "err_id": "u7vwrg2ck", "err_trace": "reverseproxy.statusError (reverseproxy.go:1272)"}


For reference, the ip 192.168.1.70 is my nextcloud jail.
 

danb35

Hall of Famer
Joined
Aug 16, 2011
Messages
15,504
For reference, the ip 192.168.1.70 is my nextcloud jail.
...and let me guess, your proxy block looks like this:
Code:
my.b.c {
  reverse_proxy https://192.168.1.70
}


What Caddy's telling you here is that it's trying to connect securely to 192.168.1.70, but that host presented a certificate that doesn't include 192.168.1.70, so therefore it won't continue. There are a couple of ways to address this:
  • (preferred) Make my.b.c resolve to 192.168.1.70 on your LAN, or at least within this jail
    • I'd do this by setting a DNS host override on my local DNS server, which is also my OPNsense router. You can do similarly with Pi-Hole if you're running that, or perhaps with your router.
    • As an alternative, you can edit /etc/hosts inside the jail to make this association
  • (less-preferred) Tell Caddy to ignore the cert validation error using the tls_insecure_skip_verify option. See https://caddyserver.com/docs/caddyfile/directives/reverse_proxy#the-http-transport for more on this.
 

toxikat

Dabbler
Joined
Nov 3, 2022
Messages
27
Hi Danb35, I really appreciate the reply.

I tried the suggestion to edit the `/etc/hosts` file on my caddy jail but it did not fix the issue post system reboot. Here is that file:
Code:
root@caddy:/usr/local/www # cat /etc/hosts
# $FreeBSD$
#
# Host Database
#
# This file should contain the addresses and aliases for local hosts that
# share this file.  Replace 'my.domain' below with the domainname of your
# machine.
#
# In the presence of the domain name service or NIS, this file may
# not be consulted at all; see /etc/nsswitch.conf for the resolution order.
#
#
::1                     localhost localhost.my.domain
127.0.0.1               localhost localhost.my.domain caddy
#
# Imaginary network.
#10.0.0.2               myname.my.domain myname
#10.0.0.3               myfriend.my.domain myfriend
#
# According to RFC 1918, you can use the following IP networks for
# private nets which will never be connected to the Internet:
#
#       10.0.0.0        -   10.255.255.255
#       172.16.0.0      -   172.31.255.255
#       192.168.0.0     -   192.168.255.255
#
# In case you want to be able to connect to the Internet, you need
# real official assigned numbers.  Do not try to invent your own network
# numbers but instead get one from your network provider (if any) or
# from your regional registry (ARIN, APNIC, LACNIC, RIPE NCC, or AfriNIC.)
#
192.168.1.199   caddy

192.168.1.70    my.b.c


Your guess is correct, my Caddyfile was shown above, but I will show it again:
Caddyfile for caddy jail:

Code:
root@caddy:/usr/local/www # cat Caddyfile
{
        #acme_ca https://acme-staging-v02.api.letsencrypt.org/directory
        #acme_ca https://acme-v02.api.letsencrypt.org/directory
        email myemail@gmail.com
}

:2020 {
        respond "uwu nya"
}

#bruh.b.c {
#       #tls {
#       #       dns cloudflare xxx
#       #}
#       reverse_proxy https://192.168.1.70 # nextcloud
#
#       log {
#
#               output file "test.log"
#       }
#}


my.b.c {
        log {
                output file /var/log/caddy/nextcloud-access.log
        }

        reverse_proxy https://192.168.1.70 # nextcloud test please work
}


nya.b.c {
        respond "pomf pomf boom"
        #reverse_proxy http://192.168.1.95:9091 # Transmission
}

whatthedogdoin.b.c {
        tls {
                dns cloudflare xxx
        }

        redir https://www.youtube.com/watch?v=dQw4w9WgXcQ
}

watch.b.c {
        reverse_proxy 192.168.1.101:8096 # jellyfin
}



my Caddyfile in nextcloud:
Code:
root@nextcloud:/usr/local/www # cat Caddyfile
{
        # debug
        #acme_ca https://acme-staging-v02.api.letsencrypt.org/directory
        email myemail@gmail.com
        default_sni my.b.c
}

my.b.c {
        root * /usr/local/www/nextcloud
        file_server
        log {
                output file /var/log/my.b.c.log
        }

        php_fastcgi 127.0.0.1:9000 {
                env front_controller_active true
        }

        tls {
#               dns cloudflare xxx
                dns cloudflare xxxdifferentone
        }

        header {
                # enable HSTS
                Strict-Transport-Security max-age=31536000;
        }

        # client support (e.g. os x calendar / contacts)
        redir /.well-known/carddav /remote.php/dav 301
        redir /.well-known/caldav /remote.php/dav 301
        redir /.well-known/webfinger /index.php/.well-known/webfinger 301
        redir /.well-known/nodeinfo /index.php/.well-known/nodeinfo 301

        # .htaccess / data / config / ... shouldn't be accessible from outside
        @forbidden {
                path /.htaccess
                path /data/*
                path /config/*
                path /db_structure
                path /.xml
                path /README
                path /3rdparty/*
                path /lib/*
                path /templates/*
                path /occ
                path /console.php
        }

        respond @forbidden 404
}


#other.b.c {
#       tls {
#               dns cloudflare
#       }

        #respond "please work"
#}



Unfortunately, it seems like accessing the site `my.b.c` still produces an 502. My `caddy run debug` isn't spitting out any useful logs this time...

I tried `curl -v my.b.c` from an external computer and this is what showed up:

Code:
 curl -v my.b.c                                                                                                                                                                                          ✔  3.0.3  
*   Trying 260xxxx80...
*   Trying 17xxxxxx0...
* Connected to my.b.c (17xxxxxx9) port 80 (#0)
> GET / HTTP/1.1
> Host: my.b.c
> User-Agent: curl/7.79.1
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 308 Permanent Redirect
< Date: Sun, 27 Nov 2022 04:45:09 GMT
< Content-Length: 0
< Connection: keep-alive
< Location: https://my.b.c/
< CF-Cache-Status: DYNAMIC
< Report-To: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=DIqNLc7Mg0HNT3AmeWwD95Fz9vSw8z5IPD2zpm43X6uypEAz5RXkE0YEjJAF%2BS6KQZo7r31%2F5Wr%2FRKuFl5IjoBNY9AQ6UFbPVOGyOIVRS9945Zp7Y7dhX%2BrhOxRBHg8lsQ5IGN7z"}],"group":"cf-nel","max_age":604800}
< NEL: {"success_fraction":0,"report_to":"cf-nel","max_age":604800}
< Server: cloudflare
< CF-RAY: 77083xxxxx891-SEA
< alt-svc: h3=":443"; ma=86400, h3-29=":443"; ma=86400
<
* Connection #0 to host my.b.c left intact




Same error as before in `caddy run debug`: "x509: cannot validate certificate for 192.168.1.70 because it doesn't contain any IP SANs" when accessing the site from an external network.

So I'm still seeing the 502s but I'm unsure what the issue is. Jellyfin is accessible from external networks, and nextcloud is accessible from internal networks. My router is provided by the ISP so I don't think I'm able to perform your other suggested preferred action.

Thankful for any guidance. Thanks :)


Edit: I tried the unrecommended suggestion, and now this is working on external networks. Is this really bad to do? It'll be used by friends, not government files or anything lol. Just wanted to understand how bad this is.
Code:
my.b.c {
        log {
                output file /var/log/caddy/nextcloud-access.log
        }

        #reverse_proxy https://192.168.1.70 # nextcloud test please work
        reverse_proxy https://192.168.1.70 {
                transport http {
                        tls_insecure_skip_verify
                }
        }
}
 
Last edited:

danb35

Hall of Famer
Joined
Aug 16, 2011
Messages
15,504
"x509: cannot validate certificate for 192.168.1.70 because it doesn't contain any IP SANs" when accessing the site from an external network.
After changing the hosts file, I neglected to mention updating the Caddyfile to proxy to proxy to that FQDN rather than the IP address--that was the purpose of changing the hosts file, so that the name would resolve to the correct (local) IP address.
I tried the unrecommended suggestion, and now this is working on external networks. Is this really bad to do?
It means that your Caddy instance will proxy connections to anything on your LAN that can talk HTTPS, regardless of the cert's validity or what it says--which kind of defeats the purpose of using HTTPS in the first place.
 

toxikat

Dabbler
Joined
Nov 3, 2022
Messages
27
to that FQDN rather than the IP address
Yes I changed it and now it works with /etc/hosts edited with the ip alias and Caddyfile changed to `reverse_proxy https://my.b.c` without `tls_insecure_skip_verify`.


I was wondering what that did but I thought it worked the other way around. Thanks for the clarification and cheers for all the help. Finally got this resolved so I will be closing this thread down for now.
 
Top