Jail cannot access network from datacenter

[owenfi]

Hello,
I've tried following the guide linked here:

How to set up an nginx reverse proxy with SSL termination in a jail

Instructions are available here: https://www.samueldowling.com/2020/01/18/nginx-reverse-proxy-freenas-ssl-tls/ Recently I went through the process of standing up a Bitwarden server. As this is the second service that I plan to make available...

www.ixsystems.com

and various other attempts at setting up jails for my situation without much luck yet.

Specifically, my TrueNAS Core 12.0 install is hosted in a colo-facility. I had everything (a nextcloud and plex jail, for instance) working decently at home, when I could assign local IPs to the jails, with my mediocre understanding of local networking. After research and trials I ordered a 2nd static IP with the intent of having an nginx-reverse-proxy at that IP make my other jails available externally.

What I'm struggling with is what settings to apply to the jail(s) and maybe base system's network to get them to have network connectivity. Specifically, in the above guide it mentions vnet which I tried at first (with my public IP and route in place of the private ones listed in the guide). Then I tried enabling DHCP, and finally no vnet (just connected to eth3, which is my normal port).

One slight oddity, when I start the jail, for ~60 seconds pinging google.com works (so I thought I fixed my problem), but after a bit it stops working. I can no longer ping domains or IPs until I stop and restart it.

I also tried to make another jail with private addresses, but wasn't really sure what to do in terms of routing, so I couldn't even ping a dns server.

I can access the server's dashboard and ssh via the original IP address just fine, and when I'm in the default shell have access to internet/dns/etc.

Jails are on 12.1-release-p10 (in general is it best to keep these aligned with the underlying system version, or doesn't matter?)

I can't tell a difference between `ifconfig` or `netstat -nr` during the period where connectivity works, vs when it doesn't. So I'm fairly stumped on what to look into to debug it further. I don't have any static routes defined, and haven't made much if any modifications to etc/conf files in either the base system or the jails.

Show : host ifconfig

Code:

igb0: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=e527bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
        ether 0c:c4:7a:33:c2:60
        media: Ethernet autoselect
        status: no carrier
        nd6 options=1<PERFORMNUD>
igb1: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=e507bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
        ether 0c:c4:7a:33:c2:61
        media: Ethernet autoselect
        status: no carrier
        nd6 options=1<PERFORMNUD>
igb2: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=e507bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
        ether 0c:c4:7a:33:c2:62
        media: Ethernet autoselect
        status: no carrier
        nd6 options=1<PERFORMNUD>
igb3: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=8100b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWFILTER>
        ether 0c:c4:7a:33:c2:63
        inet 107.xxx.yyy.199 netmask 0xfffffff0 broadcast 107.xxx.yyy.207
        inet 107.xxx.yyy.198 netmask 0xfffffff0 broadcast 107.xxx.yyy.207
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=9<PERFORMNUD,IFDISABLED>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x5
        inet 127.0.0.1 netmask 0xff000000
        groups: lo
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
pflog0: flags=0<> metric 0 mtu 33160
        groups: pflog
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        ether 02:e3:88:26:25:00
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto stp-rstp maxaddr 2000 timeout 1200
        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
        member: vnet0.5 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 8 priority 128 path cost 2000
        member: igb3 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 4 priority 128 path cost 20000
        groups: bridge
        nd6 options=1<PERFORMNUD>
vnet0.5: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: associated with jail: folsom as nic: epair0b
        options=8<VLAN_MTU>
        ether 0e:c4:7a:e3:2c:0f
        hwaddr 02:2c:cc:35:24:0a
        groups: epair
        media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
        status: active
        nd6 options=1<PERFORMNUD>

Show : jails ifconfig

Code:

root@reverse-proxy:/ # ifconfig
igb0: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=e527bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
        ether 0c:c4:7a:33:c2:60
        media: Ethernet autoselect
        status: no carrier
igb1: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=e507bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
        ether 0c:c4:7a:33:c2:61
        media: Ethernet autoselect
        status: no carrier
igb2: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=e507bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
        ether 0c:c4:7a:33:c2:62
        media: Ethernet autoselect
        status: no carrier
igb3: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=8100b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWFILTER>
        ether 0c:c4:7a:33:c2:63
        inet 107.xxx.yyy.198 netmask 0xfffffff0 broadcast 107.xxx.yyy.207
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        groups: lo
pflog0: flags=0<> metric 0 mtu 33160
        groups: pflog
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        ether 02:e3:88:26:25:00
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
        member: vnet0.5 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 8 priority 128 path cost 2000
        member: igb3 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 4 priority 128 path cost 20000
        groups: bridge
vnet0.5: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: associated with jail: folsom as nic: epair0b
        options=8<VLAN_MTU>
        ether 0e:c4:7a:e3:2c:0f
        hwaddr 02:2c:cc:35:24:0a
        groups: epair
        media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
        status: active

Show : host netstat -nr

Code:

Routing tables

Internet:
Destination        Gateway            Flags     Netif Expire
default            107.148.224.206    UGS        igb3
107.148.224.192/28 link#4             U          igb3
107.148.224.198    link#4             UHS         lo0
107.148.224.199    link#4             UHS         lo0
127.0.0.1          link#5             UH          lo0

Internet6:
Destination                       Gateway                       Flags     Netif Expire
::/96                             ::1                           UGRS        lo0
::1                               link#5                        UH          lo0
::ffff:0.0.0.0/96                 ::1                           UGRS        lo0
fe80::/10                         ::1                           UGRS        lo0
fe80::%lo0/64                     link#5                        U           lo0
fe80::1%lo0                       link#5                        UHS         lo0
ff02::/16                         ::1                           UGRS        lo0

Show : jails netstat -nr

Code:

Routing tables

Internet:
Destination        Gateway            Flags     Netif Expire
107.148.224.198    link#4             UHS         lo0

 

[jgreco]

This has nothing to do with data centers, other than the fact that your data center probably has real networks and your home network is very likely to be an RFC1918 network of some sort.

Debugging networking by trying to interpret ifconfig and netstat results is somewhat difficult, especially since it provides no context as to what sort of networking you have at the data center. This can range anywhere from single-IP-on-a-shared-ethernet, small routed network like a /28, all the way on up to ISP-grade BGP with an ASN and globally routed IP space.

 

[owenfi]

This has nothing to do with data centers, other than the fact that your data center probably has real networks and your home network is very likely to be an RFC1918 network of some sort.

Debugging networking by trying to interpret ifconfig and netstat results is somewhat difficult, especially since it provides no context as to what sort of networking you have at the data center. This can range anywhere from single-IP-on-a-shared-ethernet, small routed network like a /28, all the way on up to ISP-grade BGP with an ASN and globally routed IP space.

Right on all accounts afaict. The “local IP” above would probably be better said as “private” or as you say RFC1918.

I’m not sure how to find out too much more about my colo’s routing, other than that it is a /28 subnet, and they gave me a couple adjacent IPs, and a gateway address.

I guess my question is how to set up a jail to be accessible when the system has a “public” IP and I don’t have control over the gateway/router. (And would prefer not to add additional hardware or buy too many IPs… ideally just the original and sub domains would work, but I understand it might be easier/smarter to keep them a bit isolated)?

 

[jgreco]

Bearing in mind that it is an awful idea to expose FreeNAS directly to the Internet---

From a networking perspective, the basic option would be to have your FreeNAS have an interface on the /28 as a primary IP address on its ethernet interface, and then have any other IP's you have been allocated as secondary (or "alias") IP addresses.

I don't know how exactly this interplays with the jail subsystem, as I do not use FreeNAS's jails (though I do extensive work with FreeBSD jails). FreeNAS has introduced complications into the mix with vnet and bridging and other stuff that might not be adding value in this scenario.

I don't have any idea what your goals are, but it might be worth picking up a small pfSense host or Ubiquiti EdgeRouter type device to act as a NAT gateway and port forwarding engine. This would do a better job of isolating your FreeNAS from the perils of the Internet, and would also give you a design more similar to the home based setup. This allows you to use a single public IP address, but slightly increases the complexity of configuration and introduces dependency on a new device.

Perhaps someone else can comment on the jail aspects here. I mainly stuck my nose in because I approved the original post.

 

[owenfi]

My main goal was to move my home server to a safer (from wildfires, theft, etc) location for minimal cost. It's purpose is as a backup server, and ideally it would run a few services such as plex, a sync server to pull in some backups automatically, some periodic workers, etc.

Bearing in mind that it is an awful idea to expose FreeNAS directly to the Internet---

My host has an option to include a firewall...but it doubles the monthly rate (and I'm not sure what routing capabilities/configurability it includes). I'll ask, and see if they have routers available, or what the cost might be if I add another unit.

It feels like it should be possible to run SSH+nginx; perhaps disable or block the TrueNAS web interface (and shut off other external services?). Maybe one of the services behind nginx would need to be a VPN to re-introduce some of the "ease-of-use". It seems this wouldn't ultimately be less secure than a FreeBSD host on the internet, which feels like it should be able to run with a low likelihood of getting pwnd....but maybe my intuition is off base.

 

[jgreco]

FreeBSD hosts have lived for years very successfully on the Internet, it's mostly a matter of keeping exposed services secure. Those of us who do this professionally may run enhancements like firewalls and host hardening.

The problem with FreeNAS is that the onboard services were designed to do things like share files and manage the server, and the design didn't include security as a day-one consideration. The NAS management and API are promiscuous by default, and services such as SMB aren't easy to protect in a meaningful way. It runs trash-grade services like Bonjour/Avahi in order to do the things that people want a NAS to do, but which are dangerous to expose to the general Internet. You can run "netstat -an" yourself to see.

It's possible to run an ipfw setup on the FreeNAS host itself to restrict access, but there is some danger there because the default firewall fails open, so a failure in the firewall script can result in unexpected open access to the Internet. A true firewall needs to fail secure.

A better way to handle this is to have an ESXi hypervisor running the host. ESXi has a firewall that can be configured to limit management access, and then you can run a pfSense host and your NAS as virtual machines, along with anything else you want to do. This is still a little bit sketchy, and requires you to have a static IP address to be able to reach the ESXi hypervisor in a crisis.

 

[owenfi]

Duly noted, and thanks for all the advice. Would've loved to figure out an ESXi approach with pfSense in front of TrueNAS, but given it's already deployed with no/sporadic physical access I'll probably be leaving that for a new setup down the road.

I was able to order a Unifi Dream Machine Pro, which seems to fit the bill (though their actual technical details available before setting it up are scant...what with no user manual to be found...I assume it'll have the basic necessities such as NAT, and appears to support more via SSH) in terms of remote configurability, throughput (including with IDS/IPS), and only increases my monthly datacenter cost 67% (after the breakeven of ~2 years, vs their firewall).

I'd still be keen to hear from folks who have experience or resource pointers on setting up jails that utilize a public IP, and what might be causing the brief period of connectivity I was seeing. Or a from scratch guide on how to set up routes for the host and the jails, given slightly unconventional networking approaches; but it seems like my long term solution will be to put it all behind a router and I think I should be able to figure things out.

 

[owenfi]

Well. I think this issue might've originally been outside my control.

I bought a Unifi Dream Machine Pro (rack mount firewall) and installed it, had difficulty getting it onto the network as well (similar to the issues I had setting up the new IP for the jails, so maybe the same root cause in the end). Eventually I filed a ticket and the staff fixed the configuration, so now my TrueNAS is fully behind a firewall, and all is working well.

Not sure this will be helpful to anyone, since it seems fairly situational, but I wanted to give an update in case somebody runs across a similar issue down the road. Good luck!