NAT-based jails cannot connect to each other

izomiac

Dabbler
Joined
May 3, 2018
Messages
19
I just switched my plugins/jails to use the relatively new NAT port forwarding feature. While they can access the internet, and LAN hosts can access them, they are unable to talk to each other. For example, I'm forwarding port 9091 on my FreeNAS host (192.168.99.5) to Transmission and I would like Radarr to be able to connect to it.

Radarr cannot connect to Transmission, but it can see port 80 on the FreeNAS host and access the internet.
Code:
root@radarr:/ # curl http://192.168.99.5:9091
curl: (7) Failed to connect to 192.168.99.5 port 9091: Connection refused
root@radarr:/ # curl http://192.168.99.5:80
<html>
<head><title>302 Found</title></head>
<body>
<center><h1>302 Found</h1></center>
<hr><center>nginx</center>
</body>
</html>
root@radarr:/ # curl http://www.example.com
<!doctype html>
<html>
<head>
<title>Example Domain</title>


Any other device on my LAN can access Transmission.
Code:
C:\>curl http://192.168.99.5:9091
<h1>301: Moved Permanently</h1>


And any jail that uses DHCP can access Transmission as well.
Code:
root@reverseproxy:/ # curl http://192.168.99.5:9091
<h1>301: Moved Permanently</h1>


Here's the config.json for Transmission:
Code:
{
"CONFIG_VERSION": "26",
"allow_chflags": 0,
"allow_mlock": 0,
"allow_mount": 0,
"allow_mount_devfs": 0,
"allow_mount_fusefs": 0,
"allow_mount_nullfs": 0,
"allow_mount_procfs": 0,
"allow_mount_tmpfs": 0,
"allow_mount_zfs": 0,
"allow_quotas": 0,
"allow_raw_sockets": 0,
"allow_set_hostname": 1,
"allow_socket_af": 0,
"allow_sysvipc": 0,
"allow_tun": 0,
"allow_vmm": 0,
"assign_localhost": 0,
"available": "readonly",
"basejail": 1,
"boot": 1,
"bpf": 0,
"children_max": "0",
"cloned_release": "11.2-RELEASE-p9",
"comment": "none",
"compression": "lz4",
"compressratio": "readonly",
"coredumpsize": "off",
"count": "1",
"cpuset": "off",
"cputime": "off",
"datasize": "off",
"dedup": "off",
"defaultrouter": "172.16.0.21",
"defaultrouter6": "auto",
"depends": "none",
"devfs_ruleset": "4",
"dhcp": 0,
"enforce_statfs": "2",
"exec_clean": 1,
"exec_created": "/usr/bin/true",
"exec_fib": "0",
"exec_jail_user": "root",
"exec_poststart": "/usr/bin/true",
"exec_poststop": "/usr/bin/true",
"exec_prestart": "/usr/bin/true",
"exec_prestop": "/usr/bin/true",
"exec_start": "/bin/sh /etc/rc",
"exec_stop": "/bin/sh /etc/rc.shutdown",
"exec_system_jail_user": "0",
"exec_system_user": "root",
"exec_timeout": "60",
"host_domainname": "none",
"host_hostname": "transmission",
"host_hostuuid": "transmission",
"host_time": 1,
"hostid": "2ff269df-73c1-11e9-b13e-0cc47acdefe7",
"hostid_strict_check": 0,
"interfaces": "vnet0:bridge0",
"ip4": "new",
"ip4_addr": "vnet0|172.16.0.22/30",
"ip4_saddrsel": 1,
"ip6": "new",
"ip6_addr": "none",
"ip6_saddrsel": 1,
"ip_hostname": 0,
"jail_zfs": 0,
"jail_zfs_dataset": "iocage/jails/transmission/data",
"jail_zfs_mountpoint": "none",
"last_started": "2020-04-02 09:49:07",
"localhost_ip": "none",
"login_flags": "-f root",
"mac_prefix": "0cc47a",
"maxproc": "off",
"memorylocked": "off",
"memoryuse": "off",
"mount_devfs": 1,
"mount_fdescfs": 1,
"mount_linprocfs": 0,
"mount_procfs": 0,
"mountpoint": "readonly",
"msgqqueued": "off",
"msgqsize": "off",
"nat": 1,
"nat_backend": "ipfw",
"nat_forwards": "tcp(9091:9091),tcp(12929:12929)",
"nat_interface": "none",
"nat_prefix": "172.16",
"nmsgq": "off",
"notes": "none",
"nsem": "off",
"nsemop": "off",
"nshm": "off",
"nthr": "off",
"openfiles": "off",
"origin": "readonly",
"owner": "root",
"pcpu": "off",
"plugin_name": "transmission",
"plugin_repository": "https://github.com/freenas/iocage-ix-plugins.git",
"priority": "99",
"pseudoterminals": "off",
"quota": "none",
"readbps": "off",
"readiops": "off",
"release": "11.2-RELEASE-p9",
"reservation": "none",
"resolver": "/etc/resolv.conf",
"rlimits": "off",
"rtsold": 0,
"securelevel": "2",
"shmsize": "off",
"stacksize": "off",
"stop_timeout": "30",
"swapuse": "off",
"sync_state": "none",
"sync_target": "none",
"sync_tgt_zpool": "none",
"sysvmsg": "new",
"sysvsem": "new",
"sysvshm": "new",
"template": 0,
"type": "pluginv2",
"used": "readonly",
"vmemoryuse": "off",
"vnet": 1,
"vnet0_mac": "0cc47a03aa46 0cc47a03aa47",
"vnet1_mac": "none",
"vnet2_mac": "none",
"vnet3_mac": "none",
"vnet_default_interface": "auto",
"vnet_interfaces": "none",
"wallclock": "off",
"writebps": "off",
"writeiops": "off"
}

and my config.json for Radarr:
Code:
{
"CONFIG_VERSION": "26",
"allow_chflags": 0,
"allow_mlock": 0,
"allow_mount": 0,
"allow_mount_devfs": 0,
"allow_mount_fusefs": 0,
"allow_mount_nullfs": 0,
"allow_mount_procfs": 0,
"allow_mount_tmpfs": 0,
"allow_mount_zfs": 0,
"allow_quotas": 0,
"allow_raw_sockets": 0,
"allow_set_hostname": 1,
"allow_socket_af": 0,
"allow_sysvipc": 0,
"allow_tun": 0,
"allow_vmm": 0,
"assign_localhost": 0,
"available": "readonly",
"basejail": 1,
"boot": 1,
"bpf": 0,
"children_max": "0",
"cloned_release": "11.2-RELEASE-p9",
"comment": "none",
"compression": "lz4",
"compressratio": "readonly",
"coredumpsize": "off",
"count": "1",
"cpuset": "off",
"cputime": "off",
"datasize": "off",
"dedup": "off",
"defaultrouter": "172.16.0.17",
"defaultrouter6": "auto",
"depends": "none",
"devfs_ruleset": "4",
"dhcp": 0,
"enforce_statfs": "2",
"exec_clean": 1,
"exec_created": "/usr/bin/true",
"exec_fib": "0",
"exec_jail_user": "root",
"exec_poststart": "/usr/bin/true",
"exec_poststop": "/usr/bin/true",
"exec_prestart": "/usr/bin/true",
"exec_prestop": "/usr/bin/true",
"exec_start": "/bin/sh /etc/rc",
"exec_stop": "/bin/sh /etc/rc.shutdown",
"exec_system_jail_user": "0",
"exec_system_user": "root",
"exec_timeout": "60",
"host_domainname": "none",
"host_hostname": "radarr",
"host_hostuuid": "radarr",
"host_time": 1,
"hostid": "2ff269df-73c1-11e9-b13e-0cc47acdefe7",
"hostid_strict_check": 0,
"interfaces": "vnet0:bridge0",
"ip4": "new",
"ip4_addr": "vnet0|172.16.0.18/30",
"ip4_saddrsel": 1,
"ip6": "new",
"ip6_addr": "none",
"ip6_saddrsel": 1,
"ip_hostname": 0,
"jail_zfs": 0,
"jail_zfs_dataset": "iocage/jails/radarr/data",
"jail_zfs_mountpoint": "none",
"last_started": "2020-04-02 09:44:24",
"localhost_ip": "none",
"login_flags": "-f root",
"mac_prefix": "0cc47a",
"maxproc": "off",
"memorylocked": "off",
"memoryuse": "off",
"mount_devfs": 1,
"mount_fdescfs": 1,
"mount_linprocfs": 0,
"mount_procfs": 0,
"mountpoint": "readonly",
"msgqqueued": "off",
"msgqsize": "off",
"nat": 1,
"nat_backend": "ipfw",
"nat_forwards": "tcp(7878:7878)",
"nat_interface": "none",
"nat_prefix": "172.16",
"nmsgq": "off",
"notes": "none",
"nsem": "off",
"nsemop": "off",
"nshm": "off",
"nthr": "off",
"openfiles": "off",
"origin": "readonly",
"owner": "root",
"pcpu": "off",
"plugin_name": "radarr",
"plugin_repository": "https://github.com/freenas/iocage-ix-plugins.git",
"priority": "99",
"pseudoterminals": "off",
"quota": "none",
"readbps": "off",
"readiops": "off",
"release": "11.2-RELEASE-p9",
"reservation": "none",
"resolver": "/etc/resolv.conf",
"rlimits": "off",
"rtsold": 0,
"securelevel": "2",
"shmsize": "off",
"stacksize": "off",
"stop_timeout": "30",
"swapuse": "off",
"sync_state": "none",
"sync_target": "none",
"sync_tgt_zpool": "none",
"sysvmsg": "new",
"sysvsem": "new",
"sysvshm": "new",
"template": 0,
"type": "pluginv2",
"used": "readonly",
"vmemoryuse": "off",
"vnet": 1,
"vnet0_mac": "0cc47ac1dfca 0cc47ac1dfcb",
"vnet1_mac": "none",
"vnet2_mac": "none",
"vnet3_mac": "none",
"vnet_default_interface": "auto",
"vnet_interfaces": "none",
"wallclock": "off",
"writebps": "off",
"writeiops": "off"
}

and reverseproxy for comparison:
Code:
{
"CONFIG_VERSION": "26",
"allow_chflags": 0,
"allow_mlock": 0,
"allow_mount": 0,
"allow_mount_devfs": 0,
"allow_mount_fusefs": 0,
"allow_mount_nullfs": 0,
"allow_mount_procfs": 0,
"allow_mount_tmpfs": 0,
"allow_mount_zfs": 0,
"allow_quotas": 0,
"allow_raw_sockets": 0,
"allow_set_hostname": 1,
"allow_socket_af": 0,
"allow_sysvipc": 0,
"allow_tun": 0,
"allow_vmm": 0,
"assign_localhost": 0,
"available": "readonly",
"basejail": 0,
"boot": 1,
"bpf": 1,
"children_max": "0",
"cloned_release": "11.2-RELEASE-p9",
"comment": "none",
"compression": "lz4",
"compressratio": "readonly",
"coredumpsize": "off",
"count": "1",
"cpuset": "off",
"cputime": "off",
"datasize": "off",
"dedup": "off",
"defaultrouter": "auto",
"defaultrouter6": "auto",
"depends": "none",
"devfs_ruleset": "4",
"dhcp": 1,
"enforce_statfs": "2",
"exec_clean": 1,
"exec_created": "/usr/bin/true",
"exec_fib": "0",
"exec_jail_user": "root",
"exec_poststart": "/usr/bin/true",
"exec_poststop": "/usr/bin/true",
"exec_prestart": "/usr/bin/true",
"exec_prestop": "/usr/bin/true",
"exec_start": "/bin/sh /etc/rc",
"exec_stop": "/bin/sh /etc/rc.shutdown",
"exec_system_jail_user": "0",
"exec_system_user": "root",
"exec_timeout": "60",
"host_domainname": "none",
"host_hostname": "reverseproxy",
"host_hostuuid": "reverseproxy",
"host_time": 1,
"hostid": "2ff269df-73c1-11e9-b13e-0cc47acdefe7",
"hostid_strict_check": 0,
"interfaces": "vnet0:bridge0",
"ip4": "new",
"ip4_addr": "none",
"ip4_saddrsel": 1,
"ip6": "new",
"ip6_addr": "none",
"ip6_saddrsel": 1,
"ip_hostname": 0,
"jail_zfs": 0,
"jail_zfs_dataset": "iocage/jails/reverseproxy/data",
"jail_zfs_mountpoint": "none",
"last_started": "2020-03-21 04:30:48",
"localhost_ip": "none",
"login_flags": "-f root",
"mac_prefix": "0cc47a",
"maxproc": "off",
"memorylocked": "off",
"memoryuse": "off",
"mount_devfs": 1,
"mount_fdescfs": 1,
"mount_linprocfs": 0,
"mount_procfs": 0,
"mountpoint": "readonly",
"msgqqueued": "off",
"msgqsize": "off",
"nat": 0,
"nat_backend": "ipfw",
"nat_forwards": "none",
"nat_interface": "none",
"nat_prefix": "172.16",
"nmsgq": "off",
"notes": "none",
"nsem": "off",
"nsemop": "off",
"nshm": "off",
"nthr": "off",
"openfiles": "off",
"origin": "readonly",
"owner": "root",
"pcpu": "off",
"plugin_name": "none",
"plugin_repository": "none",
"priority": "99",
"pseudoterminals": "off",
"quota": "none",
"readbps": "off",
"readiops": "off",
"release": "11.2-RELEASE-p9",
"reservation": "none",
"resolver": "/etc/resolv.conf",
"rlimits": "off",
"rtsold": 0,
"securelevel": "2",
"shmsize": "off",
"stacksize": "off",
"stop_timeout": "30",
"swapuse": "off",
"sync_state": "none",
"sync_target": "none",
"sync_tgt_zpool": "none",
"sysvmsg": "new",
"sysvsem": "new",
"sysvshm": "new",
"template": 0,
"type": "jail",
"used": "readonly",
"vmemoryuse": "off",
"vnet": 1,
"vnet0_mac": "0cc47a549309 0cc47a54930a",
"vnet1_mac": "none",
"vnet2_mac": "none",
"vnet3_mac": "none",
"vnet_default_interface": "auto",
"vnet_interfaces": "none",
"wallclock": "off",
"writebps": "off",
"writeiops": "off"
}

So, short of going back to DHCP, does anyone have any ideas for how to get my jails back on speaking terms with each other?
 

sretalla

Powered by Neutrality
Moderator
Joined
Jan 1, 2016
Messages
9,702
Only a guess, but since docker is based on the jails concept and the docker network is somehow like the NAT jail network (I think), perhaps the only way NAT jails can see each other is with their private-side addresses, not the public-side one.

What I don't know (since I don't use NAT for jails) is how to work out that address.

Perhaps try ifconfig in the jail CLI?

Or try by the hostname of the jail?
 

izomiac

Dabbler
Joined
May 3, 2018
Messages
19
That actually did the trick, although I was sure I'd tried that last night... No matter.

Any idea if those IP addresses are static or dynamic?
 

sretalla

Powered by Neutrality
Moderator
Joined
Jan 1, 2016
Messages
9,702
Which was it? (name or ifconfig showed you the IPs)

I would guess the NAT is doing some kind of DHCP, but probably hands the same address back to the same MAC address all the time, so it will be long-standing unless you re-create the jail.
 

izomiac

Dabbler
Joined
May 3, 2018
Messages
19
I used ifconfig from the jail.

Earlier I used the reported IP address from the jail section of the Web GUI, no clue why it didn't work before. I'll double check that they're the same, and if not then I'll file a bug report.
 

krylic

Cadet
Joined
Nov 22, 2020
Messages
2
Any idea if those IP addresses are static or dynamic?

They do seem to change if you reboot TrueNAS. Maybe not all the time because they have some lease like sretalla said but wait a day or so and reboot and all your jails have new ips.
Have you ran into this and if so found a solution?

I'm simply trying to get Tautulli to talk to Plex with either a hostname or static IP when using VNET + NAT on both jails.
 

sretalla

Powered by Neutrality
Moderator
Joined
Jan 1, 2016
Messages
9,702
I guess looking for a DNS/name resolution option would be the way forward rather than trying to work out how the DHCP is done.

If the jails are able to see each other by name, then IP is irrelevant.
 

krylic

Cadet
Joined
Nov 22, 2020
Messages
2
Yea I couldn't figure out how to get them to see each other by hostname without having to do some kind of script that injects it into their configs on boot.
I've just pointed those services to public reverse proxy, basically taking "the long way around". Works for now though
 

nasrrat

Cadet
Joined
Jun 30, 2021
Messages
1
I'm experiencing this exact problem. But would like to find a solution that would allow me to access my other jails via the port setup in the NAT port forwarding rules for the jail.

It does work for me to get access using the 172.16.0.0/24 subnet, but in my case that's not ideal. I'd rather access http://mynas:8000 instead of http://172.16.0.2

Does anyone have any ideas how to continue to work out what is happening here? Figure out where the traffic is getting dropped?

How can I find on my host TrueNas server where the port forwarding rules are set up and used? Are they part of iocage? ipfw? route tables? I want to see and understand what is happening. i.e. where do the port forwarding rules live under the hood?
 

Hackslash

Cadet
Joined
Jun 2, 2021
Messages
9
I ran in to this too. For some reason it works when there is only one server under NAT and the others have their own IPs. This is unfortunate because I don't want to use a bunch of IPs for something I could just use ports for.

In normal networking terms; there would only need to be a route from the private virtual NAT network in to your own LAN subnet. Normally, routers use a routing protocol, like OSPF, to dynamically make all the routes they need. There appears to be a virtual default gateway which passes packets on out through the LAN to the WAN. It also passes packets from the LAN in to the port you've specified in your port forward. What I don't understand is why packets that can get out on the LAN wouldn't be able to come back in on the same port forward that you use to communicate with these services on the LAN.

I do know this situation is called "Double NAT" and it has some pitfalls: https://dongknows.com/double-nat-vs-single-nat/

Because I don't know enough about how this virtual networking functions in TrueNAS and I don't know how to configure routing protocols in FreeBSD: I recommend just avoiding the double NAT. Configure all of your jailed services with unique static IPs so they live in your regular LAN.

I just changed my jail that couldn't connect to have it's own static IP and it's able to communicate to the other jail which lives in double NAT land. They go in through the same port forward that you use to access the service.
 

Patrick M. Hausen

Hall of Famer
Joined
Nov 25, 2013
Messages
7,740
You can set up a private bridge interface, connect each jail to it, and use static addresses. Any reason why you prefer NAT over VNET? The latter is the much more recent architecture and definitely the direction FreeBSD is heading. Also NAT on TrueNAS looks pretty intransparent to me and tends to break when people try to add their own NAT rules for e.g. VPN access.

You can of course add a "NAT jail" with one exposed interface, one on the private bridge, and all the ipfw rules restricted to that particular jail.
 

Hackslash

Cadet
Joined
Jun 2, 2021
Messages
9
I don't prefer. I used Plugins to install and they came configured this way. In jails I build myself I am making them regular VNET with their own IP as I described. I have one plugin left that lives in it's little NAT world.
 
Top