Jails w/ NAT not handling incoming packets

[subwire]

I will soon be moving to an environment where I don't control my LAN, and can't have jails with their own IPs.
I enabled NAT on all my jails using the checkbox in the WebUI. Most everything was configured as expected, except there's no internet connectivity in the jails. Upon further inspection, it appears that packets are escaping the NAT just fine, but the replies (eg. DNS) arrive but do not make it onto the bridge.
The ipfw rules look correct (see below) and I would expect the return packets to hit the nat in recv rule.
Am I missing something? Thanks in advance!


Here's my setup:

My LAN (let's say this is out of my control) is 192.168.0.0/24
My FreeNAS's internal LAN is 192.168.1.0/24

Ifconfig from the host outside jails says:

Code:

igb0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
   options=400b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWTSO>
   ether d0:50:99:1c:32:d7
   inet 192.168.0.105 netmask 0xffffff00 broadcast 192.168.0.255
   nd6 options=9<PERFORMNUD,IFDISABLED>
   media: Ethernet autoselect (100baseTX <full-duplex>)
   status: active
igb1: flags=8c02<BROADCAST,OACTIVE,SIMPLEX,MULTICAST> metric 0 mtu 1500
   options=403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO>
   ether d0:50:99:1c:32:d8
   nd6 options=9<PERFORMNUD,IFDISABLED>
   media: Ethernet autoselect (100baseTX <full-duplex>)
   status: active
ipfw0: flags=8801<UP,SIMPLEX,MULTICAST> metric 0 mtu 65536
   nd6 options=9<PERFORMNUD,IFDISABLED>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
   options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
   inet6 ::1 prefixlen 128
   inet6 fe80::1%lo0 prefixlen 64 scopeid 0x6
   inet 127.0.0.1 netmask 0xff000000
   nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
   ether 02:fe:4a:c8:9c:00
   inet 192.168.1.254 netmask 0xffffffff broadcast 192.168.1.254
   nd6 options=1<PERFORMNUD>
   id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
   maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
   root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
   member: epair9a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
    ifmaxaddr 0 port 17 priority 128 path cost 2000
   member: epair8a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
    ifmaxaddr 0 port 16 priority 128 path cost 2000
   member: epair7a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
    ifmaxaddr 0 port 15 priority 128 path cost 2000
   member: epair6a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
    ifmaxaddr 0 port 14 priority 128 path cost 2000
   member: epair5a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
    ifmaxaddr 0 port 13 priority 128 path cost 2000
   member: epair4a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
    ifmaxaddr 0 port 12 priority 128 path cost 2000
   member: epair3a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
    ifmaxaddr 0 port 11 priority 128 path cost 2000
   member: epair2a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
    ifmaxaddr 0 port 10 priority 128 path cost 2000
   member: epair1a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
    ifmaxaddr 0 port 9 priority 128 path cost 2000
   member: epair0a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
    ifmaxaddr 0 port 8 priority 128 path cost 2000
   member: igb0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
    ifmaxaddr 0 port 2 priority 128 path cost 200000
epair0a: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
   options=8<VLAN_MTU>
   ether 02:c5:bc:00:08:0a
   nd6 options=1<PERFORMNUD>
   media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
   status: active
epair1a: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
   options=8<VLAN_MTU>
   ether 02:11:3c:00:09:0a
   nd6 options=1<PERFORMNUD>
   media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
   status: active
epair2a: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
   options=8<VLAN_MTU>
   ether 02:fa:d6:00:0a:0a
   nd6 options=1<PERFORMNUD>
   media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
   status: active
epair3a: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
   options=8<VLAN_MTU>
   ether 02:67:89:00:0b:0a
   nd6 options=1<PERFORMNUD>
   media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
   status: active
epair4a: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
   options=8<VLAN_MTU>
   ether 02:e8:f9:00:0c:0a
   nd6 options=1<PERFORMNUD>
   media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
   status: active
epair5a: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
   options=8<VLAN_MTU>
   ether 02:e1:f9:00:0d:0a
   nd6 options=1<PERFORMNUD>
   media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
   status: active
epair6a: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
   options=8<VLAN_MTU>
   ether 02:9a:db:00:0e:0a
   nd6 options=1<PERFORMNUD>
   media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
   status: active
epair7a: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
   options=8<VLAN_MTU>
   ether 02:3c:cb:00:0f:0a
   nd6 options=1<PERFORMNUD>
   media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
   status: active
epair8a: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
   options=8<VLAN_MTU>
   ether 02:83:cf:00:10:0a
   nd6 options=1<PERFORMNUD>
   media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
   status: active
epair9a: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
   options=8<VLAN_MTU>
   ether 02:82:35:00:11:0a
   nd6 options=1<PERFORMNUD>
   media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
   status: active

ifconfig from within one of the jail looks like:

Code:

[root@zetta ~]# jexec 2 ifconfig -a
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
   options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
   inet6 ::1 prefixlen 128
   inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1
   inet 127.0.0.1 netmask 0xff000000
   nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
epair1b: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
   options=8<VLAN_MTU>
   ether 02:38:58:00:0b:0b
   inet 192.168.1.22 netmask 0xffffff00 broadcast 192.168.1.255
   nd6 options=9<PERFORMNUD,IFDISABLED>
   media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)

Here's the current ipfw setup, as configured automatically by freenas:

Code:

00100 allow ip from any to any via lo0
00200 nat 100 ip from any to 192.168.0.105 in recv igb0
00300 nat 100 ip from 192.168.1.23 to any out xmit igb0
00400 nat 100 ip from 192.168.1.22 to any out xmit igb0
00500 nat 100 ip from 192.168.1.26 to any out xmit igb0
00600 nat 100 ip from 192.168.1.27 to any out xmit igb0
00700 nat 100 ip from 192.168.1.25 to any out xmit igb0
00800 nat 100 ip from 192.168.1.19 to any out xmit igb0
00900 nat 100 ip from 192.168.1.24 to any out xmit igb0
01000 nat 100 ip from 192.168.1.28 to any out xmit igb0
01100 nat 100 ip from 192.168.1.21 to any out xmit igb0
01200 nat 100 ip from 192.168.1.20 to any out xmit igb0
65535 allow ip from any to any

Here's what I'm seeing while resolving google.com from inside one of the jails, as monitored by the bridge. Note no replies here:

Code:

# tcpdump -i bridge0 -vv udp port 53
tcpdump: listening on bridge0, link-type EN10MB (Ethernet), capture size 65535 bytes
14:21:20.688341 IP (tos 0x0, ttl 64, id 6689, offset 0, flags [none], proto UDP (17), length 56)
  192.168.1.20.24455 > 192.168.0.1.domain: [udp sum ok] 38074+ A? google.com. (28)
14:21:25.689020 IP (tos 0x0, ttl 64, id 6690, offset 0, flags [none], proto UDP (17), length 56)
  192.168.1.20.24455 > 192.168.0.1.domain: [udp sum ok] 38074+ A? google.com. (28)
14:21:30.690015 IP (tos 0x0, ttl 64, id 6691, offset 0, flags [none], proto UDP (17), length 56)
  192.168.1.20.24455 > 192.168.0.1.domain: [udp sum ok] 38074+ A? google.com. (28)

Here's what the interface sees. Note the proper replies are here:

Code:

# tcpdump -i igb0 -vv udp port 53
tcpdump: listening on igb0, link-type EN10MB (Ethernet), capture size 65535 bytes
14:21:45.722908 IP (tos 0x0, ttl 63, id 6692, offset 0, flags [none], proto UDP (17), length 56)
  192.168.0.105.58878 > 192.168.0.1.domain: [udp sum ok] 14538+ A? google.com. (28)
14:21:45.726879 IP (tos 0x0, ttl 64, id 2593, offset 0, flags [DF], proto UDP (17), length 232)
  192.168.0.1.domain > 192.168.0.105.58878: [udp sum ok] 14538 q: A? google.com. 11/0/0 google.com. A 74.125.239.136, google.com. A 74.125.239.137, google.com. A 74.125.239.132, google.com. A 74.125.239.129, google.com. A 74.125.239.135, google.com. A 74.125.239.128, google.com. A 74.125.239.133, google.com. A 74.125.239.131, google.com. A 74.125.239.134, google.com. A 74.125.239.142, google.com. A 74.125.239.130 (204)
14:21:45.726884 IP (tos 0x0, ttl 63, id 2593, offset 0, flags [DF], proto UDP (17), length 232)
  192.168.0.1.domain > 192.168.1.20.58878: [udp sum ok] 14538 q: A? google.com. 11/0/0 google.com. A 74.125.239.136, google.com. A 74.125.239.137, google.com. A 74.125.239.132, google.com. A 74.125.239.129, google.com. A 74.125.239.135, google.com. A 74.125.239.128, google.com. A 74.125.239.133, google.com. A 74.125.239.131, google.com. A 74.125.239.134, google.com. A 74.125.239.142, google.com. A 74.125.239.130 (204)
14:21:46.582113 IP (tos 0x0, ttl 64, id 32949, offset 0, flags [none], proto UDP (17), length 70)
  192.168.0.105.30648 > 192.168.0.1.domain: [udp sum ok] 37971+ PTR? 1.0.168.192.in-addr.arpa. (42)
14:21:46.596459 IP (tos 0x0, ttl 64, id 2594, offset 0, flags [DF], proto UDP (17), length 70)
  192.168.0.1.domain > 192.168.0.105.30648: [udp sum ok] 37971 NXDomain q: PTR? 1.0.168.192.in-addr.arpa. 0/0/0 (42)
14:21:46.596530 IP (tos 0x0, ttl 64, id 32950, offset 0, flags [none], proto UDP (17), length 72)
  192.168.0.105.40794 > 192.168.0.1.domain: [udp sum ok] 37972+ PTR? 105.0.168.192.in-addr.arpa. (44)
14:21:46.610952 IP (tos 0x0, ttl 64, id 2595, offset 0, flags [DF], proto UDP (17), length 72)
  192.168.0.1.domain > 192.168.0.105.40794: [udp sum ok] 37972 NXDomain q: PTR? 105.0.168.192.in-addr.arpa. 0/0/0 (44)
14:21:46.611019 IP (tos 0x0, ttl 64, id 32953, offset 0, flags [none], proto UDP (17), length 71)
  192.168.0.105.11138 > 192.168.0.1.domain: [udp sum ok] 37973+ PTR? 20.1.168.192.in-addr.arpa. (43)
14:21:46.624888 IP (tos 0x0, ttl 64, id 2596, offset 0, flags [DF], proto UDP (17), length 71)
  192.168.0.1.domain > 192.168.0.105.11138: [udp sum ok] 37973 NXDomain q: PTR? 20.1.168.192.in-addr.arpa. 0/0/0 (43)
14:21:50.723028 IP (tos 0x0, ttl 63, id 6693, offset 0, flags [none], proto UDP (17), length 56)
  192.168.0.105.58878 > 192.168.0.1.domain: [udp sum ok] 14538+ A? google.com. (28)
14:21:50.724996 IP (tos 0x0, ttl 64, id 2598, offset 0, flags [DF], proto UDP (17), length 232)
  192.168.0.1.domain > 192.168.0.105.58878: [udp sum ok] 14538 q: A? google.com. 11/0/0 google.com. A 74.125.239.130, google.com. A 74.125.239.136, google.com. A 74.125.239.137, google.com. A 74.125.239.132, google.com. A 74.125.239.129, google.com. A 74.125.239.135, google.com. A 74.125.239.128, google.com. A 74.125.239.133, google.com. A 74.125.239.131, google.com. A 74.125.239.134, google.com. A 74.125.239.142 (204)
14:21:50.725001 IP (tos 0x0, ttl 63, id 2598, offset 0, flags [DF], proto UDP (17), length 232)
  192.168.0.1.domain > 192.168.1.20.58878: [udp sum ok] 14538 q: A? google.com. 11/0/0 google.com. A 74.125.239.130, google.com. A 74.125.239.136, google.com. A 74.125.239.137, google.com. A 74.125.239.132, google.com. A 74.125.239.129, google.com. A 74.125.239.135, google.com. A 74.125.239.128, google.com. A 74.125.239.133, google.com. A 74.125.239.131, google.com. A 74.125.239.134, google.com. A 74.125.239.142 (204)
14:21:55.724022 IP (tos 0x0, ttl 63, id 6694, offset 0, flags [none], proto UDP (17), length 56)
  192.168.0.105.58878 > 192.168.0.1.domain: [udp sum ok] 14538+ A? google.com. (28)
14:21:55.725978 IP (tos 0x0, ttl 64, id 2599, offset 0, flags [DF], proto UDP (17), length 232)
  192.168.0.1.domain > 192.168.0.105.58878: [udp sum ok] 14538 q: A? google.com. 11/0/0 google.com. A 74.125.239.142, google.com. A 74.125.239.130, google.com. A 74.125.239.136, google.com. A 74.125.239.137, google.com. A 74.125.239.132, google.com. A 74.125.239.129, google.com. A 74.125.239.135, google.com. A 74.125.239.128, google.com. A 74.125.239.133, google.com. A 74.125.239.131, google.com. A 74.125.239.134 (204)
14:21:55.725986 IP (tos 0x0, ttl 63, id 2599, offset 0, flags [DF], proto UDP (17), length 232)
  192.168.0.1.domain > 192.168.1.20.58878: [udp sum ok] 14538 q: A? google.com. 11/0/0 google.com. A 74.125.239.142, google.com. A 74.125.239.130, google.com. A 74.125.239.136, google.com. A 74.125.239.137, google.com. A 74.125.239.132, google.com. A 74.125.239.129, google.com. A 74.125.239.135, google.com. A 74.125.239.128, google.com. A 74.125.239.133, google.com. A 74.125.239.131, google.com. A 74.125.239.134 (204)

 

[subwire]

After more fiddling, I seem to have resolved the issue, although it seems like my fix should be something the WebUI does automatically. Here's my routing table now. Line 6 is the line I just added that solved the issue. Line 8 looks like the line added by WebUI to do what my line 6 does, but since the packets have already been translated to endpoint jail IPs, they merely need to get pointed at the bridge submet to get to their destination, and not to the bridge's IP itself.

Also, while I'm no stranger to UNIXes, I'm not yet familiar with how FreeNAS does configuration internally. How do I make this change persist?

Code:

Destination  Gateway  Flags  Refs  Use  Netif Expire
default  192.168.0.1  UGS  0  94505  igb0
127.0.0.1  link#6  UH  0  4249  lo0
192.168.0.0/24  link#2  U  0  15749  igb0
192.168.0.105  link#2  UHS  0  173  lo0
192.168.1.0/24  192.168.1.254  US  0  32 bridge
192.168.1.254  link#7  UHS  0  35  lo0 =>
192.168.1.254/32  link#7  U  0  0 bridge

 

[berlin]

Have you tried to create a startup script using the GUI to have the route command run post-init to add that route? Never used those scripts but seems like it should work.

 

[Whattteva]

Pretty sure the web UI has a tab to add static routes. I've never used it, but it should do what you want.

 

[subwire]

After much struggling, I finally got it all working!
Indeed there is the Static Routes page. That did the trick. This should probably get done for us though, so perhaps I'll file it as a bug.

Three other tips for those attempting the same setup: Set the DNS server used by each jail in /etc/resolv.conf to something not related to DHCP. This fouled me up for a bit until I realized the jails weren't getting updated when I switched networks. Using 8.8.8.8 does the trick, as usual. If you're connecting to potentially mismanaged networks in the first place, you probably want to set this system-wide anyway, as their DNS will probably be a performance issue.

Port forwards through the in-kernel ipfw NAT that FreeNAS is using are not particularly intuitive to write and not very well documented. I wanted to basically flatten the services running in my jails behind one IP, so this was an important step. I created a startup script that sets up all my forwards, and have that get kicked off as a post-init script via the WebUI's section for startup scripts.
Here's an example of the important part:

Code:

# Set IFACE to your external interface
ipfw nat 100 config if $IFACE reset same_ports \
  redirect_port tcp 192.168.1.21:9091 9091 \
  redirect_port tcp 192.168.1.19:8080 8080 \
  redirect_port tcp 192.168.1.27:7000 7000 \
  redirect_port tcp 192.168.1.23:5050 5050 \
  redirect_port tcp 192.168.1.25:8081 8081 \
...
(more forwards here)
...

The forwards get setup in one big long ipfw command, which I broke up with newlines for clarity. The first line sets up some basic options for the nat mechanism (nat number 100 seems to be the FreeNAS default), and the remaining redirect_port directives setup the forwards. They are backwards from how you'd expect, as the first one means "forward all packets coming in on interface $IFACE on port 9091 to 192.168.1.21:9091"; the source port is last.
This setup is pretty convenient for DHCP use, as it'll automatically work for whatever address $IFACE has.

Lastly, I had one jail in particular that for whatever reason would not send any traffic outside of the local subnet. Routing tables, DNS, etc etc all looked fine. After banging on it for hours, restarting the jail, recreating the jail etc without any success, I found it: The wrong entry for the NAT gateway had been permanently stuck in that jail's ARP table. I have absolutely no idea how this happened, as I've done nothing that would have bothered the ARP table in a persistent manner (who knows! some bizarre race condition? sunspots?) The quick fix was to jump into the jail and use arp -S <your bridge0 IP here>, followed by arp -s <bridge0 IP> <actual bridge0 MAC> The problem was instantaneously solved, and all is now well.

After all that, and having not seen much FreeNAS/NAT-related documentation out there, I may create some, compile my experience into a how-to. I can't be the only one attempting this, as such a setup is necessary for places that get angry about bringing your own network gear (eg. most universities)

 

[Whattteva]

Well, that's certainly true about restrictive networks, but the solution to that is rather simple. You stick that connection to your router wan port and let it do the ip masquerading. It is pretty darn cheap and painless way than that round about way of yours.

I imagine you'd want to connect your own personal machine anyway, so either way you'd be needing another port anyway.

 

[subwire]

I would be less worried about them finding my router on their network, and more concerned about them finding it physically. In the past, the enforcement end of things were looking for and confiscating consumer network gear, regardless of whether it was being used or not, as having the stuff at all was against their policy. Yes that's stupid, but it's cheap livin' :P
And I did/will have an extra port, so this is mainly about keeping the multiple IPs off their network so it doesn't look like I'm using a switch that I'm not actually using.

 

[eminguez]

I've just registered to say that after a few tests and reading this post, it worked for me.

Basically I wanted to have some jails without 'real' IPs (so no 192.168.X.Y/24) but having an internal only network (like 10.0.0.0/24), but those jails be able to reach 'the outside world' (aka NAT). So I configured the jail from the GUI as:

• IPv4: 10.0.1.5 (some IP I choose)
• IPv4 netmask: /24
• IPv4 bridge: 10.0.1.1
• IPv4 bridge netmask: /24
• IPv4 default gateway: 192.168.1.1 (so, the real router)
  
• Marked 'NAT'

Then, after the jail booted I wasn't able to reach the outside world... but after setting the 'defaultrouter' variable in /etc/rc.conf INSIDE the jail to the bridge IP, everything worked.

\o/

My questions are:

• Is there any official documentation that I missed on how to do it?
  
• Shouldn't this be much simpler? In other environments (like virt-manager for example) you can just create a 'Virtual Network' where you can specify a network range, NAT/isolated/bridged and some other parameters and that's it. It will create under the hood the pipes (bridges, firewall rules, dhcpd/dnsmasq entries, etc.) and you just select your network when you are booting your VM (I know jail != vm, it was just an example)

I now need to figure it out how to do it with iocage... :)

--- Edited ---

I've been able to do everything from the the GUI with the following jail settings:

• IPv4: 10.0.2.99 (some IP I choose)
• IPv4 netmask: /24
• IPv4 bridge: 10.0.2.1
• IPv4 bridge netmask: /24
• IPv4 default gateway: 10.0.2.1 (bridge IP)
• Sysctls: "allow.raw_sockets=true" (configured automatically by FreeNAS)
  
• Marked 'VIMAGE'
• Marked 'NAT'

Under the hood I've seen in /mnt/storage/jails/.jailname/ there are some files created with those settings as well as firewall rules created automatically for NAT (ipfw list):

Code:

00300 nat 100 ip from 10.0.2.99 to any out xmit bge0

Now my question is, how do translate that into iocage commands? I've been looking in the new ui but couldn't find any parameter to specify those settings...

Thanks!