Running jails behind IPv4 NAT

[Alec Edworthy]

Hi,

I'm hoping someone on here can help me with what sounds like it should be a really easy thing but has baffled me so far. I have a server running FreeNAS 9.2.1.3 with one ethernet interface which has a public IPv4 address and a public IPv6 address associated with it. I want to create some jails to run other software in but rather than them getting IPv4 addresses from the public range the box is on I want to run up a private network and NAT them out onto the public network as necessary. I am happy to allocate them public IPv6 addresses (I've plenty of them spare) but their IPv4 addresses need to be private. I have succeeded in getting a jail up and running with another IPv4 address from the public range allocated to it but have struggled to create one using a NAT'd range instead. For the sake of arguments let's assume my FreeNAS's external IP address is 10.10.128.252 (it isn't, that's clearly a private IP address but I didn't really want to put my real public IP in a posting ;)) and I want to use 192.168.123.0/24 as the IP address range for the jail(s).

I've tried setting the jail IP address range to 192.168.123.0/24 with .1 and .254 as the upper and lower IP address limits in the Jails -> Configuration section and then adding a jail with an address from that range but when it comes up it has no connectivity. I am guessing I'm doing one little bit wrong but I'll be damned if I can work out what it is. Please can someone hold my hand through getting this up and running? I am sure I could do it back hacking around with FreeNAS to enable NAT manually and enabling packet forwarding in ipfw etc. but I'd rather do it through more approved methods so that it is done in a FreeNAS friendly way, and hopefully one which will easily survive upgrades (and where possible a clean re-install and configuration file restore) rather than hacking the base OS.

Am I asking for the moon on a stick and/or expecting more from FreeNAS than I should? Is this beyond the scope of a normal FreeNAS set up?

Lastly a bit of background about me: I am very much at home working on the command line of a Linux server (Linux user for over 17 years, looking after Linux server in one form or another for around 12 of those) but am relatively new to FreeBSD et al. I work for the networks team at a university in the UK and so am also very much at home with IP addresses, ethernet etc. This is in part why I am so frustrated that I'm having to resort to a posting here asking for help, this should be well within my skill set but it's eluded me so far Please don't be afraid to be technical in any responses (but also don't be afraid to teach me to suck eggs if I'm really being that dumb).

Thank you in advance for your help.

Kind regards, Alec

 

[Alec Edworthy]

OK, so it turns out the only missing part was the default route! Adding the route manually in the jail makes connectivity appear.

Code:

root@nattest-6:/ # netstat -nr
Routing tables
 
Internet:
Destination        Gateway            Flags    Refs      Use  Netif Expire
127.0.0.1          link#1            UH          0        0    lo0
192.168.254.0/24  link#2            U          0        0 epair6
192.168.254.8      link#2            UHS        0        0    lo0
 
Internet6:
Destination                      Gateway                      Flags      Netif Expire
::/96                            ::1                          UGRS        lo0
::1                              link#1                        UH          lo0
::ffff:0.0.0.0/96                ::1                          UGRS        lo0
fe80::/10                        ::1                          UGRS        lo0
fe80::%lo0/64                    link#1                        U          lo0
fe80::1%lo0                      link#1                        UHS        lo0
ff01::%lo0/32                    ::1                          U          lo0
ff02::/16                        ::1                          UGRS        lo0
ff02::%lo0/32                    ::1                          U          lo0
root@nattest-6:/ # ping -c 1 131.231.128.1
PING 131.231.128.1 (131.231.128.1): 56 data bytes
ping: sendto: No route to host
^C
--- 131.231.128.1 ping statistics ---
1 packets transmitted, 0 packets received, 100.0% packet loss
root@nattest-6:/ # route add default 192.168.254.1
add net default: gateway 192.168.254.1
root@nattest-6:/ # ping -c 1 131.231.128.1
PING 131.231.128.1 (131.231.128.1): 56 data bytes
64 bytes from 131.231.128.1: icmp_seq=0 ttl=254 time=0.393 ms
 
--- 131.231.128.1 ping statistics ---
1 packets transmitted, 1 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.393/0.393/0.393/0.000 ms
root@nattest-6:/ #

Adding a defaultrouter line to /etc/rc.conf adds the route every time the jail starts,

Code:

root@nattest-6:/ # grep defaultrouter /etc/rc.conf
root@nattest-6:/ # echo 'defaultrouter="192.168.254.1"' >> /etc/rc.conf
root@nattest-6:/ # grep defaultrouter /etc/rc.conf
defaultrouter="192.168.254.1"
root@nattest-6:/ # exit
[root@ccae-freenas /mnt/Vol-3TB/jails/.testnat.meta]# warden stop nattest-6; warden start nattest-6
Stoppingthejail....192.168.254.8 (192.168.254.8) deleted
Unmounting user-supplied file-systems
umount: fstab reading failure
Stopping jail with: /etc/rc.shutdown
...DoneMounting user-supplied file-systems
jail -c path=/mnt/Vol-3TB/jails/nattest-6 name=nattest-6 host.hostname=nattest-6 allow.raw_sockets=true persist vnet
Setting IP4 address: 192.168.254.8/24
ipfw nat 100 config if bge0 reset same_ports unreg_only log
ipfw nat 100 config if bge0 log same_ports unreg_only reset
ipfw add nat 100 all from 192.168.254.8 to any out xmit bge0
01300 nat 100 ip from 192.168.254.8 to any out xmit bge0
Starting jail with: /etc/rc
[root@ccae-freenas /mnt/Vol-3TB/jails/.testnat.meta]# jls
  JID  IP Address      Hostname                      Path
..snip..
  13  -              nattest-6                    /mnt/Vol-3TB/jails/nattest-6
[root@ccae-freenas /mnt/Vol-3TB/jails/.testnat.meta]# jexec 13 csh
root@nattest-6:/ # netstat -nr
Routing tables
 
Internet:
Destination        Gateway            Flags    Refs      Use  Netif Expire
default            192.168.254.1      UGS        0        0 epair6
127.0.0.1          link#1            UH          0        0    lo0
192.168.254.0/24  link#2            U          0        0 epair6
192.168.254.8      link#2            UHS        0        0    lo0
 
Internet6:
Destination                      Gateway                      Flags      Netif Expire
::/96                            ::1                          UGRS        lo0
::1                              link#1                        UH          lo0
::ffff:0.0.0.0/96                ::1                          UGRS        lo0
fe80::/10                        ::1                          UGRS        lo0
fe80::%lo0/64                    link#1                        U          lo0
fe80::1%lo0                      link#1                        UHS        lo0
ff01::%lo0/32                    ::1                          U          lo0
ff02::/16                        ::1                          UGRS        lo0
ff02::%lo0/32                    ::1                          U          lo0
root@nattest-6:/ # ping -c 1 131.231.128.1
PING 131.231.128.1 (131.231.128.1): 56 data bytes
64 bytes from 131.231.128.1: icmp_seq=0 ttl=254 time=0.501 ms
 
--- 131.231.128.1 ping statistics ---
1 packets transmitted, 1 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.501/0.501/0.501/0.000 ms
 
root@nattest-6:/ #

I'm guessing at this stage I need to file a bug for the missing default route when creating a jail using NAT.

 

[jgreco]

Sorry you didn't get any responses, but you seem to have managed very well. Good job.

 

[Alec Edworthy]

Sorry you didn't get any responses, but you seem to have managed very well. Good job.

Thank you, fortunately I have a friend who is very good with FreeBSD and so he was able to set me on the right route for fixing it.

Do you agree that the lack of a default route sounds like a bug in FreeNAS?

Alec

 

[jgreco]

Isn't clear to me one way or another. We would never put a non-storage app on a filer so basically we don't use FreeNAS jails here. I see them as more of a home user thing.

As you appear to be a University sysadmin you'll be familiar with the complexities of modern computing systems and so I encourage you to come to your own conclusion as to whether it is a bug or just something that you maybe didn't configure in the way the developers expected or something like that.