OpenVPN Gateway Jail, cannot pass traffic for other jails

[KeithW]

Hi all,

I have been trying to set up an OpenVPN Jail to act as a Gateway for other Jails, namely sabnzbd and transmission. I have Open VPN Set up and running and it does connect to my VPN Service but it won't routemy other jails through the VPN.

I have searched for days trying to get this working. Some post mention setting up IPFW with details of the rules but none I have tried works, some stop my OpenVPN Jail from accessing the internet and some allow the OpenVPN but not the other Jails.

As far as I can tell the OpenVPN jail just isnt routing the traffic at all.

My /etc/rc.conf looks like this:

Show : rc.conf

Code:

hostname="openvpn"
cron_flags="$cron_flags -J 15"

# Disable Sendmail by default
sendmail_enable="NO"
sendmail_submit_enable="NO"
sendmail_outbound_enable="NO"
sendmail_msp_queue_enable="NO"

# Run secure syslog
syslogd_flags="-c -ss"

# Enable IPv6
ipv6_activate_all_interfaces="YES"

# Addition OpenVPN stuff
openvpn_enable="YES"
openvpn_configfile="/usr/local/etc/openvpn/openvpn.conf"
openvpn_if="tun"
gateway_enable="YES"
firewall_enable="YES"
firewall_script="/usr/local/etc/ipfw.rules"

My ipfw.rules look like this:

Show : ipfw.rules

Code:

ipfw -q -f flush
ipfw -q add nat 1 all from any to any via any

I have also tried several others like this:

Show : ipfw.rules alt

Code:

ipfw -q -f flush
ipfw -q nat 1 config if tun0
ipfw -q add nat 1 all from any to any via any

My OpenVPN Server is set up like this:

Show : Jail Config

Code:

CONFIG_VERSION:14
allow_chflags:0
allow_mlock:0
allow_mount:0
allow_mount_devfs:0
allow_mount_nullfs:0
allow_mount_procfs:0
allow_mount_tmpfs:0
allow_mount_zfs:0
allow_quotas:0
allow_raw_sockets:1
allow_set_hostname:1
allow_socket_af:0
allow_sysvipc:0
allow_tun:1
available:readonly
basejail:no
boot:on
bpf:no
children_max:0
cloned_release:11.2-RELEASE
comment:none
compression:lz4
compressratio:readonly
coredumpsize:off
count:1
cpuset:off
cputime:off
datasize:off
dedup:off
defaultrouter:192.168.1.1
defaultrouter6:none
depends:none
devfs_ruleset:4
dhcp:off
enforce_statfs:2
exec_clean:1
exec_fib:0
exec_jail_user:root
exec_poststart:/usr/bin/true
exec_poststop:/usr/bin/true
exec_prestart:/usr/bin/true
exec_prestop:/usr/bin/true
exec_start:/bin/sh /etc/rc
exec_stop:/bin/sh /etc/rc.shutdown
exec_system_jail_user:0
exec_system_user:root
exec_timeout:60
host_domainname:none
host_hostname:openvpn
host_hostuuid:openvpn
host_time:yes
hostid:00000000-0000-0000-0000-448A5B8ADA25
hostid_strict_check:off
interfaces:vnet0:bridge0
ip4:new
ip4_addr:vnet0|192.168.1.30/24
ip4_saddrsel:1
ip6:new
ip6_addr:none
ip6_saddrsel:1
jail_zfs:off
jail_zfs_dataset:iocage/jails/openvpn/data
jail_zfs_mountpoint:none
last_started:2019-03-03 17:34:17
login_flags:-f root
mac_prefix:448a5b
maxproc:off
memorylocked:off
memoryuse:off
mount_devfs:1
mount_fdescfs:1
mount_linprocfs:0
mount_procfs:0
mountpoint:readonly
msgqqueued:off
msgqsize:off
nmsgq:off
notes:none
nsemop:off
nshm:off
nthr:off
openfiles:off
origin:readonly
owner:root
pcpu:off
priority:99
pseudoterminals:off
quota:none
release:11.2-RELEASE-p9
reservation:none
resolver:/etc/resolv.conf
rlimits:off
securelevel:2
shmsize:off
stacksize:off
state:up
stop_timeout:30
swapuse:off
sync_state:none
sync_target:none
sync_tgt_zpool:none
sysvmsg:new
sysvsem:new
sysvshm:new
template:no
type:jail
used:readonly
vmemoryuse:off
vnet:on
vnet0_mac:448a5b363831 448a5b363832
vnet1_mac:none
vnet2_mac:none
vnet3_mac:none
vnet_default_interface:auto
vnet_interfaces:none
wallclock:off

If anyone could point me in the right direction I would be very grateful. I could just run up a VM with Ubuntu on it and run everything from there but it seems like that is taking the easy way out and also wasting the features of FreeNAS.

 

[Heracles]

Hi again Keith,

What are you trying to achieve here ?

Remote access from outside to your jails ?
Providing Internet access to your jails over that VPN ?
Both ?
Something else ?

We need to know what you try to do before telling you how to do it

 

[KeithW]

Hi Heracles,

I am trying to Provide internet access to the jails over the VPN Service which is IPVanish. OpenVPN does connect to IPVanish and gets a valid IP Address and reports a different external IP to my ISP's so that is working, it is just getting the routing working that I am struggling with. I have set the default gateway on the other jails to point to the OpenVPN IP Address.

This is the result of ifconfig:

Code:

lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1
        inet 127.0.0.1 netmask 0xff000000
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        groups: lo
epair0b: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=8<VLAN_MTU>
        ether 44:8a:5b:36:38:32
        hwaddr 02:39:d0:00:07:0b
        inet 192.168.1.30 netmask 0xffffff00 broadcast 192.168.1.255
        nd6 options=1<PERFORMNUD>
        media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
        status: active
        groups: epair
tun0: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> metric 0 mtu 1500
        options=80000<LINKSTATE>
        inet 172.21.21.194 --> 172.21.20.1 netmask 0xfffffe00
        nd6 options=1<PERFORMNUD>
        groups: tun
        Opened by PID 22743

 

[John Doe]

I am not too sure if FreeNAS can do this. I think the routing will be handled by freenas, not from one jail to another. I assume you would need something on the layer under freenas.

An option would be a router with custom firmware a la ddwrt, openwrt etc.
or
You just set up everything in the same jail.

one point, since there is transmission and an Ip vanischer... consider implementing a "kill switch".
what is a killswitch?

 

[Heracles]

Hey Keith,

First, you need to realize that there will always be at least one entity that will be able to trace you at network level. With the setup you are doing, you are transferring that privilege from your ISP to that VPN provider. But still, there is and will always be an entity who will be able to trace you at network level, by design.

Second, what are you doing for all other tracking information that are way more reliable than IP source ? Do you know how unique your browser is ? With all details of which plugins are installed, their exact version, the browser and its version, the OS and its version and more... Usually, a browser has a fingerprint of 1 in 100 000.

Add to that all the tracking that you offer on a silver plate like cookies and location services in your smartphone and tablet. Are you really sure IP Vanish is of any benefit ?

So before working that hard on an illusion of privacy, to understand the global situation may be helpful and turn the entire exercise of IP Vanish to not needed or even desirable at all.

As for your config, are you sure that your OpenVPN as routing turned on ? Did you check your layer 2 connectivity between your jails and OpenVPN ?

Usually, it is easier to do VPN on the router facing Internet than on a FreeNAS. What kind of router do you have ?

 

[KeithW]

I am not too sure if FreeNAS can do this. I think the routing will be handled by freenas, not from one jail to another. I assume you would need something on the layer under freenas.

An option would be a router with custom firmware a la ddwrt, openwrt etc.
or
You just set up everything in the same jail.

one point, since there is transmission and an IP vanischer... consider implementing a "kill switch".
what is a killswitch?

There are a few posts where people have it working on older version of FreeNAS but their configs do not work for me in the latest version. I would hope to setup a killswitch with ipfw rules once I get it working at all. I would set this up on my router but from my experience a router is the worst place to try to do VPN due to the poor performance, I have a Linksys WRT1900ACS which I bough specifically for this purpose and it simply wasn't powerful enough. If necessary I will build a VPN Gateway server but I am trying to consolidate my hardware due to impending space limitations.

 

[KeithW]

Hey Keith,

First, you need to realize that there will always be at least one entity that will be able to trace you at network level. With the setup you are doing, you are transferring that privilege from your ISP to that VPN provider. But still, there is and will always be an entity who will be able to trace you at network level, by design.

Second, what are you doing for all other tracking information that are way more reliable than IP source ? Do you know how unique your browser is ? With all details of which plugins are installed, their exact version, the browser and its version, the OS and its version and more... Usually, a browser has a fingerprint of 1 in 100 000.

Add to that all the tracking that you offer on a silver plate like cookies and location services in your smartphone and tablet. Are you really sure IP Vanish is of any benefit ?

So before working that hard on an illusion of privacy, to understand the global situation may be helpful and turn the entire exercise of IP Vanish to not needed or even desirable at all.

As for your config, are you sure that your OpenVPN as routing turned on ? Did you check your layer 2 connectivity between your jails and OpenVPN ?

Usually, it is easier to do VPN on the router facing Internet than on a FreeNAS. What kind of router do you have ?

The VPN is less for privacy and more for the fact my ISP Throttles certain traffic to very low speeds and a VPN solves this. Also my browser isn't in the equation as I am only trying to allow downloads to work at sane speeds.

As mentioned in my previous post, routers are terrible for VPN due to the speed issues.

 

[Heracles]

Remember that your Internet pipe has a limited upload capacity and that speed is usually way below the maximum download speed. If your upload is 10 Mbits, you do not need more than 10 Mbits of VPN throughput to saturate it.

Here, I used DD-WRT on my D-Link DIR-880L and it works very well. DD-WRT can be OpenVPN client, server or both.

To flash your router to a Linux kernel like DD-WRT may be an option for a much cleaner networking setup.

Other way, how is your DNS configured ? Your setup may be working at IP level but without DNS, not work at user level...

 

[KeithW]

Remember that your Internet pipe has a limited upload capacity and that speed is usually way below the maximum download speed. If your upload is 10 Mbits, you do not need more than 10 Mbits of VPN throughput to saturate it.

Here, I used DD-WRT on my D-Link DIR-880L and it works very well. DD-WRT can be OpenVPN client, server or both.

To flash your router to a Linux kernel like DD-WRT may be an option for a much cleaner networking setup.

Other way, how is your DNS configured ? Your setup may be working at IP level but without DNS, not work at user level...

I am not uploading with my VPN, just downloading so I can download over VPN at 80Mbps but my ISP throttles that traffic to 2Mbps and a router set up with VPN is far far slower than a computer due to the low powered CPU. I have a Linksys WRT 1900ACS which has a 1.6 GHz dual-core processor and when configure with DD_WRT and using OpenVPN not only is painfully slow for downloading but also slows down non VPN traffic.

I am not sure what you mean about how my DNS is configured. Do you mean on my Router or on my FreeNAS?

 

[Heracles]

If you are routing all trafic through the VPN and your DNS is not be reached over VPN, then you are routing your DNS trafic wrong and your clients can not resolve DNS names. Without DNS resolution, they will basically not access anything.

So, I will wait for your layer 2 connectivity results and your DNS and will try to work from there after that.

As for your ISP, should mine do something like that, I would just run away to another provider ;-)

 

[Heracles]

As for the speed you can achieve on a router, know that your router can do well over 90 Mbits...
https://forum.dd-wrt.com/phpBB2/viewtopic.php?t=308285&sid=67449e64abf71de2147003ce0b172ed4

 

[KeithW]

As for the speed you can achieve on a router, know that your router can do well over 90 Mbits...
https://forum.dd-wrt.com/phpBB2/viewtopic.php?t=308285&sid=67449e64abf71de2147003ce0b172ed4

That is the forum post I based my purchase on but apart from "speed tests" it never reaches anywhere near those speeds.

 

[LeoSum]

Hey Keith,
I think I managed to do what you are looking for.
My setup is working and described here:
https://forums.freebsd.org/threads/jail-as-openvpn-gateway-with-killswitch-95-done.69913/

However I am still missing one final step.

I would like the jail to no longer route any traffic through, if the OpenVPN connection is down.

Still looking for the last bit in my ipfw config.

 

[KeithW]

Hi LeoSum,

Thanks, I had seen your post while searching for a solution but I didn't understand what you were doing to make it work as a gateway and I thought there might be something different on FreeNAS compared to FreeBSD which makes what you are doing work in FreeBSD but not FreeNAS. Is it the NAT Part that makes it work as a gateway?

In your IPFW script you aren't using any rule numbers so I wonder if the deny rule at the end is being applied first or something?

 

[LeoSum]

Hi Keith,
my setup is running FreeNAS too.

I'm no expert, but from what I have learned so far, to work as a gateway you need the gateway_enable="YES" part in your rc.conf, which you do.
Also it is important that you enable VIMAGE for the jail, so that it has its own ipstack. I can't see that from your jail config, as I just configure my jails via the web-GUI.

As for the rule numbers: If you just add rules from a script without assigning numbers, ipfw will assign numbers in ascending order with steps of 100 between. So the outcoming ruleset that my script applies looks like this:

Code:

root# ipfw list

00100 nat 1 ip from 192.xxx.xxx.0/24 to any out via tun2
00200 nat 1 ip from any to any in via tun2
00300 allow log udp from 192.xxx.xxx.0/24 to 208.67.222.222 dst-port 53 keep-state
00400 allow log udp from 192.xxx.xxx.0/24 to 91.239.100.100 dst-port 53 keep-state
00500 allow ip from 192.xxx.xxx.0/24 to 192.xxx.xxx.0/24 keep-state
00600 allow ip from 192.xxx.xxx.0/24 to yyy.yyy.yyy.yyy keep-state
00700 allow ip from 127.0.0.1 to any
00800 allow ip from zzz.zzz.aa.0/23 to any
00900 allow ip from zzz.zzz.bb.0/23 to any
01000 allow ip from zzz.zzz.cc.0/23 to any
01100 allow ip from zzz.zzz.dd.0/23 to any
01200 allow ip from zzz.zzz.ee.0/23 to any
01300 allow ip from zzz.zzz.ff.0/23 to any
01400 allow ip from any to zzz.zzz.aa.0/23
01500 allow ip from any to zzz.zzz.bb.0/23
01600 allow ip from any to zzz.zzz.cc.0/23
01700 allow ip from any to zzz.zzz.dd.0/23
01800 allow ip from any to zzz.zzz.ee.0/23
01900 allow ip from any to zzz.zzz.ff.0/23
65535 allow ip from any to any

now when I enable the last line, I get an additional 02000 allow ip from any to any, but then only traffic from within the jail can get to the internet, the gateway part stops working.

I guess that I need to additionally catch the inbound, to-be-gatewayed packets from the local network with some additional allow rule, but I couldn't figure that out so far. I also don't know how to "look" at the denied packets in some log file to see their "from" and "to" addresses to try and figure it out from there ...

On other Idea I have is to create a down.sh ipfw script that denies all traffic and is trigered by OpenVPN via the down command, but that wouldn't be too robust against OpenVPN crashing or something.

But if this part is not important for you and you don't mind if packets are still gatewayed untunneled in case the VPN drops, you should be able to copy my setup.

 

[Sasquatch]

Did you find solution?
Im trying to setup Openvpn jail client to enable remote replication via secure channel.
As soon as I open port 22 for ssh it gets spammed to oblivion, hence openvpn approach.
Router VPN won't run above 5mb/s on 20mb/s connection, router CPU is bottlenecking.
I have Opnevpn setup in jail but it won't route anything.
tried setting up ipfw.rules, but anything I could find on here didn't work, usually was killing all network access.

last setup befor i gave up:

Code:

root@VPN:~ # ifconfig
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1
        inet 127.0.0.1 netmask 0xff000000
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        groups: lo
epair0b: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=8<VLAN_MTU>
        ether 44:8a:5b:bd:91:b7
        hwaddr 02:86:d0:00:07:0b
        inet 192.168.1.119 netmask 0xffffff00 broadcast 192.168.1.255
        nd6 options=1<PERFORMNUD>
        media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
        status: active
        groups: epair
tun0: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> metric 0 mtu 1500
        options=80000<LINKSTATE>
        inet 10.8.0.9 --> 10.8.0.1 netmask 0xffffff00
        nd6 options=1<PERFORMNUD>
        groups: tun
        Opened by PID 26146

rc.conf

Code:

root@VPN:~ # cat /etc/rc.conf
ifconfig_epair0b="DHCP"
hostname="VPN"
cron_flags="$cron_flags -J 15"

# Disable Sendmail by default
sendmail_enable="NO"
sendmail_submit_enable="NO"
sendmail_outbound_enable="NO"
sendmail_msp_queue_enable="NO"

# Run secure syslog
syslogd_flags="-c -ss"

# Enable IPv6
ipv6_activate_all_interfaces="YES"
openvpn_enable="YES"
openvpn_configfile="/usr/local/etc/openvpn/openvpn.conf"
gateway_enable="YES"
firewall_enable="YES"
firewall_script="/usr/local/etc/ipfw.rules"

ipfw.rules

Code:

root@VPN:~ # cat /usr/local/etc/ipfw.rules
ipfw -q -f flush
ipfw -q nat 1 config if tun0
ipfw -q add nat 1 all from any to any out via any

ping response(same for all ip granges):

Code:

PING 192.168.xx.xx (192.168.xx.xx): 56 data bytes
ping: sendto: Permission denied
ping: sendto: Permission denied

What's strange is that if i manually add same ipfw rules ping is just not getting any responses.

 

[KeithW]

Did you find solution?
Im trying to setup Openvpn jail client to enable remote replication via secure channel.
As soon as I open port 22 for ssh it gets spammed to oblivion, hence openvpn approach.
Router VPN won't run above 5mb/s on 20mb/s connection, router CPU is bottlenecking.
I have Opnevpn setup in jail but it won't route anything.
tried setting up ipfw.rules, but anything I could find on here didn't work, usually was killing all network access.

last setup befor i gave up:

Code:

root@VPN:~ # ifconfig
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1
        inet 127.0.0.1 netmask 0xff000000
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        groups: lo
epair0b: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=8<VLAN_MTU>
        ether 44:8a:5b:bd:91:b7
        hwaddr 02:86:d0:00:07:0b
        inet 192.168.1.119 netmask 0xffffff00 broadcast 192.168.1.255
        nd6 options=1<PERFORMNUD>
        media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
        status: active
        groups: epair
tun0: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> metric 0 mtu 1500
        options=80000<LINKSTATE>
        inet 10.8.0.9 --> 10.8.0.1 netmask 0xffffff00
        nd6 options=1<PERFORMNUD>
        groups: tun
        Opened by PID 26146

rc.conf

Code:

root@VPN:~ # cat /etc/rc.conf
ifconfig_epair0b="DHCP"
hostname="VPN"
cron_flags="$cron_flags -J 15"

# Disable Sendmail by default
sendmail_enable="NO"
sendmail_submit_enable="NO"
sendmail_outbound_enable="NO"
sendmail_msp_queue_enable="NO"

# Run secure syslog
syslogd_flags="-c -ss"

# Enable IPv6
ipv6_activate_all_interfaces="YES"
openvpn_enable="YES"
openvpn_configfile="/usr/local/etc/openvpn/openvpn.conf"
gateway_enable="YES"
firewall_enable="YES"
firewall_script="/usr/local/etc/ipfw.rules"

ipfw.rules

Code:

root@VPN:~ # cat /usr/local/etc/ipfw.rules
ipfw -q -f flush
ipfw -q nat 1 config if tun0
ipfw -q add nat 1 all from any to any out via any

ping response(same for all IP granges):

Code:

PING 192.168.xx.xx (192.168.xx.xx): 56 data bytes
ping: sendto: Permission denied
ping: sendto: Permission denied

What's strange is that if i manually add same ipfw rules ping is just not getting any responses.

Sorry, no. I eventually gave up and installed Proxmox with XFS support and am using that with a mix of VM's and containers for my needs. After my efforts I have come to the conclusion that if you want to do something with FreeNAS that is not an official feature you may as well forget it as even if you do get it to work any update may break it and you will get no support from here if it is not a supported feature.

If you want a NAS and only a NAS then FreeNAS may be an option but otherwise look else where.

 

[LeoSum]

As soon as I open port 22 for ssh it gets spammed to oblivion, hence openvpn approach.

Have you thought about using a non-standard port other than 22 for ssh?

If you want a NAS and only a NAS then FreeNAS may be an option but otherwise look else where.

I think that is a little harsh. I find the whole jail concept very suitable for lots of things with way less overhead and the ability to easily snapshot and roll back when messing around with stuff. I have come to find that more convenient than running several full fledged VMs.

 

[Sasquatch]

Have you thought about using a non-standard port other than 22 for ssh?

Yes, after 1-3 days it gets spammed too, only openvpn (nonstandard) port is fairly save, gets probed couple times a day, but that's all.
I'm on dynamic IP no idea how or why i'm a target. Used to mine some crypto currencies, but it must be 10'th IP since I stopped and scrapped all rigs.

I figured Jail routing out:
1: Openvpn server needed restarting, no config changes so...???
2: configuring client jail firewall as open and leaving ipfw.rules empty worked for me.
/etc/rc.conf

Code:

(...)
gateway_enable="YES"
firewall_enable="YES"
firewall_type="open"

generated automatically:

Code:

# ipfw list
00100 allow ip from any to any via lo0
00200 deny ip from any to 127.0.0.0/8
00300 deny ip from 127.0.0.0/8 to any
00400 deny ip from any to ::1
00500 deny ip from ::1 to any
00600 allow ipv6-icmp from :: to ff02::/16
00700 allow ipv6-icmp from fe80::/10 to fe80::/10
00800 allow ipv6-icmp from fe80::/10 to ff02::/16
00900 allow ipv6-icmp from any to any ip6 icmp6types 1
01000 allow ipv6-icmp from any to any ip6 icmp6types 2,135,136
65000 allow ip from any to any
65535 allow ip from any to any

3: Server side routing was wonky, traffic back from server's subnet had one extra hop.