[HowTo] Install ZeroTier in Jail as Gateway

[Vislike]

Hello all, I am a long time reader, frist time poster. I have been running a Freenas system for about 5 years. I have only ever used the WebGUI and accessed my Freenas via Samba and SSH. So this is my frist attempt to setup a VPN service.

I settled on ZeroTier, since that is what i have been using between my desktop and my laptop and it has worked flawlessly. But when searching i have had a hard time finding a guide how to do it, so i had to figure it out from a lot of different bits and pieces. And now i decided to create my own guide to help others who are on the same mission.

Disclaimer
This is the first time i use a Jail ever, so i am not sure of what i am doing is considered good practice or possible opposite, so i would happily receive some feedback of how i could improve.

Goals

• Setup ZeroTier in VNET Iocage Jail.
• Setup Jail as Gateway between ZeroTier network and LAN.
• Setup static route for split-tunnel to send LAN traffic over ZeroTier, but use local connection for internet.

Prerequisite

• Create ZeroTier Network in Central
• Consider setting up LAN to use a more unique range instead of standard 192.168.0.0/24 or 192.168.1.0/24 to avoid split-tunnel static route conflicts.
• Split tunnel will be configured using trick to set net mask to /23, see ZeroTier links in references.

Step 1 - Add Init Devfs Script to create RuleSet for ZeroTier Jail
Default ruleset will hide tap* and zt* devices, so we need to create a new ruleset for this jail:
# Tasks -> Init/Shutdown Scripts
Description -> Zerotier - Devfs Rule 20
Type -> Command
When -> Post Init
Command:

Code:

devfs rule -s 20 add include 1 && devfs rule -s 20 add include 2 && devfs rule -s 20 add include 3 && devfs rule -s 20 add include 4 && devfs rule -s 20 add path 'bpf*' unhide && devfs rule -s 20 add path 'tap*' unhide && devfs rule -s 20 add path 'zt*' unhide

To continue either reboot for the rules to take effect, or enter the command manually in shell.
Verify with:
devfs rule -s 20 show

Step 2 - Create ZeroTier Jail
I only hade to change one setting from the defaults for it to work, so this is how i created the jail:
# Jails -> Add
Name -> ZeroTier-Gateway
Release -> 12.3-RELEASE
# Next
VNET
DHCP # Feel free to config the network however you want.
# Next
# Submit
# Edit Jail
Auto-start
# Jail Properties
devfs_ruleset -> 20
# Save

Step 3 - Install ZeroTier in Jail and join network
# Start and login to jail
iocage console ZeroTier-Gateway

# Install ZeroTier and Nano text editor
pkg install zerotier nano

# Enable auto-start of ZeroTier
nano /etc/rc.conf

Code:

# ZeroTier
zerotier_enable="YES"

# Time to start ZeroTier and join our network
service zerotier start
zerotier-cli join *networkID*

# Login to ZeroTier Central and auth the Jail
# Now its time to test the connection
ifconfig
ping *some other host in zeronet*
ping *jail ip from other host*

Now is a good time to check so everything persists, with jail auto-start and devfs rules.
So i suggest a reboot of truenas and see if everything comes up automatically.
ZeroTier announces to the network every 120 seconds, so can take some time before it is reachable after reboot.
But it should be reachable from another host on zeronet within a few minutes.

Step 4 - Configure Gateway
There are multiple ways to configure the gateway, each with its own pros and cons:

• Bridged - VPN and LAN is placed on same sub-net.
• Routed - VPN and LAN use different sub-net and static routes are added on both sides to route traffic.
• NAT - VPN and LAN use different sub-net static route is configured via ZeroTier for VPN, and traffic to LAN is NAT:ed at the gateway.

The following steps are for NAT method using IPFW:

# In ZeroTier Central add split-tunnel route:
# Advanced -> Managed Routes -> Add Routes
# To home network via ip of ZeroTier jail.
Destination -> 192.168.78.0/23 # Example, use your own (this is the /23 trick)
(via) -> 10.147.17.1 # Enter ZeroTier ip of Jail

# Back in Jail we need to enable autostart of IPFW and Gateway
nano /etc/rc.conf

Code:

# Gateway
gateway_enable="YES"

# IPFW
firewall_enable="YES"
firewall_nat_enable="YES"
firewall_script="/etc/ipfw.rules"

# According to FreeBSD Handbook to use kernel NAT we must turn off TCP segmentation offloading
nano /etc/sysctl.conf

Code:

# Disable TCP segmentation offloading (TSO) for Kernel NAT.
net.inet.tcp.tso="0"

# Restart Jail for changes to take effect

This is the firewall script i have come up with while learning IPFW, feel free to make your own adjustments:
nano /etc/ipfw.rules

Code:

#!/bin/sh

# Flush out old rules
ipfw -q -f flush

# Shorthand alias to add rule
add="ipfw -q add"

# Interfaces
lan_if="epair0b"
zt_if="zt*"

# Configure NAT
# Required if firewall should process NATed packets
ipfw disable one_pass
# Setup NAT service 1
ipfw -q nat 1 config if $lan_if same_ports unreg_only reset

# Antispoof for Network Hygiene
$add deny ip from any to any not antispoof in

# Allow everything on Loopback & ZeroTier (both in and out)
$add allow all from any to any via lo0
$add allow all from any to any via $zt_if

# Because of the behavior of in-kernel NAT it is advised to place a reassemble rule just before the first NAT rule and after the rules that allow traffic on trusted interface
$add reass all from any to any in

# NAT any inbound packets from LAN and immediately allow them
$add nat 1 ip from any to any in via $lan_if
$add allow all from any to any in via $lan_if

# NAT any outbound packets to LAN and immediately allow them
$add nat 1 ip from any to any out via $lan_if
$add allow all from any to any out via $lan_if

# Should not be reachable
$add deny all from any to any

# Reload and check rules
service ipfw restart
ipfw list
ipfw show
ipfw nat 1 show config
ipfw zero

# Once again everything should survive a reboot

This config allows all other nodes on the ZeroTier network to access your local LAN, if you need bidirectional access, i suggest trying one of the other methods, Routed or Bridged.

References

Route between ZeroTier and Physical Networks - ZeroTier Knowledge Base - Confluence

zerotier.atlassian.net

Bridge your ZeroTier and local network with a RaspberryPi - ZeroTier Knowledge Base - Confluence

zerotier.atlassian.net

https://community.openvpn.net/openvpn/wiki/BridgingAndRouting

How-To: Setup a Wireguard VPN Server in a Jail

[DRAFT] open to comments Goal To setup a VPN server based on the Wireguard technology and running from within a Jail. The VPN server would allow remote devices to connect and access resources in the local network All remote traffic should be...

www.truenas.com

[HowTo] Install Tailscale in a Jail

So in the past few days I have been playing around with Tailscale for the one day when I finally get back on the road for work and I want to access things at home like my files or my Plex instance when I finally get around to uploading media to my NAS. I looked around and fond that the...

www.truenas.com

Chapter 33. Firewalls

FreeBSD has three firewalls built into the base system: PF, IPFW, and IPFILTER. This chapter covers how to define packet filtering rules, the differences between the firewalls built into FreeBSD and how to use them

docs.freebsd.org

ipfw(8)

www.freebsd.org

iocage

www.freebsd.org

 

[sretalla]

Nice writeup. For those having trouble with it, I would recommend Tailscale as linked above as an alternative.

 

[ilium007]

Can anyone add the steps to get a routed connection working, I'm not using NAT with the jail. I have got ZeroTier installed and connected. I have a network route setup in my.zerotier.com for my internal network but can't get traffic in or out of the TrueNAS jail.