OpenVPN iocage jail stopped working...

[spotcatbug]

I had a working OpenVPN iocage jail. It stopped working. Unfortunately, I hadn't used the VPN in a while (a couple of months), so I don't know exactly when it stopped working - that is, I'm not sure what I did to make it break. I've been spending a lot of time on this. I need help! Please.

What does "stopped working" mean? I couldn't connect the VPN on my iPhone (attempting to connect to the server timed-out). I discovered that the jail had no networking. It failed the ping test. After much fiddling ping works and I can connect to the server from the client (my iPhone on the cell network, will connect to the OpenVPN server on my LAN). However, even though the connection looks good and established, I can't connect to anything on my LAN through the VPN.

I remember having this issue when I first set-up the jail. Same symptoms: connection looked good but no traffic through the VPN. Turned out I needed a static route on my router to send the VPN subnet (10.8.0.0/24) traffic to the OpenVPN server (10.0.1.52/24). I added that static route to my home router and that fixed it. Not this time, though. The static route is there still. I tried removing it and putting it back, in case the router was having an issue - no change.

I'm guessing there's some other networking issue at play, but I'm not good enough at this to know what that is. I'm hoping somebody looking at my ifconfig listings might see a problem or have ideas.

host ifconfig:


igb0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM>
ether d0:50:99:c2:f0:1f
hwaddr d0:50:99:c2:f0:1f
nd6 options=9<PERFORMNUD,IFDISABLED>
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
em0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=98<VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM>
ether d0:50:99:c2:f0:1f
hwaddr d0:50:99:c2:f0:20
nd6 options=9<PERFORMNUD,IFDISABLED>
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
inet6 ::1 prefixlen 128
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3
inet 127.0.0.1 netmask 0xff000000
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
groups: lo
lagg0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=98<VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM>
ether d0:50:99:c2:f0:1f
inet 10.0.1.114 netmask 0xffffff00 broadcast 10.0.1.255
inet 10.0.1.63 netmask 0xffffff00 broadcast 10.0.1.255
nd6 options=9<PERFORMNUD,IFDISABLED>
media: Ethernet autoselect
status: active
groups: lagg
laggproto loadbalance lagghash l2,l3,l4
laggport: igb0 flags=4<ACTIVE>
laggport: em0 flags=4<ACTIVE>
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
ether 02:81:c5:8a:b2:00
nd6 options=1<PERFORMNUD>
groups: bridge
id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
bridge1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
ether 02:81:c5:8a:b2:01
nd6 options=9<PERFORMNUD,IFDISABLED>
groups: bridge
id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
member: vnet0:5 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 6 priority 128 path cost 2000
member: epair2a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 9 priority 128 path cost 2000
member: epair1a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 8 priority 128 path cost 2000
member: lagg0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 4 priority 128 path cost 10000
epair1a: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=8<VLAN_MTU>
ether 02:9d:50:00:08:0a
hwaddr 02:9d:50:00:08:0a
nd6 options=1<PERFORMNUD>
media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
status: active
groups: epair
epair2a: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=8<VLAN_MTU>
ether 02:9d:50:00:09:0a
hwaddr 02:9d:50:00:09:0a
nd6 options=1<PERFORMNUD>
media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
status: active
groups: epair
vnet0:5: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
description: associated with jail: OpenVPN
options=8<VLAN_MTU>
ether 02:ff:60:9b:c8:7f
hwaddr 02:9d:50:00:06:0a
nd6 options=1<PERFORMNUD>
media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
status: active
groups: epair


OpenVPN jail ifconfig:


lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
inet6 ::1 prefixlen 128
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1
inet 127.0.0.1 netmask 0xff000000
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
groups: lo
epair0b: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=8<VLAN_MTU>
ether 02:ff:60:9b:c8:80
hwaddr 02:9d:a0:00:0a:0b
inet 10.0.1.52 netmask 0xffffff00 broadcast 10.0.1.255
nd6 options=1<PERFORMNUD>
media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
status: active
groups: epair
tun0: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> metric 0 mtu 1500
options=80000<LINKSTATE>
inet 10.8.0.1 --> 10.8.0.2 netmask 0xffffffff
nd6 options=1<PERFORMNUD>
groups: tun


Anything else I should supply?

Thanks for any help.

 

[m0nkey_]

What is the contents of your OpenVPN server configuration?

 

[spotcatbug]

What is the contents of your OpenVPN server configuration?

I'm not sure if you meant the jail config or the openvpn.conf file from the jail. Here's both:

iocage get -a OpenVPN:


CONFIG_VERSION:10
allow_chflags:0
allow_mount:0
allow_mount_devfs:1
allow_mount_nullfs:0
allow_mount_procfs:0
allow_mount_tmpfs:0
allow_mount_zfs:0
allow_quotas:0
allow_raw_sockets:0
allow_set_hostname:1
allow_socket_af:0
allow_sysvipc:0
available:readonly
basejail:no
boot:on
bpf:no
children_max:0
cloned_release:11.1-RELEASE-p9
comment:none
compression:lz4
compressratio:readonly
coredumpsize:off
count:1
cpuset:off
cputime:off
datasize:off
dedup:off
defaultrouter:10.0.1.254
defaultrouter6:none
depends:none
devfs_ruleset:4
dhcp:off
enforce_statfs:2
exec_clean:1
exec_fib:0
exec_jail_user:root
exec_poststart:/usr/bin/true
exec_poststop:/usr/bin/true
exec_prestart:/usr/bin/true
exec_prestop:/usr/bin/true
exec_start:/bin/sh /etc/rc
exec_stop:/bin/sh /etc/rc.shutdown
exec_system_jail_user:0
exec_system_user:root
exec_timeout:60
host_domainname:none
host_hostname:OpenVPN
host_hostuuid:OpenVPN
host_time:yes
hostid:03DE0294-0480-056B-CC06-690700080009
hostid_strict_check:off
interfaces:vnet0:bridge1
ip4:new
ip4_addr:vnet0|10.0.1.52/24
ip4_saddrsel:1
ip6:new
ip6_addr:none
ip6_saddrsel:0
jail_zfs:off
jail_zfs_dataset:iocage/jails/OpenVPN/data
jail_zfs_mountpoint:none
last_started:2018-06-22 18:43:22
login_flags:-f root
mac_prefix:02ff60
maxproc:off
memorylocked:off
memoryuse:off
mount_devfs:1
mount_fdescfs:1
mount_linprocfs:0
mount_procfs:0
mountpoint:readonly
msgqqueued:off
msgqsize:off
nmsgq:off
notes:none
nsemop:off
nshm:off
nthr:off
openfiles:off
origin:readonly
owner:root
pcpu:off
priority:99
pseudoterminals:off
quota:none
release:11.1-RELEASE-p9
reservation:none
resolver:/etc/resolv.conf
rlimits:off
securelevel:2
shmsize:off
stacksize:off
state:up
stop_timeout:30
swapuse:off
sync_state:none
sync_target:none
sync_tgt_zpool:none
sysvmsg:new
sysvsem:new
sysvshm:new
template:no
type:jail
used:readonly
vmemoryuse:off
vnet:on
vnet0_mac:02ff609bc87f,02ff609bc880
vnet1_mac:none
vnet2_mac:none
vnet3_mac:none
vnet_interfaces:none
wallclock:off


openvpn.conf from the jail (this didn't change at all from when the VPN was working - the client config also hasn't changed):


#################################################
# Sample OpenVPN 2.0 config file for #
# multi-client server. #
# #
# This file is for the server side #
# of a many-clients <-> one-server #
# OpenVPN configuration. #
# #
# OpenVPN also supports #
# single-machine <-> single-machine #
# configurations (See the Examples page #
# on the web site for more info). #
# #
# This config should work on Windows #
# or Linux/BSD systems. Remember on #
# Windows to quote pathnames and use #
# double backslashes, e.g.: #
# "C:\\Program Files\\OpenVPN\\config\\foo.key" #
# #
# Comments are preceded with '#' or ';' #
#################################################

# Which local IP address should OpenVPN
# listen on? (optional)
;local a.b.c.d
local 10.0.1.52

# Which TCP/UDP port should OpenVPN listen on?
# If you want to run multiple OpenVPN instances
# on the same machine, use a different port
# number for each one. You will need to
# open up this port on your firewall.
port 1194

# TCP or UDP server?
;proto tcp
proto udp

# "dev tun" will create a routed IP tunnel,
# "dev tap" will create an ethernet tunnel.
# Use "dev tap0" if you are ethernet bridging
# and have precreated a tap0 virtual interface
# and bridged it with your ethernet interface.
# If you want to control access policies
# over the VPN, you must create firewall
# rules for the the TUN/TAP interface.
# On non-Windows systems, you can give
# an explicit unit number, such as tun0.
# On Windows, use "dev-node" for this.
# On most systems, the VPN will not function
# unless you partially or fully disable
# the firewall for the TUN/TAP interface.
;dev tap
dev tun

# Windows needs the TAP-Win32 adapter name
# from the Network Connections panel if you
# have more than one. On XP SP2 or higher,
# you may need to selectively disable the
# Windows firewall for the TAP adapter.
# Non-Windows systems usually don't need this.
;dev-node MyTap

# SSL/TLS root certificate (ca), certificate
# (cert), and private key (key). Each client
# and the server must have their own cert and
# key file. The server and all clients will
# use the same ca file.
#
# See the "easy-rsa" directory for a series
# of scripts for generating RSA certificates
# and private keys. Remember to use
# a unique Common Name for the server
# and each of the client certificates.
#
# Any X509 key management system can be used.
# OpenVPN can also use a PKCS #12 formatted key file
# (see "pkcs12" directive in man page).
ca /usr/local/etc/openvpn/keys/ca.crt
cert /usr/local/etc/openvpn/keys/openvpn-server.crt
key /usr/local/etc/openvpn/keys/openvpn-server.key # This file should be kept secret

# Diffie hellman parameters.
# Generate your own with:
# openssl dhparam -out dh2048.pem 2048
dh /usr/local/etc/openvpn/keys/dh.pem

# Network topology
# Should be subnet (addressing via IP)
# unless Windows clients v2.0.9 and lower have to
# be supported (then net30, i.e. a /30 per client)
# Defaults to net30 (not recommended)
;topology subnet

# Configure server mode and supply a VPN subnet
# for OpenVPN to draw client addresses from.
# The server will take 10.8.0.1 for itself,
# the rest will be made available to clients.
# Each client will be able to reach the server
# on 10.8.0.1. Comment this line out if you are
# ethernet bridging. See the man page for more info.
server 10.8.0.0 255.255.255.0

# Maintain a record of client <-> virtual IP address
# associations in this file. If OpenVPN goes down or
# is restarted, reconnecting clients can be assigned
# the same virtual IP address from the pool that was
# previously assigned.
ifconfig-pool-persist ipp.txt

# Configure server mode for ethernet bridging.
# You must first use your OS's bridging capability
# to bridge the TAP interface with the ethernet
# NIC interface. Then you must manually set the
# IP/netmask on the bridge interface, here we
# assume 10.8.0.4/255.255.255.0. Finally we
# must set aside an IP range in this subnet
# (start=10.8.0.50 end=10.8.0.100) to allocate
# to connecting clients. Leave this line commented
# out unless you are ethernet bridging.
;server-bridge 10.8.0.4 255.255.255.0 10.8.0.50 10.8.0.100

# Configure server mode for ethernet bridging
# using a DHCP-proxy, where clients talk
# to the OpenVPN server-side DHCP server
# to receive their IP address allocation
# and DNS server addresses. You must first use
# your OS's bridging capability to bridge the TAP
# interface with the ethernet NIC interface.
# Note: this mode only works on clients (such as
# Windows), where the client-side TAP adapter is
# bound to a DHCP client.
;server-bridge

# Push routes to the client to allow it
# to reach other private subnets behind
# the server. Remember that these
# private subnets will also need
# to know to route the OpenVPN client
# address pool (10.8.0.0/255.255.255.0)
# back to the OpenVPN server.
;push "route 192.168.10.0 255.255.255.0"
;push "route 192.168.20.0 255.255.255.0"
push "route 10.0.1.0 255.255.255.0"

# To assign specific IP addresses to specific
# clients or if a connecting client has a private
# subnet behind it that should also have VPN access,
# use the subdirectory "ccd" for client-specific
# configuration files (see man page for more info).

# EXAMPLE: Suppose the client
# having the certificate common name "Thelonious"
# also has a small subnet behind his connecting
# machine, such as 192.168.40.128/255.255.255.248.
# First, uncomment out these lines:
;client-config-dir ccd
;route 192.168.40.128 255.255.255.248
# Then create a file ccd/Thelonious with this line:
# iroute 192.168.40.128 255.255.255.248
# This will allow Thelonious' private subnet to
# access the VPN. This example will only work
# if you are routing, not bridging, i.e. you are
# using "dev tun" and "server" directives.

# EXAMPLE: Suppose you want to give
# Thelonious a fixed VPN IP address of 10.9.0.1.
# First uncomment out these lines:
;client-config-dir ccd
;route 10.9.0.0 255.255.255.252
# Then add this line to ccd/Thelonious:
# ifconfig-push 10.9.0.1 10.9.0.2

# Suppose that you want to enable different
# firewall access policies for different groups
# of clients. There are two methods:
# (1) Run multiple OpenVPN daemons, one for each
# group, and firewall the TUN/TAP interface
# for each group/daemon appropriately.
# (2) (Advanced) Create a script to dynamically
# modify the firewall in response to access
# from different clients. See man
# page for more info on learn-address script.
;learn-address ./script

# If enabled, this directive will configure
# all clients to redirect their default
# network gateway through the VPN, causing
# all IP traffic such as web browsing and
# and DNS lookups to go through the VPN
# (The OpenVPN server machine may need to NAT
# or bridge the TUN/TAP interface to the internet
# in order for this to work properly).
;push "redirect-gateway def1 bypass-dhcp"

# Certain Windows-specific network settings
# can be pushed to clients, such as DNS
# or WINS server addresses. CAVEAT:
# http://openvpn.net/faq.html#dhcpcaveats
# The addresses below refer to the public
# DNS servers provided by opendns.com.
;push "dhcp-option DNS 208.67.222.222"
;push "dhcp-option DNS 208.67.220.220"

# Uncomment this directive to allow different
# clients to be able to "see" each other.
# By default, clients will only see the server.
# To force clients to only see the server, you
# will also need to appropriately firewall the
# server's TUN/TAP interface.
;client-to-client

# Uncomment this directive if multiple clients
# might connect with the same certificate/key
# files or common names. This is recommended
# only for testing purposes. For production use,
# each client should have its own certificate/key
# pair.
#
# IF YOU HAVE NOT GENERATED INDIVIDUAL
# CERTIFICATE/KEY PAIRS FOR EACH CLIENT,
# EACH HAVING ITS OWN UNIQUE "COMMON NAME",
# UNCOMMENT THIS LINE OUT.
;duplicate-cn

# The keepalive directive causes ping-like
# messages to be sent back and forth over
# the link so that each side knows when
# the other side has gone down.
# Ping every 10 seconds, assume that remote
# peer is down if no ping received during
# a 120 second time period.
keepalive 10 120

# For extra security beyond that provided
# by SSL/TLS, create an "HMAC firewall"
# to help block DoS attacks and UDP port flooding.
#
# Generate with:
# openvpn --genkey --secret ta.key
#
# The server and each client must have
# a copy of this key.
# The second parameter should be '0'
# on the server and '1' on the clients.
tls-auth /usr/local/etc/openvpn/keys/ta.key 0 # This file is secret
remote-cert-tls client

# Select a cryptographic cipher.
# This config item must be copied to
# the client config file as well.
# Note that v2.4 client/server will automatically
# negotiate AES-256-GCM in TLS mode.
# See also the ncp-cipher option in the manpage
cipher AES-256-CBC

# Enable compression on the VPN link and push the
# option to the client (v2.4+ only, for earlier
# versions see below)
;compress lz4-v2
;push "compress lz4-v2"

# For compression compatible with older clients use comp-lzo
# If you enable it here, you must also
# enable it in the client config file.
comp-lzo

# The maximum number of concurrently connected
# clients we want to allow.
;max-clients 100

# It's a good idea to reduce the OpenVPN
# daemon's privileges after initialization.
#
# You can uncomment this out on
# non-Windows systems.
user nobody
group nobody

# The persist options will try to avoid
# accessing certain resources on restart
# that may no longer be accessible because
# of the privilege downgrade.
persist-key
persist-tun

# Output a short status file showing
# current connections, truncated
# and rewritten every minute.
status openvpn-status.log

# By default, log messages will go to the syslog (or
# on Windows, if running as a service, they will go to
# the "\Program Files\OpenVPN\log" directory).
# Use log or log-append to override this default.
# "log" will truncate the log file on OpenVPN startup,
# while "log-append" will append to it. Use one
# or the other (but not both).
;log openvpn.log
;log-append openvpn.log

# Set the appropriate level of log
# file verbosity.
#
# 0 is silent, except for fatal errors
# 4 is reasonable for general usage
# 5 and 6 can help to debug connection problems
# 9 is extremely verbose
verb 3

# Silence repeating messages. At most 20
# sequential messages of the same message
# category will be output to the log.
;mute 20

# Notify the client that when the server restarts so it
# can automatically reconnect.
explicit-exit-notify 1



 

[dlavigne]

Did you recently update to U5?

 

[spotcatbug]

Did you recently update to U5?

I did, but I've since reverted back to U4. The VPN was broken before I upgraded, however.

 

[spotcatbug]

I'm still trying to get this working.

Here's my /etc/rc.conf. I thought this might clue somebody into what's going on with my VPN connection.


host_hostname="OpenVPN"
cron_flags="$cron_flags -J 15"

# Disable Sendmail by default
sendmail_enable="NONE"
sendmail_submit_enable="NO"
sendmail_outbound_enable="NO"
sendmail_msp_queue_enable="NO"

# Run secure syslog
syslogd_flags="-c -ss"

openvpn_enable="YES"
openvpn_if="tun"
openvpn_configfile="/usr/local/etc/openvpn/openvpn.conf"
openvpn_dir="/usr/local/etc/openvpn/"
cloned_interfaces="tun"
gateway_enable="YES"
firewall_enable="YES"
firewall_script="/usr/local/etc/ipfw.rules"


Here's /usr/local/etc/ipfw.rules:


#!/bin/sh
EPAIR=$(/sbin/ifconfig -l | tr " " "\n" | /usr/bin/grep epair)
ipfw -q -f flush
ipfw -q nat 1 config if ${EPAIR}
ipfw -q add nat 1 all from 10.8.0.0/24 to any out via ${EPAIR}
ipfw -q add nat 1 all from any to any in via ${EPAIR}

TUN=$(/sbin/ifconfig -l | tr " " "\n" | /usr/bin/grep tun)
ifconfig ${TUN} name tun0


Any help will be greatly appreciated!

 

[KenNashua]

Did you ever get this working? I'm having the same issue...trying to finish converting all my warden jails to iocage. openvpn is the last one... Client shows I have a route, but can't hit any internal LAN addresses. 11.2-U5.

 

[spotcatbug]

I did get it working. It turned out to be a bug where the FreeNAS network interface wasn't getting added to the bridge interface for the OpenVPN jail, resulting in no network for the jail. I believe this happened because my main interface is a link aggregation.

It appears to be fixed in 11.2 BETA2 (it's bug #38155). I won't know if the fix works for me until I upgrade to 11.2 release (I'm sticking with BETA1 until the actual 11.2 release). Until then, what I've had to do is manually add the lagg interface to the iocage bridge interface (basically do what bug #38155 was preventing) after any reboot - rare enough that it's no big deal.

 

[KenNashua]

Deleted....working now...had to reboot Google Wifi