jail config: tun0 interface not showing up in vnet_default_interface despite allow_tun

[hadr0n]

Hello,

I would need some help configuring a jail to listen on a tun interface (trying for a few full days now but I'm new to Truenas core and Freebsd). Here are some details about my setup:

- Truenas core 13.0-I4
- Openvpn server used through the Truenas core "services", working, I can connect to it from my openvpn client and ping the tun interface
- Application jail (for instance emby - but does not matter I think)

What I want to do: being able to access my application through openvpn, either through NAT (like 10.8.2.1:port) or through a separate IP (10.8.2.31 for example) whatever works.

- I gave up on the NAT solution, it works with the physical interface, but I found no way to "setup" the NAT on the tun interface, and whatever I do it tells me I cannot change the default interface which is the physical one (found nothing online seemed like a lost cause). And apparently NAT is not advised to be used with Jail.
- So I tried to go back to vnet, I found the "allow_tun" in "Custom Properties", however even after enabled it, the tun0 interface does not show up in the list "vnet_default_interface", and when I set it to none and configure some 10.8.2.x IPs it refuses to start the jail.
- I read somewhere I needed to create a bridge interface on top of the tun interface, but Truenas core refuses to create this bridge (if someone tells me it is indeed the way to go will post the error message).

I achieved to make it work at some point by linking the jail to the physical interface, then pushing some routes to it in the openvpn config but it seem ugly and might conflict with client local IPs.

Am I missing something obvious?

Thanks for your help

 

[Patrick M. Hausen]

The tun interface needs to be brought up by some service inside the jail. It is not a vnet interface. So it doen't show up in vnet_default_interface.

Did you manually create a bridge interface as recommended? If yes, then set vnet_default_interface to "none" and interfaces to "vnet0:bridge0" if bridge0 is the bridge interface you want to use.

 

[hadr0n]

Thanks for trying to help, I tried, but the thing is each time I try to create a bridge interface I get the following error:

Code:

Error: Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/middlewared/main.py", line 139, in call_method
    result = await self.middleware._call(message['method'], serviceobj, methodobj, params, app=self)
  File "/usr/local/lib/python3.9/site-packages/middlewared/main.py", line 1236, in _call
    return await methodobj(*prepared_call.args)
  File "/usr/local/lib/python3.9/site-packages/middlewared/schema.py", line 975, in nf
    return await f(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/middlewared/plugins/network.py", line 900, in commit
    await self.sync()
  File "/usr/local/lib/python3.9/site-packages/middlewared/plugins/network.py", line 1991, in sync
    await self.middleware.call('interface.bridge_setup', bridge)
  File "/usr/local/lib/python3.9/site-packages/middlewared/main.py", line 1279, in call
    return await self._call(
  File "/usr/local/lib/python3.9/site-packages/middlewared/main.py", line 1247, in _call
    return await self.run_in_executor(prepared_call.executor, methodobj, *prepared_call.args)
  File "/usr/local/lib/python3.9/site-packages/middlewared/main.py", line 1152, in run_in_executor
    return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs))
  File "/usr/local/lib/python3.9/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/local/lib/python3.9/site-packages/middlewared/plugins/interface/bridge.py", line 57, in bridge_setup
    iface.add_member(member)
  File "netif.pyx", line 1211, in netif.BridgeInterface.add_member
  File "netif.pyx", line 1229, in netif.BridgeInterface.bridge_cmd
OSError: [Errno 22] Invalid argument

The bridge is created with the following settings:
- members: I tried putting both igc0 and tun0 as bridge members and only tun0, same error pops up. I admit I have no idea what I'm doing with this bridge though, but I feel like I tried every possible combination to create it (not that much parameters to play with). Never had to create a bridge between tun and physical on Linux for openvpn server to work as expected for instance, and if there is only one interface as bridge member, I'm not sure what it bridges together.
- I have set bridge IP to 10.8.3.1/24. My physical subnet is 192.168.0.X/24 and my tun 10.8.2.X/24 - and it wants a different subnet .

 

[hadr0n]

Made a few more tests, when creating a bridge containing only igc0, it says "device is busy" which I guess is normal, but the error message is different than when trying to build the bridge containing the tun0 interface.

When looking at the code the line where the cursor is fails when the member parameter is tun0.

In the middlewared logs, the interfaces tun0 and igc0 are found in database. Seems like a bug since the code retrieve itself the parameter member, which it achieves to find, and it passes this parameter to the method add_member of the BridgedInterface object iface and it fails with OSError 22 "invalid argument". Tried to delve further in code netif.pyx but have not found this source code yet.

 

[Patrick M. Hausen]

IMHO we need to start over from the ground up to resolve this. I'll be writing more tomorrow.

Do you have any more jails than this one?

 

[hadr0n]

Thank you for looking into it, yes, I also have a good old transmission jail. Here are my current jails (emby is back to nat mode for now, I use it in local without openvpn for now):

Here is a screenshot of my IF, type and link state of tun0 is unknown btw, don't remember if it was the case initially

Nothing in static routes.

I remember I followed a tutorial a few months ago to setup open vpn server and had to add these weird things in Tunables (don't remember why exactly, maybe that's where/when things got messed up):

 

[hadr0n]

I could not make this work, I will just install openvpn server on my openwrt in front of the truenas core server and set up the jail in nat mode on the igc0 interface.

 

[hadr0n]

In case somebody needs help trying to use jails through openvpn, I finally managed to get things working by installing openvpn client inside each jail (after resetting my broken truenas config from the CLI interface).

1) Configure certificates CA, server cert, client cert (lot of details about this on the web...). There are some tricky things however,
- openvpn requires "Common name /CN" to be set to a unique value for each certificate, otherwise connections will fail (while the GUI allows you to create these certificates without specifying a CN)
- for the CA and for client certificates, if you use "TLS Crypt Auth", you must when creating the certificates add the usage "ANY_EXTENDED_KEY_USAGE".

2) OpenVPN server configuration: go the services, edit "OpenVPN server".
- device type TUN, topology subnet (might also be possible using TAP but some next steps might differ then)
- Additional parameters
* "client-to-client": enable direct networking between openvpn clients
* "client-config-dir *some path not destroyed by truenas after startup*" : this directory will contain files specifying static IP addresses for your jails (would be annoying if these change randomly). In this dir, filename must be the "CN" of the client and the content "ifconfig-push *staticIP* *subnet*" for example "ifconfig-push 10.8.0.3 255.255.255.0". Create one such file for each jail you want to access through VPN. As *some path not destroyed by truenas after startup* I have selected a path to my system dataset in /mnt/...

When using static IPs for some clients, it is advised to restrict the server DHCP pool to reject these static IPs from it. However, ifconfig-pool option conflicts with the fact that the Truenas GUI forces you to set the "server" directory with a subnet.

What works on typical setup (not Truenas) is to set server directive to a an IP "server 10.8.0.1" (or to use the nopool option after the server directive) and "ifconfig-pool 10.8.0.2 10.8.0.100 255.255.255.0", but GUI behaviour makes it not possible, it forces server directive to have the form "server 10.8.0.0 255.255.255.0". Directly changing the openvpn conf file using the shell does not work either since truenas overwrites the conf file (even tried to chmod 444 the conf file without success, Truenas also chmod automatically as root apparently). I don't have a solution for this yet, and just cross fingers for DHCP and static IPs not to conflict.

3) Jail configuration through GUI
- VNET, Berkely Packet Filter enabled (NAT, DHCP disabled)
- IPv4 inteface to vnet0 with typical local physical IP like 192.168.0.30
- IPv4 default Router set to auto
- In Jail properties, enabled allow_set_hostname, allow_raw_sockets
- in custom properties, enabled allow_tun
4) Jail configuration through shell
- Open jail shell going to section "Jails", right arrow, shell
- install openvpn client inside jail:
* pkg update && pkg install openvpn
* mkdir /usr/local/etc/openvpn && cd /usr/local/etc/openvpn
* Create a new file openvpn.conf in that dir, and paste the content of your ovpn file inside, then save the file. You can create the ovpn file using truenas GUI, in the openvpn server service configuration, after selecting the corresponding client certificate.
* Add to the file /etc/rc.conf the following two lines (will enable openvpn auto start at jail boot):
openvpn_enable="YES"
openvpn_configfile="/usr/local/etc/openvpn/openvpn.conf"
* Command "service openvpn start" should start openvpn client, on my side I needed a reboot before the tun interface to work properly
* Error message can be found in /var/log/messages (using jail shell)

Jail should be accessible from one vpn client using the static IP in the files written in dir *some path not destroyed by truenas after startup* . A drawback of this config is that all clients to clients communications are allowed, there are ways to restrict this to improve security, but it is fine like this for my home/personal use.