Could you explain further this part:
This is all outside the FreeNAS/TrueNAS context, of course. I am not quite sure if TN can be forced to configure networking that way, but possibly. Would have to experiment in a VM a bit and I have other things on my "almost-but-not-quite-work-because-fun" list of hacking to do, first.
But generally: vnet relies on the epair(4) interface, which is essentially a pair of virtual interfaces that are connected by a virtual wire. If I look at the end in my Nextcloud jail, it looks like this:
Code:
epair0b: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
[...]
ether 02:ff:60:07:4c:d7
hwaddr 02:cd:ad:3d:16:0b
inet 192.168.1.53 netmask 0xffffff00 broadcast 192.168.1.255
[...]
On the host there is the other end of that one:
Code:
vnet0.2: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
description: associated with jail: cloud as nic: epair0b
[...]
This interface does not have any IP address!
It's just the other end of that "virtual cable" and it is plugged into a "virtual switch", namely bridge0:
Code:
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
[...]
member: vnet0.2 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
[...]
member: vlan100 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
[...]
So the "bridge0" connects the jail's interface with my local LAN on VLAN 100 - could be anything like em0, igb0, ... instead. Physical network. So the jail participates in the LAN's broadcast domain. ARP, IPv6 neighbour discovery, has got it's own default gateway(s), can have additional routes and firewall rules etc.
The downside is that with 100s of jails that one "LAN" becomes crowded. So more and more bandwidth is wasted for all these broadcast/multicast protocols: ARP, neighbour discovery, ... the "virtual switch", i.e. the FreeBSD "bridge" interface needs to forward every single ARP request on the wire to every single jail. Because broadcast ...
So why not get rid of this bridging and broadcasting altogether? If not in the TrueNAS context, a jail created with an epair interface get's e.g. epair0b inside the jail while the host gets epair0a as its "end" of the interface.
So picture:
em0/igb0/whatever on the host: 192.168.0.1/24
epair0a on the host: 192.168.1.1/30
epair0b inside the jail: 192.168.1.2/30
That way the host and the jail can talk on layer 3 over that epair "wire" with the two configured IP addresses. For the jail to be able to reach the outside world the host must be configured to be a router (trivial, any Unix can do that) and the jail's default gateway must point to 192.168.1.1.
Then we need to tell the rest of the world that this jail exists on this host, so we need a route known in all other hosts and/or routers on our network that reads:
192.168.1.0/30 (that's .0 .1 .2 .3 of which .1 and .2 can be used) goes to ---> 192.168.0.1 (the IP address of our jail host).
Tadah! Jail with vnet and its own address without bridging. Now the downside is that with bridging if you move a jail from one host to a new one, if all are on the same LAN you have IP/jail mobility. At the cost of all this broadcast/ARP mumbo jumbo. But it's trivial to implement and it "just works" for reasonably small environments.
If you want that mobility with routed connectivity as laid out above, then you need a dynamic routing protocol like OSPF or BGP and you need to trust your jail hosts in participating in that routing protocol without messing up your network.
But essentially that's what all larger cloud infrastructures do. Use dynamic routing to connect containers or VMs. Docker is no different. If you want container mobility you need to publish those IP addresses somehow. Only in the case of a single docker host it looks simpler than our jails but actually isn't. Because you can do the very same with jails. This is what TN does when you run a jail/plugin with NAT.
There is the bridge interface (again) but it's not connected to the outside world. Alle the jails are put on that bridge/switch. And the bridge interface itself has got an IP address and is the default gateway for the jails. And then you NAT on the host to get the traffic flowing.
Phew, in German we ask tounge-in-cheek "alle Klarheiten beseitigt?", i.e. "all clarity successfully eliminated?" after such a lecture
. Hope that shed some light on the topic for you (and possibly others).
