How to force jail to bridge epair with different interfaces? (different one from default GW)

[asimov-solensan]

Hello,

I have been trying this for several days and I can't find a solution. Mostly the problem is described here:

https://forums.freenas.org/index.php?threads/tap-interface-shows-status-no-carrier.59572/

But I want try a different approximation here, on this thread, therefore I think it applies to have a new thread.

Basically I have got freenas on a trunk interface VLANs 13 - 17. Freenas must be on vlan17 (everything ok here) and then I want to have different jails on different vlans. So far the only I found is to disable VTNET and then force the NIC to use, vlan15 for instance, and then configure an IP in that subnet.

VLAN different from freenas. In this case it tries to use freenas default gateway and since it has no direct access to it there isn't any kind of connectivity to the outside.
VLAN same as freenas. In this case the jail can reach other network and even one can reach the service running on that jail. The problem is that technically the IP belongs to freenas and it will answer and bind all services to this address also. Some things can be configured and others not but still I find the solution kind of a kludge.

What I think it should be done is to bridge every epair with a different vlan interface, then it's possible also to configure a different gateway on every jail. But I have go all jails briged with default gateway vlan17 following the example. Is there a way do this?

Thanks in advance.

 

[asimov-solensan]

Seems that this guy here got it working but not in my case (or freenas version):

https://www.reddit.com/r/freenas/comments/4c398j/jails_in_vlan_have_no_default_route/

Seems that everything is ok on the host:

Code:

epair0a: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1496
		options=8<VLAN_MTU>
		ether 02:ff:20:00:14:0a
		nd6 options=1<PERFORMNUD>
		media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
		status: active

bridge1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1496
		ether 02:c8:7a:06:44:01
		nd6 options=1<PERFORMNUD>
		id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
		maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
		root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
		member: epair0a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
				ifmaxaddr 0 port 20 priority 128 path cost 2000
		member: vlan17 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
				ifmaxaddr 0 port 15 priority 128 path cost 2000000

vlan17: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1496
		ether 00:bd:21:d2:00:c8
		inet 192.168.17.10 netmask 0xffffff00 broadcast 192.168.17.255
		nd6 options=9<PERFORMNUD,IFDISABLED>
		media: Ethernet autoselect
		status: active
		vlan: 17 parent interface: tap200

Same on the jail:

Code:

root@jumper:/ # ifconfig														
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384			   
	   options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>				   
	   inet6 ::1 prefixlen 128												 
	   inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1							 
	   inet 127.0.0.1 netmask 0xff000000									   
	   nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>							   
epair0b: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1496   
	   options=8<VLAN_MTU>													 
	   ether 02:ff:70:00:15:0b												 
	   inet 192.168.17.12 netmask 0xffffff00 broadcast 192.168.17.255		 
	   nd6 options=9<PERFORMNUD,IFDISABLED>									
	   media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)					 
	   status: active														 
root@jumper:/ #	 

root@jumper:/ # netstat -rn													 
Routing tables																 
																			   
Internet:																	   
Destination		Gateway			Flags	  Netif Expire				   
default			192.168.17.1	   UGS	 epair0b						   
127.0.0.1		  link#1			 UH		  lo0						   
192.168.17.0/24	link#2			 U	   epair0b						   
192.168.17.12	  link#2			 UHS		 lo0						   
						   

But still if I run a ping from inside the jail to the default gateway:

Code:

[root@lavochkin] /mnt/RAID2x3a/jails/.jumper.meta# tcpdump -n -i tap200 | grep 192.168.17.12
tcpdump: WARNING: tap200: no IPv4 address assigned
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on tap200, link-type EN10MB (Ethernet), capture size 65535 bytes
22:35:32.600944 IP 192.168.17.12 > 192.168.17.1: ICMP echo request, id 27, seq 20, length 64
22:35:32.600993 IP 192.168.17.1 > 192.168.17.12: ICMP echo reply, id 27, seq 20, length 64
22:35:33.638563 IP 192.168.17.12 > 192.168.17.1: ICMP echo request, id 27, seq 21, length 64
22:35:33.638612 IP 192.168.17.1 > 192.168.17.12: ICMP echo reply, id 27, seq 21, length 64
22:35:34.678251 IP 192.168.17.12 > 192.168.17.1: ICMP echo request, id 27, seq 22, length 64
22:35:34.678298 IP 192.168.17.1 > 192.168.17.12: ICMP echo reply, id 27, seq 22, length 64
22:35:35.680154 IP 192.168.17.12 > 192.168.17.1: ICMP echo request, id 27, seq 23, length 64
22:35:35.680250 IP 192.168.17.1 > 192.168.17.12: ICMP echo reply, id 27, seq 23, length 64
^C112 packets captured
117 packets received by filter
0 packets dropped by kernel

[root@lavochkin] /mnt/RAID2x3a/jails/.jumper.meta# tcpdump -e -n -i tap200 | grep 192.168.17.12
tcpdump: WARNING: tap200: no IPv4 address assigned
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on tap200, link-type EN10MB (Ethernet), capture size 65535 bytes
22:35:56.996676 02:ff:70:00:15:0b > 00:a0:98:ef:ae:3a, ethertype 802.1Q (0x8100), length 102: vlan 17, p 0, ethertype IPv4, 192.168.17.12 > 192.168.17.1: ICMP echo request, id 27, seq 44, length 64
22:35:56.996752 00:a0:98:ef:ae:3a > 02:ff:70:00:15:0b, ethertype 802.1Q (0x8100), length 102: vlan 17, p 0, ethertype IPv4, 192.168.17.1 > 192.168.17.12: ICMP echo reply, id 27, seq 44, length 64
22:35:58.001340 02:ff:70:00:15:0b > 00:a0:98:ef:ae:3a, ethertype 802.1Q (0x8100), length 102: vlan 17, p 0, ethertype IPv4, 192.168.17.12 > 192.168.17.1: ICMP echo request, id 27, seq 45, length 64
22:35:58.001409 00:a0:98:ef:ae:3a > 02:ff:70:00:15:0b, ethertype 802.1Q (0x8100), length 102: vlan 17, p 0, ethertype IPv4, 192.168.17.1 > 192.168.17.12: ICMP echo reply, id 27, seq 45, length 64
^C38 packets captured
51 packets received by filter
0 packets dropped by kernel

[root@lavochkin] /mnt/RAID2x3a/jails/.jumper.meta#

Traffic correctly reaches the firewall and it answers. But for some reason the parent interface doesn't forward this to the vlan interface:

Code:

[root@lavochkin] /mnt/RAID2x3a/jails/.jumper.meta# tcpdump -e -n -i vlan17 icmp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on vlan17, link-type EN10MB (Ethernet), capture size 65535 bytes
22:38:43.860253 02:ff:70:00:15:0b > 00:a0:98:ef:ae:3a, ethertype IPv4 (0x0800), length 98: 192.168.17.12 > 192.168.17.1: ICMP echo request, id 27, seq 208, length 64
22:38:44.883199 02:ff:70:00:15:0b > 00:a0:98:ef:ae:3a, ethertype IPv4 (0x0800), length 98: 192.168.17.12 > 192.168.17.1: ICMP echo request, id 27, seq 209, length 64
22:38:45.913242 02:ff:70:00:15:0b > 00:a0:98:ef:ae:3a, ethertype IPv4 (0x0800), length 98: 192.168.17.12 > 192.168.17.1: ICMP echo request, id 27, seq 210, length 64
22:38:46.922355 02:ff:70:00:15:0b > 00:a0:98:ef:ae:3a, ethertype IPv4 (0x0800), length 98: 192.168.17.12 > 192.168.17.1: ICMP echo request, id 27, seq 211, length 64
22:38:47.922943 02:ff:70:00:15:0b > 00:a0:98:ef:ae:3a, ethertype IPv4 (0x0800), length 98: 192.168.17.12 > 192.168.17.1: ICMP echo request, id 27, seq 212, length 64

Someone is stealing my ping replies. Please, I hope someone helps me because this is becoming an obsession.

By the way, another solution would be to just create the epair against the tagged interface and then create the vlan interface inside the jail, using the epair as it's parent. But I see no way to do this.

Found this article but no idea if it can apply to freenas.

https://gist.github.com/sdebnath/086874c5df8b68e0df69

I hope someone lend me a hand because this is becoming an obsession.

 

[Patrick M. Hausen]

That article you linked looks like the author got it the wrong way round to me. I trust it works, but it just doesn't feel right with more than 30 years of networking experience.
I would not bridge the physical interface into the jail or VM and manage the tags there. I would do all the topology setup on the host system.

I don't have the time to try and configure what you need in FreeNAS right now, but generally and on a stock FreeBSD system I would use this setup:

configure e.g. em0 up
configure all tagged VLAN interfaces on the host system with vlandev em0
configure N bridge interfaces, one for each VLAN device on the host, then addm vlanX
add each epair interface to the desired bridge

On a stock FreeBSD and using iocage, this should be as simple as picking the right vlanX for iocage to attach the jail to.

If needed, I could try to setup a couple of jails this way later next week ...

HTH,
Patrick

 

[asimov-solensan]

I already tried and got this problem:

Lets think I create bridge10 including vlan10 interface. Then I force the jail to use vlan10 interface. Then all I get is a new bridge with vlan10 and epairx. And it seemed that somehow packets reached destination (starting communication from the jail), but on the way back something was lost.

Can't test now but took this notes on my notepad. I may have been doing something wrong.

 

[Patrick M. Hausen]

OK, so on stock FreeBSD we do pre-create all the bridge interfaces in our production hosting environment. The iocage (new py-iocage as in FreeNAS) settings for a particular jail are like this:

Code:

iocage get all vpro0121
[...]
interfaces:vnet0:bridge0
ip4_addr:vnet0|217.29.41.121/24
[...]

I assume you would create the jails with iocage on the command line, not via the new beta UI.
Patrick

P.S. You need to assign the jail to the existing bridge10 interface, of course. Not the vlan10 interface.

 

[asimov-solensan]

Well, now I'm totally lost here. My experience with freebsd and jails comes only from freenas.

I'm using the graphical interface because I read it's not advised to abuse CLI configurations.

Still I wouldn't know how to try your solution, as far as I see neither iocage nor py-iocage are present in freenas 9.10.

Right now this is the solution I'm using right now. Since the firewall is also virtulized in freenas I created a tap interface for every vlan and then all those interfaces bridged to the firewall. Then jails are assigned to the tap interface that fits (as explained in that reddit thread) and everything is managed by the firewall.

As for bhyve i tried many things but finally used a physical interface as a trunk to the physical switch. This reaches the firewall from the outside/physical world as any other network and it works. Still my guests OS must ssupport vlan in order to work this way.

This works but I really think is dirty kludge and complicates the firewall configuration.

I also think that if freenas sells itself as a container and a hypervisor it should include better network management. I already opened a feature request to include openvswitch. This could work absolutely seamless, I think you would only create as many tap interfaces as you need and have openvswitch manage them. Freenas wouldn't even notice it being in the middle.

The conclusion I came after several days tinkering is that bridges and taps aren't enough.

I want to leave the question opened because maybe someone has better ideas.

 

[Patrick M. Hausen]

Oh ... I'm so sorry. I didn't think anyone would run anything but FreeNAS 11 today. In FreeNAS 9 there is AFAIK no proper command line interface to the jail/vnet configuration and jails are of course still based on the warden wrapper ... ugh :) Rather limited degrees of freedom, here.

Patrick

 

[asimov-solensan]

Indeed. It has been my first year with freenas and never faced a major upgrade.

I guess once I have a few free days I will try to go to version 11.

Anyhow thanks a lot for your interest. I learned a couple things.