SOLVED bridge from jails and iohyve guests to outside network fails

[wurzeladministrator]

Hi,

I have seen similar questions on this forum but no real solutions that worked for me so I decided to post a new one. Any help appreciated.

We are running Freenas 9.10.2-U1 on a Dell R530 with Broadcom Corporation NetXtreme BCM5720 NICs of which only one is connected to the LAN.

Interface name is bge0 and configured with a static IP in a /22 private network:

Code:

root@freenashost# ifconfig bge0
bge0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
		options=c0099<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,VLAN_HWTSO,LINKSTATE>
		ether 18:66:da:85:bb:05
		inet 10.6.0.25 netmask 0xfffffc00 broadcast 10.6.3.255
		nd6 options=9<PERFORMNUD,IFDISABLED>
		media: Ethernet autoselect (1000baseT <full-duplex>)
		status: active

I have tried to setup jails and iohyve guests but I never get them to be able to connect to the LAN other than the host. DHCP requests from guests fail and when I setup static IPs I can ping the FreeNAS host IP but not the gateway or any other host on the LAN:

Code:

root@freenashost# arp -a
? (10.6.0.1) at 54:75:d0:ba:54:f5 on bge0 expires in 822 seconds [ethernet]
? (10.6.0.25) at 18:66:da:85:bb:05 on bge0 permanent [ethernet]
? (10.6.0.26) at 00:a0:98:79:64:d0 on bge0 expires in 1064 seconds [ethernet]
root@freenashost# ping 10.6.0.26
PING 10.6.0.26 (10.6.0.26): 56 data bytes
64 bytes from 10.6.0.26: icmp_seq=0 ttl=64 time=0.138 ms
64 bytes from 10.6.0.26: icmp_seq=1 ttl=64 time=0.086 ms
^C
--- 10.6.0.26 ping statistics ---
2 packets transmitted, 2 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.086/0.112/0.138/0.026 ms

root@freenashost # ssh crasher@10.6.0.26
crasher@10.6.0.26's password:
Welcome to Ubuntu 16.04.1 LTS (GNU/Linux 4.4.0-31-generic x86_64)

* Documentation:  https://help.ubuntu.com
* Management:	 https://landscape.canonical.com
* Support:		https://ubuntu.com/advantage

0 packages can be updated.
0 updates are security updates.


Last login: Fri Jan 20 07:17:10 2017 from 10.6.0.25
crasher@crashub:~$ ifconfig
enp0s3	Link encap:Ethernet  HWaddr 00:a0:98:79:64:d0
		  inet addr:10.6.0.26  Bcast:10.6.3.255  Mask:255.255.252.0
		  inet6 addr: fe80::2a0:98ff:fe79:64d0/64 Scope:Link
		  UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
		  RX packets:45 errors:0 dropped:0 overruns:0 frame:0
		  TX packets:51 errors:0 dropped:0 overruns:0 carrier:0
		  collisions:0 txqueuelen:1000
		  RX bytes:7288 (7.2 KB)  TX bytes:7373 (7.3 KB)

lo		Link encap:Local Loopback
		  inet addr:127.0.0.1  Mask:255.0.0.0
		  inet6 addr: ::1/128 Scope:Host
		  UP LOOPBACK RUNNING  MTU:65536  Metric:1
		  RX packets:160 errors:0 dropped:0 overruns:0 frame:0
		  TX packets:160 errors:0 dropped:0 overruns:0 carrier:0
		  collisions:0 txqueuelen:1
		  RX bytes:11840 (11.8 KB)  TX bytes:11840 (11.8 KB)

crasher@crashub:~$ ping 10.6.0.25
PING 10.6.0.25 (10.6.0.25) 56(84) bytes of data.
64 bytes from 10.6.0.25: icmp_seq=1 ttl=64 time=0.067 ms
64 bytes from 10.6.0.25: icmp_seq=2 ttl=64 time=0.097 ms
^C
--- 10.6.0.25 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 999ms
rtt min/avg/max/mdev = 0.067/0.082/0.097/0.015 ms
crasher@crashub:~$ ping 10.6.0.1
PING 10.6.0.1 (10.6.0.1) 56(84) bytes of data.
From 10.6.0.26 icmp_seq=1 Destination Host Unreachable
From 10.6.0.26 icmp_seq=2 Destination Host Unreachable
From 10.6.0.26 icmp_seq=3 Destination Host Unreachable
^C
--- 10.6.0.1 ping statistics ---
4 packets transmitted, 0 received, +3 errors, 100% packet loss, time 3012ms
pipe 3
crasher@crashub:~$ arp -a
? (10.6.0.1) at <incomplete> on enp0s3
? (10.6.0.25) at 18:66:da:85:bb:05 [ether] on enp0s3
crasher@crashub:~$

The freebsd jail was created from the WebUI, the iohyve guest is an Ubuntu with all the steps taken into consideration from the FreeNAS Doc (https://doc.freenas.org/9.10/jails.html#using-iohyve) and the iohyve github README (https://github.com/pr1ntf/iohyve)

Here is the ifconfig for the bridge and tap (iohyve) and epair (jail) devices. tap1 is from a second iohyve guest currently not running (same network problems)

Code:

bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
		description: iohyve-bridge
		ether 02:42:fa:be:08:00
		inet 0.0.0.0 netmask 0xff000000 broadcast 255.255.255.255
		nd6 options=1<PERFORMNUD>
		id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
		maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
		root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
		member: epair0a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
				ifmaxaddr 0 port 8 priority 128 path cost 2000
		member: tap1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
				ifmaxaddr 0 port 9 priority 128 path cost 2000000
		member: tap2 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
				ifmaxaddr 0 port 7 priority 128 path cost 2000000
		member: bge0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
				ifmaxaddr 0 port 1 priority 128 path cost 20000
tap2: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
		description: iohyve-crashub
		options=80000<LINKSTATE>
		ether 00:bd:ca:23:03:02
		nd6 options=9<PERFORMNUD,IFDISABLED>
		media: Ethernet autoselect
		status: active
		Opened by PID 8058
tap1: flags=8902<BROADCAST,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
		description: iohyve-crasharch
		options=80000<LINKSTATE>
		ether 00:bd:4b:47:13:01
		nd6 options=9<PERFORMNUD,IFDISABLED>
		media: Ethernet autoselect
		status: no carrier
epair0a: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
		options=8<VLAN_MTU>
		ether 02:ff:20:00:08:0a
		nd6 options=1<PERFORMNUD>
		media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
		status: active

When I run tcpdump on the host interface bge0, I see the arp requests from the guest but no replies:

Code:

root@freenashost# tcpdump arp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on bge0, link-type EN10MB (Ethernet), capture size 65535 bytes
16:03:47.879511 ARP, Request who-has 10.6.0.1 tell 10.6.0.26, length 46
16:03:48.880417 ARP, Request who-has 10.6.0.1 tell 10.6.0.26, length 46

How do I set up the bridge and NIC interfaces so that replies from LAN hosts actually reach the guestVMs / jails ?

Thanks,
Ben

 

[dlavigne]

Were you able to figure this out?

 

[wurzeladministrator]

No, not completely yet. I still can not get the jails or iohyve vm connect to a different address than the host.

As no one pointed out an obvious error in my posted configuration yet I was starting to wonder if the problem could lay on the other side of the connection. A recent conversation with my co-workers revealed that some switch ports in our network might be configured to only serve one IP or MAC at a time. They had to jump some hurdles with the network administrators to be able to install a small switch to serve more machines from only one available socket in their office. I guess this could be the same for the socket the FreeNAS host is connected to. The FreeNAS host and clients currently reside in a sub segment of our partner and we have no control over that network infrastructure. In fact FreeNAS is the first step of our work group to be more independent. I have contacted the network admin of our partner about this but will have to wait a few days for a proper reply.

TL;DR: This seems to be a problem on the LAN side. Likely not a FreeNAS bug or a configuration problem.

Ben

 

[wurzeladministrator]

The issue has been resolved. It was indeed a restriction from the LAN side. I now am able to connect jails and iohyve with separate DHCP or static IPs. :)

 

[Jonas Bülow]

Can you explain how you solved this problem?

 

[wurzeladministrator]

As I said, it didn't need to be solved on the FreenNAS side. The configuration on the network switch had to be changed to allow multiple IP addresses or hosts behind the same drop.