iocage VMs and multiple bridges problems

Status
Not open for further replies.
M

Mark WatIng

Explorer
Joined
Mar 6, 2017
Messages
53
Hi all,

Very new to networking so I might be doing something wrong here..... I'm on FreeNAS 11.1-Release

Anyway setup 2 VM's and using iocage for plex and a basic FreeBSD.

When setting up, FreeNAS creates two bridges one for iocage (includes vnets) and one for the VM's (includes tap0, tap1).

Problem is that my physical ethernet adapter can only be assigned to one bridge (which FreeNAS assigns to the VM bridge), and this means that my jails cannot access the internet.

The only solution I have found is once FreeNAS has booted, destroy the bridge containing tap0, tap1, bge1(ethernet adapter), and then assign them to bridge0 which is where my vnets are.

Question is - is this normal, or am I missing something here - appreciate any help please? Thank you!

My ifconfig output after boot :
Code:
bge0: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=c019b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,VLAN_HWTSO,LINKSTATE>
	ether d0:bf:9c:45:0c:88
	hwaddr d0:bf:9c:45:0c:88
	nd6 options=9<PERFORMNUD,IFDISABLED>
	media: Ethernet autoselect
bge1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=c0099<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,VLAN_HWTSO,LINKSTATE>
	ether d0:bf:9c:45:0c:89
	hwaddr d0:bf:9c:45:0c:89
	inet 192.168.1.64 netmask 0xffffff00 broadcast 192.168.1.255 
	nd6 options=9<PERFORMNUD,IFDISABLED>
	media: Ethernet autoselect (100baseTX <full-duplex>)
	status: active
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
	options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
	inet6 ::1 prefixlen 128 
	inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3 
	inet 127.0.0.1 netmask 0xff000000 
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
	groups: lo 
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	ether 02:7e:d7:d1:a7:00
	nd6 options=1<PERFORMNUD>
	groups: bridge 
	id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
	maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
	root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
	member: vnet0:2 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
			ifmaxaddr 0 port 6 priority 128 path cost 2000
	member: vnet0:1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
			ifmaxaddr 0 port 5 priority 128 path cost 2000
vnet0:1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
	description: associated with jail: plex
	options=8<VLAN_MTU>
	ether 02:ff:60:14:fa:09
	hwaddr 02:d4:50:00:05:0a
	nd6 options=1<PERFORMNUD>
	media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
	status: active
	groups: epair 
vnet0:2: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
	description: associated with jail: FreeBSD_iocage
	options=8<VLAN_MTU>
	ether 02:ff:60:4a:02:51
	hwaddr 02:d4:50:00:06:0a
	nd6 options=1<PERFORMNUD>
	media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
	status: active
	groups: epair 
bridge1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	ether 02:7e:d7:d1:a7:01
	nd6 options=1<PERFORMNUD>
	groups: bridge 
	id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
	maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
	root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
	member: tap1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
			ifmaxaddr 0 port 10 priority 128 path cost 2000000
	member: tap0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
			ifmaxaddr 0 port 9 priority 128 path cost 2000000
	member: bge1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
			ifmaxaddr 0 port 2 priority 128 path cost 200000
tap0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=80000<LINKSTATE>
	ether 00:bd:d1:96:f8:00
	hwaddr 00:bd:d1:96:f8:00
	nd6 options=1<PERFORMNUD>
	media: Ethernet autoselect
	status: active
	groups: tap 
	Opened by PID 6482
tap1: flags=8903<UP,BROADCAST,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=80000<LINKSTATE>
	ether 00:bd:2c:97:f8:01
	hwaddr 00:bd:2c:97:f8:01
	nd6 options=1<PERFORMNUD>
	media: Ethernet autoselect
	status: no carrier
	groups: tap


My modified working ifconfig :
Code:
bge0: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=c019b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,VLAN_HWTSO,LINKSTATE>
	ether d0:bf:9c:45:0c:88
	hwaddr d0:bf:9c:45:0c:88
	nd6 options=9<PERFORMNUD,IFDISABLED>
	media: Ethernet autoselect
bge1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=c0099<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,VLAN_HWTSO,LINKSTATE>
	ether d0:bf:9c:45:0c:89
	hwaddr d0:bf:9c:45:0c:89
	inet 192.168.1.64 netmask 0xffffff00 broadcast 192.168.1.255 
	nd6 options=9<PERFORMNUD,IFDISABLED>
	media: Ethernet autoselect (100baseTX <full-duplex>)
	status: active
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
	options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
	inet6 ::1 prefixlen 128 
	inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3 
	inet 127.0.0.1 netmask 0xff000000 
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
	groups: lo 
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	ether 02:7e:d7:d1:a7:00
	nd6 options=1<PERFORMNUD>
	groups: bridge 
	id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
	maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
	root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
	member: bge1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
			ifmaxaddr 0 port 2 priority 128 path cost 200000
	member: tap1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
			ifmaxaddr 0 port 9 priority 128 path cost 2000000
	member: tap0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
			ifmaxaddr 0 port 7 priority 128 path cost 2000000
	member: vnet0:2 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
			ifmaxaddr 0 port 6 priority 128 path cost 2000
	member: vnet0:1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
			ifmaxaddr 0 port 5 priority 128 path cost 2000
vnet0:1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
	description: associated with jail: plex
	options=8<VLAN_MTU>
	ether 02:ff:60:14:fa:09
	hwaddr 02:d4:50:00:05:0a
	nd6 options=1<PERFORMNUD>
	media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
	status: active
	groups: epair 
vnet0:2: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
	description: associated with jail: FreeBSD_iocage
	options=8<VLAN_MTU>
	ether 02:ff:60:4a:02:51
	hwaddr 02:d4:50:00:06:0a
	nd6 options=1<PERFORMNUD>
	media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
	status: active
	groups: epair 
tap0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=80000<LINKSTATE>
	ether 00:bd:6c:8d:f8:00
	hwaddr 00:bd:6c:8d:f8:00
	nd6 options=1<PERFORMNUD>
	media: Ethernet autoselect
	status: active
	groups: tap 
	Opened by PID 5276
tap1: flags=8903<UP,BROADCAST,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=80000<LINKSTATE>
	ether 00:bd:fe:8d:f8:01
	hwaddr 00:bd:fe:8d:f8:01
	nd6 options=1<PERFORMNUD>
	media: Ethernet autoselect
	status: no carrier
	groups: tap

Thanks,
Mark.
 
D

dlavigne

Guest
Sounds like a bug. Please create a report at bugs.freenas.org and post the issue number here.
 
M

Mark WatIng

Explorer
Joined
Mar 6, 2017
Messages
53
Ok Thanks - I have logged the bug here :-

Issue #27780 created.
 
lukyjay

lukyjay

Contributor
Joined
May 13, 2016
Messages
134
Can you please shed some light on how you fixed this issue, i.e. what commands you used to fix your ifconfig? I'm currently experiencing this problem.
 
M

Mark WatIng

Explorer
Joined
Mar 6, 2017
Messages
53
Sure - health warning though, I'm not a network admin so don't know if my changes are correct from a purist point of view - but it does work.... see below the changes I made

Code:
ifconfig bridge1 destroy
ifconfig bridge0 addm tap1
ifconfig bridge0 addm tap0
ifconfig bridge0 addm bge1


I basically added all the adapters to a single bridge and destroyed the other bridge.
 
H

holt

Cadet
Joined
Mar 21, 2017
Messages
2
Mark WatIng said:
Sure - health warning though, I'm not a network admin so don't know if my changes are correct from a purist point of view - but it does work.... see below the changes I made

Code:
ifconfig bridge1 destroy
ifconfig bridge0 addm tap1
ifconfig bridge0 addm tap0
ifconfig bridge0 addm bge1


I basically added all the adapters to a single bridge and destroyed the other bridge.
Click to expand...

Just want to say thanks. Your solution helped me :)

Had created a VM and after a freenas reboot all my vnet jails had no internet access.
 
M

Mark WatIng

Explorer
Joined
Mar 6, 2017
Messages
53
cool :smile:
 
short-stack

short-stack

Explorer
Joined
Feb 28, 2017
Messages
80
This is a known bug, and is scheduled to be fixed soon. https://redmine.ixsystems.com/issues/30276#change-224078

But now (11.2-BETA1 and 11.2-BETA2) it does breaks in a totally different way.

This is because of a few things, Plex requires a VNET interface in the jail because it needs it's own MAC address to function. FreeNAS has/had an issue where as it was booting the VNET jail would be put on it's own bridge instead of the existing one, basically breaking everything.

The easiest thing I do to workaround it is set the Plex jail to not boot automatically. Let FreeNAS boot up completely and then once it's running, manually start the VNET jail, it then gets added to the bridge appropriately. I have a ticket in for this as well, but it's currently private while they evaluate it.
 
D

dlavigne

Guest
It wouldn't hurt to open another ticket for the new breakage as we'd like to shake out most of the use cases before we go to RC status. If you create one at bugs.freenas.org, post the issue number here.
 
short-stack

short-stack

Explorer
Joined
Feb 28, 2017
Messages
80
dlavigne said:
It wouldn't hurt to open another ticket for the new breakage as we'd like to shake out most of the use cases before we go to RC status. If you create one at bugs.freenas.org, post the issue number here.
Click to expand...

I did, the second ticket is #40484.

Thanks!


Sent from my iPhone using Tapatalk
 
Status
Not open for further replies.

Similar threads

rodofrn
  • Locked
Very slow transfer speeds.
Replies
17
Views
4K
Elliot Dierksen
Elliot Dierksen
N
Help getting 10 gig Peer to Peer working
Replies
16
Views
3K
Redcoat
Redcoat
phynias
upgrade to 11.2-U3 and now jails have no network access
Replies
2
Views
1K
phynias
phynias
kdragon75
  • Locked
Broken Jail Networking 11.1U6
Replies
2
Views
1K
kdragon75
kdragon75
K
FreeNAS kernel arp attempts to modify permanent
Replies
2
Views
2K
dlavigne
D
Top