Warden jails and iocage jails make separate bridges, and one doesn't work...

Status
Not open for further replies.
spotcatbug

spotcatbug

Dabbler
Joined
Nov 6, 2017
Messages
43
I feel like this was discussed before, in relation to iocage networking, but I didn't quite follow and now I can't find what I thought I saw.

Here's my situation:

I have two working warden jails and one broken iocage jail. The non-working iocage jail is for OpenVPN and must use the "VNET" option (so that it can create its tunnel interface). Although ifconfig in the jail looks fine, I can't ping anything - the jail has no network connectivity.

ifconfig on the server shows two bridge interfaces - one for the warden jails and one for the iocage jail. The bridge for the warden jails looks right. It has the two interfaces from the two warden jails and the interface that it's supposed to bridge to. The iocage bridge, however, has only the interface from the iocage jail - it isn't bridging anything.

When I try to specify the warden jails' bridge in the iocage jail's interface setting, it causes a reboot of the server when I restart the jail.

Anybody have any idea what's going on and how to fix this? Do I need to bite the bullet and bring my warden jails to iocage? That would really suck.

Please help! Thanks.
 
Chris Moore

Chris Moore

Hall of Famer
Joined
May 2, 2015
Messages
10,080
Mlovelace

Mlovelace

Guru
Joined
Aug 19, 2014
Messages
1,111
spotcatbug said:
I feel like this was discussed before, in relation to iocage networking, but I didn't quite follow and now I can't find what I thought I saw.

Here's my situation:

I have two working warden jails and one broken iocage jail. The non-working iocage jail is for OpenVPN and must use the "VNET" option (so that it can create its tunnel interface). Although ifconfig in the jail looks fine, I can't ping anything - the jail has no network connectivity.

ifconfig on the server shows two bridge interfaces - one for the warden jails and one for the iocage jail. The bridge for the warden jails looks right. It has the two interfaces from the two warden jails and the interface that it's supposed to bridge to. The iocage bridge, however, has only the interface from the iocage jail - it isn't bridging anything.

When I try to specify the warden jails' bridge in the iocage jail's interface setting, it causes a reboot of the server when I restart the jail.

Anybody have any idea what's going on and how to fix this? Do I need to bite the bullet and bring my warden jails to iocage? That would really suck.

Please help! Thanks.
Click to expand...
You don't have to convert the warden jails, yet. Post the output of ifconfig -a in code tags, please.
 
spotcatbug

spotcatbug

Dabbler
Joined
Nov 6, 2017
Messages
43
spotcatbug

spotcatbug

Dabbler
Joined
Nov 6, 2017
Messages
43
Mlovelace said:
You don't have to convert the warden jails, yet. Post the output of ifconfig -a in code tags, please.
Click to expand...

This is inside the OpenVPN iocage jail:

Code:
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
	options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
	inet6 ::1 prefixlen 128
	inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1
	inet 127.0.0.1 netmask 0xff000000
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
	groups: lo
epair0b: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=8<VLAN_MTU>
	ether 02:ff:60:9b:c8:80
	hwaddr 02:f7:d0:00:07:0b
	inet 10.0.1.52 netmask 0xff000000 broadcast 10.255.255.255
	nd6 options=9<PERFORMNUD,IFDISABLED>
	media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
	status: active
	groups: epair
tun0: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> metric 0 mtu 1500
	options=80000<LINKSTATE>
	inet 10.8.0.1 --> 10.8.0.2  netmask 0xffffffff
	nd6 options=9<PERFORMNUD,IFDISABLED>
	groups: tun
	Opened by PID 3717


Outside the jail (the jail-related interfaces start at bridge0):

Code:
igb0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM>
	ether d0:50:99:c2:f0:1f
	hwaddr d0:50:99:c2:f0:1f
	nd6 options=9<PERFORMNUD,IFDISABLED>
	media: Ethernet autoselect (1000baseT <full-duplex>)
	status: active
em0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=98<VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM>
	ether d0:50:99:c2:f0:1f
	hwaddr d0:50:99:c2:f0:20
	nd6 options=9<PERFORMNUD,IFDISABLED>
	media: Ethernet autoselect (1000baseT <full-duplex>)
	status: active
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
	options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
	inet6 ::1 prefixlen 128 
	inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3 
	inet 127.0.0.1 netmask 0xff000000 
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
	groups: lo 
lagg0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=98<VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM>
	ether d0:50:99:c2:f0:1f
	inet 10.0.1.114 netmask 0xffffff00 broadcast 10.0.1.255 
	inet 10.0.1.63 netmask 0xffffff00 broadcast 10.0.1.255 
	nd6 options=9<PERFORMNUD,IFDISABLED>
	media: Ethernet autoselect
	status: active
	groups: lagg 
	laggproto loadbalance lagghash l2,l3,l4
	laggport: igb0 flags=4<ACTIVE>
	laggport: em0 flags=4<ACTIVE>
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	ether 02:81:c5:8a:b2:00
	nd6 options=1<PERFORMNUD>
	groups: bridge 
	id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
	maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
	root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
	member: vnet0:2 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
			ifmaxaddr 0 port 6 priority 128 path cost 2000
vnet0:2: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
	description: associated with jail: OpenVPN
	options=8<VLAN_MTU>
	ether 02:ff:60:9b:c8:7f
	hwaddr 02:f7:d0:00:06:0a
	nd6 options=1<PERFORMNUD>
	media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
	status: active
	groups: epair 
bridge1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	ether 02:81:c5:8a:b2:01
	nd6 options=1<PERFORMNUD>
	groups: bridge 
	id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
	maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
	root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
	member: epair2a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
			ifmaxaddr 0 port 9 priority 128 path cost 2000
	member: epair1a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
			ifmaxaddr 0 port 8 priority 128 path cost 2000
	member: lagg0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
			ifmaxaddr 0 port 4 priority 128 path cost 10000
epair1a: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=8<VLAN_MTU>
	ether 02:f7:d0:00:08:0a
	hwaddr 02:f7:d0:00:08:0a
	nd6 options=1<PERFORMNUD>
	media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
	status: active
	groups: epair 
epair2a: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=8<VLAN_MTU>
	ether 02:f7:d0:00:09:0a
	hwaddr 02:f7:d0:00:09:0a
	nd6 options=1<PERFORMNUD>
	media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
	status: active
	groups: epair


OK, I feel dumb. Putting these in here, I realize I did this before, in another thread.
 
Mlovelace

Mlovelace

Guru
Joined
Aug 19, 2014
Messages
1,111
spotcatbug said:
This is inside the OpenVPN iocage jail:

Code:
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
	options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
	inet6 ::1 prefixlen 128
	inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1
	inet 127.0.0.1 netmask 0xff000000
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
	groups: lo
epair0b: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=8<VLAN_MTU>
	ether 02:ff:60:9b:c8:80
	hwaddr 02:f7:d0:00:07:0b
	inet 10.0.1.52 netmask 0xff000000 broadcast 10.255.255.255
	nd6 options=9<PERFORMNUD,IFDISABLED>
	media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
	status: active
	groups: epair
tun0: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> metric 0 mtu 1500
	options=80000<LINKSTATE>
	inet 10.8.0.1 --> 10.8.0.2  netmask 0xffffffff
	nd6 options=9<PERFORMNUD,IFDISABLED>
	groups: tun
	Opened by PID 3717


Outside the jail (the jail-related interfaces start at bridge0):

Code:
igb0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM>
	ether d0:50:99:c2:f0:1f
	hwaddr d0:50:99:c2:f0:1f
	nd6 options=9<PERFORMNUD,IFDISABLED>
	media: Ethernet autoselect (1000baseT <full-duplex>)
	status: active
em0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=98<VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM>
	ether d0:50:99:c2:f0:1f
	hwaddr d0:50:99:c2:f0:20
	nd6 options=9<PERFORMNUD,IFDISABLED>
	media: Ethernet autoselect (1000baseT <full-duplex>)
	status: active
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
	options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
	inet6 ::1 prefixlen 128
	inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3
	inet 127.0.0.1 netmask 0xff000000
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
	groups: lo
lagg0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=98<VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM>
	ether d0:50:99:c2:f0:1f
	inet 10.0.1.114 netmask 0xffffff00 broadcast 10.0.1.255
	inet 10.0.1.63 netmask 0xffffff00 broadcast 10.0.1.255
	nd6 options=9<PERFORMNUD,IFDISABLED>
	media: Ethernet autoselect
	status: active
	groups: lagg
	laggproto loadbalance lagghash l2,l3,l4
	laggport: igb0 flags=4<ACTIVE>
	laggport: em0 flags=4<ACTIVE>
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	ether 02:81:c5:8a:b2:00
	nd6 options=1<PERFORMNUD>
	groups: bridge
	id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
	maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
	root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
	member: vnet0:2 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
			ifmaxaddr 0 port 6 priority 128 path cost 2000
vnet0:2: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
	description: associated with jail: OpenVPN
	options=8<VLAN_MTU>
	ether 02:ff:60:9b:c8:7f
	hwaddr 02:f7:d0:00:06:0a
	nd6 options=1<PERFORMNUD>
	media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
	status: active
	groups: epair
bridge1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	ether 02:81:c5:8a:b2:01
	nd6 options=1<PERFORMNUD>
	groups: bridge
	id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
	maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
	root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
	member: epair2a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
			ifmaxaddr 0 port 9 priority 128 path cost 2000
	member: epair1a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
			ifmaxaddr 0 port 8 priority 128 path cost 2000
	member: lagg0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
			ifmaxaddr 0 port 4 priority 128 path cost 10000
epair1a: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=8<VLAN_MTU>
	ether 02:f7:d0:00:08:0a
	hwaddr 02:f7:d0:00:08:0a
	nd6 options=1<PERFORMNUD>
	media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
	status: active
	groups: epair
epair2a: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=8<VLAN_MTU>
	ether 02:f7:d0:00:09:0a
	hwaddr 02:f7:d0:00:09:0a
	nd6 options=1<PERFORMNUD>
	media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
	status: active
	groups: epair


OK, I feel dumb. Putting these in here, I realize I did this before, in another thread.
Click to expand...
So, bridge0 is not getting lagg0 as a member, you'll want to add these tunables.

Code:
Variable: cloned_interfaces
Value: bridge0 bridge1
Type: rc.conf

Code:
Variable: ifconfig_bridge0
Value: addm lagg0 up
Type: rc.conf


I ran into an issue with 11.1-U5 where my interface wasn't being placed into promiscuous mode. If this also ends up being the case for you add promisc to the interface options in the networking tab.
 
spotcatbug

spotcatbug

Dabbler
Joined
Nov 6, 2017
Messages
43
Mlovelace said:
So, bridge0 is not getting lagg0 as a member, you'll want to add these tunables.

Code:
Variable: cloned_interfaces
Value: bridge0 bridge1
Type: rc.conf

Code:
Variable: ifconfig_bridge0
Value: addm lagg0 up
Type: rc.conf


I ran into an issue with 11.1-U5 where my interface wasn't being placed into promiscuous mode. If this also ends up being the case for you add promisc to the interface options in the networking tab.
Click to expand...

I added the tunables, now I can't get into the web GUI. I can get to a shell at the physical server. I did an ifconfig and I see that two new bridges have been created (bridge2 and bridge3). I can't copy and paste for better analysis, but it looks pretty wrong. Bridge0 and bridge1 are still there (if I'm remembering right - I can't be here and at the server at the same time). Each warden jail interface ended up in one of the two new bridge interfaces all by themselves (like the iocage jail interface was). I can't remember where the iocage interface ended up.

How do I remove these tunables from the shell? I'm Googling and not finding anything. My FreeNAS box has no network right now. There's got to be a non-GUI way to edit the tunables, right?
 
Mlovelace

Mlovelace

Guru
Joined
Aug 19, 2014
Messages
1,111
spotcatbug said:
I added the tunables, now I can't get into the web GUI. I can get to a shell at the physical server. I did an ifconfig and I see that two new bridges have been created (bridge2 and bridge3). I can't copy and paste for better analysis, but it looks pretty wrong. Bridge0 and bridge1 are still there (if I'm remembering right - I can't be here and at the server at the same time). Each warden jail interface ended up in one of the two new bridge interfaces all by themselves (like the iocage jail interface was). I can't remember where the iocage interface ended up.

How do I remove these tunables from the shell? I'm Googling and not finding anything. My FreeNAS box has no network right now. There's got to be a non-GUI way to edit the tunables, right?
Click to expand...
I don't know that you can edit the gui tunables from the cli, however you can try restarting the gui or reboot into a previous boot environment.
 
spotcatbug

spotcatbug

Dabbler
Joined
Nov 6, 2017
Messages
43
Thanks for your help. My problem was that my network connection is aggregated. This caused those tunables to make the network connection not work at all. It's all good now.

To fix my jail networking issue, I ended up with a fairly simple manual process that I can perform every time I have to reboot - which is pretty rare. After a reboot, I have to add my aggregated network connection to bridge0 ( ifconfig bridge0 addm lagg0) and then manually start my warden jails after the iocage jails have already started.
 
Status
Not open for further replies.

Similar threads

kdragon75
  • Locked
FreeNAS keeps redoing all of my jail networking!
Replies
1
Views
667
kdragon75
kdragon75
A
  • Locked
Changing Network settings causes loss of connection to iocage jails
Replies
2
Views
1K
wintered
wintered
M
ZeroTier in iocage jail won't work
Replies
6
Views
3K
Bigsby
Bigsby
eminguez
  • Locked
iocage NAT cli?
Replies
7
Views
3K
kdragon75
kdragon75
D
iocage plex metadata and transcoding issue
2 3
Replies
47
Views
15K
Giovani
Giovani
Top