iocage, vnet0, and bridge0

[aschmitz]

I'm getting started with FreeNAS 11.1, and I want to setup a few jails. I understand that warden jails are going away soon, so I should use iocage jails instead. I have the network working mostly the way I want it to using ip4_addr="vnet0|172.16.16.201/24". This sets up the vnet0 interface in the jail and connects it to bridge0.

I'm having a problem adding the physical Ethernet interface as a member of the bridge. bridge0 doesn't exist until I start the first jail, so I don't think I'm supposed to configure the bridge on system startup. If I run ifconfig bridge0 addm igb0 manually after I start the jail, everything works the way I want it to. I have to run this once each time I reboot FreeNAS, but after I start the first jail that creates bridge0.

In FreeNAS 11.1 what's the correct way to tell iocage that igb0 should be added as a member interface when it creates bridge0?

 

[xlf]

I think there's a bug filed for that already.

To make things really complicated, you cannot set things up permanently from the gui. You need physical access/console access, since, with the old gui you have to klick through stuff one by one, and then inbetween you lose you network connectivity if you have only one network card attached....

 

[Allan Wilmath]

If you have the console commands figured out, seems like you can just put those in a bash script and execute it from the GUI as a Cron job under tasks.

 

[aschmitz]

I would never add commands to configure a bridge interface to a cron job. These commands shouldn't be run more than once. When the machine is rebooted, the system shouldn't wait for a scheduled job to run, before the bridge is connected to the Ethernet interface.

For now I've added a post init script with the following command:
ifconfig bridge create; ifconfig bridge0 addm igb0

It's a hack but it works. I haven't found a related bug report yet, so I'll probably create one.

 

[Patrick M. Hausen]

Code:

root@freenas-pmh:~ # cat /mnt/zfs/scripts/bridge0.sh 
#! /bin/sh

PAUSE="10"
BRIDGE="bridge0"
INTERFACES="igb1 igb2 igb3"

(
	until /sbin/ifconfig | /usr/bin/grep -q "^${BRIDGE}:"; do sleep 2; done
	sleep ${PAUSE}

	for if in ${INTERFACES}
	do
		/sbin/ifconfig ${if} up
		/sbin/ifconfig ${BRIDGE} addm ${if}
	done
) &

exit 0

 

[Celena]

Hello,

i had the same problem but after a few research i have added 2 tunable in the rc.conf :

Code:

Variable : cloned_interfaces ; Value : bridge0
Variable : ifconfig_bridge0 ; Value : addm igb0 up

 

[aschmitz]

i had the same problem but after a few research i have added 2 tunable in the rc.conf :

Code:

Variable : cloned_interfaces ; Value : bridge0
Variable : ifconfig_bridge0 ; Value : addm igb0 up

Thanks. I saw these in some FreeBSD documentation that described adding them to rc.conf. I quickly learned that the rc. conf on FreeNAS can't be edited in the same way. Using tunables adds them to the configuration in a way that survives reboots, and it only configures the bridge once at the right time.

 

[kazooless]

I'm running version FreeNAS-11.1-RELEASE and can confirm the same problem. Adding tunables as mentioned above fixed it.

 

[Beep]

Still not getting iocage jails to work in latest nightly... no IP-adress


Edit: After a reboot the jail isn't working anymore.

Edit2: It's working now.

cloned_interface : bridge0
ifconfig_bridge0 : addm em0 up

It's important to use your real interface (ifconfig from shell and the first interface connected to your home network)

 

[Michael De Cou]

I'm running version FreeNAS-11.1-RELEASE and can confirm the same problem. Adding tunables as mentioned above fixed it.

Can anyone tell me what my tunables should look like if my network connection is an aggregation? The interface is named "lagg0". The problem is am finding is that if I replace the "igb0" above with "lagg0", I am unable to access the entire freenas system after reboot (until i reconfigure the interface from the local kb/monitor).

Here is what I have tried that causes the issue:

and here is my interface view:

Thanks

 

[Michael De Cou]

Ok... this is maddening! As noted, the tunables seem to hose my network connection until i go into the console and configure an interface (any interface). All I need to do is answer "y" when it asks if I want to reset the network interface. It states that the reset failed, but then all my connectivity is back.

Here's where I am going crazy...
I can create an iocage jail, and get no connectivity. However, if I enter the following command manually, the system freezes for a few seconds, then seems to restart the network.

ifconfig bridge0 addm lagg0

At that point, I have normal connectivity to the jail.
Is there a way to clean this up without having to add a custom script? I dont want to have to add that, and then have a mess when the next update comes (and hopefully fixes this). I assume the tunable entry is supposed to accomplish this, but doesnt seem work on my system.

 

[Michael De Cou]

And one follow-up question:

Could all of this be caused if I am running a Warden jail at the same time I am running iocage?
Thanks

 

[Patrick M. Hausen]

Why don't you use the script I posted above as a startup script?

Change this line:

Code:

INTERFACES="igb1 igb2 igb3"

into this:

Code:

INTERFACES="lagg0"

HTH,
Patrick

 

[Michael De Cou]

I would gladly give it a shot, but I am unfamiliar with startup scripts, including where they need to be located and how to call them.

 

[Michael De Cou]

There may be another issue here.... I am currently running a Warden jail on this system as well. When I issue the following manually:

ifconfig bridge0 addm lagg0

The iocage jail gets network connectivity, but the Warden jail loses connectivity. Could there be a conflict with the bridge? If so, is there a way to specify "bridge1" for iocage?

 

[Patrick M. Hausen]

What is the output of ifconfig -a when the system is up and running and warden jails have connectivity as they should?

Patrick

 

[Michael De Cou]

What is the output of ifconfig -a when the system is up and running and warden jails have connectivity as they should?

Patrick

See below. The lagg0 interface is the aggregation of igb1 and igb2.


root@freenas:~ # ifconfig -a
igb0: flags=8c02<BROADCAST,OACTIVE,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=6403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
ether 00:1b:21:67:78:c8
hwaddr 00:1b:21:67:78:c8
nd6 options=9<PERFORMNUD,IFDISABLED>
media: Ethernet autoselect
status: no carrier
igb1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=2400b9<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWTSO,RXCSUM_IPV6>
ether 00:1b:21:67:78:c9
hwaddr 00:1b:21:67:78:c9
nd6 options=9<PERFORMNUD,IFDISABLED>
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
igb2: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=2400b9<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWTSO,RXCSUM_IPV6>
ether 00:1b:21:67:78:c9
hwaddr 00:1b:21:67:78:cc
nd6 options=9<PERFORMNUD,IFDISABLED>
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
igb3: flags=8c02<BROADCAST,OACTIVE,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=6403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
ether 00:1b:21:67:78:cd
hwaddr 00:1b:21:67:78:cd
nd6 options=9<PERFORMNUD,IFDISABLED>
media: Ethernet autoselect
status: no carrier
bge0: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=c019b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,VLAN_HWTSO,LINKSTATE>
ether 5c:b9:01:3b:2a:20
hwaddr 5c:b9:01:3b:2a:20
nd6 options=9<PERFORMNUD,IFDISABLED>
media: Ethernet autoselect
bge1: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=c019b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,VLAN_HWTSO,LINKSTATE>
ether 5c:b9:01:3b:2a:21
hwaddr 5c:b9:01:3b:2a:21
nd6 options=9<PERFORMNUD,IFDISABLED>
media: Ethernet autoselect
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
inet6 ::1 prefixlen 128
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x7
inet 127.0.0.1 netmask 0xff000000
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
groups: lo
lagg0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=2400b9<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWTSO,RXCSUM_IPV6>
ether 00:1b:21:67:78:c9
inet 192.168.1.22 netmask 0xffffff00 broadcast 192.168.1.255
nd6 options=9<PERFORMNUD,IFDISABLED>
media: Ethernet autoselect
status: active
groups: lagg
laggproto lacp lagghash l2,l3,l4
laggport: igb1 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
laggport: igb2 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
ether 02:98:5f:32:82:00
nd6 options=1<PERFORMNUD>
groups: bridge
id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
member: epair0a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 10 priority 128 path cost 2000
member: lagg0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 8 priority 128 path cost 10000
epair0a: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=8<VLAN_MTU>
ether 02:85:50:00:0a:0a
hwaddr 02:85:50:00:0a:0a
nd6 options=1<PERFORMNUD>
media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
status: active
groups: epair
root@freenas:~ #

 

[Michael De Cou]

What is the output of ifconfig -a when the system is up and running and warden jails have connectivity as they should?

Patrick

And... here is the ifconfig output with the Warden jail running correctly as well as the iocage jail, which has no connectivity.

igb0: flags=8c02<BROADCAST,OACTIVE,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=6403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
ether 00:1b:21:67:78:c8
hwaddr 00:1b:21:67:78:c8
nd6 options=9<PERFORMNUD,IFDISABLED>
media: Ethernet autoselect
status: no carrier
igb1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=2400b9<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWTSO,RXCSUM_IPV6>
ether 00:1b:21:67:78:c9
hwaddr 00:1b:21:67:78:c9
nd6 options=9<PERFORMNUD,IFDISABLED>
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
igb2: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=2400b9<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWTSO,RXCSUM_IPV6>
ether 00:1b:21:67:78:c9
hwaddr 00:1b:21:67:78:cc
nd6 options=9<PERFORMNUD,IFDISABLED>
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
igb3: flags=8c02<BROADCAST,OACTIVE,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=6403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
ether 00:1b:21:67:78:cd
hwaddr 00:1b:21:67:78:cd
nd6 options=9<PERFORMNUD,IFDISABLED>
media: Ethernet autoselect
status: no carrier
bge0: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=c019b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,VLAN_HWTSO,LINKSTATE>
ether 5c:b9:01:3b:2a:20
hwaddr 5c:b9:01:3b:2a:20
nd6 options=9<PERFORMNUD,IFDISABLED>
media: Ethernet autoselect
bge1: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=c019b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,VLAN_HWTSO,LINKSTATE>
ether 5c:b9:01:3b:2a:21
hwaddr 5c:b9:01:3b:2a:21
nd6 options=9<PERFORMNUD,IFDISABLED>
media: Ethernet autoselect
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
inet6 ::1 prefixlen 128
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x7
inet 127.0.0.1 netmask 0xff000000
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
groups: lo
lagg0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=2400b9<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWTSO,RXCSUM_IPV6>
ether 00:1b:21:67:78:c9
inet 192.168.1.22 netmask 0xffffff00 broadcast 192.168.1.255
nd6 options=9<PERFORMNUD,IFDISABLED>
media: Ethernet autoselect
status: active
groups: lagg
laggproto lacp lagghash l2,l3,l4
laggport: igb1 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
laggport: igb2 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
ether 02:98:5f:32:82:00
nd6 options=1<PERFORMNUD>
groups: bridge
id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
member: vnet0:4 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 10 priority 128 path cost 2000
bridge1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
ether 02:98:5f:32:82:01
nd6 options=9<PERFORMNUD,IFDISABLED>
groups: bridge
id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
member: epair1a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 12 priority 128 path cost 2000
member: lagg0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 8 priority 128 path cost 10000
epair1a: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=8<VLAN_MTU>
ether 02:85:50:00:0c:0a
hwaddr 02:85:50:00:0c:0a
nd6 options=1<PERFORMNUD>
media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
status: active
groups: epair
vnet0:4: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
description: associated with jail: Plexmediaserver_2
options=8<VLAN_MTU>
ether 02:ff:60:70:24:cb
hwaddr 02:85:50:00:0a:0a
nd6 options=1<PERFORMNUD>
media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
status: active
groups: epair
root@freenas:~ #

 

[Michael De Cou]

So best i can tell, bridge1 is the bridge for my functioning Warden jail, with epair1a and lagg0 as members. In that jail, the interface is epair1b.
In the iocage jail, the interface is epair0b. The bridge for this iocage is bridge0, which only has the member vnet0:4. When i try to add the lagg0 to bridge0, i get:

root@freenas:~ # ifconfig bridge0 addm lagg0
ifconfig: BRDGADD lagg0: Device busy

Is it possible to have lagg0 as a member of 2 different bridges?

 

[GreyMatters]

I just updated to FreeNAS-11.1-U2 and got this network issue, but I'm not sure it's the same bug.
My situation is delicate because I have no easy physical access to the machine, so I am esithant to do network changes.

My situation is this: I access two jails from zeroconfig network. One jail is a warden one, the other is a newer iocage one. This is very lucky! I created the iocage jail because I am unable to update the warden one (and this is shameful). After the last update the iocage one lost the ability to manage its network interface.
Zerotier is not functioning and even sshd can't open the port:

sshd[49118]: error: Bind to port 22 on :: failed: Can't assign requested address

I can access the outside network from the jail, but the daemons can't manage their connections.

This is my ifconfig from the freenas host:

bge0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=c0099<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,VLAN_HWTSO,LINKSTATE>
ether 98:f2:b3:f7:28:b4
hwaddr 98:f2:b3:f7:28:b4
inet 192.168.1.136 netmask 0xffffff00 broadcast 192.168.1.255
nd6 options=9<PERFORMNUD,IFDISABLED>
media: Ethernet autoselect (100baseTX <full-duplex>)
status: active
bge1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=c019b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,VLAN_HWTSO,LINKSTATE>
ether 98:f2:b3:f7:28:b5
hwaddr 98:f2:b3:f7:28:b5
nd6 options=9<PERFORMNUD,IFDISABLED>
media: Ethernet autoselect (none)
status: no carrier
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
inet6 ::1 prefixlen 128
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3
inet 127.0.0.1 netmask 0xff000000
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
groups: lo
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
ether 02:a5:da:0c:e5:00
nd6 options=9<PERFORMNUD,IFDISABLED>
groups: bridge
id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
member: vnet0:5 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 5 priority 128 path cost 2000
member: epair3a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 8 priority 128 path cost 2000
member: epair2a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 7 priority 128 path cost 2000
member: epair1a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 6 priority 128 path cost 2000
member: bge0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 1 priority 128 path cost 200000
epair1a: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=8<VLAN_MTU>
ether 02:54:50:00:06:0a
hwaddr 02:54:50:00:06:0a
nd6 options=1<PERFORMNUD>
media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
status: active
groups: epair
epair2a: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=8<VLAN_MTU>
ether 02:54:50:00:07:0a
hwaddr 02:54:50:00:07:0a
nd6 options=1<PERFORMNUD>
media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
status: active
groups: epair
epair3a: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=8<VLAN_MTU>
ether 02:54:50:00:08:0a
hwaddr 02:54:50:00:08:0a
nd6 options=1<PERFORMNUD>
media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
status: active
groups: epair
vnet0:5: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
description: associated with jail: PortRoyal
options=8<VLAN_MTU>
ether 02:ff:60:33:95:eb
hwaddr 02:54:50:00:05:0a
nd6 options=1<PERFORMNUD>
media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
status: active
groups: epair

In the gui I can see that the iocage jail has the "Interfaces" field set to: vnet0:bridge0 .

What I'm missing?

Thank you for your help!