em0 missing on bridge0 after upgrade to FreeNAS 11.1

[Ravefiend]

So decided to upgrade my FreeNAS to 11.1-U1 and started to recreate my jails using iocage. All went well initially, till after a reboot I appear to have lost em0 on my bridge0. As a result, my new jails (configured with static IP addresses) have no more network connectivity to my LAN.

# ifconfig
em0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=2098<VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC>
ether 00:25:90:30:71:b4
hwaddr 00:25:90:30:71:b4
inet 192.168.1.40 netmask 0xffffff00 broadcast 192.168.1.255
nd6 options=9<PERFORMNUD,IFDISABLED>
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
em1: flags=8c02<BROADCAST,OACTIVE,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC>
ether 00:25:90:30:71:b5
hwaddr 00:25:90:30:71:b5
nd6 options=9<PERFORMNUD,IFDISABLED>
media: Ethernet autoselect
status: no carrier
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
inet6 ::1 prefixlen 128
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3
inet 127.0.0.1 netmask 0xff000000
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
groups: lo
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
ether 02:12:27:d0:2e:00
nd6 options=1<PERFORMNUD>
groups: bridge
id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
member: vnet0:4 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 8 priority 128 path cost 2000
member: vnet0:3 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 7 priority 128 path cost 2000
member: vnet0:2 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 6 priority 128 path cost 2000
member: vnet0:1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 5 priority 128 path cost 2000
vnet0:1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
description: associated with jail: plex2
options=8<VLAN_MTU>
ether 02:ff:60:7e:eb:bb
hwaddr 02:bc:90:00:05:0a
nd6 options=1<PERFORMNUD>
media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
status: active
groups: epair
vnet0:2: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
description: associated with jail: transmission
options=8<VLAN_MTU>
ether 02:ff:60:03:aa:46
hwaddr 02:bc:90:00:06:0a
nd6 options=1<PERFORMNUD>
media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
status: active
groups: epair
vnet0:3: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
description: associated with jail: sabnzbd
options=8<VLAN_MTU>
ether 02:ff:60:ef:4d:38
hwaddr 02:bc:90:00:07:0a
nd6 options=1<PERFORMNUD>
media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
status: active
groups: epair
vnet0:4: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
description: associated with jail: plex
options=8<VLAN_MTU>
ether 02:ff:60:14:fa:09
hwaddr 02:bc:90:00:08:0a
nd6 options=1<PERFORMNUD>
media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
status: active
groups: epair
bridge1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
ether 02:12:27:d0:2e:01
nd6 options=9<PERFORMNUD,IFDISABLED>
groups: bridge
id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
member: epair11a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 17 priority 128 path cost 2000
member: epair10a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 16 priority 128 path cost 2000
member: epair9a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 15 priority 128 path cost 2000
member: epair8a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 14 priority 128 path cost 2000
member: epair7a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 13 priority 128 path cost 2000
member: epair6a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 12 priority 128 path cost 2000
member: epair5a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 11 priority 128 path cost 2000
member: epair4a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 10 priority 128 path cost 2000
member: em0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 1 priority 128 path cost 20000
epair4a: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=8<VLAN_MTU>
ether 02:bc:90:00:0a:0a
hwaddr 02:bc:90:00:0a:0a
nd6 options=1<PERFORMNUD>
media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
status: active
groups: epair
epair5a: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=8<VLAN_MTU>
ether 02:bc:90:00:0b:0a
hwaddr 02:bc:90:00:0b:0a
nd6 options=1<PERFORMNUD>
media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
status: active
groups: epair
epair6a: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=8<VLAN_MTU>
ether 02:bc:90:00:0c:0a
hwaddr 02:bc:90:00:0c:0a
nd6 options=1<PERFORMNUD>
media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
status: active
groups: epair
epair7a: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=8<VLAN_MTU>
ether 02:bc:90:00:0d:0a
hwaddr 02:bc:90:00:0d:0a
nd6 options=1<PERFORMNUD>
media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
status: active
groups: epair
epair8a: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=8<VLAN_MTU>
ether 02:bc:90:00:0e:0a
hwaddr 02:bc:90:00:0e:0a
nd6 options=1<PERFORMNUD>
media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
status: active
groups: epair
epair9a: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=8<VLAN_MTU>
ether 02:bc:90:00:0f:0a
hwaddr 02:bc:90:00:0f:0a
nd6 options=1<PERFORMNUD>
media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
status: active
groups: epair
epair10a: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=8<VLAN_MTU>
ether 02:bc:90:00:10:0a
hwaddr 02:bc:90:00:10:0a
nd6 options=1<PERFORMNUD>
media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
status: active
groups: epair
epair11a: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=8<VLAN_MTU>
ether 02:bc:90:00:11:0a
hwaddr 02:bc:90:00:11:0a
nd6 options=1<PERFORMNUD>
media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
status: active
groups: epair

As you can see, I've still got my old warden jails aside a couple new iocage jails. The later ones are on bridge0 but it's here that I'm missing the em0 which as I assume, require to hook the bridge to my LAN.

Can anyone advise what tunables (rc.conf) I should use to get em0 onto bridge0?

 

[BigDave]

This thread is very similar to your issue
There was a bug report filed on it, which you can keep tabs on.
In the mean time, I recommend you go back to version 11.1
until you see a fix or an update.

 

[Ravefiend]

This thread is very similar to your issue
There was a bug report filed on it, which you can keep tabs on.
In the mean time, I recommend you go back to version 11.1
until you see a fix or an update.

It is somewhat related though in that case, though @bwegrzyn does not have any older jails running on his system from what is appears based on his ifconfig output.

The iocage doc does advise the following:
https://iocage.readthedocs.io/en/latest/networking.html

# set up two bridge interfaces for iocage
cloned_interfaces="bridge0 bridge1"

# plumb interface em0 into bridge0
ifconfig_bridge0="addm em0 up"
ifconfig_em0="up"

I already got both bridges and it does not seem possible to add the em0 to bridge0 as well when trying it from the command line:

Code:

# ifconfig bridge0 addm em0
ifconfig: BRDGADD em0: Device busy

 

[fracai]

I'm in the same situation. Old jails connected to bridge1, new iocage jail connected to bridge0 via vnet0, bridge0 isn't connected to em0.
For now I've solved it by changing the iocage jail interface to connect through bridge1.

Code:

$ iocage get interfaces new-iocage-jail
vnet0:bridge0
$ iocage stop new-iocage-jail
$ iocage set interfaces=vnet0:bridge1 new-iocage-jail
$ iocage start new-iocage-jail

I'm typing that from memory so I might have the vnet/bridge order incorrect, but that's the gist of what I did to solve my issue.

 

[bendl]

I'm in the same situation. Old jails connected to bridge1, new iocage jail connected to bridge0 via vnet0, bridge0 isn't connected to em0.
For now I've solved it by changing the iocage jail interface to connect through bridge1.

Code:

$ iocage get interface new-iocage-jail
vnet0:bridge0
$ iocage stop new-iocage-jail
$ iocage set interface=vnet0:bridge1 new-iocage-jail
$ iocage start new-iocage-jail

I'm typing that from memory so I might have the vnet/bridge order incorrect, but that's the gist of what I did to solve my issue.

For anyone having the same problem and stumbling upon this thread: It is interfaces not interface.
So the correct series of commands which worked for me (for now) is:

Code:

$ iocage get interfaces new-iocage-jail
vnet0:bridge0
$ iocage stop new-iocage-jail
$ iocage set interfaces=vnet0:bridge1 new-iocage-jail
$ iocage start new-iocage-jail

Thanks @fracai

 

[fracai]

And thank you @bendl, I've updated my post with your correction.

 

[corky876]

I am having the same issue. This solution does not survive a reboot though. Any suggestions on something that will survive?

 

[fracai]

I just performed a reboot and noticed the interface change didn't survive. Or rather, it did survive, but the bridge interfaces switched. I switched them back to bridge0, restarted the jails, and they were fine.
It's like iocage is processed first, sees the references to bridge1, creates that but doesn't make the connection to the network device, then processes warden jails, creates bridge0 as the first open name and does connect that bridge to the network device. So, no matter what iocage interfaces are set to, they'll fail and need to be switched to the warden interface.
I think some of the other threads had different solutions adding tunables to create and connect the bridges. Those would probably survive the reboot.