Rebooted, iocage jails have no network access, and old tricks aren't fixing this anymore

Status
Not open for further replies.

Stranded Camel

Explorer
Joined
May 25, 2017
Messages
79
Hi all!

After rebooting my system (all details in sig), I lost all network connectivity in my three iocage jails (FreeNAS itself, however, does have network access, as does my bhyve VM). This happens every single time I reboot, and I used to be able to fix it by going into the new UI, opening the jail settings, heading to the network tab, and changing the bridge number from 0 to 1 (or vice versa).

But this time, the hack isn't working. I haven't updated FreeNAS, so I have no idea what the problem could be. This is what I see when I try to ping out from inside the jail:

Code:
# ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8): 56 data bytes
ping: sendto: No route to host
ping: sendto: No route to host
ping: sendto: No route to host
# 


Here's my FreeNAS ifconfig output:

Code:
$ ifconfig -a
em0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
		options=2098<VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC>
		ether xx:xx:xx:xx:xx:xx
		hwaddr xx:xx:xx:xx:xx:xx
		inet 192.168.0.250 netmask 0xffffff00 broadcast 192.168.0.255
		nd6 options=9<PERFORMNUD,IFDISABLED>
		media: Ethernet autoselect (1000baseT <full-duplex>)
		status: active
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
		options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
		inet6 ::1 prefixlen 128
		inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2
		inet 127.0.0.1 netmask 0xff000000
		nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
		groups: lo
bridge11: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
		ether xx:xx:xx:xx:xx:xx
		nd6 options=9<PERFORMNUD,IFDISABLED>
		groups: bridge
		id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
		maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
		root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
		ether xx:xx:xx:xx:xx:xx
		nd6 options=9<PERFORMNUD,IFDISABLED>
		groups: bridge
		id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
		maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
		root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
bridge1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
		ether xx:xx:xx:xx:xx:xx
		nd6 options=9<PERFORMNUD,IFDISABLED>
		groups: bridge
		id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
		maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
		root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
		member: vnet0:5 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
				ifmaxaddr 0 port 6 priority 128 path cost 2000
tap0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
		options=80000<LINKSTATE>
		ether xx:xx:xx:xx:xx:xx
		hwaddr xx:xx:xx:xx:xx:xx
		nd6 options=1<PERFORMNUD>
		media: Ethernet autoselect
		status: active
		groups: tap
		Opened by PID 16450
bridge2: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
		ether xx:xx:xx:xx:xx:xx
		nd6 options=9<PERFORMNUD,IFDISABLED>
		groups: bridge
		id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
		maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
		root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
		member: em0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
				ifmaxaddr 0 port 1 priority 128 path cost 20000
		member: tap0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
				ifmaxaddr 0 port 7 priority 128 path cost 2000000
vnet0:5: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
		description: associated with jail: plex
		options=8<VLAN_MTU>
		ether xx:xx:xx:xx:xx:xx
		hwaddr xx:xx:xx:xx:xx:xx
		nd6 options=1<PERFORMNUD>
		media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
		status: active
		groups: epair
$


And here's the output of ifconfig -a inside my jail:

Code:
# ifconfig -a
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
		options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
		inet6 ::1 prefixlen 128
		inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1
		inet 127.0.0.1 netmask 0xff000000
		nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
		groups: lo
epair0b: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
		options=8<VLAN_MTU>
		ether xx:xx:xx:xx:xx:xx
		hwaddr xx:xx:xx:xx:xx:xx
		inet 192.168.0.251 netmask 0xffffff00 broadcast 192.168.0.255
		nd6 options=1<PERFORMNUD>
		media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
		status: active
		groups: epair
root@plex:~ #


Can anyone help me get my jails connected? I'm at my wit's end here...
 
Last edited:

Stranded Camel

Explorer
Joined
May 25, 2017
Messages
79
By the way, I've already tried the suggestion here, but it doesn't change anything.

I've also tried the suggestion here, involving doing iocage exec <jail> 'sysrc ifconfig_epair0_name="epair0b"', but no joy.

It should be noted, too, that the IP address the jail gets, inet 192.168.0.251, is *correct*. I don't understand how this can be the case if I can't even ping out.
 
Last edited:

Stranded Camel

Explorer
Joined
May 25, 2017
Messages
79
Is there really no one with any suggestions? I know FreeNAS is riddled with bugs, but this does seem to be one of the more common and persistent ones.
 

Stranded Camel

Explorer
Joined
May 25, 2017
Messages
79
Continuing to bang my head against the wall. I tried the solution here, which boills down to this:

Code:
iocage set vnet=off plex
iocage set "ip4_addr=em0|192.168.0.251/24" plex


But while it does allow the jail to connect to the network and internet, it's not really a solution -- Plex doesn't work properly with it, as others have reported. Network access spontaneously disconnects and reconnects for no reason every two to three minutes (which makes it a lot more than a Plex problem, but this causes Plex to jump to the next item in a series, for example).

So I'm still looking for a way to unbreak network access in iocage jails.
 
Last edited:

Stranded Camel

Explorer
Joined
May 25, 2017
Messages
79
Your active interface on the host must be in the bridge with the vnet interface, eg make em0 and vnet0.x member of bridge0 in a postinit command. How to do that is explained here https://www.freebsd.org/doc/handbook/network-bridging.html

Thanks for weighing in! If I understand the manual you linked to, I should run this in FreeNAS:

Code:
ifconfig bridge0 addm em0
ifconfig bridge0 addm vnet0:5


And then change the jail's configuration like this:
Code:
iocage set vnet=on plex
iocage set ip4_addr="vnet0:5|192.168.0.251" plex


Unfortunately, this doesn't work -- I still don't have network access from inside the jail.

What am I doing wrong? One thing that is definitely off is that the jail doesn't seem to respect the `ip4_addr` I set for it -- when I stop it, I get the following:

Code:
$ iocage stop plex
* Stopping plex
 + Running prestop OK
 + Stopping services OK
 + Tearing down VNET FAILED
 ifconfig: interface vnet0:22 does not exist
 + Removing jail process OK
 + Running poststop OK

$ iocage get ip4_addr plex
vnet0:12|192.168.0.251


So even though vnet is set to 0:12, the jail is looking for vnet:22. And this changes, with the number after the colon increasing every time I start the jail. I can't make heads or tails of that.
 
Last edited:

Stranded Camel

Explorer
Joined
May 25, 2017
Messages
79
Oh well... more broken FreeNAS crap. No surprise there. It looks like I'm going to have to move from iocage jails to one or more Linux VMs to get this working properly.
 

KrisBee

Wizard
Joined
Mar 20, 2017
Messages
1,288
@Stranded Camel Don't let FreeNAS give you the hump! Joking apart, jails can work in FreeNAS 11.1-U6. I've just started making us of jails again after a long break and it took a bit of time to check what workarounds/fixes/bugs are still relevant in FN11.1-U6 compared to earlier in the year.

The network config your first post shows for the FreeNAS host seems overly complex, with several bridges which could well be redundant. Here's my config:

Code:
root@freenasB:~ # ifconfig
em0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
		options=98<VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM>
		ether 68:05:ca:16:f3:fe
		hwaddr 68:05:ca:16:f3:fe
		inet 192.168.0.99 netmask 0xffffff00 broadcast 192.168.0.255 
		nd6 options=9<PERFORMNUD,IFDISABLED>
		media: Ethernet autoselect (1000baseT <full-duplex>)
		status: active
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
		options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
		inet6 ::1 prefixlen 128 
		inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3 
		inet 127.0.0.1 netmask 0xff000000 
		nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
		groups: lo 
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
		ether 02:bf:32:fa:02:00
		nd6 options=9<PERFORMNUD,IFDISABLED>
		groups: bridge 
		id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
		maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
		root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
		member: vnet0:5 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
				ifmaxaddr 0 port 6 priority 128 path cost 2000
		member: vnet0:1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
				ifmaxaddr 0 port 5 priority 128 path cost 2000
		member: em0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
				ifmaxaddr 0 port 1 priority 128 path cost 55
vnet0:1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
		description: associated with jail: dns
		options=8<VLAN_MTU>
		ether 02:ff:60:fd:33:60
		hwaddr 02:51:90:00:05:0a
		nd6 options=1<PERFORMNUD>
		media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
		status: active
		groups: epair 
vnet0:5: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
		description: associated with jail: kdc
		options=8<VLAN_MTU>
		ether 02:ff:60:ba:94:ae
		hwaddr 02:51:90:00:06:0a
		nd6 options=1<PERFORMNUD>
		media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
		status: active
		groups: epair 


There are two jails running and no VMs. I have one external NIC em0, which is a member of the bridge0. To ensure the bridge0 exists with the member em0 post boot, I added two tunables ( that might not strictly be needed now). See here: https://forums.freenas.org/index.php?threads/iocage-vnet0-and-bridge0.59964/#post-425631

As you are creating jails with VNET , the ip4_addr property should be set with statements like iocage set ip4_addr="vnet0|192.168.0.251" plex , don't use vnet0:x. Iocage should be adding/removing the appropriate "vnet0:x" as a member of the bridge in use, as jails are started and stopped.
 
Last edited:
Status
Not open for further replies.
Top