No networking in iocage jail after upgrade to 11.1 U5

[faminebadger]

I upgraded to 11.1-U5 this morning, and after reboot found I had no networking to/from my jails - I couldn't ping them from outside, nor reach outside from within.

After checking a few things, I noticed some differences in the networking setup from before the reboot.

1) The ethernet device on the NAS was no longer in promiscuous mode, which I think should be required for VNET.

2) The ethernet device on the NAS had different options (+RXCSUM/TXCSUM) - could be related to promiscious toggle

3) The ethernet device on the NAS was no longer on the bridge created for the jails.

4) The tunnel device on the jail was missing completely.

These all seem related, and undoubtedly the reason I can't get network access, but why has this suddenly happened as part of what should be a fairly minor OS upgrade? Does anyone know what's going on here?

Thanks,

Famine

 

[joeschmuck]

Sorry I can't answer your question but please create a bug report and post the bug report number here.

 

[faminebadger]

Not sure how to - all I could find is https://redmine.ixsystems.com/projects/freenas which doesn't seem to have any clear report a bug option, just bug lists - and doesn't seem to accept the same user credentials as these forums. I'm fed up of having to create multiple new accounts every time I need to discuss a piece of software :)

 

[joeschmuck]

Yup, it is a new account as it is a separate system. Once you have created an account then you can report the bug. That is the best way for the developers to know you have an issue as they do not spend much time on the forums, they use the bug report system.

 

[danb35]

all I could find is https://redmine.ixsystems.com/projects/freenas which doesn't seem to have any clear report a bug option

The best way to report a bug is using the Support button in the web GUI, as that auto-populates your environment information and allows you to attach a debug file (which I'd recommend). But you'll still need to register for an account there.

 

[faminebadger]

Ok, YAAAC (yet another annoying account created) :)

https://redmine.ixsystems.com/issues/34339 is the bug

 

[MarcusJ]

We are experiencing the same result. If I change the IOCAGE jails from VNET=ON to VNET=OFF and ip4_addr="vnet0|192.168.0.31/24" to ip4_addr="em1|192.168.0.31/24". All is well, except plex doesn't like running like that :)

11.1-u5 iocage get -a output:

Code:

CONFIG_VERSION:11
allow_chflags:0
allow_mount:0
allow_mount_devfs:0
allow_mount_nullfs:0
allow_mount_procfs:0
allow_mount_tmpfs:0
allow_mount_zfs:0
allow_quotas:0
allow_raw_sockets:1
allow_set_hostname:1
allow_socket_af:0
allow_sysvipc:0
available:readonly
basejail:no
boot:on
bpf:no
children_max:0
cloned_release:11.1-RELEASE-p10
comment:none
compression:lz4
compressratio:readonly
coredumpsize:off
count:1
cpuset:off
cputime:off
datasize:off
dedup:off
defaultrouter:192.168.0.1
defaultrouter6:none
depends:none
devfs_ruleset:4
dhcp:off
enforce_statfs:2
exec_clean:1
exec_fib:0
exec_jail_user:root
exec_poststart:/usr/bin/true
exec_poststop:/usr/bin/true
exec_prestart:/usr/bin/true
exec_prestop:/usr/bin/true
exec_start:/bin/sh /etc/rc
exec_stop:/bin/sh /etc/rc.shutdown
exec_system_jail_user:0
exec_system_user:root
exec_timeout:60
host_domainname:none
host_hostname:pms
host_hostuuid:pms
host_time:yes
hostid:d74f84e1-601f-11e8-b060-001f295d6450
hostid_strict_check:off
interfaces:vnet0:bridge0
ip4:new
ip4_addr:vnet0|192.168.0.31/24
ip4_saddrsel:1
ip6:new
ip6_addr:none
ip6_saddrsel:1
jail_zfs:off
jail_zfs_dataset:iocage/jails/pms/data
jail_zfs_mountpoint:none
last_started:2018-05-31 13:14:43
login_flags:-f root
mac_prefix:02ff60
maxproc:off
memorylocked:off
memoryuse:off
mount_devfs:1
mount_fdescfs:1
mount_linprocfs:0
mount_procfs:0
mountpoint:readonly
msgqqueued:off
msgqsize:off
nmsgq:off
notes:none
nsemop:off
nshm:off
nthr:off
openfiles:off
origin:readonly
owner:root
pcpu:off
priority:99
pseudoterminals:off
quota:none
release:11.1-RELEASE-p10
reservation:none
resolver:/etc/resolv.conf
rlimits:off
securelevel:2
shmsize:off
stacksize:off
state:up
stop_timeout:30
swapuse:off
sync_state:none
sync_target:none
sync_tgt_zpool:none
sysvmsg:new
sysvsem:new
sysvshm:new
template:no
type:jail
used:readonly
vmemoryuse:off
vnet:on
vnet0_mac:02ff60577ec0,02ff60577ec1
vnet1_mac:none
vnet2_mac:none
vnet3_mac:none
vnet_interfaces:none
wallclock:off

FYI: I rolled back to 11.1-u4 and all is well and here is the output from iocage get -a pms:

Code:

CONFIG_VERSION:10
allow_chflags:0
allow_mount:0
allow_mount_devfs:0
allow_mount_nullfs:0
allow_mount_procfs:0
allow_mount_tmpfs:0
allow_mount_zfs:0
allow_quotas:0
allow_raw_sockets:1
allow_set_hostname:1
allow_socket_af:0
allow_sysvipc:0
available:readonly
basejail:no
boot:on
bpf:no
children_max:0
cloned_release:11.1-RELEASE-p10
comment:none
compression:lz4
compressratio:readonly
coredumpsize:off
count:1
cpuset:off
cputime:off
datasize:off
dedup:off
defaultrouter:192.168.0.1
defaultrouter6:none
depends:none
devfs_ruleset:4
dhcp:off
enforce_statfs:2
exec_clean:1
exec_fib:0
exec_jail_user:root
exec_poststart:/usr/bin/true
exec_poststop:/usr/bin/true
exec_prestart:/usr/bin/true
exec_prestop:/usr/bin/true
exec_start:/bin/sh /etc/rc
exec_stop:/bin/sh /etc/rc.shutdown
exec_system_jail_user:0
exec_system_user:root
exec_timeout:60
host_domainname:none
host_hostname:pms
host_hostuuid:pms
host_time:yes
hostid:d74f84e1-601f-11e8-b060-001f295d6450
hostid_strict_check:off
interfaces:vnet0:bridge0
ip4:new
ip4_addr:vnet0|192.168.0.31/24
ip4_saddrsel:1
ip6:new
ip6_addr:none
ip6_saddrsel:1
jail_zfs:off
jail_zfs_dataset:iocage/jails/pms/data
jail_zfs_mountpoint:none
last_started:2018-05-31 13:34:00
login_flags:-f root
mac_prefix:02ff60
maxproc:off
memorylocked:off
memoryuse:off
mount_devfs:1
mount_fdescfs:1
mount_linprocfs:0
mount_procfs:0
mountpoint:readonly
msgqqueued:off
msgqsize:off
nmsgq:off
notes:none
nsemop:off
nshm:off
nthr:off
openfiles:off
origin:readonly
owner:root
pcpu:off
priority:99
pseudoterminals:off
quota:none
release:11.1-RELEASE-p10
reservation:none
resolver:/etc/resolv.conf
rlimits:off
securelevel:2
shmsize:off
stacksize:off
state:up
stop_timeout:30
swapuse:off
sync_state:none
sync_target:none
sync_tgt_zpool:none
sysvmsg:new
sysvsem:new
sysvshm:new
template:no
type:jail
used:readonly
vmemoryuse:off
vnet:on
vnet0_mac:02ff60577ec0,02ff60577ec1
vnet1_mac:none
vnet2_mac:none
vnet3_mac:none
vnet_interfaces:none
wallclock:off

 

[faminebadger]

Unfortunately, I required vnet as I run the jail as an openvpn client, such that the software in the jail uses a paid vpn service to access the internet.

 

[Jurgen Segaert]

Check ifconfig in your FreeNAS host and make sure the host interface is added to the vnet bridge.

https://forums.freenas.org/index.ph...e-guidance-on-iocage.63831/page-2#post-458030

This was recently changed in iocage.

 

[faminebadger]

Per my post at the top, my host interface wasn't on the bridge, so this like it could be relevant.

Given there was no mention of jails or iocage in any way in the list of changes for this update, this seems a pretty significant change to happen unannounced, and will surely affect many users.

I'll try the rc.conf change, and post back here.

 

[faminebadger]

Well, on the plus side - manually running "ifconfig bridge0 addm em0 up" makes networking functional again.

On the negative side - adding an rc.conf tunable "ifconfig_bridge0=addm em0 up" (per the thread mentioned by Jurgen above) does nothing - em0 is not added, and there's no networking until I add it manually.

 

[MarcusJ]

Well.. interesting note here. I confirmed that my interface was a member (and added in the tunables):

Code:

bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	   ether 02:85:3b:b5:3c:00
	   nd6 options=1<PERFORMNUD>
	   groups: bridge
	   id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
	   maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
	   root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
	   member: vnet0:7 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
			   ifmaxaddr 0 port 12 priority 128 path cost 2000
	   member: vnet0:6 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
			   ifmaxaddr 0 port 11 priority 128 path cost 2000
	   member: em1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
			   ifmaxaddr 0 port 2 priority 128 path cost 20000

rebooted and all was fine. I then updated to 11.1-u5 again and interestly my interface em1 got removed again:

Code:

bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	   ether 02:85:3b:b5:3c:00
	   nd6 options=1<PERFORMNUD>
	   groups: bridge
	   id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
	   maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
	   root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
	   member: vnet0:7 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
			   ifmaxaddr 0 port 12 priority 128 path cost 2000
	   member: vnet0:6 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
			   ifmaxaddr 0 port 11 priority 128 path cost 2000
	   member: vnet0:5 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
			   ifmaxaddr 0 port 10 priority 128 path cost 2000
	   member: vnet0:4 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
			   ifmaxaddr 0 port 9 priority 128 path cost 2000
	   member: vnet0:3 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
			   ifmaxaddr 0 port 8 priority 128 path cost 2000
	   member: vnet0:2 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
			   ifmaxaddr 0 port 7 priority 128 path cost 2000
	   member: vnet0:1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
			   ifmaxaddr 0 port 6 priority 128 path cost 2000

I ran "ifconfig bridge0 addm em1 up" and it added it back and my iocage jails are functioning normal now. Any reason why my tunable entry wouldn't be working (or do I need to remove and readd):
Variable: ifconfig_bridge0
Value: addm em1 up
Type: rc.conf

 

[Jurgen Segaert]

The tunable is to make this setting upon (re)boot so you wouldn't have to do this manually again.

 

[faminebadger]

Well, don't know about MarcusJ, but as I said above, for me that tunable isn't working - after a reboot, it still needs manually adding despite the tunable being in place.

MarcusJ - is the tunable actually present on the 11.1-U5 boot for you, and not just on the 11.1-U4 before you rebooted? I thought from the wiki that configuration and boot environments were separate, but quite a few of my settings (e.g. cron jobs, tunables) seem tied to the particular boot environment selected.

 

[MarcusJ]

Yup the tunable is there. But I need to enter it manually each reboot. Funny it is listed as a member before the upgrade to U5 then after its gone and the tunable doesn't seem to work.

 

[MarcusJ]

I have three NICs in my box (two fo which are unplugged). Not sure if this has an impact.

 

[faminebadger]

Well, at least it's not just me then :)

FWIW, I have two NICs - the other one is igb0, but I don't use it and it's disabled in the BIOS (but still found by the kernel).

 

[Michael De Cou]

I have the same problem, in that my main interface (lagg0) is not added to bridge0 after updating to U5. U4 works perfectly. I have added frustration in that when I add the tunable, the lagg0 interface has no connectivity upon reboot. The only way i can get it back is to go into the local terminal, configure the network, and say "yes" when asked if i want to reset the configuration. The process fails, but then my lagg0 interface gets it IP and is functional again.

Staying on U4 for the time being I guess.

 

[RootBeer]

Updated to U5. An iocage jail set up in U4 still did work correctly. A new iocage jail set up in U5 had no internet connection. I tried everything I could find on the internet: no success! I compared al the relevant config files for networking between the old and the new iocage jail: no difference.
So I went back to U4: After the reboot of my system the new iocage jail had internet connection right away.

So if you have the same problem, don't waste time on it. Just return to U4.
It's definitely a bug deep down in U5.

 

[dureal99d]

Variable: ifconfig_bridge0
Value: addm em1 up
Type: rc.conf[/QUOTE]

Hum I would try this

Code:

Variable: ifconfig_bridge0 addm em1 up
Value:YES
Type: rc.conf

Works for me though when booting is says not available it somehow kick starts the iocage jail networking feature