iocage jail with shared ip reset host connection when stopped

[AndersG]

Hi,

I've set up an iocage jail which shares IP with the FreeNAS host. This seem to work fine although when I stop the jail the network stops working on the host as well. Do I need to change some jail property when using shared IP?

FreeNAS version: 11.1-U5
Host ip: 172.18.0.89/24

Commands to configure the jail:

iocage create -r 11.1-RELEASE --name mysql_ng
iocage set ip4_addr="em0|172.18.0.89/24" mysql_ng
iocage set allow_raw_sockets="1" mysql_ng
iocage set jail_zfs=on mysql_ng
iocage fstab -a mysql_ng "/mnt/pool1/backup /mnt/backup nullfs rw 0 0"


ifconfig before jail stop:

Code:

em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
   options=209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC>
   ether 00:25:90:23:fa:a8
   hwaddr 00:25:90:23:fa:a8
   inet 172.18.0.89 netmask 0xffffff00 broadcast 172.18.0.255
   nd6 options=9<PERFORMNUD,IFDISABLED>
   media: Ethernet autoselect (1000baseT <full-duplex>)
   status: active

ifconfig after jail stop:

Code:

em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
   options=209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC>
   ether 00:25:90:23:fa:a8
   hwaddr 00:25:90:23:fa:a8
   nd6 options=9<PERFORMNUD,IFDISABLED>
   media: Ethernet autoselect (1000baseT <full-duplex>)
   status: active

Jail properties

Code:

CONFIG_VERSION:11
allow_chflags:0
allow_mount:0
allow_mount_devfs:0
allow_mount_nullfs:0
allow_mount_procfs:0
allow_mount_tmpfs:0
allow_mount_zfs:0
allow_quotas:0
allow_raw_sockets:1
allow_set_hostname:1
allow_socket_af:0
allow_sysvipc:0
available:readonly
basejail:no
boot:off
bpf:no
children_max:0
cloned_release:11.1-RELEASE
comment:none
compression:lz4
compressratio:readonly
coredumpsize:off
count:1
cpuset:off
cputime:off
datasize:off
dedup:off
defaultrouter:none
defaultrouter6:none
depends:none
devfs_ruleset:4
dhcp:off
enforce_statfs:2
exec_clean:1
exec_fib:0
exec_jail_user:root
exec_poststart:/usr/bin/true
exec_poststop:/usr/bin/true
exec_prestart:/usr/bin/true
exec_prestop:/usr/bin/true
exec_start:/bin/sh /etc/rc
exec_stop:/bin/sh /etc/rc.shutdown
exec_system_jail_user:0
exec_system_user:root
exec_timeout:60
host_domainname:none
host_hostname:mysql_ng
host_hostuuid:mysql_ng
host_time:yes
hostid:24c2d80f-5f36-11e8-a91e-00259023faa8
hostid_strict_check:off
interfaces:vnet0:bridge0
ip4:new
ip4_addr:em0|172.18.0.89/24
ip4_saddrsel:1
ip6:new
ip6_addr:none
ip6_saddrsel:1
jail_zfs:on
jail_zfs_dataset:iocage/jails/mysql_ng/data
jail_zfs_mountpoint:none
last_started:2018-06-15 08:26:07
login_flags:-f root
mac_prefix:02ff60
maxproc:off
memorylocked:off
memoryuse:off
mount_devfs:1
mount_fdescfs:1
mount_linprocfs:0
mount_procfs:0
mountpoint:readonly
msgqqueued:off
msgqsize:off
nmsgq:off
notes:none
nsemop:off
nshm:off
nthr:off
openfiles:off
origin:readonly
owner:root
pcpu:off
priority:99
pseudoterminals:off
quota:none
release:11.1-RELEASE-p10
reservation:none
resolver:/etc/resolv.conf
rlimits:off
securelevel:2
shmsize:off
stacksize:off
state:up
stop_timeout:30
swapuse:off
sync_state:none
sync_target:none
sync_tgt_zpool:none
sysvmsg:new
sysvsem:new
sysvshm:new
template:no
type:jail
used:readonly
vmemoryuse:off
vnet:off
vnet0_mac:none
vnet1_mac:none
vnet2_mac:none
vnet3_mac:none
vnet_interfaces:none
wallclock:off

 

[Jailer]

I've set up an iocage jail which shares IP with the FreeNAS host.

Why would you do that? Having your network break is expected behavior if your sharing your web GUI IP with a jail. Assign your jail it's own IP address outside your router's DHCP range.

 

[AndersG]

Well, I'm quite new to freebsd so I'm not familiar with how things are usually configured. Especially not with jails which I started testing out about a week ago.

Reading this documentation I was under the impression that IP could be shared with the host, but apparently not. From what you say I guess I should assign another IP to the same interface and then use that one for my jail. In that case, why is it called shared IP?
http://iocage.readthedocs.io/en/latest/networking.html

This setup was not what I initially wanted. If I describe my use case, maybe there are better suggestions for how to set thing up. What I would like to run mysql in a jail and expose it to services on another vlan. I initially did a setup like this with a warden jail, and a dedicated nic for that jail. It worked well with the exception for freebsd-11.0 missing fdatasync, which broke some functions related to backup management.

What I would like to have is an iocage jail with mysql exposed on that other interface, but I'm under the impression that it's not possible. https://redmine.ixsystems.com/issues/31008
I would not like some messy home brew internal routing, just as simple configuration as possible. Another requirement is to use as few IP's as possible on the service vlan, where I want mysql exposed. Preferably only one since the address space left is very limited.

The setup I have right now is one warden jail for the mysql service and an iocage jail for backup management. Everything is working except for the iocage jail messing with the interface configuration when stopped.

Any suggestions for how to set this up are welcome.

 

[Jailer]

Turn on vnet and assign an IP address to your iocage jail as I suggested.

 

[AndersG]

Ok! Can you point me to an example for how to configure vnet on a FreeNAS host. The documentation for iocage seems add network configuration for bridges etc to hosts config files. Can I expect such changes to survive system updates?

Also can I configure just an address to use for my iocage jail without having one configured for the interface, or configure a virtual subnet and port forwarding from the interface on our service vlan?

 

[sretalla]

You might want to look at the last couple of posts in this thread

By default iocage jails will join vnet0 when you set them to use vnet and vnet0 will join bridge0 (so you don't need to do that). "addm igb0 up" on "ifconfig_bridge0" works well for me. You can test that before creating the tunable by just using:

ifconfig bridge0 addm igb0 up

use ifconfig alone to see the outcome (look for members under bridge0 and you should see your nic and vnet0 there when a jail has been started)

in a shell