Jail on dedicated network

[mpfusion]

Hi,

I've problems setting up a jail's network configuration. The server has two network adapters:

Code:

em0: 10.0.11.15  - main interface
em1: 10.0.91.15  - only for jails

I created a jail using the GUI. Settings:

Code:

Release: 11.2-RELEASE
VNET: unchecked
IPv4 Interface: em1
IPv4 Address: 10.0.91.100
IPv4 Netmask: 24
allow.raw_sockets: checked
interfaces: vnet0:bridge0 (default setting)
resolver: nameserver 10.0.91.10

Pinging the gateway 10.0.91.10 works from within the jail. However, pinging external sites (e.g. 8.8.8.8) fails. Runnig a packet capture on the router reveals that the gateway ping to 10.0.91.10 went through em1, which is correct. However, the ping to 8.8.8.8 came in through em0, which is the wrong interface and doens't work. Here's a screenshot from the packets the router received:

pinging 10.0.91.10:

pinging 8.8.8.8:

Why is the jail communicating using em0 when pinging 8.8.8.8? Is that a misconfiguration on my side?

Another thing I don't understand: Why does it autofill vnet0:bridge0 when em1 is selected and VNET unchecked? bridge0 has one member: em0 If vnet is unchecked I would imagine this whole line to be greyed out. Maybe this is related to the issue, maybe not.

ifconfig (host):

Code:

em0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC>
        ether 00:25:90:01:4e:ff
        hwaddr 00:25:90:01:4e:ff
        inet 10.0.11.15 netmask 0xffffff00 broadcast 10.0.11.255
        nd6 options=9<PERFORMNUD,IFDISABLED>
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
em1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC>
        ether 00:25:90:01:4e:fe
        hwaddr 00:25:90:01:4e:fe
        inet 10.0.91.15 netmask 0xffffff00 broadcast 10.0.91.255
        inet 10.0.91.100 netmask 0xffffff00 broadcast 10.0.91.255
        nd6 options=9<PERFORMNUD,IFDISABLED>
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3
        inet 127.0.0.1 netmask 0xff000000
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        groups: lo
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        ether 02:55:28:63:d5:00
        nd6 options=1<PERFORMNUD>
        groups: bridge
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
        member: em0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 1 priority 128 path cost 20000

Code:

FreeNAS-11.2-RELEASE-U1
Intel(R) Xeon(R) CPU L3426 @ 1.87GHz (8 cores)
12 GiB

 

[millst]

What is the jail gateway set to? Does ifconfig output look right in the jail?

 

[DrKK]

If the VNET is the same thing as we used to call VIMAGE, then by unchecking it, you are sharing the network stack with the host.......

 

[mpfusion]

From the jail:

Code:

netstat -r
Routing tables

Internet:
Destination   Gateway   Flags   Netif Expire
10.0.91.100   link#2    UHS       lo0

 

[mpfusion]

@DrKK I know what Basic Properties → VNET is for. What I'm wondering is, is the meaning of Network Properties → interfaces → vnet0:bridge0 if VNET is unchecked (which it is). If VNET is unchecked, this line makes no sense to me. But leaving it empty and FreeNAS complains. So I left it on the default value.

 

[mpfusion]

@millst Here's the ifconfig from the jail:

Code:

em0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC>
        ether 00:25:90:01:4e:ff
        hwaddr 00:25:90:01:4e:ff
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
em1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC>
        ether 00:25:90:01:4e:fe
        hwaddr 00:25:90:01:4e:fe
        inet 10.0.91.100 netmask 0xffffff00 broadcast 10.0.91.255
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
        groups: lo
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        ether 02:55:28:63:d5:00
        groups: bridge
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
        member: em0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 1 priority 128 path cost 20000

 

[millst]

Looks like you are using the jail's IP for the default route. You should be using the router's IP (10.0.91.1, maybe?).

Yes, VNET=VIMAGE, so OP is sharing the network stack with the NAS. That is confirmed by the ifconfig output showing all the networking interfaces: em0, em1, lo0, bridge0.

 

[mpfusion]

@millst Where to configure the default route for a jail? It's deactivated in the jails Basic Properties (IPv4 Default Router).

 

[millst]

Oh, that's right, you can't set the gateway unless you use VNET/VIMAGE. Without that, it uses the default gateway of the system that is configured in the regular "Network" part of the UI. Your default gateway is probably on the same subnet as em0, causing your traffic for 8.8.8.8 to go out through that NIC. Changing the default gateway or adjusting static routes might help.

Alternatively, you'll need to enable VNET/VIMAGE and get your bridges setup correctly.

 

[mpfusion]

@millst True, the default gateway is on em0, as I said, that's the main network for the FreeNAS, em1 is only supposed to be for jails. I cannot change the default gateway for the freenas box. Traffic takes a wrong path then.

I don't quite know which static route I would need to set. The freenas box should use em0's network including gateway, the jails em1's network including gateway. But AFAIK I can only specify one default gateway. Or can/should I configure the route in the jail?

Being able to select an interface and no gateway makes the interface selection kind of useless, since networking doesn't work if the wrong gateway is used. I don't quite understand how that's supposed to work.

I tried using VNET, but only got a loopback interface in the jail. I searched the manual for bridges, but it mentions nowhere where/how to set up bridges. There's no bridge setup in the networking section. I have one bridge “bridge0”, but its member is em0, so that won't work either.

 

[millst]

There is only one default route for the system, thus it being termed default. The routes are always destination-based, so I don't think there is any easy way to get the other NIC traffic to go somewhere else. You'll probably need to muddle through the VNET/VIMAGE method.

You need to get another bridge (bridge1) that connects your jail and em1. Check out this thread for some ideas:
https://forums.freenas.org/index.php?threads/correctly-using-iocage-with-vnet-and-bridge.60181/

 

[mpfusion]

@millst Thanks for your response. I tried but didn't quite suceed. The first thing I did was to create a bridge that has one member: em1

System → Tunables → Add

Variable: cloned_interface
Value: bridge0
Type: rc

Variable: ifconfig_bridge0
Value: addm em1 up
Type: rc

ifconfig on the host after reboot before the jail was started:

Code:

em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC>
        ether 00:25:90:01:4e:ff
        hwaddr 00:25:90:01:4e:ff
        inet 10.0.11.15 netmask 0xffffff00 broadcast 10.0.11.255
        nd6 options=9<PERFORMNUD,IFDISABLED>
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
em1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC>
        ether 00:25:90:01:4e:fe
        hwaddr 00:25:90:01:4e:fe
        inet 10.0.91.15 netmask 0xffffff00 broadcast 10.0.91.255
        nd6 options=9<PERFORMNUD,IFDISABLED>
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3
        inet 127.0.0.1 netmask 0xff000000
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        groups: lo
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        ether 02:55:28:63:d5:00
        nd6 options=9<PERFORMNUD,IFDISABLED>
        groups: bridge
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
        member: em1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 2 priority 128 path cost 55

This looks good to me. Bridge bridge0 with one member: em1

Then I configured the jail:

Basic Properties → VNET → checked
Basic Properties → IPv4 Interface → Vnet0
Basic Properties → IPv4 Address → 10.0.91.100
Basic Properties → IPv4 Default Router → 10.0.91.10
Jail Properties → interfaces → vnet0:bridge0

After starting the jail, ifconfig on the host:

Code:

em0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=2098<VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC>
        ether 00:25:90:01:4e:ff
        hwaddr 00:25:90:01:4e:ff
        inet 10.0.11.15 netmask 0xffffff00 broadcast 10.0.11.255
        nd6 options=9<PERFORMNUD,IFDISABLED>
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
em1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=2098<VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC>
        ether 00:25:90:01:4e:fe
        hwaddr 00:25:90:01:4e:fe
        inet 10.0.91.15 netmask 0xffffff00 broadcast 10.0.91.255
        nd6 options=9<PERFORMNUD,IFDISABLED>
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3
        inet 127.0.0.1 netmask 0xff000000
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        groups: lo
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        ether 02:55:28:63:d5:00
        nd6 options=9<PERFORMNUD,IFDISABLED>
        groups: bridge
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
        member: vnet0:1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 5 priority 128 path cost 2000
        member: em0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 1 priority 128 path cost 20000
        member: em1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 2 priority 128 path cost 55
vnet0:1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: associated with jail: test
        options=8<VLAN_MTU>
        ether 02:ff:60:ae:1b:75
        hwaddr 02:c2:10:00:05:0a
        nd6 options=1<PERFORMNUD>
        media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
        status: active
        groups: epair

This doesn't look good. bridge0 not only has em1 and vnet0 but also em0. Why? How was is added? Did I misconfigure the jail or the bridge somehow?

ifconfig from this jail:

Code:

lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1
        inet 127.0.0.1 netmask 0xff000000
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        groups: lo
epair0b: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=8<VLAN_MTU>
        ether 02:ff:60:ae:1b:76
        hwaddr 02:c2:10:00:06:0b
        inet 10.0.91.100 netmask 0xffffff00 broadcast 10.0.91.255
        nd6 options=1<PERFORMNUD>
        media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
        status: active
        groups: epair

Any ideas what's wrong here?

 

[millst]

I believe iocage is automatically added that interface to bridge0. Does it work? I wouldn't expect that having the extra NIC in there necessarily hurts anything. All the appropriate things are bridged and on the same subnet with your proper default route in the jail. If not, this command should remove it:

ifconfig bridge0 deletem em0

 

[millst]

The FreeNAS manual indicates there is a vnet_default_interface in the jail configuration:

Default network interface used for the VNET bridge interface in the jail. Only takes effect when VNET is set and bridge interfaces are not active.​

Setting that to em1 might correct the situation during system startup.

 

[mpfusion]

I believe iocage is automatically added that interface to bridge0. Does it work?

No, it doesn't. It's still trying to send data over em0 (according to wireshark on the router).

I wouldn't expect that having the extra NIC in there necessarily hurts anything.

Maybe you're right and the problem is something else.

All the appropriate things are bridged and on the same subnet with your proper default route in the jail. If not, this command should remove it:

ifconfig bridge0 deletem em0

I tried that and em0 got removed. After starting the jail it reappeared. I removed it again. But same outcome regarding the network connectivity. So that probably wasn't it.

 

[mpfusion]

The FreeNAS manual indicates there is a vnet_default_interface in the jail configuration:

Default network interface used for the VNET bridge interface in the jail. Only takes effect when VNET is set and bridge interfaces are not active.​

Setting that to em1 might correct the situation during system startup.

I've seen that, too. However, I have to note that the descriptions in the manual don't agree with the GUI naming, which is unfortunate and makes the manual (or GUI) much less useful. Where to find this setting? I tried setting

Network Properties → interfaces → vnet0:em1

but got

Code:

RuntimeError

Stopped test due to VNET failure

when trying to start the jail.

 

[mpfusion]

I've seen that, too. However, I have to note that the descriptions in the manual don't agree with the GUI naming, which is unfortunate and makes the manual (or GUI) much less useful. Where to find this setting?

Found it.

 

[mpfusion]

I tried setting

Network Properties → vnet default interface → em1

as well as

Network Properties → vnet default interface → bridge0

but with the same result. Pinging 10.0.91.10 works, 8.8.8.8 doesn't.

 

[millst]

I would recommend keeping the system and jail up while trying to get things working (as much as possible, obviously some changes will necessitate start/stop). Then, you are just determining the proper config rather than fighting auto-configuration at the same time. Once it is working, move on to getting it to boot right.

You should be able to manually use ifconfig to get your bridge configured. Try it again with just the jail interface and em1. Ping may not be the best test as it requires raw sockets (make sure it is enabled in your jail config). Doublecheck all your IPs from the host and inside the jail.

 

[mpfusion]

I would recommend keeping the system and jail up while trying to get things working (as much as possible, obviously some changes will necessitate start/stop).

The jail has to be stopped, otherwise the following error appears when trying to change the settings:

“Jails cannot be changed while running. Stop the jail to make changes.”

Then, you are just determining the proper config…

Easier said than done. If only I knew what the “proper config” was!

You should be able to manually use ifconfig to get your bridge configured. Try it again with just the jail interface and em1.

I'll try to get things working, but frankly I'm just randomly trying every combination because I don't really know what the “working” configuration is supposed to look like.

Ping may not be the best test as it requires raw sockets (make sure it is enabled in your jail config). Doublecheck all your IPs from the host and inside the jail.

Raw sockets are enabled (see OP). I posted all IPs, so you can have a look. I didn't see any obvious issues, but that doesn't mean it's all correct. What tool would you suggest rather than ping?