VNET in jail works only when NAT option is specified

[expr]

I created the jail with default parameters and only checked DHCP Autoconfigure IPv4 checkbox.
Despite the fact the jail successfully leases IP from DHCP server (192.168.1.41) it doesn't have access to the Internet.
I tried to ping different IP addresses from the Jail:
1. I can ping IP of FreeNAS server (192.168.1.8).
2. I cannot ping other devices in my local network (for example 192.168.1.7) and even my gateway/DHCP server.

When I check NAT checkbox, the jail has access to local network and the Internet.

I have 'Ethernet Connection X722 for 1GbE' 8086:37d1 internal NIC.
FreeNAS loads this driver: <Intel(R) Ethernet Connection 700 Series PF Driver, Version - 1.11.9-k>

I've never used FreeNAS before. How can I diagnose the issue and maybe I should report a bug?

 

[sretalla]

I don't think you understand what VNET means... you can't have both VNET and NAT enabled.

What you should have is VNET enables as it's how jails work best (with their own IP address).

You need to make sure the bridge is properly configured for it to work, so look at the output from ifconfig both on the FreeNAS host and in the jail.

 

[expr]

It's interesting. When I check NAT checkbox and trying to uncheck VNET checkbox I see "VNET is required." error in UI. So I a bit confused when you say "you can't have both VNET and NAT enabled." Could you please explain what I get wrong?
As for ifconfig in jail - it looks ok. As I said before the jail leases IP from my DHCP server and I can ping FreeNAS IP.
As for FreeNAS, I don't know what to check:

Code:

ixl1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM>
        inet 192.168.1.8 netmask 0xffffff00 broadcast 192.168.1.255
        nd6 options=9<PERFORMNUD,IFDISABLED>
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3
        inet 127.0.0.1 netmask 0xff000000
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        groups: lo
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        ether 02:f4:3d:0f:99:00
        nd6 options=1<PERFORMNUD>
        groups: bridge
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
        member: vnet0.4 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 6 priority 128 path cost 2000
        member: ixl1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 2 priority 128 path cost 200000
vnet0.4: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: associated with jail: test2 as nic: epair0b
        options=8<VLAN_MTU>
        ether 3e:ec:ef:4b:48:3c
        hwaddr 02:94:50:00:06:0a
        nd6 options=1<PERFORMNUD>
        media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
        status: active
        groups: epair

 

[sretalla]

OK, so that bridge looks fine.

What do you get from this?

iocage get defaultrouter test2

 

[expr]

Code:

root@freenas[~]# iocage get defaultrouter test2
auto
root@freenas[~]#

 

[expr]

P.S. I tried to change it to ixl1, but it changed nothing.

 

[sretalla]

You want to set that to whatever your network's router is... my guess:
iocage set defaultrouter="192.168.1.1" test2

 

[expr]

Oh, I'm sorry I didn't say that routing in jail looks ok even with "auto":

Code:

root@test2:~ # netstat -r
Routing tables

Internet:
Destination        Gateway            Flags     Netif Expire
default            192.168.1.125      UGS     epair0b
localhost          link#1             UH          lo0
192.168.1.0/24     link#2             U       epair0b
192.168.1.244      link#2             UHS         lo0

Moreover without default route I wouldn't have access to any network but 192.168.1.0/24. But as I said before I can ping only FreeNAS IP :(

Also I forget to mention one thing: in jail it takes about 20 seconds to print those four routes. I mean I run netstat -r in jail and only after 20 seconds I see those routes. Is it ok? Or maybe this and the main issue have common roots?

 

[sretalla]

Well, netstat -r takes less than a second for me... I note that my gateway address is resolved in DNS to its name, so perhaps that's giving us a clue... do you have DNS configured correctly on the host and in the jail?

 

[expr]

I forget that netstat -r resolves IP. As jail doesn't have access to the whole network it doesn't have access to the name server. I have name server configured properly that's why jail tries to resolve IP and waits the response.
If I get it right the problem is in VNET system or maybe driver?

 

[sretalla]

Could it be that your FreeNAS server IP is 192.168.1.125? It will not act as a gateway in the bridged network setup... you should use the router (apologies if I'm wasting time with this one and you somehow have your router at .125)

 

[expr]

I'm sorry that I haven't said that 192.168.1.125 is my gateway and name server. But I thought that it doesn't matter because the gateway is used only when packet is sent to unknown network what is not my case.

 

[sretalla]

Perhaps it's a switching thing... does your switch allow impersonation/promiscuous mode? The FreeNAS NIC needs to be able to impersonate additional MACs for each jail in addition to the primary MAC of the NIC.

This can be an issue on some managed switches, but is more often an issue in Virtual environments like VMware with vSwitches.

 

[expr]

I'm sorry that I've disappeared for two days. You gave me a brilliant idea and I decided to check switch. It took some time to setup another router to monitor all ports in switch. Finally I found out that it mirrored all traffic from one port to another. It's strange that this thing affected FreeNAS, because it haven't created loops and something like this and the host should just ignore those packages. But when I removed mirroring and reboot FreeNAS the jail had access to the local network and the Internet.

 

[sretalla]

It's possible that if you're mirroring the port FreeNAS is connected to to another port, then the port originating the mirror probably can't be in promiscuous mode, whereas the target port getting the mirrored traffic must be in promiscuous mode.

FreeNAS needs promiscuous mode to do both the host and jail networks on the same NIC.

 

[expr]

In fact I don't understand what you mean when say that switch should allow promiscuous mode. Do you mean that switch should support more than one mac address on each port? A believe every switch supports it. If we are saying about switches (hardware device working on layer 2 osi), they have forwarding table which stores what mac is seen in which port. Maybe do you think that the mirroring somehow damaged this table? Maybe, but I have no idea why that damage was critical only for FreeNAS.

 

[sretalla]

Do you mean that switch should support more than one mac address on each port?

Yes, that's what that means... actually it's the NIC on the attached device that's in promiscuous mode, but the switch needs to support that when managing the ARP table.

Not every switch supports port mirroring and I'm certainly no expert on that, I'm just trying to explain what you're seeing with a theory. I'm not sure if it's a bug/problem or by design.