VLANs and Jails

[ProbieK]

Hello Everyone,

I've searched, YouTube'd, and Googled... And I can't seem to figure this out.
I am trying to set up a plugin/jail to use a VLAN.

Here are the details:
I have a Freenas Mini XL running TrueNAS-12.0-U2.1.
There are two NICs; igb0 and igb1... igb0 is dedicated for everything non-jail/plugin (webui, shares, etc)
igb1 has no IP, but is physically connected to a unifi switch (more on that later).
I created a vlan (vlan180) and has igb1 as a member. (no IP assigned)
Then I created a bridge (bridge180) and it has vlan180 as a member. (no IP assigned)

For my jail, after initial installation, I stop it > edit > Network Properties > interfaces > and set "vnet0:bridge180"

For the switch, I have a Unifi switch, with a network created that is configured for "vlan only" with a vlan ID of 180. The switch port is assigned the "All" profile, so that it gets access to the Native and 180 VLAN.

My router is a pfSense firewall, VLAN interface enabled and assigned to the appropriate downlink to the switch.
pfSense and the Unify switch are LAGG'd together across 4 ports in LACP.

---
Now, when i start the jail, I receive the following error:
Error: [EFAULT] + Acquiring DHCP address: FAILED, address received: 0.0.0.0/8 Stopped [plugin name] due to DHCP failure

In the pfSense system logs, I can see the following:
DHCPDISCOVER from d0:50:99:03:aa:47 (transmission) via lagg0.180
DHCPOFFER on 192.168.8.100 to d0:50:99:03:aa:47 (transmission) via lagg0.180

---
So, it appears as though the DHCP request is making it all the way to the router and it is responding back.
If I manually set an IP on the jail to something in the vlan180 network, the jail will show as up, but it has no connectivity.
When I try to ping either the gateway for the network, or 8.8.8.8, I get the following error:
ping: sendto: Host is down

Any idea what I'm missing in order to get this to work?

Thanks in advance!

 

[ProbieK]

Also, for what it's worth.... I can add other VM's (from a different VM server) to this VLAN and get fully working connectivity.

 

[ProbieK]

Here is the output I'm getting from ifconfig:

Code:

root@freenas[~]# ifconfig
igb0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=a520b9<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6>
        ether d0:50:99:d4:47:1c
        inet 192.168.0.7 netmask 0xffffff00 broadcast 192.168.0.255
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=9<PERFORMNUD,IFDISABLED>
igb1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: igb1
        options=a520b9<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6>
        ether d0:50:99:d4:47:1d
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=9<PERFORMNUD,IFDISABLED>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3
        inet 127.0.0.1 netmask 0xff000000
        groups: lo
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
pflog0: flags=0<> metric 0 mtu 33160
        groups: pflog
vlan140: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: ***** VLAN140
        ether d0:50:99:d4:47:1d
        groups: vlan
        vlan: 140 vlanpcp: 0 parent interface: igb1
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=9<PERFORMNUD,IFDISABLED>
bridge140: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        ether 02:a5:1b:da:3d:8c
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto stp-rstp maxaddr 2000 timeout 1200
        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
        member: vnet2 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 17 priority 128 path cost 2000000
        member: vlan140 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 5 priority 128 path cost 20000
        groups: bridge
        nd6 options=9<PERFORMNUD,IFDISABLED>
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        ether 02:a5:1b:da:3d:00
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto stp-rstp maxaddr 2000 timeout 1200
        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
        member: vnet0.36 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 19 priority 128 path cost 2000
        member: vnet0.5 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 14 priority 128 path cost 2000
        member: vnet0.4 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 13 priority 128 path cost 2000
        member: igb1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 2 priority 128 path cost 20000
        member: vnet0.3 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 12 priority 128 path cost 2000
        member: vnet0.2 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 11 priority 128 path cost 2000
        member: vnet0.1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 10 priority 128 path cost 2000
        member: vnet1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 9 priority 128 path cost 2000000
        member: vnet0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 8 priority 128 path cost 2000000
        member: igb0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 1 priority 128 path cost 20000
        groups: bridge
        nd6 options=1<PERFORMNUD>
vnet0: flags=8942<BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=80000<LINKSTATE>
        ether fe:a0:98:6f:69:8e
        hwaddr 58:9c:fc:10:ff:fd
        groups: tap
        media: Ethernet autoselect
        status: active
        nd6 options=1<PERFORMNUD>
        Opened by PID 2375
vnet1: flags=8942<BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=80000<LINKSTATE>
        ether fe:a0:98:5d:83:76
        hwaddr 58:9c:fc:10:a6:2f
        groups: tap
        media: Ethernet autoselect
        status: active
        nd6 options=1<PERFORMNUD>
        Opened by PID 2441
vnet0.1: flags=8942<BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: associated with jail: asigra as nic: epair0b
        options=8<VLAN_MTU>
        ether d2:50:99:f3:d0:55
        hwaddr 02:c4:56:97:56:0a
        groups: epair
        media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
        status: active
        nd6 options=1<PERFORMNUD>
vnet0.2: flags=8942<BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: associated with jail: couchpotato as nic: epair0b
        options=8<VLAN_MTU>
        ether d0:50:99:1c:24:41
        hwaddr 02:cb:4d:78:8f:0a
        groups: epair
        media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
        status: active
        nd6 options=1<PERFORMNUD>
vnet0.3: flags=8942<BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: associated with jail: duplicati as nic: epair0b
        options=8<VLAN_MTU>
        ether d0:50:99:e2:92:b5
        hwaddr 02:5c:c7:43:da:0a
        groups: epair
        media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
        status: active
        nd6 options=1<PERFORMNUD>
vnet0.4: flags=8942<BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: associated with jail: nextcloud as nic: epair0b
        options=8<VLAN_MTU>
        ether d0:50:99:ba:b5:81
        hwaddr 02:f8:8f:0f:29:0a
        groups: epair
        media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
        status: active
        nd6 options=1<PERFORMNUD>
vnet0.5: flags=8942<BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: associated with jail: plex-plexpass as nic: epair0b
        options=8<VLAN_MTU>
        ether d0:50:99:30:63:77
        hwaddr 02:e9:e8:f8:71:0a
        groups: epair
        media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
        status: active
        nd6 options=1<PERFORMNUD>
vlan180: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: ***** VLAN 180
        ether d0:50:99:d4:47:1d
        groups: vlan
        vlan: 180 vlanpcp: 0 parent interface: igb1
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=9<PERFORMNUD,IFDISABLED>
bridge180: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: ***** Bridge180
        ether 02:a5:1b:da:3d:b4
        inet 192.168.8.99 netmask 0xffffff00 broadcast 192.168.8.255
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto stp-rstp maxaddr 2000 timeout 1200
        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
        member: vlan180 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 15 priority 128 path cost 20000
        groups: bridge
        nd6 options=9<PERFORMNUD,IFDISABLED>
vnet0.36: flags=8942<BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: associated with jail: vault as nic: epair0b
        options=8<VLAN_MTU>
        ether d2:50:99:09:5d:70
        hwaddr 02:a7:b8:25:7e:0a
        groups: epair
        media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
        status: active
        nd6 options=1<PERFORMNUD>
vnet2: flags=8942<BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=80000<LINKSTATE>
        ether fe:a0:98:06:5c:c3
        hwaddr 58:

 

[ProbieK]

I also have turned off Hardware Offloading on the physical interface (igb1) and the vlan interface (vlan180). I tried to disable it on the bridge interface as well, but i got the following error: "Offloading capabilities is not supported for bridge interfaces"

Does anyone have any ideas of what I could try or what I might be missing?

Thanks

 

[ProbieK]

Figured I'd try again... Can anyone suggest something that I could try or have any idea what might be causing the issue? I'm still not able to figure this out.

 

[Samuel Tai]

I don't know how your Ubiquiti switch works, but the vlan interface you've created is expecting an 802.1q tagged packet, if I understand your set up correctly. Your switch will need to provide tagged packets, but I suspect it's providing untagged packets, which are then getting dropped.

 

[Vertigo 7]

Did you ever find a solution? I'm running into a similar problem. I'm passing 3 VLANS to igb1. I have vlan interfaces set up for vlans 20 and 7, 40 is the native on this switch interface. I assigned the vlan20 interface to a jail and by default, it's getting an IP in vlan 40. If I change anything in the "interfaces" field on the jail's netwoking properties from vnet0:bridge0, I get no IP address no matter what I set it to.

 

[Vertigo 7]

Persistence paid off. I did solve this.

Doesn't look like TrueNAS plays well with tagged and untagged traffic on the same interface. After much trial and error the final solution was to drop the native network from the trunk I was sending to TrueNAS and tag every network and then creating pairs of VLAN and bridge interfaces for each of the networks in the trunk (the parent interface all being igb1 in my case). In each of the jails the default interface was set to "none" and the interfaces field set to vnet0:bridgeXX (xx=bridge number of matching VLAN tag).

This has even solved all the problems i was having in another post with packet loss and other network oddities.

Jails and VMs are now able to run on boot and get the proper IP address for their network and their connection is stable.

I'd do a friggin backflip if I were physically able to...