VM attached to VLAN - is it actually possible?

[seb101]

Hi folks,

There are numerous posts on this but all have ended with either the OP giving up and doing something else or are just open ended without a solution.

I have a server software than I need to run on a linux VM (it won't run on FreeBSD so a jail is not an option). The VM needs to be on a specific VLAN.

My interfaces are as follows:
ix0 carries untagged LAN traffic and is the primary interface for the NAS
ix1 carries tagged VLAN packets for all my other networks
vlan600 is the vlan interface for the required VLAN and is bound to ix1
bridge1 is a bridge with ix0 as the only member
bridge600 is a bridge with vlan600 as the only member

Setting up a VLAN interface is FreeNAS is simple, it receives a DHCP address in the correct IP range and I can ping the NAS address locally from other devices within that VLAN and access the GUI. So L2/3 connectivity is working fine for vlan600.

However when I try to bring VMs into the picture it goes downhill.

Lets start with what does work. If I set the NIC of the VM to attach to bridge1 - everything works, DHCP address instantly, all connectivity works as expected on the main LAN.

However, if I set the VM NIC to attach to bridge600 - nothing works. No DHCP, no connectivity.

Why is the behaviour so different between a bridge to a physical interface (bridge1) vs a bridge to a vlan interface (bridge600)?

Is it possible to fix this? Or is it a limitation of FreeBSD/FreeNAS? Does anyone have this working?

FYI - several posts in other threads have suggested that when working with VLANs you must assign an IP address to the bridge and not the vlan itself, I can confirm this makes absolutely no difference to the outcome.

 

[seb101]

Also - can confirm that when the VM is brought up with the NIC attached to bridge600 - is does correctly add the VM tap adaptor to the vlan bridge. So everything looks good - there is something hidden going on.

Code:

bridge600: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
        ether 02:33:51:7a:91:58
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto stp-rstp maxaddr 2000 timeout 1200
        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
        member: vnet1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 12 priority 128 path cost 2000000
        member: vlan600 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 8 priority 128 path cost 2000
        groups: bridge
        nd6 options=9<PERFORMNUD,IFDISABLED>
vnet1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 9000
        options=80000<LINKSTATE>
        ether fe:a0:98:ff:ff:ff
        hwaddr 58:9c:fc:10:ff:83
        groups: tap
        media: Ethernet autoselect
        status: active
        nd6 options=1<PERFORMNUD>
        Opened by PID 34451

 

[Patrick M. Hausen]

We run VMs and jails attached to VLANs all the time. In precisely the way you configured the system, i.e. creating the bridge interfaces in advance with the VLAN interfaces as members, attaching the VMs to the bridge interfaces.

Did you disable hardware offloading for the physical interface carrying the VLANs?

Second you must move the IP configuration from the VLAN to the bridge. This has been documented in FreeBSD since the bridge interface was first introduced. Never put IP addresses on member interfaces.

 

[seb101]

Thanks. The only thing I hadn't done was disable hardware offload. So have implemented that also.

However when I move the IP config to bridge600 instead of vlan600 as you suggested, even basic connectivity no longer works, even if I hard-code an IP my NAS is no longer accessible from the VLAN. It's seemingly like the bridge is unable to actually bridge to the vlan600 interface.

 

[Patrick M. Hausen]

Code:

bridge1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: Privat
    ether ac:1f:6b:76:64:1c
    inet6 fe80::ae1f:6bff:fe76:641c%bridge1 prefixlen 64 scopeid 0x8
    inet6 2003:a:d59:3800:ae1f:6bff:fe76:641c prefixlen 64 autoconf
    inet 192.168.1.10 netmask 0xffffff00 broadcast 192.168.1.255
    id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
    maxage 20 holdcnt 6 proto stp-rstp maxaddr 2000 timeout 1200
    root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
    member: vnet2 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 13 priority 128 path cost 2000000
    member: vnet0.7 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 23 priority 128 path cost 2000
    member: vnet0.6 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 21 priority 128 path cost 2000
    member: vnet0.5 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 20 priority 128 path cost 2000
    member: vnet0.4 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 19 priority 128 path cost 2000
    member: vnet0.3 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 18 priority 128 path cost 2000
    member: vnet0.2 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 17 priority 128 path cost 2000
    member: vnet4 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 15 priority 128 path cost 2000000
    member: vnet3 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 14 priority 128 path cost 2000000
    member: vnet1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 12 priority 128 path cost 2000000
    member: vnet0.1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 11 priority 128 path cost 2000
    member: vnet0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 10 priority 128 path cost 2000000
    member: vlan1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 6 priority 128 path cost 2000000
    groups: bridge
    nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>

 

[Patrick M. Hausen]

It's seemingly like the bridge is unable to actually bridge to the vlan600 interface.

Did you configure the physical interface "up"? You can try that with ifconfig. If that helps, simply put "up" into the Options field for the physical.

There is a known bug in the bnxt(4) driver that requires the interface to be set to "promisc" to carry VLAN traffic, but as far as I am aware this does not apply to any ix(4) variant.

And there are various posts by myself, where I successfully guide people through exactly this setup.
E.g.

SOLVED - freenas mini XL+ vnet Jail bridged to vlan extremely slow

I have a brand new freenas mini XL+. My network switch is gigabit. I have a couple of vlans configured on the network. transfers to host system is very near gigabit speeds. I created a jail with vnet/vimage which is bridged to one of the vlans. Doing installs of packages such as 'pkg install...

www.truenas.com

TrueNAS 12.0-STABLE - Set up VLANs

Folks, From my post Are JAIL VLANs broken (since upgrading from 11.3 to 12.0) I cannot (despite incantations to the computer gods, making up new swear words, etc., etc.) get VLANs to work. I started from scratch deleting my Emby jails, deleting the bridges to the VLANs and deleting the VLANs...

www.truenas.com

 

[seb101]

It's not a problem with the physical interface - because if I remove the bridge from the equation, VLAN traffic is correctly handled on the vlan600 interface via ix1 - i.e. I can have the NAS as a DHCP client on that VLAN with no issues at all, gets an IP, communicates fine.

My config is identical to yours, so it's very confusing why the bridge fails to work.

 

[Patrick M. Hausen]

If neither the physical nor the VLAN carries an IP address, in some versions TrueNAS "forgets" to enable the physical interface. So you need to explicitly ifconfig ix1 up. Have you tried that?

 

[seb101]

Good lord. In desperation I rebooted the NAS.... and it's started working immediately. I've been going crazy on this for a few hours now.

I guess I made one too many changes and it just needed to bounce.

 

[Patrick M. Hausen]

Changing the bridging configuration always disconnects all my VMs, too. Reboot required. Make sure you follow my "ip address on the bridge" advice. It really is mandatory, see

CORE bridging setup still in need of improvement

Hi all, after repeatedly hinting people at "Disable Hardware Offloading" when bridging, i.e. plugins, jails and VMs are involved, I really think we need to either make this automatic of at least try to "solve it in documentation". This setting needs to be featured way more prominently...

www.truenas.com

 

[seb101]

Thanks for all your help Patrick. Working in the end - with the IP on the bridge. The reboot required is definitely an undocumented feature!