FreeNAS drops tagged packets addressed to bHYVE VM (tags: lagg vlan vm bhyve 802.1q networking)

[IOSonic]

Hello,

I am running into a very strange issue in which FreeNAS seems to drop frames destined for a bHyve VM running inside of it. Here is my setup, with IP addresses and MACs generalized. I use a DHCP request here to illustrate the problem.

• My FreeNAS host is an ASUS P11C-i with two Intel I210AT Gigabit Ethernet NICs. I have configured these NICs as a LAG in Freenas 11.3-U4.1
• On this LAG, I have a native, untagged VLAN (vlan1 in this example) and a tagged VLAN subinterface (vlan2).
• I create a virtual machine (Alpine LInux) and add two virtual NICs -- vNIC#1 & vNIC#2. The parent interface for vNIC #1 is vlan1; the parent for vNIC #2 is vlan2.
• To make a long story short, traffic sent from vNIC #2 successfully exits onto the physical network, but does not return to the virtual machine.
• Using packet captures on multiple devices, I can 100% confirm where things go wrong--any return traffic appears to be dropped by the freenas at the host level.

To be perfectly clear, I can perform the following without issue:

• From the router, ping the address of Freenas' vlan2 interface.
• From the vm, ping the address of Freenas' vlan2 interface.
• From Freenas, ping the address of the vm's vlan2 interface.

What's clear is that as soon as traffic from vNIC #2 leaves the Freenas box, it can't get back again. This indicates that the kernel is dropping the packet for some reason.

Does anyone have any ideas? Can anyone reproduce this? I would be extremely grateful for any help! Thanks!


SYSTEM DETAILS
---------------------

• Version: Freenas 11.3-U4.1
• Motherboard: ASUS P11C-i
• Physical NICS: Intel I210AT Gigabit Ethernet NICs
• Virtual NICS: VirtuIO
• Virtual machine OS: Alpine Linux (latest)

 

[IOSonic]

Nobody?

 

[IOSonic]

FWIW to anyone stymied by this who comes across this thread, my issue was due to my (poor) understanding of FreeNAS networking. I knew what I wanted to do, but was not doing it properly. This guide helped me a lot. Hope it helps someone else.

How to setup VLANs within FreeNAS 11.3

This post is a follow-up solution for solving how to setup VLANS with 11.2U7 (https://forums.lawrencesystems.com/t/freenas-jails-with-vlan/3046/11). Both 11.2 and 11.3 are VLAN able, however the documentation for getting a working setup is lacking. Pre-requisites: 1. This guide assumes your...

www.ixsystems.com

 

[Stephane-Geek]

Hello,
I have a similar setup as mentioned at the beginning and I am experiencing a similar problem.
However, I have 2 network interfaces for which the 1st is connected to vlan 1 (untagged) and the second with a trunk (all vlan tagged). I created the bridges and connected them to the vlanX interfaces as described in the post.

My virtual machine is connected to bridge10. Tested with the 2 types of cards (AHCI or Virt0) I obtain a DHCP address from my router and from the correct vlan10.

Observations:
from vm
1-I get a DHCP address from the router
2-Failed to ping router gateway
3-impossible to ping other IPs (server for example) on vlan10
4-I can Ping the static address of the vlan10 card

from trueNas CLI
1-I can ping the router gateway (vlan10)
2-I can not ping the vm IP
3-arp table showed the vm IP and MAC

TrueNas release: TrueNAS-12.0
VirtualMachine OS: Windows 10

Any idea what's wrong?

 

[Patrick M. Hausen]

VMs are connected to physical interfaces via the FreeBSD bridge interface.
The bridge interface does not support tagged frames.

Create the VLAN interfaces in TrueNAS and create a separate bridge interface for each VLAN. Then use as many virtual interfaces in your VMs connected to the matching bridge interface as necessary.

Don't run untagged frames on the same physical interface, specifically don't put a bridge interface on that untagged interface. See above. Bridge on interface --> no VLANs.

 

[Stephane-Geek]

VMs are connected to physical interfaces via the FreeBSD bridge interface.
The bridge interface does not support tagged frames.

Create the VLAN interfaces in TrueNAS and create a separate bridge interface for each VLAN. Then use as many virtual interfaces in your VMs connected to the matching bridge interface as necessary.

Don't run untagged frames on the same physical interface, specifically don't put a bridge interface on that untagged interface. See above. Bridge on interface --> no VLANs.

As you can see in my post, this is what I set up. Check vlan10 interface configured to use tag-id 10 and vlan10 and vnet3 (vm nic) are member of bridge10 (separate and dedicated bridge for vlan ID 10).
-ag0 10G Physical interface has no IP configured
-vlan10 interface using ag0 as principal interface
-the switch port where ag0 is connected to use only tagged vlan. No untag vlan present in my config.

Looking for the needle in the haystack 8)

 

[Patrick M. Hausen]

Did you disable hardware offload for interface aq0? Can you try to add "up" to the options field of interface aq0?

 

[Stephane-Geek]

I only applied theses parameters on the physical interface. Unfortunatly no effect.

 

[Samuel Tai]

I'm not surprised. The Aquantia driver in TrueNAS is alpha quality. It's included only for testing, and is not feature-complete. There has been no further development of this driver, so far as I know. You'll need to switch to a more supported 10G NIC, like an Intel or Chelsio.

 

[Stephane-Geek]

I'm not surprised. The Aquantia driver in TrueNAS is alpha quality. It's included only for testing, and is not feature-complete. There has been no further development of this driver, so far as I know. You'll need to switch to a more supported 10G NIC, like an Intel or Chelsio.

IT'S WORKING!!!
I did few tests to confirm the hypothesis. I have 2 network cards of different brands and I swapped their roles in order to test the configuration with vlan and bridge. It seems that I am unlucky or there is a real bug in TrueNas 12.0-U8. Still have exactly the same behavior.
Here's my network interfaces

Code:

root@truenas[~]#  pciconf -lv | grep -A1 -B3 network
re0@pci0:3:0:0:    class=0x020000 card=0x05b71028 chip=0x816810ec rev=0x0c hdr=0x00
    vendor     = 'Realtek Semiconductor Co., Ltd.'
    device     = 'RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller'
    class      = network
    subclass   = ethernet
aq0@pci0:5:0:0:    class=0x020000 card=0x00011d6a chip=0xd1071d6a rev=0x02 hdr=0x00
    vendor     = 'Aquantia Corp.'
    device     = 'AQC107 NBase-T/IEEE 802.3bz Ethernet Controller [AQtion]'
    class      = network
    subclass   = ethernet

So, still searching and testing and I decided to;
1- reset all my network interfaces (aq0 and re0).
2- configure re1 with vlan 1 but without vlan tag
3- configure vlan10 interface on aq0
4- configure bridge10 with vlan10
5- reboot
6- reassign VNC and NIC vm devices to the appropriate NIC/Bridge.
7- boot my vm
Now's working well!

I suspecting a bug intruduced with a previous problematic deployment (vm or jail).
If it helps developers, I noticed that bridge0 remained in place even after removing all associations.

 

[Patrick M. Hausen]

Unfortunately you did not post your complete ifconfig output initially. If there was a bridge0 with aq0 as a member, VLANs could not work as I already stated. Next time please post complete command output as text, not as screen shots. Thanks. Glad it's working now.

 

[Stephane-Geek]

Unfortunately you did not post your complete ifconfig output initially. If there was a bridge0 with aq0 as a member, VLANs could not work as I already stated. Next time please post complete command output as text, not as screen shots. Thanks. Glad it's working now.

Your request is an order Master. 8)

My observation is that when the bridge0 is active, no matter on which interface, the bridge does not work on the other interfaces. My VM is on bridge10 (vlan10 linked to aq0) and was working last night. I had only started this VM. And when I started my jail which is on the re0 interface (with vnet0), the bridge0 was added and broke the bridge10. Despite that they are on different physical interfaces.

Here's my current ifconfig and not functionnal:

Code:

root@truenas[~]# ifconfig
re0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=82099<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC,LINKSTATE>
    ether b8:ca:3a:80:90:52
    inet 192.168.2.245 netmask 0xffffff00 broadcast 192.168.2.255
    media: Ethernet autoselect (1000baseT <full-duplex>)
    status: active
    nd6 options=9<PERFORMNUD,IFDISABLED>
aq0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=8103bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWFILTER>
    ether 88:c9:b3:bf:a2:05
    media: Ethernet autoselect <full-duplex,rxpause,txpause> (10Gbase-T <full-duplex>)
    status: active
    nd6 options=1<PERFORMNUD>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
    options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
    inet6 ::1 prefixlen 128
    inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3
    inet 127.0.0.1 netmask 0xff000000
    groups: lo
    nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
pflog0: flags=0<> metric 0 mtu 33160
    groups: pflog
vlan10: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=1<RXCSUM>
    ether 88:c9:b3:bf:a2:05
    inet 10.0.10.2 netmask 0xffffff00 broadcast 10.0.10.255
    groups: vlan
    vlan: 10 vlanpcp: 0 parent interface: aq0
    media: Ethernet autoselect <full-duplex,rxpause,txpause> (10Gbase-T <full-duplex>)
    status: active
    nd6 options=9<PERFORMNUD,IFDISABLED>
bridge10: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    ether 02:18:d2:c0:9f:0a
    id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
    maxage 20 holdcnt 6 proto stp-rstp maxaddr 2000 timeout 1200
    root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
    member: vnet0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 7 priority 128 path cost 2000000
    member: vlan10 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 5 priority 128 path cost 2000000
    groups: bridge
    nd6 options=9<PERFORMNUD,IFDISABLED>
vnet0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=80000<LINKSTATE>
    ether fe:a0:98:ff:ff:ff
    hwaddr 58:9c:fc:10:7d:2c
    groups: tap
    media: Ethernet autoselect
    status: active
    nd6 options=1<PERFORMNUD>
    Opened by PID 1454
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    ether 02:18:d2:c0:9f:00
    id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
    maxage 20 holdcnt 6 proto stp-rstp maxaddr 2000 timeout 1200
    root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
    member: vnet0.1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 9 priority 128 path cost 2000
    member: re0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 1 priority 128 path cost 20000
    groups: bridge
    nd6 options=1<PERFORMNUD>
vnet0.1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: associated with jail: plex-server as nic: epair0b
    options=8<VLAN_MTU>
    ether ba:ca:3a:91:18:6f
    hwaddr 02:4b:84:45:44:0a
    groups: epair
    media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
    status: active
    nd6 options=1<PERFORMNUD>
root@truenas[~]#

So I'll conclude that it is a bug with TrueNas 12.0-U8.

 

[Patrick M. Hausen]

Manually create the bridge0 with re0 as the only member. Remove IP address information from re0 and put it on bridge0. Assign your jail to bridge0 instead of re0.

This will work. If you run anything more complex than one interface, one bridge0 (automatically created), you must manually create all bridge interfaces in advance and assign IP addresses to the bridge interfaces (if needed) according to the FreeBSD documentation.

Kind regards,
Patrick

P.S. The best way to get rid of the bridge0 so you can create it in the UI is to shut down the jail, then use ifconfig bridge0 destroy.