SOLVED Losing packets between LAGG and VLAN

[hexley]

Hi,

I would like to get networking working for a VM under FreeNAS 11.1-U2, starting with DHCP.

So far I've used only the FreeNAS GUI to create the following setup:
Router [DHCP server]
Switch [DHCP relay > LACP trunk]
FreeNAS [lagg0{igb0,igb1,igb2,igb3} > vlan74 > bridge1 > tap0]
VM [eth0]

It seems like the Router and Switch half of the setup are working, because a tcpdump on lagg0 reveals both the DHCP discover and returning offer:


tcpdump -i lagg0 -env vlan 74

14:31:42.211581 00:a0:98:3f:60:a7 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 346: vlan 74, p 0, ethertype IPv4, (tos 0x0, ttl 64, id 0, offset 0, flags [none], proto UDP (17), length 328)
0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 00:a0:98:3f:60:a7, length 300, xid 0x67ce440b, secs 3063, Flags [none]
Client-Ethernet-Address 00:a0:98:3f:60:a7
Vendor-rfc1048 Extensions
Magic Cookie 0x63825363
DHCP-Message Option 53, length 1: Discover
Client-ID Option 61, length 7: ether 00:a0:98:3f:60:a7
MSZ Option 57, length 2: 576
Parameter-Request Option 55, length 7:
Subnet-Mask, Default-Gateway, Domain-Name-Server, Hostname
Domain-Name, BR, NTP
Vendor-Class Option 60, length 3: "d-i"

14:31:42.231221 98:f2:b3:65:19:be > 00:a0:98:3f:60:a7, ethertype 802.1Q (0x8100), length 357: vlan 74, p 0, ethertype IPv4, (tos 0x0, ttl 64, id 59652, offset 0, flags [none], proto UDP (17), length 339)
10.7.127.254.67 > 10.7.103.191.68: BOOTP/DHCP, Reply, length 311, hops 1, xid 0x67ce440b, secs 3063, Flags [none]
Your-IP 10.7.103.191
Server-IP 192.168.7.1
Gateway-IP 10.7.127.254
Client-Ethernet-Address 00:a0:98:3f:60:a7
Vendor-rfc1048 Extensions
Magic Cookie 0x63825363
DHCP-Message Option 53, length 1: Offer
Server-ID Option 54, length 4: 192.168.7.1
Lease-Time Option 51, length 4: 900
RN Option 58, length 4: 450
RB Option 59, length 4: 787
Subnet-Mask Option 1, length 4: 255.255.224.0
BR Option 28, length 4: 10.7.127.255
Domain-Name-Server Option 6, length 4: 192.168.7.1
Domain-Name Option 15, length 11: "local.lan"
NTP Option 42, length 4: 192.168.7.1
Default-Gateway Option 3, length 4: 10.7.127.254


But a tcpdump on vlan74 (or bridge1, tap0, eth0) shows the DHCP discover (plus some ICMPv6 chatter not shown), but no returning offer:


tcpdump -i vlan74 -env

14:37:51.519823 00:a0:98:3f:60:a7 > ff:ff:ff:ff:ff:ff, ethertype IPv4 (0x0800), length 342: (tos 0x0, ttl 64, id 0, offset 0, flags [none], proto UDP (17), length 328)
0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 00:a0:98:3f:60:a7, length 300, xid 0x669bd1f, secs 376, Flags [none]
Client-Ethernet-Address 00:a0:98:3f:60:a7
Vendor-rfc1048 Extensions
Magic Cookie 0x63825363
DHCP-Message Option 53, length 1: Discover
Client-ID Option 61, length 7: ether 00:a0:98:3f:60:a7
MSZ Option 57, length 2: 576
Parameter-Request Option 55, length 7:
Subnet-Mask, Default-Gateway, Domain-Name-Server, Hostname
Domain-Name, BR, NTP
Vendor-Class Option 60, length 3: "d-i"


What I'm expecting is that vlan74 would receive the offer (since the packet is tagged 74) and that bridge1 would share the packet with tap0 and eventually eth0.


ifconfig

igb0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=2400b9<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWTSO,RXCSUM_IPV6>
ether 0c:c4:7a:db:fc:48
hwaddr 0c:c4:7a:db:fc:48
nd6 options=9<PERFORMNUD,IFDISABLED>
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
igb1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=2400b9<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWTSO,RXCSUM_IPV6>
ether 0c:c4:7a:db:fc:48
hwaddr 0c:c4:7a:db:fc:49
nd6 options=9<PERFORMNUD,IFDISABLED>
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
igb2: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=2400b9<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWTSO,RXCSUM_IPV6>
ether 0c:c4:7a:db:fc:48
hwaddr 0c:c4:7a:db:fc:4a
nd6 options=9<PERFORMNUD,IFDISABLED>
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
igb3: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=2400b9<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWTSO,RXCSUM_IPV6>
ether 0c:c4:7a:db:fc:48
hwaddr 0c:c4:7a:db:fc:4b
nd6 options=9<PERFORMNUD,IFDISABLED>
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
lagg0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=2400b9<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWTSO,RXCSUM_IPV6>
ether 0c:c4:7a:db:fc:48
inet 192.168.7.7 netmask 0xffffff00 broadcast 192.168.7.255
nd6 options=9<PERFORMNUD,IFDISABLED>
media: Ethernet autoselect
status: active
groups: lagg
laggproto lacp lagghash l2,l3,l4
laggport: igb0 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
laggport: igb1 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
laggport: igb2 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
laggport: igb3 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
vlan74: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=200001<RXCSUM,RXCSUM_IPV6>
ether 0c:c4:7a:db:fc:48
nd6 options=9<PERFORMNUD,IFDISABLED>
media: Ethernet autoselect
status: active
vlan: 74 vlanpcp: 0 parent interface: lagg0
groups: vlan
bridge1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
ether 02:6b:66:2d:af:03
nd6 options=1<PERFORMNUD>
groups: bridge
id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
member: tap0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 27 priority 128 path cost 2000000
member: vlan74 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 7 priority 128 path cost 55
tap0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=80000<LINKSTATE>
ether 00:bd:18:70:af:00
hwaddr 00:bd:18:70:af:00
nd6 options=1<PERFORMNUD>
media: Ethernet autoselect
status: active
groups: tap


Hopefully I'm just missing something silly...

Thanks for your help.

 

[dlavigne]

There is an open bug for this: https://redmine.ixsystems.com/issues/28778.

 

[hexley]

I appreciate the response.

 

[sean.nutech]

This is still an issue with 11.2-U8. Bug report 28778 is related to GUI and the ability to build a VM using a VLAN etc and AFAIK not the underlying issue of the packet disappearing or not being delivered to the vlanXX interface on the bridge.

I can confirm the exact same behaviour where packets going to the bridge vlan interface never arrive but they do exist on the lagg.