VLAN Tag issues with Intel I350 (igb) cards

[jlw52761]

I'm having an issue with VLAN tagged traffic on my FreeNAS with these cards, and am pretty sure it's something in the driver that's causing my issues. Let me step back and detail my config and my testing to show my conclusions.

My box has two, Intel I350 Quad Port Gigabit NIC's that use the igb driver. They are connected to a managed switch. I have igb0 connected to an access port on my server VLAN, 30, and all is well. I have igb4 connected to a trunk port with VLAN 30 as my Port/Native VLANID. Any jail or VM that does not use a bridge attached to a VLAN interface, all is well.

The problems comes in when trying to use a VLAN interface to tag the traffic. My VLAN 30 does not have DHCP enabled, but my VLAN 34 does, and my configuration in FreeNAS is as follows:

bridge34 -> vlan34 -> igb4 -> managedswitchtrunk
I have a test jail, called Test, that has DHCP enabled, and under the Network Properties of the jail, I have the interfaces set to "vnet0:bridge34". When I start the jail, I can see the DHCPDISCOVER hitting my DHCP server, and DHCPOFFER being sent back out, but nothing on the jail.

So using tcpdump, I started the Test jail and watched traffic on igb4, and I can see the DHCPDISCOVERY going out, and the DHCPOFFER coming back in, with the VLAN tag of 34 set correctly. I'm using Wireshark to analyze the tcpdump file. I then moved tcpdump to VLAN34, and I can see the DHCPDISCOVERY going out, but I do not see the DHCPOFFER packet coming back.

This pretty much concludes that the DHCPOFFER is making it to the physical interface, but the driver is not translating it over to the VLAN VIF, essentially just dropping it.

I have another jail, let's call it Static, stat does not have DHCP enabled, and has the interfaces set to vnet0:bridge0. I can see the packets to and from this jail traversing igb4, which honestly I don't know how it picked that one since I haven't specified that anywhere, but ifconfig confirms that finding. Anyway, so this further concludes that the driver is not handling the vlan tagging correctly.

Now, when I do an ifconfig, here's what I see

Code:

igb0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
        description: connected to 4 peers
        options=6403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
        ether 00:1e:67:98:42:67
        hwaddr 00:1e:67:98:42:67
        inet n.n.n.n netmask 0x00000000 broadcast n.n.n.n
        nd6 options=9<PERFORMNUD,IFDISABLED>
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
igb4: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: connected to 4 peers
        options=b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM>
        ether 00:1e:67:ab:6f:58
        hwaddr 00:1e:67:ab:6f:58
        nd6 options=9<PERFORMNUD,IFDISABLED>
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        ether 02:f0:12:21:4b:00
        nd6 options=1<PERFORMNUD>
        groups: bridge
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
        member: vnet0.22 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 12 priority 128 path cost 2000
        member: igb4 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 5 priority 128 path cost 20000
vlan34: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        ether 00:1e:67:ab:6f:58
        nd6 options=9<PERFORMNUD,IFDISABLED>
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        vlan: 34 vlanpcp: 0 parent interface: igb4
        groups: vlan
bridge34: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: Bridge for VLAN34
        ether 02:f0:12:21:4b:22
        nd6 options=9<PERFORMNUD,IFDISABLED>
        groups: bridge
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
        member: vlan34 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
        ifmaxaddr 0 port 13 priority 128 path cost 20000                                                                                                                                                                media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
        status: active
        groups: epair
vnet0.22: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: associated with jail: Static as nic: epair0b
        options=8<VLAN_MTU>
        ether 00:1e:67:a7:dd:dc
        hwaddr 02:98:d0:00:0c:0a
        nd6 options=1<PERFORMNUD>
        media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
        status: active
        groups: epair

It is sanitized, and I did not include the tap0 or tap1 interfaces, or any of the down interfaces.

What I'm inferring from this is that bridge34 does have vlan34 as a member, but I'm not seeing vlan34 actually attaching to igb4. Also, I did disable HW Offloading on igb4, as per best practice of not using hardware offloading with jails and VMs.

I have also added the hw.igb.fc_setting=0 tunable and the kern.ipc.nmbclusters=1000000 tunables as per the IXSystems documentation for tuning of the igb based cards. Plus, I don't like flow control for NAS/storage things.

So as it stands, I have to only use access ports with my rig and have to eat a interface for each VLAN I want to use. To me, this is horrible and not sure why it would exist this way. I'm open to any and all conversation and advice, save the "Don't use jails or VMs on FreeNAS". If that's you're only answer, then I don't have time for that.

 

[Kcaj]

Hi jlw52761,

What release of FreeNAS are you on? Some things changed between 11.2 & In 11.3. @KevDog did a nice write up on 11.3 which might help.

From your ifconfig where is the vnet interface for the test jail? That will need to be a member of the bridge34 for it to work....

Edit:

What I'm inferring from this is that bridge34 does have vlan34 as a member, but I'm not seeing vlan34 actually attaching to igb4.

From your ifconfig the vlan34 has parent igb4 so that looks fine.

and under the Network Properties of the jail, I have the interfaces set to "vnet0:bridge34".

If your on 11.2 have you set the jail IPv4 Interface under 'Basic Properties' to vnet0?
In 11.3 as a summary for vlan in jail:

1. Create a bridge with vlan34 as a member under network->interfaces called bridge34
2. Set IPv4 Interface to none in jail
3. Set Interfaces to vnet0:bridge34 in jail
4. Set vnet_default_interface to auto or none (mines fine on auto)
5. check allow_raw_sockets in jail and try ping

Are you intending to use Jumbo Frames?

 

[jlw52761]

Hi jlw52761,

What release of FreeNAS are you on? Some things changed between 11.2 & In 11.3. @KevDog did a nice write up on 11.3 which might help.

From your ifconfig where is the vnet interface for the test jail? That will need to be a member of the bridge34 for it to work....

Edit:


From your ifconfig the vlan34 has parent igb4 so that looks fine.


If your on 11.2 have you set the jail IPv4 Interface under 'Basic Properties' to vnet0?
In 11.3:

1. Create a bridge with vlan34 as a member under network->interfaces called bridge34
2. Set IPv4 Interface to none in jail
3. Set Interfaces to vnet0:bridge34 in jail
4. check allow_raw_sockets in jail and try ping

Are you intending to use Jumbo Frames?

So the only vnet interface I have is the vnet0.22 attached to the Static jail. Didn't catch that, or the parent interface on the vlan34.

I enabled the raw_sockets in the jail config, and the Interfaces is vnet0:bridge34. Unfortunately same results. Here's some screenshots of the jail config.

Here's a screengrab of the network interface config in the GUI

As for jumbo frames, no intention to use them on the jails, I use them on my iSCSI interfaces, but I also didn't put any specific options in the interface settings so those are coming default when creating them in the GUI.

As a side note, I just figured out I can do "iocage get -a Test" to get the jail config. Still learning, old Linux admin here.

 

[Kcaj]

Can you try set the jail vnet_default_interface to auto and the post the output from ifconfig

 

[jlw52761]

Can you try set the jail vnet_default_interface to auto and the post the output from ifconfig

While the jail is trying to startup, I see this:
vnet0.29: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
description: associated with jail: Test as nic: epair0b
options=8<VLAN_MTU>
ether 00:1e:67:1b:34:d2
hwaddr 02:98:d0:00:10:0a
nd6 options=1<PERFORMNUD>
media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
status: active
groups: epair

The startup still fails, can't get DHCP. Once the startup fails, the vnet disappears, which I guess I expect because the process is not running.

Out of curiosity, I ran another tcpdump using "tcpdump -s 0 -i vlan34 -w vlan342.pcap", and still not seeing the DHCPOFFER making it back through the stack past igb4.

 

[Kcaj]

I personally dont use DHCP but maybe do a capture on the bridge interface instead and DHCP uses UDP.

Can you try the jail with a static IPs to prove that the NIC is working?

 

[jlw52761]

I personally don't use DHCP but maybe do a capture on the bridge interface instead and DHCP uses UDP.

Can you try the jail with a static IPs to prove that the NIC is working?

Switched Test to a static IP and started, pretty quick gor a return of:

* Starting Test
+ Started OK
+ Using devfs_ruleset: 6
+ Configuring VNET FAILED
route: writing to routing socket: Network is unreachable

Now remember, my other jail, Static, is currently running in igb4 on bridge0 (autocreated) so I know the NIC works, plus I can see the packets leave the NIC, just nothing going from the NIC to the VLAN interface, or the bridge.

 

[jlw52761]

Switched Test to a static IP and started, pretty quick gor a return of:

* Starting Test
+ Started OK
+ Using devfs_ruleset: 6
+ Configuring VNET FAILED
route: writing to routing socket: Network is unreachable

Now remember, my other jail, Static, is currently running in igb4 on bridge0 (autocreated) so I know the NIC works, plus I can see the packets leave the NIC, just nothing going from the NIC to the VLAN interface, or the bridge.

I also just noticed that my VM, running on tap0, is also using bridge0 and is working fine, it's one of my Docker swarm nodes.

 

[jlw52761]

So at this point, I think I have to count this as a bug with the driver and move onto not using VLAN trunking with this NIC, unless anyone has other ideas.