SOLVED Trunk port not working after upgrade from 11.3U5 to 12.0U6

[asimov-solensan]

Hello,

I really need help with this. I finally upgraded to truenas and it broke a feature critical on my setup.

I have got a VM bridged to a physical interface with no configuration in freenas. This VM manages the vlan tagging and the interface is connected to a trunk port in a switch.

Worked like this for years without issue but after upgrading to truenas this stopped working. For instance:

From the VM I try to ping a machine in VLAN13 (192.168.13.1 - 192.168.13.254) but the ARP request gets lost at some point. Check that the packet is correctly tagged in the physical interface.

root@lavochkin:~ # tcpdump -i em2 -nn -e vlan
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on em2, link-type EN10MB (Ethernet), capture size 262144 bytes
09:35:01.344641 00:a0:98:06:00:2e > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 64: vlan 13, p 0, ethertype ARP, Request who-has 192.168.13.254 tell 192.168.13.1, length 46
09:35:01.498001 00:a0:98:43:a8:14 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 64: vlan 17, p 0, ethertype ARP, Request who-has 192.168.17.20 tell 192.168.17.22, length 46

I also checked that connecting a computer to the interface it is correctly tagging packets.

root@charcoal2:/home/asimov# tcpdump -i enx00e04c30eed2 -nn -e vlan
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on enx00e04c30eed2, link-type EN10MB (Ethernet), capture size 262144 bytes
09:51:39.155214 0c:c4:7a:cf:86:d2 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 64: vlan 17, p 0, ethertype ARP, Request who-has 192.168.17.20 tell 192.168.17.10, length 46
09:51:39.323703 00:a0:98:06:00:2e > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 64: vlan 13, p 0, ethertype ARP, Request who-has 192.168.13.254 tell 192.168.13.1, length 46
09:51:40.412825 00:a0:98:06:00:2e > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 64: vlan 13, p 0, ethertype ARP, Request who-has 192.168.13.254 tell 192.168.13.1, length 46

To be sure I also connected to a VLAN13 port in the switch to confirm the problem is not in the switch.

tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on enx00e04c30eed2, link-type EN10MB (Ethernet), capture size 262144 bytes
09:59:02.672190 ARP, Request who-has 192.168.13.254 tell 192.168.13.1, length 46
09:59:03.760964 ARP, Request who-has 192.168.13.254 tell 192.168.13.1, length 46

And finally tested pinging a machine with tcpdump running (192.168.13.222). To confirm it is answering to the ARP request.

root@charcoal2:/home/asimov# tcpdump -i wlp2s0 -nn
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on wlp2s0, link-type EN10MB (Ethernet), capture size 262144 bytes
10:27:56.472361 ARP, Request who-has 192.168.13.222 tell 192.168.13.1, length 46
10:27:56.472384 ARP, Reply 192.168.13.222 is-at b0:35:9f:be:d1:ec, length 28
10:27:57.086887 ARP, Request who-has 192.168.13.254 tell 192.168.13.1, length 46
10:27:57.496316 ARP, Request who-has 192.168.13.222 tell 192.168.13.1, length 46

But the reply seems to be dropped because I cannot capture it on the freenas physical interface.

After many tests I finally decided to go back to freenas and try again. Same netwoek configuration, same VM, same switch, all exactly the same and it works again.

root@lavochkin:~ # tcpdump -i em2 -nn -e vlan | grep 192.168.13.1. | grep 192.168.13.254
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on em2, link-type EN10MB (Ethernet), capture size 262144 bytes
13:15:34.065473 00:a0:98:06:00:2e > 88:5d:fb:c9:3e:c4, ethertype 802.1Q (0x8100), length 102: vlan 13, p 0, ethertype IPv4, 192.168.13.1 > 192.168.13.254: ICMP echo request, id 18769, seq 0, length 64
13:15:34.066012 88:5d:fb:c9:3e:c4 > 00:a0:98:06:00:2e, ethertype 802.1Q (0x8100), length 102: vlan 13, p 1, ethertype IPv4, 192.168.13.254 > 192.168.13.1: ICMP echo reply, id 18769, seq 0, length 64
13:15:35.089865 00:a0:98:06:00:2e > 88:5d:fb:c9:3e:c4, ethertype 802.1Q (0x8100), length 102: vlan 13, p 0, ethertype IPv4, 192.168.13.1 > 192.168.13.254: ICMP echo request, id 18769, seq 1, length 64
13:15:35.090266 88:5d:fb:c9:3e:c4 > 00:a0:98:06:00:2e, ethertype 802.1Q (0x8100), length 102: vlan 13, p 1, ethertype IPv4, 192.168.13.254 > 192.168.13.1: ICMP echo reply, id 18769, seq 1, length 64

I fear that truenas is behaving differently but I don't know were is the problem. And getting this working is mandatory for mys setup.

The only difference I found between 11.3 and 12 is in the interface configuration:
11.3

em2: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
description: em2 TRUNK externo
options=98<VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM>
ether 00:15:17:2e:65:a3
hwaddr 00:15:17:2e:65:a3
nd6 options=9<PERFORMNUD,IFDISABLED>
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active

12

em2: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
description: em2 TRUNK externo
options=810099<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,VLAN_HWFILTER>
ether 00:15:17:2e:65:a3
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
nd6 options=9<PERFORMNUD,IFDISABLED>

Maybe the most remarkable difference is the option VLAN_HWFILTER. Not present in 11.3.

Does anyone know anything about this problem. How can I enable that option in 11.3 to confirm if it was causing the problem?

Thanks in advance

 

[Patrick M. Hausen]

Tick the "disable hardware offloading" in the interface settings. Might need a reboot for your VM to get connected.

 

[asimov-solensan]

What is the theory behind this?

I'm on 11.3 and hardware offloading is enabled (unticked option) and it is working. It was the same in 12.0.

It is very time consuming to try this because if it doesn't work I need to go back to 11.3 and last time it deleted the boot environment, so I had to reinstall and load configuration backup. I need to be sure this is worth to try.

 

[Patrick M. Hausen]

When you use VLANs and bridges you must set this to off. This disables precisely the VLAN_HWFILTER option you noticed - among others.

 

[asimov-solensan]

Ok, disabled hardware offloading in 11.3 and I do not see any difference in the network device (did a full reboot to be sure).

em2: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
description: em2 TRUNK externo
options=98<VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM>
ether 00:15:17:2e:65:a3
hwaddr 00:15:17:2e:65:a3
nd6 options=9<PERFORMNUD,IFDISABLED>
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active

Upgraded again to 12.0 with hardware offloading disabled. Can see the option ticked in the UI, but again no change in the ifconfig command:

em2: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
description: em2 TRUNK externo
options=810098<VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,VLAN_HWFILTER>
ether 00:15:17:2e:65:a3
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
nd6 options=9<PERFORMNUD,IFDISABLED>

That option isn't doing anything it seems. Can anyone confirm it is changing anything in fact?

And found a couple of bugs related to VLAN_HWFILTER, I'm using a different intel chip but very similar behavior with bhyve and vlans when upgrading to freebsd 12.

230996 – em/igb: Intel i210/i350: ifconfig: enabling "vlanhwtag" renders VLAN on i210/i350 NICs unusable

bugs.freebsd.org

I need to find a way to manually disable VLAN_HWFILTER and make the change permanent.

 

[Patrick M. Hausen]

Try putting these into the "options" field of your interface configuration:
-vlanhwtag -vlanhwfilter -vlanhwcsum

The "-" means "disabled", contrary to established Unix convention, so it's supposed to do what you need.

 

[Samuel Tai]

Try deleting the VLAN interface, disabling offload in the physical interface, and then recreating the VLAN interface.

 

[Patrick M. Hausen]

Try deleting the VLAN interface, disabling offload in the physical interface, and then recreating the VLAN interface.

He's not using VLAN interfaces. He's bridging tagged frames to his VM ...

 

[asimov-solensan]

Yes, simply running:

ifconfig em2 -vlanhwfilter

Solved the problem.

In order to make it permanent I had to include the command in a custom init script. Does anyone else know a better solution? What the hell happened to tunables? SInce 11.3 it just ignores whatever I put in there.

Anyhow thanks a lot for the feedback.

P.S
By the way can somone check if disable hardware offloading makes any diference? It seems a bug in the UI.

 

[Patrick M. Hausen]

Options field in the interface configuration page as I wrote.

 

[asimov-solensan]

Ah, yes, now I see what you meant.

Thanks.