Bridge interface no longer passing traffic in 11.3

[Yukisharu]

Ever since I upgraded my FreeNAS to 11.3 I've noticed that my two ESXi hosts that are directly connected via QSFP are not using the 40Gb iSCSI path but are only using the fallback 10Gb connections. Before the upgrade I was using the tunables on FreeNAS to build out the bridge interface:
autobridge_bridge100=mlxen0 mlxen1
autobridge_interfaces=bridge100
cloned_interfaces=bridge100
ifconfig_mlxen0=up
ifconfig_mlxen1=up
ifconfig_bridge100=up 10.1.1.1 netmask 255.255.255.248

Since FreeNAS 11.3 has built in support from the network tab to create bridge interfaces I migrated the configuration over there. Doing any kind of network connectivity test on either end with a pcap running on either the bridge interface or the actual intefaces mlxen0/mlxen1 results in 0 packets captured (ping, open port test)

Code:

root@SAN[~]# ifconfig
mlxen0: flags=8902<BROADCAST,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
 options=ed03bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWFILTER,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
    nd6 options=1<PERFORMNUD>
    media: Ethernet autoselect (autoselect <full-duplex,rxpause,txpause>)
    status: active

mlxen1: flags=8902<BROADCAST,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
 options=ed03bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWFILTER,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
    nd6 options=1<PERFORMNUD>
    media: Ethernet autoselect (autoselect <full-duplex,rxpause,txpause>)
    status: active

bridge100: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    ether 02:33:ff:9c:1a:64
    inet 10.1.1.1 netmask 0xfffffff8 broadcast 10.1.1.7
    nd6 options=9<PERFORMNUD,IFDISABLED>
    groups: bridge
    id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
    maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
    root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
    member: mlxen1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
             ifmaxaddr 0 port 5 priority 128 path cost 55
    member: mlxen0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
             ifmaxaddr 0 port 4 priority 128 path cost 55


root@SAN[~]# lspci | grep Mellanox
02:00.0 Ethernet controller: Mellanox Technologies MT27520 Family [ConnectX-3 Pro]


root@SAN[~]# dmesg | grep mlx
mlx5en: Mellanox Ethernet driver 3.5.1 (April 2019)
mlx4_core0: <mlx4_core> mem 0xdf200000-0xdf2fffff,0xdd000000-0xdd7fffff at device 0.0 on pci2
mlx4_core: Mellanox ConnectX core driver v3.5.1 (April 2019)
mlx4_core: Initializing mlx4_core
mlx4_core0: Unable to determine PCI device chain minimum BW
mlx4_en mlx4_core0: Activating port:1
mlx4_en: mlx4_core0: Port 1: Using 8 TX rings
mlxen0: link state changed to DOWN
mlx4_en: mlx4_core0: Port 1: Using 8 RX rings
mlx4_en: mlxen0: Using 8 TX rings
mlx4_en: mlxen0: Using 8 RX rings
mlx4_en: mlxen0: Initializing port
mlx4_en mlx4_core0: Activating port:2
mlx4_en: mlx4_core0: Port 2: Using 8 TX rings
mlxen1: link state changed to DOWN
mlx4_en: mlx4_core0: Port 2: Using 8 RX rings
mlx4_en: mlxen1: Using 8 TX rings
mlx4_en: mlxen1: Using 8 RX rings
mlx4_en: mlxen1: Initializing port
mlxen0: promiscuous mode enabled
mlxen1: promiscuous mode enabled
mlx4_en: mlxen1: Link Down
mlx4_en: mlxen1: Link Down

The other hosts are 10.1.1.3 and 10.1.1.2

 

[Yukisharu]

Anyone have any ideas what to check here ?

 

[Patrick M. Hausen]

Forgot to configure mlxen0 and mlxen1 "up"? Can be done via UI in the "options" field.

 

[Newfoundland.Republic]

I'm not sure if it is specific to my case (and it may likely be), but I ended up deleting my jails, bridges and VLANs that I had created in 11.x-Ux and recreating the networking before it would work. Even then, I could only use DHCP to get my networking to work (e.g., no specifying IP, etc.). This very well may be related to behyve, though.

 

[Yukisharu]

Forgot to configure mlxen0 and mlxen1 "up"? Can be done via UI in the "options" field.

Thank you!! This fixed the problem immediately. I don't understand why this required though. I have four NICs total on my FreeNAS box, 2x 10Gb and 2x 40Gb. All four NICs show a link state of UP on the Network > Interfaces page and the 2x 10Gb NICs I did not have to specify an option of "up" yet they're working just fine. I don't understand why it was necessary to add "up" to the 2x 40Gb NICs.

 

[Patrick M. Hausen]

"State up" means the IF has got a link. It does not mean it is administratively "configured up". You can ifconfig up/down independent of everything else. And if you do not assign either DHCP or an IP address or anything protocol related, FreeBSD will leave your interface in the "configured down" state.

You literally had this in your very first post in this thread ;)

Code:

ifconfig_mlxen0=up
ifconfig_mlxen1=up

Kind regards,
Patrick

 

[Yukisharu]

"State up" means the IF has got a link. It does not mean it is administratively "configured up". You can ifconfig up/down independent of everything else. And if you do not assign either DHCP or an IP address or anything protocol related, FreeBSD will leave your interface in the "configured down" state.

You literally had this in your very first post in this thread ;)

Code:

ifconfig_mlxen0=up
ifconfig_mlxen1=up

Kind regards,
Patrick

Thank you so much for the detailed explanation +1