LACP on em2/3 only works when igb0 is connected?

Browner87 · Feb 21, 2016

I'm trying to get LCAP setup, mostly as a learning experience, and I'm having an issue with it. Up until the last step, it all seemed fine. My switch says ports 22 and 24 are configured by LACP link aggregation, and FreeNAS says lagg0 is an LACP interface with IP 192.168.5.230.

Now my existing normal LAN interface is igb0 which is 192.168.5.220. So I set the WebGUI IPv4 address from 220 to 203. I can no longer browse 192.168.5.220, but I get the normal web interface on 192.168.5.230. This seems like expected behaviour to me, seems like the link agg worked because the web interface is accessible through it and only it.

Now I go an unplug the cable from igb0, and I can't access either IP. Obviously 220 should be unavailable, but now so is 230.

I don't really know how to even attack this. It seems unreasonable to me that traffic destined for 230 would be somehow going through the interface configured as 220? Does anyone see an obvious problem from the description of what's going on or know where I should start on fixing it?

Hardware:
Switch: dlink dgs-1210-28
Server: SuperMicro 2U with X8DTU-F motherboard and 12GB of Samsung RAM and 1x Xeon CPU E5620 @ 2.40GHz
NIC: I don't have a model #, but it uses the Intel 82571EB controller
Cables: 2x Cat6 (2ft)

m0nkey_ · Feb 23, 2016

Please post your hardware specifications/details as per the forum rules. It helps when trying to diagnose the issue and you will likely get a better response.

Browner87 · Feb 23, 2016

Sorry, I'll update my original post.

dlavigne · Feb 24, 2016

Now I go an unplug the cable from igb0, and I can't access either IP. Obviously 220 should be unavailable, but now so is 230.

While you're at this condiction, paste the output of ifconfig using Insert -> Code.

Browner87 · Feb 24, 2016

All connected:

Code:

FreeBSD 9.3-RELEASE-p31 (FREENAS.amd64) #0 r288272+33bb475: Wed Feb  3 02:19:35 PST 2016

        FreeNAS (c) 2009-2015, The FreeNAS Development Team
        All rights reserved.
        FreeNAS is released under the modified BSD license.

        For more information, documentation, help or support, go here:
        http://freenas.org
Welcome to FreeNAS
[browner87@freenas ~]$ ifconfig
igb0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=400b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWTSO>
        ether 00:25:90:67:e7:62
        inet 192.168.5.220 netmask 0xffffff00 broadcast 192.168.5.255
        nd6 options=9<PERFORMNUD,IFDISABLED>
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
igb1: flags=8c02<BROADCAST,OACTIVE,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO>
        ether 00:25:90:67:e7:63
        nd6 options=9<PERFORMNUD,IFDISABLED>
        media: Ethernet autoselect
        status: no carrier
em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=4219b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,WOL_MAGIC,VLAN_HWTSO>
        ether 00:1b:21:9d:2d:7d
        inet 192.168.10.10 netmask 0xfffffffc broadcast 192.168.10.11
        nd6 options=9<PERFORMNUD,IFDISABLED>
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
em1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=4019b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,VLAN_HWTSO>
        ether 00:1b:21:9d:2d:7c
        inet 192.168.10.5 netmask 0xfffffffc broadcast 192.168.10.7
        nd6 options=9<PERFORMNUD,IFDISABLED>
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
em2: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=4019b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,VLAN_HWTSO>
        ether 00:1b:21:9d:2d:7f
        nd6 options=9<PERFORMNUD,IFDISABLED>
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
em3: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=4019b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,VLAN_HWTSO>
        ether 00:1b:21:9d:2d:7f
        nd6 options=9<PERFORMNUD,IFDISABLED>
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
ipfw0: flags=8801<UP,SIMPLEX,MULTICAST> metric 0 mtu 65536
        nd6 options=9<PERFORMNUD,IFDISABLED>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x10
        inet 127.0.0.1 netmask 0xff000000
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
lagg0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=4019b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,VLAN_HWTSO>
        ether 00:1b:21:9d:2d:7f
        inet 192.168.5.230 netmask 0xffffff00 broadcast 192.168.5.255
        nd6 options=9<PERFORMNUD,IFDISABLED>
        media: Ethernet autoselect
        status: active
        laggproto lacp lagghash l2,l3,l4
        laggport: em3 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
        laggport: em2 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        ether 02:48:4a:04:32:00
        nd6 options=1<PERFORMNUD>
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
        member: epair0a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 19 priority 128 path cost 2000
        member: igb0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 1 priority 128 path cost 20000
epair0a: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=8<VLAN_MTU>
        ether 02:b1:94:00:13:0a
        nd6 options=1<PERFORMNUD>
        media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
        status: active

After disconnect:

Code:

[browner87@freenas ~]$ sleep 10 && ifconfig
igb0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=400b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWTSO>
        ether 00:25:90:67:e7:62
        inet 192.168.5.220 netmask 0xffffff00 broadcast 192.168.5.255
        nd6 options=9<PERFORMNUD,IFDISABLED>
        media: Ethernet autoselect
        status: no carrier
igb1: flags=8c02<BROADCAST,OACTIVE,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO>
        ether 00:25:90:67:e7:63
        nd6 options=9<PERFORMNUD,IFDISABLED>
        media: Ethernet autoselect
        status: no carrier
em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=4219b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,WOL_MAGIC,VLAN_HWTSO>
        ether 00:1b:21:9d:2d:7d
        inet 192.168.10.10 netmask 0xfffffffc broadcast 192.168.10.11
        nd6 options=9<PERFORMNUD,IFDISABLED>
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
em1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=4019b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,VLAN_HWTSO>
        ether 00:1b:21:9d:2d:7c
        inet 192.168.10.5 netmask 0xfffffffc broadcast 192.168.10.7
        nd6 options=9<PERFORMNUD,IFDISABLED>
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
em2: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=4019b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,VLAN_HWTSO>
        ether 00:1b:21:9d:2d:7f
        nd6 options=9<PERFORMNUD,IFDISABLED>
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
em3: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=4019b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,VLAN_HWTSO>
        ether 00:1b:21:9d:2d:7f
        nd6 options=9<PERFORMNUD,IFDISABLED>
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
ipfw0: flags=8801<UP,SIMPLEX,MULTICAST> metric 0 mtu 65536
        nd6 options=9<PERFORMNUD,IFDISABLED>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x10
        inet 127.0.0.1 netmask 0xff000000
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
lagg0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=4019b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,VLAN_HWTSO>
        ether 00:1b:21:9d:2d:7f
        inet 192.168.5.230 netmask 0xffffff00 broadcast 192.168.5.255
        nd6 options=9<PERFORMNUD,IFDISABLED>
        media: Ethernet autoselect
        status: active
        laggproto lacp lagghash l2,l3,l4
        laggport: em3 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
        laggport: em2 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        ether 02:48:4a:04:32:00
        nd6 options=1<PERFORMNUD>
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
        member: epair0a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 19 priority 128 path cost 2000
        member: igb0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 1 priority 128 path cost 20000
epair0a: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=8<VLAN_MTU>
        ether 02:b1:94:00:13:0a
        nd6 options=1<PERFORMNUD>
        media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
        status: active

dlavigne · Mar 3, 2016

Were you able to figure this out?

Browner87 · Mar 3, 2016

Nope, never got it. I can't fathom why it seems to work until igb0 is disconnected. Unless there is magic secret behind-the-scenes bridging going on.

m0nkey_ · Mar 3, 2016

I think I know why. On Supermicro boards the IPMI is bonded with the Intel NIC. I had the same issue on an ASRock when I wanted to use link aggregation. You need to break the bond in the IPMI and set an IP on eth1. That should allow it to work.

Browner87 · Mar 3, 2016

I'm 99.9% sure eth1 isn't IPMI. My board has 1 IPMI NIC, 2 normal NICs, and a 4-port NIC expansion card.

m0nkey_ · Mar 4, 2016

When I say eth1, that is within the IPMI configuration, not FreeNAS. In my IPMI I had a 'Network Bonding' configuration item. This is what I had to disable in order for link aggregation to work in FreeNAS. I had the exact same symptoms when I connected the second NIC it would lose connectivity.

Browner87 · Mar 4, 2016

Hmmm, the closest thing I can find to this in the IPMI is "LAN Select Setting", and it's disabled. Which I believe means only use the dedicated IPMI port for IPMI. The only stuff under Networking is DHCP and VLAN. :(

Important Announcement for the TrueNAS Community.

LACP on em2/3 only works when igb0 is connected?

Browner87

Cadet

m0nkey_

MVP

Browner87

Cadet

dlavigne

Guest

Browner87

Cadet

dlavigne

Guest

Browner87

Cadet

m0nkey_

MVP

Browner87

Cadet

m0nkey_

MVP

Browner87

Cadet

Similar threads