LACP/LAGG Issues with 9.3

[Elliot Dierksen]

I am running FreeNAS-9.3-STABLE-201412240734 on an HP DL380 G5. I recently added the following 10G NIC to increase performance.

bxe0@pci0:11:0:0: class=0x020000 card=0x121314e4 chip=0x166214e4 rev=0x01
hdr=0x00
vendor = 'Broadcom Corporation'
device = 'NetXtreme II BCM57712 10 Gigabit Ethernet'
class = network
subclass = ethernet
bxe1@pci0:11:0:1: class=0x020000 card=0x121314e4 chip=0x166214e4 rev=0x01
hdr=0x00
vendor = 'Broadcom Corporation'
device = 'NetXtreme II BCM57712 10 Gigabit Ethernet'
class = network
subclass = ethernet

These NIC's connect to the following switch:

HP Comware Platform Software
Comware Software, Version 5.20.105, Release 1809P01
Copyright (c) 2010-2014 Hewlett-Packard Development Company, L.P.
HP 5820X-14XG-SFP+ Switch with 2 Interface Slots uptime is 0 week, 1 day, 15 hours, 16 minutes

HP 5820X-14XG-SFP+ Switch with 2 Interface Slots with 2 Processors
1024M bytes SDRAM
4M bytes Nor Flash Memory
512M bytes Nand Flash Memory
Config Register points to Nand Flash

Hardware Version is Ver.B
CPLDA Version is 003, CPLDB Version is 003
BootRom Version is 301
[SubSlot 0] 14SFP Plus+4GE Hardware Version is Ver.B
[SubSlot 1] No Module
[SubSlot 2] No Module
[SubSlot 3] No Module


This server also has 4 gigabit ports as follows:

em0@pci0:6:0:0: class=0x020000 card=0x7044103c chip=0x105e8086 rev=0x06 hdr=0x00
vendor = 'Intel Corporation'
device = '82571EB Gigabit Ethernet Controller'
class = network
subclass = ethernet
em1@pci0:6:0:1: class=0x020000 card=0x7044103c chip=0x105e8086 rev=0x06 hdr=0x00
vendor = 'Intel Corporation'
device = '82571EB Gigabit Ethernet Controller'
class = network
subclass = ethernet
bce0@pci0:3:0:0: class=0x020000 card=0x7038103c chip=0x164c14e4 rev=0x12
hdr=0x00
vendor = 'Broadcom Corporation'
device = 'NetXtreme II BCM5708 Gigabit Ethernet'
class = network
subclass = ethernet
bce1@pci0:5:0:0: class=0x020000 card=0x7038103c chip=0x164c14e4 rev=0x12
hdr=0x00
vendor = 'Broadcom Corporation'
device = 'NetXtreme II BCM5708 Gigabit Ethernet'
class = network
subclass = ethernet

These NIC's connect via an LACP LAG to this switch:

HP ProCurve 2848
Image stamp: /sw/code/build/mako
Sep 24 2013 16:28:07
I.10.103
1674
Boot Image: Primary


The twisted pair NIC's are all configured as an LACP LAG, and that works fine. I attempted to configure the two 10G ports as a LAG, but I cannot get that to work. The switch reports that it sees the successful LACP neogtiation, and I think it even learned a MAC address, but FreeNAS reports the LAG interface as down. I tried a number of combinations, but nothing was successful. I can get it to work when I configure it as static and the switch side and FEC on the FreeNAS side. I would really prefer to have LACP work. I am not very familiar with the CommWorks (H2C) flavor of switch, but I can gather any other information that might be relevant. Any suggestions?

 

[Elliot Dierksen]

I have some more information from the 10G switch side. Here is what the switch logs when FreeNAS boots up and brings up the LAGG

Dec 28 08:49:50 192.168.253.40 2014 HP-5820X-14XG %%10IFNET/3/LINK_UPDOWN(l): Ten-GigabitEthernet1/0/14 link status is UP.
Dec 28 08:49:51 192.168.253.40 2014 HP-5820X-14XG %%10IFNET/3/LINK_UPDOWN(l): Ten-GigabitEthernet1/0/13 link status is UP.
Dec 28 08:49:56 192.168.253.40 2014 HP-5820X-14XG %%10LAGG/5/LAGG_ACTIVE(l): Member port Ten-GigabitEthernet1/0/13 of aggregation group BAGG1 becomes ACTIVE.
Dec 28 08:49:57 192.168.253.40 2014 HP-5820X-14XG %%10LAGG/5/LAGG_ACTIVE(l): Member port Ten-GigabitEthernet1/0/14 of aggregation group BAGG1 becomes ACTIVE.

And here is what I can extract from the FreeNAS logs around the same time regarding the relevant innterfaces.

Dec 28 08:44:13 freenas bxe0: <QLogic NetXtreme II BCM57712 10GbE (A1) BXE v:1.78.78
Dec 28 08:44:13 freenas bxe0: PCI BAR0 [10] memory allocated: 0xf7000000-0xf77fffff (8388608) -> 0xfffffe00f7000000
Dec 28 08:44:13 freenas bxe0: PCI BAR2 [18] memory allocated: 0xf6800000-0xf6ffffff (8388608) -> 0xfffffe00f6800000
Dec 28 08:44:13 freenas bxe0: PCI BAR4 [20] memory allocated: 0xf67f0000-0xf67fffff (65536) -> 0xfffffe00f67f0000
Dec 28 08:44:13 freenas bxe0: Media not present.
Dec 28 08:44:13 freenas bxe0: Ethernet address: 00:10:18:af:f6:3c
Dec 28 08:44:13 freenas bxe0: MSI-X vectors Requested 5 and Allocated 5
Dec 28 08:44:13 freenas bxe1: <QLogic NetXtreme II BCM57712 10GbE (A1) BXE v:1.78.78
Dec 28 08:44:13 freenas bxe1: PCI BAR0 [10] memory allocated: 0xf5800000-0xf5ffffff (8388608) -> 0xfffffe00f5800000
Dec 28 08:44:13 freenas bxe1: PCI BAR2 [18] memory allocated: 0xf5000000-0xf57fffff (8388608) -> 0xfffffe00f5000000
Dec 28 08:44:13 freenas bxe1: PCI BAR4 [20] memory allocated: 0xf4ff0000-0xf4ffffff (65536) -> 0xfffffe00f4ff0000
Dec 28 08:44:13 freenas bxe1: Media not present.
Dec 28 08:44:13 freenas bxe1: Ethernet address: 00:10:18:af:f6:3e
Dec 28 08:44:13 freenas bxe1: MSI-X vectors Requested 5 and Allocated 5
Dec 28 08:44:13 freenas bxe0: NIC Link is Down
Dec 28 08:44:13 freenas bxe0: NIC Link is Up, 10000 Mbps full duplex, Flow control: ON - receive & transmit
Dec 28 08:44:13 freenas bxe1: NIC Link is Down
Dec 28 08:44:13 freenas bxe1: NIC Link is Up, 10000 Mbps full duplex, Flow control: ON - receive & transmit
Dec 28 08:44:13 freenas bxe1: 2 link states coalesced
Dec 28 08:45:52 freenas notifier: Starting Network: lo0 bxe0 bxe1 em0 em1 bce0 bce1 ipfw0 lagg0 lagg1 vlan1.
Dec 28 08:45:52 freenas notifier: bxe0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
Dec 28 08:45:52 freenas notifier: bxe1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
Dec 28 08:45:52 freenas notifier: lagg1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
Dec 28 08:45:52 freenas notifier: laggport: bxe1 flags=0<>
Dec 28 08:45:52 freenas notifier: laggport: bxe0 flags=0<>
Dec 28 08:50:11 freenas bxe0: <QLogic NetXtreme II BCM57712 10GbE (A1) BXE v:1.78.78
Dec 28 08:50:11 freenas bxe0: PCI BAR0 [10] memory allocated: 0xf7000000-0xf77fffff (8388608) -> 0xfffffe00f7000000
Dec 28 08:50:11 freenas bxe0: PCI BAR2 [18] memory allocated: 0xf6800000-0xf6ffffff (8388608) -> 0xfffffe00f6800000
Dec 28 08:50:11 freenas bxe0: PCI BAR4 [20] memory allocated: 0xf67f0000-0xf67fffff (65536) -> 0xfffffe00f67f0000
Dec 28 08:50:11 freenas bxe0: Media not present.
Dec 28 08:50:11 freenas bxe0: Ethernet address: 00:10:18:af:f6:3c
Dec 28 08:50:11 freenas bxe0: MSI-X vectors Requested 5 and Allocated 5
Dec 28 08:50:11 freenas bxe1: <QLogic NetXtreme II BCM57712 10GbE (A1) BXE v:1.78.78
Dec 28 08:50:11 freenas bxe1: PCI BAR0 [10] memory allocated: 0xf5800000-0xf5ffffff (8388608) -> 0xfffffe00f5800000
Dec 28 08:50:11 freenas bxe1: PCI BAR2 [18] memory allocated: 0xf5000000-0xf57fffff (8388608) -> 0xfffffe00f5000000
Dec 28 08:50:11 freenas bxe1: PCI BAR4 [20] memory allocated: 0xf4ff0000-0xf4ffffff (65536) -> 0xfffffe00f4ff0000
Dec 28 08:50:11 freenas bxe1: Media not present.
Dec 28 08:50:11 freenas bxe1: Ethernet address: 00:10:18:af:f6:3e
Dec 28 08:50:11 freenas bxe1: MSI-X vectors Requested 5 and Allocated 5
Dec 28 08:50:11 freenas bxe0: NIC Link is Down
Dec 28 08:50:11 freenas bxe0: NIC Link is Up, 10000 Mbps full duplex, Flow control: ON - receive & transmit
Dec 28 08:50:11 freenas bxe1: NIC Link is Down
Dec 28 08:50:11 freenas bxe1: NIC Link is Up, 10000 Mbps full duplex, Flow control: ON - receive & transmit
Dec 28 08:50:11 freenas bxe1: 2 link states coalesced

 

[dlavigne]

Do any of the sysctls mentioned in https://www.freebsd.org/cgi/man.cgi...=FreeBSD+9.3-RELEASE&arch=default&format=html help?

 

[Elliot Dierksen]

Do any of the sysctls mentioned in https://www.freebsd.org/cgi/man.cgi?query=bxe&apropos=0&sektion=0&manpath=FreeBSD 9.3-RELEASE&arch=default&format=html help?

I looked at those, but I don't see anything that would help. Again, this works fine when I define it as a static/FEC LAGG. It doesn't work with LACP, which makes me think it isn't part of the ifconfig/sysctl options. If there is something I am missing, please tell me, but I don't see it.

 

[Elliot Dierksen]

This appears to be an issue with the BXN driver. I replaced the 10G NIC with a Chelsio T3 based card, and now it works fine.

cxgbc0@pci0:23:0:0: class=0x020000 card=0x00011425 chip=0x00311425 rev=0x00
hdr=0x00
vendor = 'Chelsio Communications Inc'
device = 'T320 10GbE Dual Port Adapter'
class = network
subclass = ethernet

For what it is worth, enabling bi-directional flow control on the 10G switch (HP 5820) seemed to help performance.

 

[cyberjock]

Now you know why we don't recommend the Broadcom. :)