Lagg members lost after shutdown

[proto]

Hi,
currently I have a LAGG with 6 igb attached to an HP 1920 switch.
2 nics form Supermicro X10sl7 - igb[4-5]
4 nics from a quad eth HP NC365T - igb[0-3]
Usually it's working until I shutdown the server for maintenance tasks.

As soon the server is up only the mobo nics are <ACTIVE,COLLECTING,DISTRIBUTING> while the other 4 seems down.

A simple restart fix the issue and LAGG is working as expected:

Code:

lagg0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=6403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
    ether f4:ce:46:a7:b7:1c
    nd6 options=9<PERFORMNUD,IFDISABLED>
    media: Ethernet autoselect
    status: active
    groups: lagg
    laggproto lacp lagghash l2,l3,l4
    laggport: igb0 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
    laggport: igb1 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
    laggport: igb2 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
    laggport: igb3 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
    laggport: igb4 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
    laggport: igb5 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>

What else should I check?

 

[dlavigne]

Were you able to determine the cause for this?

 

[proto]

unfortunately no.
I should unplug that HP quad nic and test on another server, soon or later.

 

[proto]

Finally I gave up!
I can't test the nic on another hardware as I wanted due to pci lane incompatibility.
So I plugged an "old" dual port hp N360T (Intel 82571EB) and seems working.

But that quad should work under any recent FreeBSD.

 

[Elliot Dierksen]

What does your switch side config look look? I doubt this is a hardware issue. It sounds more like a config mismatch issue between the switch and the host (FreeNAS). I also didn't see where you mentioned what kind of LAGG is configured. If you are using LACP (which would be my recommendation), you also need some configuration on the switch side.

 

[proto]

What does your switch side config look look?

Argh! True, I didn't show it. Sorry!!!

I confirm: it's LACP. I pulled my quad nic yesterday, but I didn't touch my switch conf. You will see two failing link, now instead of four.

Here is my HP v1920 switch conf. In brief: I configured a TRUNK pvid 1001, then the two VLANs for each downstream clients. VLAN10 is my production VLAN and FreeNAS admin interface is configured on that network.
VMs and Jails get their ip from VLAN10 and VLAN20 and its working without issue.
NO ip on LAGG, NO ip on VLAN20.

Notes:
1. My VMs and Jails are set to auto-start @ boot-time. Yesterday, just before swapping the nics I did a last test: set guests auto-start OFF, shutdown the server and start again. I performed this test a couple of times: only once I had all the quad active. Again, if I restart the server the issue disappear, and the quad is working properly, even if my VMs and Jails are set to auto power on.
2. One thing that I forgot to mention: a few months ago, before moving to version 11.x, I installed VMware 6.7, and I realized that the quad wasn't working right: I wanted to virtualize FreeNAS. Then I gave up just because the NIC wasn't in HCL for that version of ESX, and said bye to virtualization...
3. Same LAGG/LACP conf worked on FreeNAS 9.x

I cannot test on other hardware now. Google showed me only old issues on FreeBSD 8 (MSI-X interrupts) and a recent one on netmap (https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=226289); neither seems related to my nic.


FreeNAS nic conf / actual conf:

Code:

em0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=98<VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM>
    ether 0c:c4:7a:32:ce:a6
    hwaddr 00:26:55:ed:6d:9c
    nd6 options=9<PERFORMNUD,IFDISABLED>
    media: Ethernet autoselect (1000baseT <full-duplex>)
    status: active
em1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=98<VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM>
    ether 0c:c4:7a:32:ce:a6
    hwaddr 00:26:55:ed:6d:9d
    nd6 options=9<PERFORMNUD,IFDISABLED>
    media: Ethernet autoselect (1000baseT <full-duplex>)
    status: active
igb0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM>
    ether 0c:c4:7a:32:ce:a6
    hwaddr 0c:c4:7a:32:ce:a6
    nd6 options=9<PERFORMNUD,IFDISABLED>
    media: Ethernet autoselect (1000baseT <full-duplex>)
    status: active
igb1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM>
    ether 0c:c4:7a:32:ce:a6
    hwaddr 0c:c4:7a:32:ce:a7
    nd6 options=9<PERFORMNUD,IFDISABLED>
    media: Ethernet autoselect (1000baseT <full-duplex>)
    status: active
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
    options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
    inet6 ::1 prefixlen 128
    inet6 fe80::1%lo0 prefixlen 64 scopeid 0x5
    inet 127.0.0.1 netmask 0xff000000
    nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
    groups: lo
lagg0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=98<VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM>
    ether 0c:c4:7a:32:ce:a6
    nd6 options=9<PERFORMNUD,IFDISABLED>
    media: Ethernet autoselect
    status: active
    groups: lagg
    laggproto lacp lagghash l2,l3,l4
    laggport: em0 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
    laggport: em1 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
    laggport: igb0 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
    laggport: igb1 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
vlan10: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    ether 0c:c4:7a:32:ce:a6
    inet 10.10.10.10 netmask 0xffffffc0 broadcast 10.10.10.63
    nd6 options=9<PERFORMNUD,IFDISABLED>
    media: Ethernet autoselect
    status: active
    vlan: 10 vlanpcp: 0 parent interface: lagg0
    groups: vlan
vlan20: flags=8842<BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    ether 0c:c4:7a:32:ce:a6
    nd6 options=9<PERFORMNUD,IFDISABLED>
    media: Ethernet autoselect
    status: active
    vlan: 20 vlanpcp: 0 parent interface: lagg0
    groups: vlan
tap0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: Attached to db00
    options=80000<LINKSTATE>
    ether 00:bd:99:81:d6:00
    hwaddr 00:bd:99:81:d6:00
    nd6 options=1<PERFORMNUD>
    media: Ethernet autoselect
    status: active
    groups: tap
    Opened by PID 44483
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    ether 02:0d:28:a5:35:00
    nd6 options=1<PERFORMNUD>
    groups: bridge
    id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
    maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
    root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
    member: tap1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 11 priority 128 path cost 2000000
    member: vlan10 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 7 priority 128 path cost 2000000
    member: tap0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 9 priority 128 path cost 2000000
tap1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: Attached to ns0
    options=80000<LINKSTATE>
    ether 00:bd:e0:fc:d6:01
    hwaddr 00:bd:e0:fc:d6:01
    nd6 options=1<PERFORMNUD>
    media: Ethernet autoselect
    status: active
    groups: tap
    Opened by PID 44552

Bagg1:

Code:

[SW0]display interface Bridge-Aggregation brief
The brief information of interface(s) under bridge mode:
Link: ADM - administratively down; Stby - standby
Speed or Duplex: (a)/A - auto; H - half; F - full
Type: A - access; T - trunk; H - hybrid
Interface            Link Speed   Duplex Type PVID Description
BAGG1                UP   4G(a)   F(a)   T    1001

Code:

[SW0]display interface Bridge-Aggregation
 Bridge-Aggregation1 current state: UP
 IP Packet Frame Type: PKTFMT_ETHNT_2, Hardware Address: d894-032b-9436
 Description: Bridge-Aggregation1 Interface
 4Gbps-speed mode, full-duplex mode
 Link speed type is autonegotiation, link duplex type is autonegotiation
 PVID: 1001
 Port link-type: trunk
  VLAN passing  : 10, 20
  VLAN permitted: 10, 20
  Trunk port encapsulation: IEEE 802.1q
 Last clearing of counters:  Never
 Last 300 seconds input:  0 packets/sec 61 bytes/sec    0%
 Last 300 seconds output:  0 packets/sec 231 bytes/sec    0%
 Input (total):  7470551 packets, 2000248620 bytes
     6472281 unicasts, 9079 broadcasts, 989191 multicasts
 Input (normal):  7470551 packets, 2000248620 bytes
     6472281 unicasts, 9079 broadcasts, 989191 multicasts
 Input:  0 input errors, 0 runts, - giants, - throttles
     0 CRC, - frame, 0 overruns, 0 aborts
     - ignored, - parity errors
 Output (total): 16717215 packets, 6878087375 bytes
     8063589 unicasts, 5222777 broadcasts, 3430849 multicasts, 0 pauses
 Output (normal): 16717215 packets, 6878087375 bytes
         8063589 unicasts, 5222777 broadcasts, 3430849 multicasts, 0 pauses
 Output: 0 output errors, - underruns, - buffer failures
     0 aborts, 0 deferred, 0 collisions, - late collisions
     - lost carrier, - no carrier

Lagg members:

Code:

SW0]display link-aggregation verbose
Loadsharing Type: Shar -- Loadsharing, NonS -- Non-Loadsharing
Port Status: S -- Selected, U -- Unselected
Flags:  A -- LACP_Activity, B -- LACP_Timeout, C -- Aggregation,
        D -- Synchronization, E -- Collecting, F -- Distributing,
        G -- Defaulted, H -- Expired

Aggregation Interface: Bridge-Aggregation1
Aggregation Mode: Dynamic
Loadsharing Type: Shar
System ID: 0x0, d894-032b-9420
Local:
  Port             Status  Priority Oper-Key  Flag
--------------------------------------------------------------------------------
  GE1/0/21         S       32768    1         {ACDEF}
  GE1/0/23         S       32768    1         {ACDEF}
  GE1/0/25         S       32768    1         {ACDEF}
  GE1/0/27         S       32768    1         {ACDEF}
  GE1/0/29         U       32768    1         {ACG}
  GE1/0/31         U       32768    1         {ACG}
Remote:
  Actor            Partner Priority Oper-Key  SystemID               Flag
--------------------------------------------------------------------------------
  GE1/0/21         3       32768    203       0x8000, 0cc4-7a32-cea6 {ACDEF}
  GE1/0/23         4       32768    203       0x8000, 0cc4-7a32-cea6 {ACDEF}
  GE1/0/25         2       32768    203       0x8000, 0cc4-7a32-cea6 {ACDEF}
  GE1/0/27         1       32768    203       0x8000, 0cc4-7a32-cea6 {ACDEF}
  GE1/0/29         0       32768    0         0x8000, 0000-0000-0000 {EF}
  GE1/0/31         0       32768    0         0x8000, 0000-0000-0000 {EF}

Code:

[SW0]display link-aggregation member-port
Flags: A -- LACP_Activity, B -- LACP_Timeout, C -- Aggregation,
       D -- Synchronization, E -- Collecting, F -- Distributing,
       G -- Defaulted, H -- Expired

GigabitEthernet1/0/21:
Aggregation Interface: Bridge-Aggregation1
Local:
    Port Number: 21
    Port Priority: 32768
    Oper-Key: 1
    Flag: {ACDEF}
Remote:
    System ID: 0x8000, 0cc4-7a32-cea6
    Port Number: 3
    Port Priority: 32768
    Oper-Key: 203
    Flag: {ACDEF}
Received LACP Packets: 165432 packet(s)
Illegal: 0 packet(s)
Sent LACP Packets: 168908 packet(s)

GigabitEthernet1/0/23:
Aggregation Interface: Bridge-Aggregation1
Local:
    Port Number: 23
    Port Priority: 32768
    Oper-Key: 1
    Flag: {ACDEF}
Remote:
    System ID: 0x8000, 0cc4-7a32-cea6
    Port Number: 4
    Port Priority: 32768
    Oper-Key: 203
    Flag: {ACDEF}
Received LACP Packets: 165414 packet(s)
Illegal: 0 packet(s)
Sent LACP Packets: 168851 packet(s)

GigabitEthernet1/0/25:
Aggregation Interface: Bridge-Aggregation1
Local:
    Port Number: 25
    Port Priority: 32768
    Oper-Key: 1
    Flag: {ACDEF}
Remote:
    System ID: 0x8000, 0cc4-7a32-cea6
    Port Number: 2
    Port Priority: 32768
    Oper-Key: 203
    Flag: {ACDEF}
Received LACP Packets: 165398 packet(s)
Illegal: 6 packet(s)
Sent LACP Packets: 174442 packet(s)

GigabitEthernet1/0/27:
Aggregation Interface: Bridge-Aggregation1
Local:
    Port Number: 27
    Port Priority: 32768
    Oper-Key: 1
    Flag: {ACDEF}
Remote:
    System ID: 0x8000, 0cc4-7a32-cea6
    Port Number: 1
    Port Priority: 32768
    Oper-Key: 203
    Flag: {ACDEF}
Received LACP Packets: 159912 packet(s)
Illegal: 0 packet(s)
Sent LACP Packets: 168821 packet(s)

GigabitEthernet1/0/29:
Aggregation Interface: Bridge-Aggregation1
Local:
    Port Number: 29
    Port Priority: 32768
    Oper-Key: 1
    Flag: {ACG}
Remote:
    System ID: 0x8000, 0000-0000-0000
    Port Number: 0
    Port Priority: 32768
    Oper-Key: 0
    Flag: {EF}
Received LACP Packets: 160922 packet(s)
Illegal: 0 packet(s)
Sent LACP Packets: 169550 packet(s)

GigabitEthernet1/0/31:
Aggregation Interface: Bridge-Aggregation1
Local:
    Port Number: 31
    Port Priority: 32768
    Oper-Key: 1
    Flag: {ACG}
Remote:
    System ID: 0x8000, 0000-0000-0000
    Port Number: 0
    Port Priority: 32768
    Oper-Key: 0
    Flag: {EF}
Received LACP Packets: 160922 packet(s)
Illegal: 0 packet(s)
Sent LACP Packets: 169550 packet(s)

Code:

Aggregation Interface Type:
BAGG -- Bridge-Aggregation, RAGG -- Route-Aggregation
Aggregation Mode: S -- Static, D -- Dynamic
Loadsharing Type: Shar -- Loadsharing, NonS -- Non-Loadsharing
Actor System ID: 0x0, d894-032b-9420

AGG         AGG       Partner ID               Select Unselect   Share
Interface   Mode                               Ports  Ports      Type
-------------------------------------------------------------------------------
BAGG1       D         0x8000, 0cc4-7a32-cea6   4      2          Shar

LACP:

Code:

[SW0]display lacp system-id
Actor System ID: 0x0, d894-032b-9420

 

[Elliot Dierksen]

Argh! True, I didn't show it. Sorry!!!

I am a bit suspicious of the PVID of 1001. That VLAN is usually reserved for translation bridging. Stuff you never see any more like ethernet to token ring, ethernet to FDDI, etc. Is your intent there to not use the native/untagged VLAN? If so, why? There isn't necessarily anything wrong with that, but I am curious.

The switch gives very similar output to mine which is really an H3C switch with an HP label on it. Here is what the BAGG looks like on my switch side.

Code:

interface Bridge-Aggregation1
description FreeNAS
port access vlan 252
link-aggregation mode dynamic
link-aggregation load-sharing mode source-mac

Do pay some heed to the load-sharing command. Most switches will normally load balance based on the destination. That does you no good since that FreeNAS is the only destination, so you have load balance by source to get any kind of use from the additional links.

Here are the BAGG members.

Code:

interface Ten-GigabitEthernet1/0/13
port link-mode bridge
description FreeNAS - P1
port access vlan 252
flow-control
flow-interval 60
port link-aggregation group 1
#
interface Ten-GigabitEthernet1/0/14
port link-mode bridge
description FreeNAS - P2
port access vlan 252
flow-control
flow-interval 60
port link-aggregation group 1

My scenario is a little different that yours because the 10G links for FreeNAS are only on the storage network. The key thing is that all of the non-BAGG related stuff must match between the BAGG and the physical links.

I hope that helps.

 

[proto]

There isn't necessarily anything wrong with that, but I am curious.

Well, it's something I could call "hystoric"... I had all my stuff in a datacenter and my network technician told me not to use VLAN1, and he configured all the switches to use TRUNKs with separate PVID. I trusted him.

Actually I cannot have benefits from 4/6 links; it's true since there are still no demanding clients. I'm still using an "old" redundant ISCSI SAN to connect 3 ESX hosts on a physical separate network. I'd like to kill that oven and have FreeNAS serving the virtual environment via NFS.

Anyway... my switch sees the quad link down.

 

[Elliot Dierksen]

Well, it's something I could call "hystoric"... I had all my stuff in a datacenter and my network technician told me not to use VLAN1, and he configured all the switches to use TRUNKs with separate PVID. I trusted him.

That isn't a bad recommendation. I will recommend the same thing, particularly with device in a less trusted security area.

Anyway... my switch sees the quad link down.

Would you mind including the output of a display current-configuration? It just seems like there is some kind of mis-match in the switch config, or that is where my gut is leaning.

 

[proto]

That isn't a bad recommendation. I will recommend the same thing, particularly with device in a less trusted security area.

Would you mind including the output of a display current-configuration? It just seems like there is some kind of mis-match in the switch config, or that is where my gut is leaning.

Now I begin to see! Ah, I have set up trunk on all bridge interfaces, while you have access, but you need just storage, while I have a mix of storage vlans and bhyve vms/jails.

Bridge ports are: 21 23 25 27 29 31.

This is my trunk conf :

Code:

interface Bridge-Aggregation1
 port link-type trunk
 undo port trunk permit vlan 1
 port trunk permit vlan 10 20
 port trunk pvid vlan 1001
 link-aggregation mode dynamic
 stp disable
 stp loop-protection
 
 [...]
 
 interface GigabitEthernet1/0/21
 port link-type trunk
 undo port trunk permit vlan 1
 port trunk permit vlan 10 20
 port trunk pvid vlan 1001
 speed auto 1000
 loopback-detection enable
 port auto-power-down
 stp disable
 stp loop-protection
 port link-aggregation group 1

And this is my current hybrid conf:

Code:

interface Bridge-Aggregation1
 port link-type hybrid
 undo port hybrid vlan 1
 port hybrid vlan 10 20 tagged
 port hybrid vlan 1001 untagged
 port hybrid pvid vlan 1001
 link-aggregation mode dynamic
 
 [...]
 
 interface GigabitEthernet1/0/21
 port link-type hybrid
 undo port hybrid vlan 1
 port hybrid vlan 10 20 tagged
 port hybrid vlan 1001 untagged
 port hybrid pvid vlan 1001
 speed auto 1000
 loopback-detection enable
 stp disable
 stp loop-protection
 port link-aggregation group 1

Full configurations are attached.
Don't ask my why I set up trunk ports all over the links: I don't remember really! But maybe it is always vmware related.

I forgot this:

Code:

<SW0>display link-aggregation load-sharing mode
Link-Aggregation Load-Sharing Mode:
Layer 2 traffic: destination-mac address, source-mac address

I've found a server with a 4-lane pci-e so I can test again that quad in the next few days.

 

[Elliot Dierksen]

I think I have an idea where the problem is. The BAGG link is configured as a link type hybrid. That is normally what you would use if you had some kind of IP where the phone sent traffic in a single tagged VLAN, and the workstation behind it sends it frames in the untagged VLAN. Your BAGG looks like this:

Code:

interface Bridge-Aggregation1
 port link-type hybrid
 undo port hybrid vlan 1
 port hybrid vlan 10 20 tagged
 port hybrid vlan 1001 untagged
 port hybrid pvid vlan 1001
 link-aggregation mode dynamic

I think it should look like this:

Code:

interface Bridge-Aggregation1
 port link-type trunk
 port trunk permit vlan 10 20
 link-aggregation mode dynamic
 link-aggregation load-sharing mode source-mac

I am also a little puzzled by this command:

Code:

lacp system-priority 0

The way you describe the problem, it sounds like the links come up physically but the LACP negotiation in not happening. I wonder (but don't know for sure) if setting that value to 0 causes the switch to not particiapte in LACP negotiations. I would suggest leaving it at the default value of 32768 unless there is some reason of which I am not aware to change it. At the very least, 0 seems like a bad option. You may have to edit both the BAGG as the physical interfaces to make those port type changes.

 

[proto]

The BAGG link is configured as a link type hybrid.

Ah, wait Elliot. I reconfigured this morning (CEST time!) that switch to test LACP in hybrid mode, and it's working "exactly" as in trunk mode, my default configuration. Please, take "exactly" as is. I only know that access will never work here.

At the very least, 0 seems like a bad option

And this sounds strange, I didn't pay attention to that!
On web console Port Priority is set (automatically) to the correct value:

But it should not a problem: console access on that switch is a hack, I usually manage via web interface and use console just to get information/debug instead of screenshots.

Tomorrow I'll try to test the quad, in trunk and hybrid mode and will report.

Thanks for your time!

 

[Elliot Dierksen]

Ah, wait Elliot. I reconfigured this morning (CEST time!) that switch to test LACP in hybrid mode, and it's working "exactly" as in trunk mode, my default configuration. Please, take "exactly" as is. I only know that access will never work here.

That is a slightly different switch model than the one I have, perhaps that is correct for that family. I am primarily a Cisco guy, and have just felt my way through enough of the H3C/HP stuff to get done what I needed.

And this sounds strange, I didn't pay attention to that!

The fact that it is showing LACP negotiations makes me think that 0 doesn't disable LACP. It just seemed a little odd to me.

Tomorrow I'll try to test the quad, in trunk and hybrid mode and will report.

Sounds good. I hope it works!

 

[proto]

I am primarily a Cisco guy

Ah, yes. I know there are different terms between Cisco and HP. I don't know why.

Anyway... this is my test report.

On my dell t605 / quad eth NC365T
Note: this server is a bad one to deal with: it has a PERC6 ctrl, it's loud, cannot "smart" hdds... so I just use to test that NIC.

lagg / lacp is configured on both FreeNAS and switch.

igbX are the only interfaces configured.
Other NICs:
bce0 is not configured
emX are not configured

Code:

lagg0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=6403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
    ether f4:ce:46:a7:b7:1c
    nd6 options=9<PERFORMNUD,IFDISABLED>
    media: Ethernet autoselect
    status: active
    groups: lagg
    laggproto lacp lagghash l2,l3,l4
    laggport: igb0 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
    laggport: igb1 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
    laggport: igb2 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
    laggport: igb3 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>

Admin interface on vlan10 / parent is lagg0 / DHCP

Code:

vlan10: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=600303<RXCSUM,TXCSUM,TSO4,TSO6,RXCSUM_IPV6,TXCSUM_IPV6>
    ether f4:ce:46:a7:b7:1c
    inet 10.10.10.61 netmask 0xffffffc0 broadcast 10.10.10.63
    nd6 options=9<PERFORMNUD,IFDISABLED>
    media: Ethernet autoselect
    status: active
    vlan: 10 vlanpcp: 0 parent interface: lagg0
    groups: vlan

Test 1 - PASSED
Rebooting the server: all links are: <ACTIVE,COLLECTING,DISTRIBUTING>

Test 2 - PASSED
Shutdown the server.
Start the server.
all links are: <ACTIVE,COLLECTING,DISTRIBUTING>

Test3 - PASSED
Configure a jail with networking.
Shutdown the server. Disconnect power cord (power failure simulation)
Start the server.
all links are: <ACTIVE,COLLECTING,DISTRIBUTING>

Test 1->3 executed both with my HP v1920 switch in hybrid and trunk ports mode.

I tried to simulate a port shutdown via switch dumping lacp packets via tcpdump:
tcpdump -i lagg0 ether proto 0x8809 -w t605_lagg0_lacp_only.pcap
All it's working as expected: affected igbX links are down on FreeNAS and defaulted on switch.

So, I think it's an hardware compatibility problem between my Supermicro Mobo X10SL7 and that particular quad HP branded nic or my mobo is dying...

 

[Elliot Dierksen]

So, I think it's an hardware compatibility problem between my Supermicro Mobo X10SL7 and that particular quad HP branded nic or my mobo is dying...

I suppose that is possible. It certainly didn't sound like you were trying to do anything that was way out there. Glad to hear it is working.