What does your switch side config look look?
Argh! True, I didn't show it. Sorry!!!
I confirm: it's LACP. I pulled my quad nic yesterday, but I didn't touch my switch conf. You will see two failing link, now instead of four.
Here is my HP v1920 switch conf. In brief: I configured a TRUNK pvid 1001, then the two VLANs for each downstream clients. VLAN10 is my production VLAN and FreeNAS admin interface is configured on that network.
VMs and Jails get their ip from VLAN10 and VLAN20 and its working without issue.
NO ip on LAGG, NO ip on VLAN20.
Notes:
1. My VMs and Jails are set to auto-start @ boot-time. Yesterday, just before swapping the nics I did a last test: set guests auto-start OFF, shutdown the server and start again. I performed this test a couple of times: only once I had all the quad active. Again, if I restart the server the issue disappear, and the quad is working properly, even if my VMs and Jails are set to auto power on.
2. One thing that I forgot to mention: a few months ago, before moving to version 11.x, I installed VMware 6.7, and I realized that the quad wasn't working right: I wanted to virtualize FreeNAS. Then I gave up just because the NIC wasn't in HCL for that version of ESX, and said bye to virtualization...
3. Same LAGG/LACP conf worked on FreeNAS 9.x
I cannot test on other hardware now.
Google showed me only old issues on FreeBSD 8 (MSI-X interrupts) and a recent one on
netmap (
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=226289); neither seems related to my nic.
FreeNAS nic conf / actual conf:
Code:
em0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=98<VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM>
ether 0c:c4:7a:32:ce:a6
hwaddr 00:26:55:ed:6d:9c
nd6 options=9<PERFORMNUD,IFDISABLED>
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
em1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=98<VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM>
ether 0c:c4:7a:32:ce:a6
hwaddr 00:26:55:ed:6d:9d
nd6 options=9<PERFORMNUD,IFDISABLED>
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
igb0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM>
ether 0c:c4:7a:32:ce:a6
hwaddr 0c:c4:7a:32:ce:a6
nd6 options=9<PERFORMNUD,IFDISABLED>
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
igb1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM>
ether 0c:c4:7a:32:ce:a6
hwaddr 0c:c4:7a:32:ce:a7
nd6 options=9<PERFORMNUD,IFDISABLED>
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
inet6 ::1 prefixlen 128
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x5
inet 127.0.0.1 netmask 0xff000000
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
groups: lo
lagg0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=98<VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM>
ether 0c:c4:7a:32:ce:a6
nd6 options=9<PERFORMNUD,IFDISABLED>
media: Ethernet autoselect
status: active
groups: lagg
laggproto lacp lagghash l2,l3,l4
laggport: em0 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
laggport: em1 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
laggport: igb0 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
laggport: igb1 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
vlan10: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
ether 0c:c4:7a:32:ce:a6
inet 10.10.10.10 netmask 0xffffffc0 broadcast 10.10.10.63
nd6 options=9<PERFORMNUD,IFDISABLED>
media: Ethernet autoselect
status: active
vlan: 10 vlanpcp: 0 parent interface: lagg0
groups: vlan
vlan20: flags=8842<BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
ether 0c:c4:7a:32:ce:a6
nd6 options=9<PERFORMNUD,IFDISABLED>
media: Ethernet autoselect
status: active
vlan: 20 vlanpcp: 0 parent interface: lagg0
groups: vlan
tap0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
description: Attached to db00
options=80000<LINKSTATE>
ether 00:bd:99:81:d6:00
hwaddr 00:bd:99:81:d6:00
nd6 options=1<PERFORMNUD>
media: Ethernet autoselect
status: active
groups: tap
Opened by PID 44483
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
ether 02:0d:28:a5:35:00
nd6 options=1<PERFORMNUD>
groups: bridge
id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
member: tap1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 11 priority 128 path cost 2000000
member: vlan10 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 7 priority 128 path cost 2000000
member: tap0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 9 priority 128 path cost 2000000
tap1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
description: Attached to ns0
options=80000<LINKSTATE>
ether 00:bd:e0:fc:d6:01
hwaddr 00:bd:e0:fc:d6:01
nd6 options=1<PERFORMNUD>
media: Ethernet autoselect
status: active
groups: tap
Opened by PID 44552
Bagg1:
Code:
[SW0]display interface Bridge-Aggregation brief
The brief information of interface(s) under bridge mode:
Link: ADM - administratively down; Stby - standby
Speed or Duplex: (a)/A - auto; H - half; F - full
Type: A - access; T - trunk; H - hybrid
Interface Link Speed Duplex Type PVID Description
BAGG1 UP 4G(a) F(a) T 1001
Code:
[SW0]display interface Bridge-Aggregation
Bridge-Aggregation1 current state: UP
IP Packet Frame Type: PKTFMT_ETHNT_2, Hardware Address: d894-032b-9436
Description: Bridge-Aggregation1 Interface
4Gbps-speed mode, full-duplex mode
Link speed type is autonegotiation, link duplex type is autonegotiation
PVID: 1001
Port link-type: trunk
VLAN passing : 10, 20
VLAN permitted: 10, 20
Trunk port encapsulation: IEEE 802.1q
Last clearing of counters: Never
Last 300 seconds input: 0 packets/sec 61 bytes/sec 0%
Last 300 seconds output: 0 packets/sec 231 bytes/sec 0%
Input (total): 7470551 packets, 2000248620 bytes
6472281 unicasts, 9079 broadcasts, 989191 multicasts
Input (normal): 7470551 packets, 2000248620 bytes
6472281 unicasts, 9079 broadcasts, 989191 multicasts
Input: 0 input errors, 0 runts, - giants, - throttles
0 CRC, - frame, 0 overruns, 0 aborts
- ignored, - parity errors
Output (total): 16717215 packets, 6878087375 bytes
8063589 unicasts, 5222777 broadcasts, 3430849 multicasts, 0 pauses
Output (normal): 16717215 packets, 6878087375 bytes
8063589 unicasts, 5222777 broadcasts, 3430849 multicasts, 0 pauses
Output: 0 output errors, - underruns, - buffer failures
0 aborts, 0 deferred, 0 collisions, - late collisions
- lost carrier, - no carrier
Lagg members:
Code:
SW0]display link-aggregation verbose
Loadsharing Type: Shar -- Loadsharing, NonS -- Non-Loadsharing
Port Status: S -- Selected, U -- Unselected
Flags: A -- LACP_Activity, B -- LACP_Timeout, C -- Aggregation,
D -- Synchronization, E -- Collecting, F -- Distributing,
G -- Defaulted, H -- Expired
Aggregation Interface: Bridge-Aggregation1
Aggregation Mode: Dynamic
Loadsharing Type: Shar
System ID: 0x0, d894-032b-9420
Local:
Port Status Priority Oper-Key Flag
--------------------------------------------------------------------------------
GE1/0/21 S 32768 1 {ACDEF}
GE1/0/23 S 32768 1 {ACDEF}
GE1/0/25 S 32768 1 {ACDEF}
GE1/0/27 S 32768 1 {ACDEF}
GE1/0/29 U 32768 1 {ACG}
GE1/0/31 U 32768 1 {ACG}
Remote:
Actor Partner Priority Oper-Key SystemID Flag
--------------------------------------------------------------------------------
GE1/0/21 3 32768 203 0x8000, 0cc4-7a32-cea6 {ACDEF}
GE1/0/23 4 32768 203 0x8000, 0cc4-7a32-cea6 {ACDEF}
GE1/0/25 2 32768 203 0x8000, 0cc4-7a32-cea6 {ACDEF}
GE1/0/27 1 32768 203 0x8000, 0cc4-7a32-cea6 {ACDEF}
GE1/0/29 0 32768 0 0x8000, 0000-0000-0000 {EF}
GE1/0/31 0 32768 0 0x8000, 0000-0000-0000 {EF}
Code:
[SW0]display link-aggregation member-port
Flags: A -- LACP_Activity, B -- LACP_Timeout, C -- Aggregation,
D -- Synchronization, E -- Collecting, F -- Distributing,
G -- Defaulted, H -- Expired
GigabitEthernet1/0/21:
Aggregation Interface: Bridge-Aggregation1
Local:
Port Number: 21
Port Priority: 32768
Oper-Key: 1
Flag: {ACDEF}
Remote:
System ID: 0x8000, 0cc4-7a32-cea6
Port Number: 3
Port Priority: 32768
Oper-Key: 203
Flag: {ACDEF}
Received LACP Packets: 165432 packet(s)
Illegal: 0 packet(s)
Sent LACP Packets: 168908 packet(s)
GigabitEthernet1/0/23:
Aggregation Interface: Bridge-Aggregation1
Local:
Port Number: 23
Port Priority: 32768
Oper-Key: 1
Flag: {ACDEF}
Remote:
System ID: 0x8000, 0cc4-7a32-cea6
Port Number: 4
Port Priority: 32768
Oper-Key: 203
Flag: {ACDEF}
Received LACP Packets: 165414 packet(s)
Illegal: 0 packet(s)
Sent LACP Packets: 168851 packet(s)
GigabitEthernet1/0/25:
Aggregation Interface: Bridge-Aggregation1
Local:
Port Number: 25
Port Priority: 32768
Oper-Key: 1
Flag: {ACDEF}
Remote:
System ID: 0x8000, 0cc4-7a32-cea6
Port Number: 2
Port Priority: 32768
Oper-Key: 203
Flag: {ACDEF}
Received LACP Packets: 165398 packet(s)
Illegal: 6 packet(s)
Sent LACP Packets: 174442 packet(s)
GigabitEthernet1/0/27:
Aggregation Interface: Bridge-Aggregation1
Local:
Port Number: 27
Port Priority: 32768
Oper-Key: 1
Flag: {ACDEF}
Remote:
System ID: 0x8000, 0cc4-7a32-cea6
Port Number: 1
Port Priority: 32768
Oper-Key: 203
Flag: {ACDEF}
Received LACP Packets: 159912 packet(s)
Illegal: 0 packet(s)
Sent LACP Packets: 168821 packet(s)
GigabitEthernet1/0/29:
Aggregation Interface: Bridge-Aggregation1
Local:
Port Number: 29
Port Priority: 32768
Oper-Key: 1
Flag: {ACG}
Remote:
System ID: 0x8000, 0000-0000-0000
Port Number: 0
Port Priority: 32768
Oper-Key: 0
Flag: {EF}
Received LACP Packets: 160922 packet(s)
Illegal: 0 packet(s)
Sent LACP Packets: 169550 packet(s)
GigabitEthernet1/0/31:
Aggregation Interface: Bridge-Aggregation1
Local:
Port Number: 31
Port Priority: 32768
Oper-Key: 1
Flag: {ACG}
Remote:
System ID: 0x8000, 0000-0000-0000
Port Number: 0
Port Priority: 32768
Oper-Key: 0
Flag: {EF}
Received LACP Packets: 160922 packet(s)
Illegal: 0 packet(s)
Sent LACP Packets: 169550 packet(s)
Code:
Aggregation Interface Type:
BAGG -- Bridge-Aggregation, RAGG -- Route-Aggregation
Aggregation Mode: S -- Static, D -- Dynamic
Loadsharing Type: Shar -- Loadsharing, NonS -- Non-Loadsharing
Actor System ID: 0x0, d894-032b-9420
AGG AGG Partner ID Select Unselect Share
Interface Mode Ports Ports Type
-------------------------------------------------------------------------------
BAGG1 D 0x8000, 0cc4-7a32-cea6 4 2 Shar
LACP:
Code:
[SW0]display lacp system-id
Actor System ID: 0x0, d894-032b-9420