I think I have something of an answer. There are a number of 'pause frames' being sent.
Code:
<HP-5820X-14XG>display interface Bridge-Aggregation3
Bridge-Aggregation3 current state: UP
IP Packet Frame Type: PKTFMT_ETHNT_2, Hardware Address: b8af-673a-bca8
Description: FreeNAS2
20Gbps-speed mode, full-duplex mode
Link speed type is autonegotiation, link duplex type is autonegotiation
PVID: 252
Port link-type: access
Tagged VLAN ID : none
Untagged VLAN ID : 252
Last clearing of counters: Never
Last 300 seconds input: 13948 packets/sec 1436336 bytes/sec 0%
Last 300 seconds output: 129592 packets/sec 192901360 bytes/sec 8%
Input (total): 38112222090 packets, 53866276538542 bytes
38111441452 unicasts, 36596 broadcasts, 743930 multicasts
Input (normal): 38112221978 packets, - bytes
38111441452 unicasts, 36596 broadcasts, 743930 multicasts
Input: 0 input errors, 0 runts, 0 giants, 0 throttles
0 CRC, 0 frame, - overruns, 0 aborts
- ignored, - parity errors
Output (total): 18182773274 packets, 21183582863857 bytes
17908002596 unicasts, 3089570 broadcasts, 13203124 multicasts, 129239012 pauses
Output (normal): 17924295290 packets, - bytes
17908002596 unicasts, 3089570 broadcasts, 13203124 multicasts, 129239012 pauses
Output: 0 output errors, - underruns, - buffer failures
0 aborts, 0 deferred, 0 collisions, 0 late collisions
0 lost carrier, - no carrier
<HP-5820X-14XG>display interface Bridge-Aggregation1
Bridge-Aggregation1 current state: UP
IP Packet Frame Type: PKTFMT_ETHNT_2, Hardware Address: b8af-673a-bcaa
Description: FreeNAS
20Gbps-speed mode, full-duplex mode
Link speed type is autonegotiation, link duplex type is autonegotiation
PVID: 252
Port link-type: access
Tagged VLAN ID : none
Untagged VLAN ID : 252
Last clearing of counters: Never
Last 300 seconds input: 311750 packets/sec 473032852 bytes/sec 19%
Last 300 seconds output: 51160 packets/sec 4413490 bytes/sec 0%
Input (total): 15286458162 packets, 18461012654191 bytes
15284589701 unicasts, 16037 broadcasts, 181336 multicasts
Input (normal): 15284787074 packets, - bytes
15284589701 unicasts, 16037 broadcasts, 181336 multicasts
Input: 0 input errors, 0 runts, 0 giants, 0 throttles
0 CRC, 0 frame, - overruns, 0 aborts
- ignored, - parity errors
Output (total): 30054386894 packets, 42018034018751 bytes
29961056380 unicasts, 25249 broadcasts, 3740025 multicasts, 44782930 pauses
Output (normal): 29964821654 packets, - bytes
29961056380 unicasts, 25249 broadcasts, 3740025 multicasts, 44782930 pauses
Output: 0 output errors, - underruns, - buffer failures
0 aborts, 0 deferred, 0 collisions, 0 late collisions
0 lost carrier, - no carrier
<HP-5820X-14XG>dis interface Ten-GigabitEthernet1/0/11
Ten-GigabitEthernet1/0/11 current state: UP
IP Packet Frame Type: PKTFMT_ETHNT_2, Hardware Address: b8af-673a-bca8
Description: FreeNAS2 - P1
Loopback is not set
Media type is optical fiber,Port hardware type is 10G_BASE_SR_SFP
10Gbps-speed mode, full-duplex mode
Link speed type is autonegotiation, link duplex type is autonegotiation
Flow-control is enabled
The Maximum Frame Length is 10000
Broadcast MAX-ratio: 100%
Unicast MAX-ratio: 100%
Multicast MAX-ratio: 100%
Allow jumbo frame to pass
PVID: 252
Mdi type: auto
Port link-type: access
Tagged VLAN ID : none
Untagged VLAN ID : 252
Port priority: 0
Last clearing of counters: Never
Peak value of input: 1067797315 bytes/sec, at 2018-07-06 14:57:20
Peak value of output: 660706356 bytes/sec, at 2018-11-04 08:54:01
Last 60 seconds input: 13377 packets/sec 1378730 bytes/sec 0%
Last 60 seconds output: 2 packets/sec 174 bytes/sec 0%
Input (total): 26286408207 packets, 37522327854688 bytes
26286007187 unicasts, 27466 broadcasts, 373365 multicasts, 0 pauses
Input (normal): 26286408018 packets, - bytes
26286007187 unicasts, 27466 broadcasts, 373365 multicasts, 0 pauses
Input: 0 input errors, 0 runts, 0 giants, 0 throttles
0 CRC, 0 frame, - overruns, 0 aborts
- ignored, - parity errors
Output (total): 6672108727 packets, 5437111560168 bytes
6602518671 unicasts, 2966113 broadcasts, 11961075 multicasts, 27331434 pauses
Output (normal): 6617445859 packets, - bytes
6602518671 unicasts, 2966113 broadcasts, 11961075 multicasts, 27331434 pauses
Output: 0 output errors, - underruns, - buffer failures
0 aborts, 0 deferred, 0 collisions, 0 late collisions
0 lost carrier, - no carrier
<HP-5820X-14XG>dis interface Ten-GigabitEthernet1/0/12
Ten-GigabitEthernet1/0/12 current state: UP
IP Packet Frame Type: PKTFMT_ETHNT_2, Hardware Address: b8af-673a-bca9
Description: FreeNAS2 - P2
Loopback is not set
Media type is optical fiber,Port hardware type is 10G_BASE_SR_SFP
10Gbps-speed mode, full-duplex mode
Link speed type is autonegotiation, link duplex type is autonegotiation
Flow-control is enabled
The Maximum Frame Length is 10000
Broadcast MAX-ratio: 100%
Unicast MAX-ratio: 100%
Multicast MAX-ratio: 100%
Allow jumbo frame to pass
PVID: 252
Mdi type: auto
Port link-type: access
Tagged VLAN ID : none
Untagged VLAN ID : 252
Port priority: 0
Last clearing of counters: Never
Peak value of input: 1111514888 bytes/sec, at 2018-10-13 10:30:09
Peak value of output: 569243839 bytes/sec, at 2018-11-09 14:48:57
Last 60 seconds input: 0 packets/sec 40 bytes/sec 0%
Last 60 seconds output: 124607 packets/sec 185479511 bytes/sec 15%
Input (total): 11826117789 packets, 16343980028720 bytes
11825738092 unicasts, 9130 broadcasts, 370567 multicasts, 0 pauses
Input (normal): 11826117789 packets, - bytes
11825738092 unicasts, 9130 broadcasts, 370567 multicasts, 0 pauses
Input: 0 input errors, 0 runts, 0 giants, 0 throttles
0 CRC, 0 frame, - overruns, 0 aborts
- ignored, - parity errors
Output (total): 11513907789 packets, 15751298543523 bytes
11308727135 unicasts, 123458 broadcasts, 1242076 multicasts, 101907578 pauses
Output (normal): 11310092669 packets, - bytes
11308727135 unicasts, 123458 broadcasts, 1242076 multicasts, 101907578 pauses
Output: 0 output errors, - underruns, - buffer failures
0 aborts, 0 deferred, 0 collisions, 0 late collisions
0 lost carrier, - no carrier
<HP-5820X-14XG>dis interface Ten-GigabitEthernet1/0/13
Ten-GigabitEthernet1/0/13 current state: UP
IP Packet Frame Type: PKTFMT_ETHNT_2, Hardware Address: b8af-673a-bcaa
Description: FreeNAS - P1
Loopback is not set
Media type is optical fiber,Port hardware type is 10G_BASE_SR_SFP
10Gbps-speed mode, full-duplex mode
Link speed type is autonegotiation, link duplex type is autonegotiation
Flow-control is enabled
The Maximum Frame Length is 10000
Broadcast MAX-ratio: 100%
Unicast MAX-ratio: 100%
Multicast MAX-ratio: 100%
Allow jumbo frame to pass
PVID: 252
Mdi type: auto
Port link-type: access
Tagged VLAN ID : none
Untagged VLAN ID : 252
Port priority: 0
Last clearing of counters: Never
Peak value of input: 369803471 bytes/sec, at 2018-11-02 09:43:40
Peak value of output: 487844986 bytes/sec, at 2018-07-07 16:14:35
Last 60 seconds input: 0 packets/sec 71 bytes/sec 0%
Last 60 seconds output: 3 packets/sec 267 bytes/sec 0%
Input (total): 846967891 packets, 825862710785 bytes
845964657 unicasts, 6602 broadcasts, 90842 multicasts, 905790 pauses
Input (normal): 846062101 packets, - bytes
845964657 unicasts, 6602 broadcasts, 90842 multicasts, 905790 pauses
Input: 0 input errors, 0 runts, 0 giants, 0 throttles
0 CRC, 0 frame, - overruns, 0 aborts
- ignored, - parity errors
Output (total): 22223874523 packets, 31368788373114 bytes
22192840289 unicasts, 4625 broadcasts, 3178073 multicasts, 13925768 pauses
Output (normal): 22196022987 packets, - bytes
22192840289 unicasts, 4625 broadcasts, 3178073 multicasts, 13925768 pauses
Output: 0 output errors, - underruns, - buffer failures
0 aborts, 0 deferred, 0 collisions, 0 late collisions
0 lost carrier, - no carrier
<HP-5820X-14XG>dis interface Ten-GigabitEthernet1/0/14
Ten-GigabitEthernet1/0/14 current state: UP
IP Packet Frame Type: PKTFMT_ETHNT_2, Hardware Address: b8af-673a-bcab
Description: FreeNAS - P2
Loopback is not set
Media type is optical fiber,Port hardware type is 10G_BASE_SR_SFP
10Gbps-speed mode, full-duplex mode
Link speed type is autonegotiation, link duplex type is autonegotiation
Flow-control is enabled
The Maximum Frame Length is 10000
Broadcast MAX-ratio: 100%
Unicast MAX-ratio: 100%
Multicast MAX-ratio: 100%
Allow jumbo frame to pass
PVID: 252
Mdi type: auto
Port link-type: access
Tagged VLAN ID : none
Untagged VLAN ID : 252
Port priority: 0
Last clearing of counters: Never
Peak value of input: 1050871408 bytes/sec, at 2018-11-09 14:12:17
Peak value of output: 591856141 bytes/sec, at 2018-11-09 15:21:27
Last 60 seconds input: 316684 packets/sec 480523762 bytes/sec 39%
Last 60 seconds output: 50993 packets/sec 4427540 bytes/sec 0%
Input (total): 14448754836 packets, 17649207961868 bytes
14447889036 unicasts, 9436 broadcasts, 90496 multicasts, 761296 pauses
Input (normal): 14447988968 packets, - bytes
14447889036 unicasts, 9436 broadcasts, 90496 multicasts, 761296 pauses
Input: 0 input errors, 0 runts, 0 giants, 0 throttles
0 CRC, 0 frame, - overruns, 0 aborts
- ignored, - parity errors
Output (total): 7831947009 packets, 10649371113747 bytes
7768967085 unicasts, 20624 broadcasts, 561981 multicasts, 31198967 pauses
Output (normal): 7769549690 packets, - bytes
7768967085 unicasts, 20624 broadcasts, 561981 multicasts, 31198967 pauses
Output: 0 output errors, - underruns, - buffer failures
0 aborts, 0 deferred, 0 collisions, 0 late collisions
0 lost carrier, - no carrier
The pauses are only displayed on the physical links, not on the bridge-aggregation interface (LAGG in H-P/H3C speak). Perhaps I need to do some iperf stuff between FreeNAS units and see what that gets. It sure looks like something is telling my network to slow down.