Weird Network Performance Problems?

[NickF]

I opened a ticket https://ixsystems.atlassian.net/browse/NAS-119508

Is anyone else seeing behavior like this?

This is a weird one. In terms of change state I’m not sure when this stopped functioning properly, but I did two things recently:

• I made a bridge with no NICs attached
• I upgraded from SCALE RC1 to 22.12.0

I noticed this problem after the sum of those two things, but I’m not sure if either or neither of them have no bearing on my problem. I can see the following information from my switch,

The bond is functioning as expected and properly negotiating LACPPDUs

Code:

Channel group 11 neighbors

                     LACP port                      Admin  Oper   Port    Port
Port          Flags  Priority  Dev ID          Age  key    Key    Number  State
Te1/1/1       FA     255       3cfd.fed7.bfcc  20s  0x0    0xF    0x1     0x3F
Te1/1/2       FA     255       3cfd.fed7.bfcc  16s  0x0    0xF    0x2     0x3F

It is reporting a 20 Gigabit connection:

Code:

#show interfaces port-channel 11
Port-channel11 is up, line protocol is up (connected)
  Hardware is EtherChannel, address is 6c6c.d3c7.54b5 (bia 6c6c.d3c7.54b5)
  Description: scale_prod_uplink
  MTU 1500 bytes, BW 20000000 Kbit/sec, DLY 10 usec,
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation ARPA, loopback not set
  Keepalive set (10 sec)
  Full-duplex, 10Gb/s, link type is auto, media type is N/A
  input flow-control is on, output flow-control is unsupported
  Members in this channel: Te1/1/1 Te1/1/2
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input never, output 00:00:01, output hang never
  Last clearing of "show interface" counters never
  Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 1986779
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 43000 bits/sec, 22 packets/sec
  5 minute output rate 125000 bits/sec, 48 packets/sec
     132187482412 packets input, 188018602975625 bytes, 0 no buffer
     Received 11549916 broadcasts (7259457 multicasts)
     0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog, 7259457 multicast, 0 pause input
     0 input packets with dribble condition detected
     119602936430 packets output, 168834842361229 bytes, 0 underruns
     Output 16934790 broadcasts (0 multicasts)
     0 output errors, 0 collisions, 5 interface resets
     0 unknown protocol drops
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 pause output
     0 output buffer failures, 0 output buffers swapped out

On SCALE in the GUI I can see the bond reports 20Gb:

In the CLI, Ethrool reports 10 Gigabit as well:

Code:

root@prod[~]# ethtool enp68s0f0
Settings for enp68s0f0:
        Supported ports: [ FIBRE ]
        Supported link modes:   10000baseT/Full
        Supported pause frame use: Symmetric Receive-only
        Supports auto-negotiation: No
        Supported FEC modes: Not reported
        Advertised link modes:  10000baseT/Full
        Advertised pause frame use: No
        Advertised auto-negotiation: No
        Advertised FEC modes: Not reported
        Speed: 10000Mb/s
        Duplex: Full
        Auto-negotiation: off
        Port: Direct Attach Copper
        PHYAD: 0
        Transceiver: internal
        Supports Wake-on: d
        Wake-on: d
        Current message level: 0x00000007 (7)
                               drv probe link
        Link detected: yes  

Code:

root@prod[~]# ethtool enp68s0f1
Settings for enp68s0f1:
        Supported ports: [ FIBRE ]
        Supported link modes:   10000baseT/Full
        Supported pause frame use: Symmetric Receive-only
        Supports auto-negotiation: No
        Supported FEC modes: Not reported
        Advertised link modes:  10000baseT/Full
        Advertised pause frame use: No
        Advertised auto-negotiation: No
        Advertised FEC modes: Not reported
        Speed: 10000Mb/s
        Duplex: Full
        Auto-negotiation: off
        Port: Direct Attach Copper
        PHYAD: 0
        Transceiver: internal
        Supports Wake-on: d
        Wake-on: d
        Current message level: 0x00000007 (7)
                               drv probe link
        Link detected: yes

But when I do an ip link show I am seeing a qlen of 1000 on the bond:

Code:

root@prod[~]# ip link show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT                                                       group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: enp68s0f0: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master b                                                      ond0 state UP mode DEFAULT group default qlen 1000
    link/ether 3c:fd:fe:d7:bf:cc brd ff:ff:ff:ff:ff:ff
3: enp68s0f1: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master b                                                      ond0 state UP mode DEFAULT group default qlen 1000
    link/ether 3c:fd:fe:d7:bf:cc brd ff:ff:ff:ff:ff:ff permaddr 3c:fd:fe:d7:bf:c                                                      d
4: usb0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group                                                       default qlen 1000
    link/ether 2e:4e:3f:19:e1:21 brd ff:ff:ff:ff:ff:ff
5: bond0: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue state                                                       UP mode DEFAULT group default qlen 1000
    link/ether 3c:fd:fe:d7:bf:cc brd ff:ff:ff:ff:ff:ff
6: vlan60@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state                                                       UP mode DEFAULT group default qlen 1000
    link/ether 3c:fd:fe:d7:bf:cc brd ff:ff:ff:ff:ff:ff
7: vlan10@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state                                                       UP mode DEFAULT group default qlen 1000
    link/ether 3c:fd:fe:d7:bf:cc brd ff:ff:ff:ff:ff:ff
8: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode D                                                      EFAULT group default qlen 1000
    link/ether ca:8b:a7:75:bc:cd brd ff:ff:ff:ff:ff:ff
9: macvtap0@vlan60: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue sta                                                      te UP mode DEFAULT group default qlen 500
    link/ether 00:a0:98:22:12:da brd ff:ff:ff:ff:ff:ff
10: macvtap1@vlan60: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue st                                                      ate UP mode DEFAULT group default qlen 500
    link/ether 00:a0:98:43:be:2a brd ff:ff:ff:ff:ff:ff
11: macvtap2@vlan60: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue st                                                      ate UP mode DEFAULT group default qlen 500
    link/ether 00:a0:98:5e:d3:8f brd ff:ff:ff:ff:ff:ff
12: macvtap3@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue sta                                                      te UP mode DEFAULT group default qlen 500
    link/ether 00:a0:98:1f:b2:c5 brd ff:ff:ff:ff:ff:ff
13: vnet0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br0 s                                                      tate UNKNOWN mode DEFAULT group default qlen 1000
    link/ether fe:a0:98:01:94:59 brd ff:ff:ff:ff:ff:ff
14: macvtap4@vlan10: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue st                                                      ate UP mode DEFAULT group default qlen 500
    link/ether 00:a0:98:38:46:98 brd ff:ff:ff:ff:ff:ff
15: macvtap5@vlan60: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue st                                                      ate UP mode DEFAULT group default qlen 500
    link/ether 00:a0:98:62:0d:b6 brd ff:ff:ff:ff:ff:ff
16: macvtap6@vlan60: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue st                                                      ate UP mode DEFAULT group default qlen 500
    link/ether 00:a0:98:35:38:0f brd ff:ff:ff:ff:ff:ff
17: macvtap7@vlan60: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue st                                                      ate UP mode DEFAULT group default qlen 500
    link/ether 00:a0:98:26:74:f9 brd ff:ff:ff:ff:ff:ff
18: macvtap8@vlan10: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue st                                                      ate UP mode DEFAULT group default qlen 500
    link/ether 00:a0:98:43:54:3b brd ff:ff:ff:ff:ff:ff
19: vnet1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br0 s                                                      tate UNKNOWN mode DEFAULT group default qlen 1000
    link/ether fe:a0:98:4a:1a:f7 brd ff:ff:ff:ff:ff:ff
20: macvtap9@vlan60: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue st                                                      ate UP mode DEFAULT group default qlen 500
    link/ether 00:a0:98:13:0f:4f brd ff:ff:ff:ff:ff:ff
21: vnet2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br0 s                                                      tate UNKNOWN mode DEFAULT group default qlen 1000
    link/ether fe:a0:98:6b:0a:70 brd ff:ff:ff:ff:ff:ff

When I attempt to do a file transfer from a computer, they seem to be limited to about 400mb, whereas before I was hitting much higher. confirming this, I ran an iperf test between the two:

Code:

C:\Users\nickf\Downloads\iperf-3.1.3-win64>iperf3 -c prod.fusco.me
Connecting to host prod.fusco.me, port 5201
[  4] local 10.69.10.67 port 44812 connected to 10.69.40.8 port 5201
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-1.00   sec  47.9 MBytes   401 Mbits/sec
[  4]   1.00-2.00   sec  47.5 MBytes   399 Mbits/sec
[  4]   2.00-3.00   sec  48.0 MBytes   402 Mbits/sec
[  4]   3.00-4.00   sec  47.8 MBytes   401 Mbits/sec
[  4]   4.00-5.00   sec  47.2 MBytes   396 Mbits/sec
[  4]   5.00-6.00   sec  48.0 MBytes   403 Mbits/sec
[  4]   6.00-7.00   sec  47.1 MBytes   395 Mbits/sec
[  4]   7.00-8.00   sec  47.9 MBytes   401 Mbits/sec
[  4]   8.00-9.00   sec  47.8 MBytes   400 Mbits/sec
[  4]   9.00-10.00  sec  47.5 MBytes   399 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-10.00  sec   477 MBytes   400 Mbits/sec                  sender
[  4]   0.00-10.00  sec   476 MBytes   400 Mbits/sec             

 

[Elliot Dierksen]

It looks like you are crossing an L3 boundary, although you didn't state that explicitly. What is doing your L3 routing? Do you have something on the same IP network where you could do an iperf test?

 

[NickF]

It looks like you are crossing an L3 boundary, although you didn't state that explicitly. What is doing your L3 routing? Do you have something on the same IP network where you could do an iperf test?

Sorry for not clarifying that.
It’s a Cisco 3850. Both devices are on the same switch. To be clear, the topology has not changed but the performance has.I don’t have previous iPerf numbers to share, but I can assure you of that. I can move my pc to that vlan and try again if you want though.

 

[Elliot Dierksen]

I think it would be worth comparing iperf throughput at L2 versus L3. No firewalls or ACL's in play? Any other deeper packet inspect or traffic shaping in the 3850?

 

[NickF]

I think it would be worth comparing iperf throughput at L2 versus L3. No firewalls or ACL's in play? Any other deeper packet inspect or traffic shaping in the 3850?

Fair enough. I did find an issue completely my bad, but I'm still having issues. I was plugged into a gigabit port with m PC when I ran the previous speed test and only saw 400mb. In either case, I should have been getting 900mb, and not 400mb, and below should be closer to 10 gig not 3 gig.

This is the fixed one.

Code:

C:\Users\nickf\Downloads\iperf-3.1.3-win64>iperf3 -c prod.fusco.me
Connecting to host prod.fusco.me, port 5201
[  4] local 10.69.10.67 port 21057 connected to 10.69.40.8 port 5201
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-1.00   sec   336 MBytes  2.81 Gbits/sec
[  4]   1.00-2.00   sec   324 MBytes  2.72 Gbits/sec
[  4]   2.00-3.00   sec   372 MBytes  3.12 Gbits/sec
[  4]   3.00-4.00   sec   352 MBytes  2.96 Gbits/sec
[  4]   4.00-5.00   sec   352 MBytes  2.96 Gbits/sec
[  4]   5.00-6.00   sec   371 MBytes  3.11 Gbits/sec
[  4]   6.00-7.00   sec   356 MBytes  2.99 Gbits/sec
[  4]   7.00-8.00   sec   359 MBytes  3.01 Gbits/sec
[  4]   8.00-9.00   sec   366 MBytes  3.07 Gbits/sec
[  4]   9.00-10.00  sec   363 MBytes  3.05 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-10.00  sec  3.47 GBytes  2.98 Gbits/sec                  sender
[  4]   0.00-10.00  sec  3.47 GBytes  2.98 Gbits/sec                  receiver

And the fixed one in the correct port for L2 adjacency:

Code:

C:\Users\nickf\Downloads\iperf-3.1.3-win64>iperf3 -c prod.fusco.me
Connecting to host prod.fusco.me, port 5201
[  4] local 10.69.40.10 port 59237 connected to 10.69.40.8 port 5201
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-1.00   sec   353 MBytes  2.96 Gbits/sec
[  4]   1.00-2.00   sec   296 MBytes  2.48 Gbits/sec
[  4]   2.00-3.00   sec   352 MBytes  2.95 Gbits/sec
[  4]   3.00-4.00   sec   348 MBytes  2.92 Gbits/sec
[  4]   4.00-5.00   sec   350 MBytes  2.93 Gbits/sec
[  4]   5.00-6.00   sec   345 MBytes  2.90 Gbits/sec
[  4]   6.00-7.00   sec   347 MBytes  2.91 Gbits/sec
[  4]   7.00-8.00   sec   351 MBytes  2.95 Gbits/sec
[  4]   8.00-9.00   sec   356 MBytes  2.98 Gbits/sec
[  4]   9.00-10.00  sec   369 MBytes  3.09 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-10.00  sec  3.38 GBytes  2.91 Gbits/sec                  sender
[  4]   0.00-10.00  sec  3.38 GBytes  2.91 Gbits/sec                  receiver


iperf Done.

All firewalling and ACLs are applied north, nothing on the Cisco or between those VLANs in any way.

Here are the port configs, I am monitoring flows FWIW:

Code:

!
interface TenGigabitEthernet1/0/44
 switchport access vlan 40
 switchport mode access
 ip flow monitor GRAVWELL-MONITOR sampler GRAVWELL-SAMPLER input
 auto qos trust dscp
 spanning-tree portfast
 service-policy input AutoQos-4.0-Trust-Dscp-Input-Policy
 service-policy output AutoQos-4.0-Output-Policy
end

Code:

!
interface Port-channel11
 description scale_prod_uplink
 switchport trunk native vlan 40
 switchport mode trunk
end

 

[Elliot Dierksen]

I am assuming the auto qos stuff and service policies were there before. I don't think the flow monitor is going to impact anything, but all of those are things to possibly take off. Here is what iperf3 looks like between my two system attached to a Nexus 3064 in 40G ports.

Code:

root@freenas:/ # iperf3 -c 192.168.252.27
Connecting to host 192.168.252.27, port 5201
[  5] local 192.168.252.23 port 64602 connected to 192.168.252.27 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  3.32 GBytes  28.5 Gbits/sec    0   5.46 MBytes
[  5]   1.00-2.00   sec  3.35 GBytes  28.7 Gbits/sec    1   1.20 MBytes
[  5]   2.00-3.00   sec  3.47 GBytes  29.8 Gbits/sec    0   3.14 MBytes
[  5]   3.00-4.00   sec  3.74 GBytes  32.1 Gbits/sec    0   4.11 MBytes
[  5]   4.00-5.00   sec  4.13 GBytes  35.4 Gbits/sec    0   5.62 MBytes
[  5]   5.00-6.00   sec  3.95 GBytes  33.9 Gbits/sec    0   6.57 MBytes
[  5]   6.00-7.00   sec  3.84 GBytes  33.0 Gbits/sec    2   3.94 MBytes
[  5]   7.00-8.00   sec  3.61 GBytes  31.0 Gbits/sec    0   4.57 MBytes
[  5]   8.00-9.00   sec  3.52 GBytes  30.2 Gbits/sec    0   4.87 MBytes
[  5]   9.00-10.00  sec  4.17 GBytes  35.8 Gbits/sec    0   6.28 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  37.1 GBytes  31.9 Gbits/sec    3             sender
[  5]   0.00-10.03  sec  37.1 GBytes  31.8 Gbits/sec                  receiver

And here are what those ports look like in the Nexus. Both the FreeNAS units are on the same VLAN/IP network.

Code:

interface Ethernet1/49
  description FreeNAS
  switchport access vlan 252
  spanning-tree link-type point-to-point

interface Ethernet1/50
  description FreeNAS2
  switchport access vlan 252
  spanning-tree link-type point-to-point

 

[NickF]

Hmmm.

I turned off the autoqos, which is just there for telephones anyway. It was there before, and it's a little better now?

Code:

C:\Users\nickf\Downloads\iperf-3.1.3-win64>iperf3 -c prod.fusco.me
Connecting to host prod.fusco.me, port 5201
[  4] local 10.69.10.60 port 32374 connected to 10.69.40.8 port 5201
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-1.00   sec   373 MBytes  3.13 Gbits/sec
[  4]   1.00-2.00   sec   366 MBytes  3.07 Gbits/sec
[  4]   2.00-3.00   sec   366 MBytes  3.07 Gbits/sec
[  4]   3.00-4.00   sec   357 MBytes  2.99 Gbits/sec
[  4]   4.00-5.00   sec   370 MBytes  3.11 Gbits/sec
[  4]   5.00-6.00   sec   367 MBytes  3.08 Gbits/sec
[  4]   6.00-7.00   sec   368 MBytes  3.08 Gbits/sec
[  4]   7.00-8.00   sec   374 MBytes  3.14 Gbits/sec
[  4]   8.00-9.00   sec   375 MBytes  3.15 Gbits/sec
[  4]   9.00-10.00  sec   373 MBytes  3.13 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-10.00  sec  3.60 GBytes  3.09 Gbits/sec                  sender
[  4]   0.00-10.00  sec  3.60 GBytes  3.09 Gbits/sec                  receiver

iperf Done.

Not seeing any errors on the ports:

Code:

TenGigabitEthernet1/0/40 is up, line protocol is up (connected)
  Hardware is Ten Gigabit Ethernet, address is 6c6c.d3c7.54a8 (bia 6c6c.d3c7.54a8)
  Description: AP
  MTU 1500 bytes, BW 10000000 Kbit/sec, DLY 10 usec,
     reliability 255/255, txload 1/255, rxload 2/255
  Encapsulation ARPA, loopback not set
  Keepalive set (10 sec)
  Full-duplex, 10Gb/s, media type is 100/1000/2.5G/5G/10GBaseTX
  input flow-control is on, output flow-control is unsupported
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input 00:07:52, output 00:00:01, output hang never
  Last clearing of "show interface" counters never
  Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 106185000 bits/sec, 8765 packets/sec
  5 minute output rate 1265000 bits/sec, 2402 packets/sec
     4543295 packets input, 6739004578 bytes, 0 no buffer
     Received 2942 broadcasts (2748 multicasts)
     0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog, 2748 multicast, 0 pause input
     0 input packets with dribble condition detected
     485180078 packets output, 604665216741 bytes, 0 underruns
     Output 2963231 broadcasts (0 multicasts)
     0 output errors, 0 collisions, 2 interface resets
     0 unknown protocol drops
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 pause output
     0 output buffer failures, 0 output buffers swapped out

Code:

show interfaces port-channel 11
Port-channel11 is up, line protocol is up (connected)
  Hardware is EtherChannel, address is 6c6c.d3c7.54b5 (bia 6c6c.d3c7.54b5)
  Description: scale_prod_uplink
  MTU 1500 bytes, BW 20000000 Kbit/sec, DLY 10 usec,
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation ARPA, loopback not set
  Keepalive set (10 sec)
  Full-duplex, 10Gb/s, link type is auto, media type is N/A
  input flow-control is on, output flow-control is unsupported
  Members in this channel: Te1/1/1 Te1/1/2
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input never, output 00:00:01, output hang never
  Last clearing of "show interface" counters never
  Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 1986779
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 2909000 bits/sec, 382 packets/sec
  5 minute output rate 72073000 bits/sec, 5997 packets/sec
     132189759276 packets input, 188020823791245 bytes, 0 no buffer
     Received 11560497 broadcasts (7266200 multicasts)
     0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog, 7266200 multicast, 0 pause input
     0 input packets with dribble condition detected
     119615525646 packets output, 168853199205464 bytes, 0 underruns
     Output 16944521 broadcasts (0 multicasts)
     0 output errors, 0 collisions, 5 interface resets
     0 unknown protocol drops
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 pause output
     0 output buffer failures, 0 output buffers swapped out