SOLVED Chelsio T580 (cxl) input drops

[Elliot Dierksen]

I have two FreeNAS systems which are both running 11.3-U3.2. Both systems are running on Cisco C240 M4 servers with dual E5-2637 v3 CPU's. On my storage network, both systems have Chelsio T580-LP-CR nic's with 9000 MTU set. The primary server has 256G RAM, and the secondary server has 160G RAM. I am seeing some packet drops while doing testing with iperf3. When the primary server is acting as the iperf3 server, I get 27.6 Gbits/sec throughput with a single thread, and I am ok with that. When the secondary server is acting as the iperf3 server, I get 21.5 Gbits/sec and the primary server show retries.

Code:

root@freenas2:/ # iperf3 -c 192.168.252.23
Connecting to host 192.168.252.23, port 5201
[  5] local 192.168.252.27 port 48237 connected to 192.168.252.23 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  2.42 GBytes  20.8 Gbits/sec   60    385 KBytes
[  5]   1.00-2.00   sec  2.50 GBytes  21.5 Gbits/sec   66    236 KBytes
[  5]   2.00-3.00   sec  2.53 GBytes  21.7 Gbits/sec   67    315 KBytes
[  5]   3.00-4.00   sec  2.51 GBytes  21.6 Gbits/sec   73    262 KBytes
[  5]   4.00-5.00   sec  2.49 GBytes  21.4 Gbits/sec   56    254 KBytes
[  5]   5.00-6.00   sec  2.64 GBytes  22.7 Gbits/sec   82    236 KBytes
[  5]   6.00-7.00   sec  2.68 GBytes  23.0 Gbits/sec   74    315 KBytes
[  5]   7.00-8.00   sec  2.54 GBytes  21.8 Gbits/sec   71    271 KBytes
[  5]   8.00-9.00   sec  2.48 GBytes  21.3 Gbits/sec   66    306 KBytes
[  5]   9.00-10.00  sec  2.49 GBytes  21.4 Gbits/sec   76    315 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  25.3 GBytes  21.7 Gbits/sec  691             sender
[  5]   0.00-10.08  sec  25.3 GBytes  21.5 Gbits/sec                  receiver

iperf Done.

I see input drops on the secondary server in the netstat output.

Code:

root@freenas:/ # netstat -in | sed -n '1p;/cxl/p'
Name    Mtu Network       Address              Ipkts Ierrs Idrop    Opkts Oerrs  Coll
cxl0   9000 <Link#3>      00:07:43:29:fc:30 223981604     0  7389 43584296     0     0
cxl0      - 192.168.252.0 192.168.252.23    14937091     -     - 42844382     -     -
cxl1   9000 <Link#4>      00:07:43:29:fc:38  5321792     0     0  8685039     0     0
cxl1      - 192.168.250.0 192.168.250.23     1462473     -     -  8306704     -     -

The switch between the two servers is a Cisco Nexus 3064, and it does not show any errors on either of the switch ports.

 

[Elliot Dierksen]

I also tested from the CLI of one of my ESXi 6.7 servers with iperf3. It is also a Cisco C240 M4 with dual E5-2697 v3 CPU's and 256G RAM. The NIC on the storage network is a Cisco VIC1387 set to 9000 MTU. I get 9.78 Gbits/sec when the secondary server is the iperf3 server, and 12.3 Gbits/sec when the primary server is the iperf3 server. I don't know if this is related to this difference in RAM amount, but it is definitely tied to the secondary server.

 

[Samuel Tai]

Are both FreeNAS servers virtualized? Or are both running on bare metal, or is there a mix so that one side is bare metal, and the other a VM?

 

[Elliot Dierksen]

Both FreeNAS servers are bare metal.

 

[Samuel Tai]

Are both UCS chasses at the same firmware level for all components? Is the Chelsio in the same slot on both chasses?

 

[Elliot Dierksen]

Yes, they are. There was a firmware update released in the last week, but I have not yet applied it.

 

[Elliot Dierksen]

Actually the update for only for the BMC, not BIOS or firmware on any of the cards. I am applying it, but find it highly unlikely it will change anything.

 

[Elliot Dierksen]

I gathered some more stats from the sysctl output.
CXL stats from secondary server

Code:

dev.cxl.0.stats.rx_trunc3: 0
dev.cxl.0.stats.rx_trunc2: 0
dev.cxl.0.stats.rx_trunc1: 0
dev.cxl.0.stats.rx_trunc0: 11402
dev.cxl.0.stats.rx_ovflow3: 0
dev.cxl.0.stats.rx_ovflow2: 0
dev.cxl.0.stats.rx_ovflow1: 0
dev.cxl.0.stats.rx_ovflow0: 11462
dev.cxl.0.stats.rx_pause: 0
dev.cxl.0.stats.tx_pause: 136859
dev.cxl.0.pause_settings: 7<RX,TX,AUTO>
dev.cxl.0.rsrv_noflowq: 0

CXL stats from primary server

Code:

dev.cxl.0.stats.rx_trunc3: 0
dev.cxl.0.stats.rx_trunc2: 0
dev.cxl.0.stats.rx_trunc1: 0
dev.cxl.0.stats.rx_trunc0: 0
dev.cxl.0.stats.rx_ovflow3: 0
dev.cxl.0.stats.rx_ovflow2: 0
dev.cxl.0.stats.rx_ovflow1: 0
dev.cxl.0.stats.rx_ovflow0: 0
dev.cxl.0.stats.rx_pause: 0
dev.cxl.0.stats.tx_pause: 0
dev.cxl.0.pause_settings: 7<RX,TX,AUTO>
dev.cxl.0.rsrv_noflowq: 0

I am trying to decide if that is a port on the NIC in the secondary, optics, switch port, or patch cables

 

[Samuel Tai]

Swap NICs between system 1 and 2, to see if the problem moves with the NIC. The sysctls indicate the MTU may not actually be 9000, which was the case in another thread showing asymmetric throughput. The switchport may also have flow control enabled.

 

[Elliot Dierksen]

It appears that I have messed up one of my patch cords. I use the 2nd port as a dedicated link between the two FreeNAS boxes, and I transposed that with the ones going to the switch port. The problem followed the patch cords. I could also be the optics in the switch port, but the patch cord seems a little wonky. Time to buy some replacements. Sigh.

 

[NickF]

Isn't it great when we problems with equipment costing tens of thousands of dollars ends up being a 3 dollar patch cable?
In the words of.my boss, at least it is explainable

 

[Elliot Dierksen]

Urgh! Turns out it was one of the NIC cards. I swapped what seemed to be the problematic one with my spare, and now I am getting more than 39G throughput no matter which one is acting as the iperf server. Good thing I already ordered the replacement patch cords..... :-(