11.2-RC2 - Network issue over Wifi: TCP Traffic Stops after Hundreds of DupACKs from FreeNAS

[milizhang]

Server Hardware: A1SAM-2550F (Atom C2550), 32GB RAM, 4x OnChip Intel GbE NIC, with Link Aggregation (LACP).
Network Hardware: Unifi GbE Switches and 11ac Wifi APs (AC-AP-Pro).

The machine is serving both as a iSCSI Device for a VM Pool, as well as a personal file server.
It gets generally good performance on wired access, and generally acceptable Wifi performance on several machines. However, when accessing from my MacBook Pro (MacOS 10.14.1, Broadcom BCM43xx series Wifi), TCP connections could get extremely unstable.
iPerf3 tests shows that in many cases that packet transmits OK in the first second or two, then it pretty much stops. A typical pattern looks like below:

Code:

Starting Test: protocol: TCP, 1 streams, 131072 byte blocks, omitting 0 seconds, 10 second test, tos 0
[ ID] Interval		   Transfer	 Bitrate
[  7]   0.00-1.00   sec  4.51 MBytes  37.8 Mbits/sec
[  7]   1.00-2.00   sec  0.00 Bytes  0.00 bits/sec
[  7]   2.00-3.00   sec  0.00 Bytes  0.00 bits/sec
[  7]   3.00-4.00   sec  0.00 Bytes  0.00 bits/sec
[  7]   4.00-5.00   sec  0.00 Bytes  0.00 bits/sec
[  7]   5.00-6.00   sec  0.00 Bytes  0.00 bits/sec
[  7]   6.00-7.00   sec  0.00 Bytes  0.00 bits/sec
[  7]   7.00-8.00   sec  0.00 Bytes  0.00 bits/sec
[  7]   8.00-9.00   sec  0.00 Bytes  0.00 bits/sec
[  7]   9.00-10.00  sec  0.00 Bytes  0.00 bits/sec

I applied packet capture and trying to understand what is happening, and what I figured out is that at some point the server is sending hundreds of DupACKs, and then pretty much stop responding to following packets until retransmission happened seconds later.

Interestingly, it seems like turning off delayed_ack in kernel does prevent the issue from happening - however it looks like doing so will cost more CPU resources with network is loaded, and computation power is quite constrained on Atom based systems.

I am not sure whether this is a FreeBSD kernel bug or somethings is wrong with my network setup. However it seems like this Macbook Pro have no problem iperf3 to other Linux servers through the same network equipments, and other Wifi devices such as a Windows tablet can hold a very stable connection with the FreeNAS server from the same AP.

A pcap file, captured from client can be found at https://www.dropbox.com/s/jet82s4jbeig5cl/Laptop.pcapng.zip. (I had another capture on the server, but it shows the same pattern)

I would really appreciate if anyone can help me to solve the mystery. Thanks.

 

[dlavigne]

Does the issue persist after updating to RELEASE?

 

[milizhang]

Sorry for not replying in time. As of 11.2 U1 the issue still persists.

 

[dlavigne]

Please create a report at bugs.freenas.org and post the issue number here.

 

[dakta]

Hey, I'd like to chime in that I have this same issue.

After upgrading to FreeNAS 11.2-RELEASE last week I have noticed significant issues with connections between FreeNAS (and jails) and my MacBook Pro 11,2 running Mojave, 10.14.2 (latest). Same symptoms: TCP connections work fine for the first few seconds, but eventually the server starts to throw lots of duplicate ACKs, eventually leading to a massive flood of around 500 duplicate ACK packets in a 10ms burst. These dup ACK packets can actually overwhelm the network interface on my MacBook and cause connectivity issues with other software. I first noticed this because I was getting inexplicable AirPlay dropouts.

After extensive investigation with Wireshark to confirm the behavior, I know the following: it's only on FreeNAS and jails, caused by massive flood of dup ACK packets, and doesn't affect any other network devices, services, or even VMs on the same server (I run FreeNAS under ESXi 6.5).

See attached pcap file, filtered to the problem conversation and time range.

I really don't want to have to downgrade FreeNAS to fix this.

Update: bug report filed: https://redmine.ixsystems.com/issues/71459