Help Tuning TCP Upload

[jmcguire525]

I've been having issues with TCP streams being limited to ~30Mbps on a 200Mb fiber connection and believe I have narrowed it down to the Freenas configuration. When I run iperf3 from freenas a single TCP stream maxes out ~30Mpbs with a window of 650k, if I run the test from a macbook with the same window size I get ~100Mbps. I'm new to all of this and so far have only enabled autotune for networking. Where should I start looking at making changes to sort this out?

 

[c32767a]

I've been having issues with TCP streams being limited to ~30Mbps on a 200Mb fiber connection and believe I have narrowed it down to the Freenas configuration. When I run iperf3 from freenas a single TCP stream maxes out ~30Mpbs with a window of 650k, if I run the test from a macbook with the same window size I get ~100Mbps. I'm new to all of this and so far have only enabled autotune for networking. Where should I start looking at making changes to sort this out?

Is your fiber some sort of WAN link? if so, what's the round trip time between the 2 endpoints?

It would be helpful if you'd post the specs of your freenas, too.

 

[jmcguire525]

Is your fiber some sort of WAN link? if so, what's the round trip time between the 2 endpoints?

It would be helpful if you'd post the specs of your freenas, too.

Server: Xeon E5-2683v3, 64GB ECC RAM, 6x8TB RaidZ2, Gb Ethernet.

My router is a Ubiquiti USG Pro with a Gigacenter (ISP) bridged over providing PPPoE WAN IP. Have to keep the gigacenter for IPTV. Right now I'm streaming the same high bitrate file from Plex on a MacBook pro to a client 150 miles away. My Freenas server struggles to stream the same file, definitely seems like I just need to tune some of the network settings.

 

[jmcguire525]

With Freenas a single TCP stream is limited but multiple parallel streams can sturate the connection, changing the window size in iperf has no affect. With the MacBook a single TCP stream can saturate the connection.

 

[dlavigne]

What's the output of ifconfig?

 

[c32767a]

With Freenas a single TCP stream is limited but multiple parallel streams can sturate the connection, changing the window size in iperf has no affect. With the MacBook a single TCP stream can saturate the connection.

FreeBSD is in general pretty good about TCP tuning for long distance links. Since you're demonstrating the issue with iperf, it's unlikely to be on the storage side of FreeNAS.

The reason I asked you what the round trip time on the link was is because that's one of the issues that will affect performance. Different OSs have different capabilities to auto tune TCP window size.

The fact you're riding a PPPoE link will also impact MTU sizing and performance.

You might be able to benefit from some of the network tuning advice here: https://calomel.org/freebsd_network_tuning.html

 

[jmcguire525]

What's the output of ifconfig?

Code:

em0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=2098<VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC>
	ether 38:d5:47:c9:5f:45
	hwaddr 38:d5:47:c9:5f:45
	inet 192.168.1.11 netmask 0xffffff00 broadcast 192.168.1.255 
	nd6 options=9<PERFORMNUD,IFDISABLED>
	media: Ethernet autoselect (1000baseT <full-duplex>)
	status: active
igb0: flags=8c02<BROADCAST,OACTIVE,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=6403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
	ether 38:d5:47:c9:5f:46
	hwaddr 38:d5:47:c9:5f:46
	nd6 options=9<PERFORMNUD,IFDISABLED>
	media: Ethernet autoselect
	status: no carrier
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
	options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
	inet6 ::1 prefixlen 128 
	inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3 
	inet 127.0.0.1 netmask 0xff000000 
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
	groups: lo 
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	ether 02:c9:c0:d9:b7:00
	nd6 options=9<PERFORMNUD,IFDISABLED>
	groups: bridge 
	id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
	maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
	root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
	member: vnet0:7 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
			ifmaxaddr 0 port 11 priority 128 path cost 2000
	member: vnet0:6 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
			ifmaxaddr 0 port 10 priority 128 path cost 2000
	member: vnet0:4 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
			ifmaxaddr 0 port 8 priority 128 path cost 2000
	member: vnet0:3 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
			ifmaxaddr 0 port 7 priority 128 path cost 2000
	member: vnet0:2 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
			ifmaxaddr 0 port 6 priority 128 path cost 2000
	member: vnet0:1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
			ifmaxaddr 0 port 5 priority 128 path cost 2000
	member: em0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
			ifmaxaddr 0 port 1 priority 128 path cost 55
vnet0:1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
	description: associated with jail: sabnzbd
	options=8<VLAN_MTU>
	ether 02:ff:60:ef:4d:38
	hwaddr 02:6f:50:00:05:0a
	nd6 options=1<PERFORMNUD>
	media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
	status: active
	groups: epair 
vnet0:2: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
	description: associated with jail: tautulli
	options=8<VLAN_MTU>
	ether 02:ff:60:46:b0:16
	hwaddr 02:6f:50:00:06:0a
	nd6 options=1<PERFORMNUD>
	media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
	status: active
	groups: epair 
vnet0:3: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
	description: associated with jail: radarr
	options=8<VLAN_MTU>
	ether 02:ff:60:c1:df:ca
	hwaddr 02:6f:50:00:07:0a
	nd6 options=1<PERFORMNUD>
	media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
	status: active
	groups: epair 
vnet0:4: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
	description: associated with jail: channels
	options=8<VLAN_MTU>
	ether 02:ff:60:be:64:88
	hwaddr 02:6f:50:00:08:0a
	nd6 options=1<PERFORMNUD>
	media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
	status: active
	groups: epair 
vnet0:6: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
	description: associated with jail: ombi
	options=8<VLAN_MTU>
	ether 02:ff:60:e9:9f:7c
	hwaddr 02:6f:50:00:0a:0a
	nd6 options=1<PERFORMNUD>
	media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
	status: active
	groups: epair 
vnet0:7: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
	description: associated with jail: sonarr
	options=8<VLAN_MTU>
	ether 02:ff:60:a5:a4:7e
	hwaddr 02:6f:50:00:0b:0a
	nd6 options=1<PERFORMNUD>
	media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
	status: active
	groups: epair 

 

[jmcguire525]

Working:
-MacBook running PMS with SMB mount to my media library on the Freenas server can stream at 50Mbps (highest bitrate file I have available)
-Macbook test with iperf3 can saturate 100Mbps upload with a 650k window size (Remote download speed is limited to 125Mbps for testing or I could determine the window size needed for my 200Mbps upload connection)

Not Working:
-Freenas iocage or warden jail running PMS with proper mounts and permissions to the media library has issue with anything over ~30Mbps on a secure connection
-Inderect (relay) streams still have a little bit of an issue getting started but can play a 50Mbps stream
-iperf3 can get ~100Mbps after a server restart for the first 5-10 seconds of the test, after that something in the settings is pulling it back to ~30Mbps. If I test for 120 seconds with a single stream it averages ~30Mbps

 

[jmcguire525]

This doesn't happen on macOS running through the same router with the same iperf server, something is making it drop off after a few seconds.

 

[jmcguire525]

I've tried making a few changes suggested in the linked guide without any success. It is odd to me though that Plex works very well with an indirect "relay" connection but buffers on the same high bitrate files when I have a secure connection.

 

[c32767a]

The fact that your TCP congestion window decreases over time may point to a tcp buffer/window size issue.. It would only drop when there's congestion or packet loss. Since it governs the amount of data the TCP stream can have "in flight" it will gate overall TCP performance. You need to figure out why that's not scaling properly and fix the issue.

Unfortunately, these things are complex to troubleshoot. You'll probably want to investigate TCP congestion control and TCP window scaling for FreeBSD on the NAS first. The defaults that FreeBSD uses are different than MacOS which is probably why it's not an issue on your mac.
It could also be packet loss, or other activity on the link, or something at the remote end.

 

[jmcguire525]

The fact that your TCP congestion window decreases over time may point to a tcp buffer/window size issue.. It would only drop when there's congestion or packet loss. Since it governs the amount of data the TCP stream can have "in flight" it will gate overall TCP performance. You need to figure out why that's not scaling properly and fix the issue.

Unfortunately, these things are complex to troubleshoot. You'll probably want to investigate TCP congestion control and TCP window scaling for FreeBSD on the NAS first. The defaults that FreeBSD uses are different than MacOS which is probably why it's not an issue on your mac.
It could also be packet loss, or other activity on the link, or something at the remote end.

Why would a connection through Plex's relay servers not have the same issue, is that handled differently than a TCP connection?

 

[c32767a]

Why would a connection through Plex's relay servers not have the same issue, is that handled differently than a TCP connection?

I've never looked closely at plex's networking, so I don't know how to answer that.
It might be something to ping some plex gurus on.