Very low TCP performance on mid-latency connection.

Status
Not open for further replies.

Bogdan

Cadet
Joined
Dec 29, 2015
Messages
3
Hi!

We have very slow outgoing TCP traffic for scp/rsync and iperf connections.

FreeNAS system: Dell R530, 32Gb RAM, 8x1Tb RAIDZ2, Broadcom 5720 QP LOM, Intel i350 DP PCI-E NIC, FreeNAS 9.10 STABLE.

Remote iperf/scp/rsync system: Ubuntu 16.04 with recent updates installed.

ping results:

Code:
100 packets transmitted, 100 received, 0% packet loss, time 19864ms
rtt min/avg/max/mdev = 49.889/51.399/58.088/1.415 ms


iperf server run on Ubuntu with following options: iperf -S
iperf client runs on FreeNAS: iperf -c REMOTE_HOST -f m -P 4 -t 90 -m -F bigfile -i 1

'bigfile' is pre-generated 900Mbyte file with random content.

Here is my benchmark results:

FreeNAS via Intel i350 to Ubuntu, single TCP connection = 6.4 Mbit/sec
FreeNAS via Intel i350 to Ubuntu, single TCP connection, offloadings disabled = 11 Mbit/sec
FreeNAS via pair of Broadcom 5720 (LACP enabled) to Ubuntu, single TCP connection, offloadings disabled = 2.1 Mbit/sec
FreeNAS via pair of Broadcom 5720 (LACP enabled) to Ubuntu, single TCP connection, offloadings disabled = 9 Mbit/sec

I performed this test out of working hours and monitored Internet connections usage on both ends to ensure that both connection was not busy during this tests.

To ensure that there are no connection issues between our locations, I repeated that test with FreeBSD 10.3 and Ubuntu 16.04 systems with same iperf options.

From FreeBSD 10.3 sender to remote Ubuntu 16.04 receiver, single TCP connection: 13.7 Mbit/sec
From Ubuntu 16.04 seder to remote Ubuntu 16.04 receiver, single TCP connection: 82.0 Mbit/sec
From Ubuntu 16.04 seder to remote Ubuntu 16.04 receiver, quad TCP connection: 115.0 Mbit/sec

To ensure thate FreeNAS itself can serve data at high speed, I setup yet another Ubuntu receiver connected to same Juniper switch with FreeNAS:

From FreeNAS to local Ubuntu receiver via pair of Broadcom 5720 (LACP enabled), single TCP connection: 941 Mbit/sec

All test was repeated few times last to weeks to ensure that no global Internet routing issues affects on my tests.

There are something wrong with my FreeNAS on global links. I would do rsync or send zfs snapshots, or copy my backup data using scp.

Please, help me with network stack tuning. I would like to have reasonable performance of single TCP thread.

I think FreeNAS is overtuned for high performance of local TCP clients and need some adjustments to works reasonably fast when connected to remote system over Internet.

Thank you in advance.
 
Status
Not open for further replies.
Top