Unreliable and slow transfers to/from a NFS share via 5GBase-T link

[psssttt]

Hi,

I've recently decided to make the connection between my laptop and FreeNAS box a bit faster.
After a bit of research I decided to get for my:

• laptop: a USB 3.0 to 5GbE adapter QNAP QNA-UC5G1T https://www.qnap.com/en-uk/product/qna-uc5g1t
• NAS: a 10GBase-T 5 speed NIC (which supports 5GBase-T) STLab N-480 https://www.amazon.co.uk/Kalea-Info...00Base-T-100Base-TX-Controller/dp/B077G5PN73/ (TEHUTI Chipset TN4010 + Marvell Alaska 88X3310)

After installing drivers on both machines and configuring a P2P connection a 5Gb link was successfully established.

The problem is that I can't reliably get AVG real data transfers:
from my laptop to NAS:

• faster than 70-110MB/s for small to medium files
• and not more than ~200-250MB/s for large files and that's both ways!

and ridiculously slow transfers ~30-50MB from my NAS to laptop

In terms of synthetic tests with iperf I'm getting around:

• 3.6 Gbits/sec from my laptop to NAS
• 2.0 Gbits/sec from NAS to laptop with TCP window size set to 128K

I tried various settings but none of them helped to achieve reliable ~3500-400MB transfers both ways.
I'd appreciate any help.

btw. Has anyone tried other 10GBase-T USB/Thunderbolt adapters, like?
* QNAP QNA-T310G1T 10 GbE NBASE-T £199.96 https://www.amazon.co.uk/QNAP-QNA-T310G1T-Thunderbolt3-NBASE-T-Adapter/dp/B07KY9Z4BD/
* QNAP QNA-T310G1S 10 GbE SFP+ £178.49 https://www.amazon.co.uk/QNAP-QNA-T310G1S-Single-Thunderbolt3-Adapter/dp/B07KY8CDKR/
* PROMISE - SANLink 3 T1 Thunderbolt 2 £435.90 https://www.amazon.co.uk/dp/B06XKWQPYB


Thanks,
P

Setup

FreeNAS box:
OS: FreeNAS-11.2-U5
ZFS filesystem version: 5
CPU: AMD A10-7870K
RAM: 32GB non-ecc
Motherboard: Asus A88XM-PLUS with 8xSATA 6Gb/s ports https://www.asus.com/Motherboards/A88XMPLUS/
Storage:

• 6x ST4000VN003 Seagate 4TB NAS HDD 6Gb/s 64MB Cache
• 2x ST4000DM000 Seagate 4TB Desktop HDD 64MB Cache

NICs:

• STLab N-480 10GBase-T 5 speed NIC inserted into PCIe 3.0/2.0 x16 slot
• (unused) onboard Realtek® 8111G 1000Base-T (RTL8251)
• (unused) Intel(R) PRO/1000 1000Base-T

I've added following options to 10Gb NIC setting: mtu 9014 rxcsum txcsum tso4 lro

I got the 10GB NIC driver for FreeBSD from chipset's manufacturer http://www.tehutinetworks.net/?t=drivers&L1=8&L2=12&L3=61

Code:

$ ifconfig tn400
tn400: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9014
    options=8d01bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,VLAN_HWFILTER,VLAN_HWTSO,LINKSTATE>
    ether 00:xxxxx
    hwaddr 00:xxxxx
    inet 192.168.3.11 netmask 0xffffff00 broadcast 192.168.3.255 
    nd6 options=9<PERFORMNUD,IFDISABLED>
    media: Ethernet autoselect (5000Base-T <full-duplex>)
    status: active

lspci | grep -i eth
01:00.0 Ethernet controller: Tehuti Networks Ltd. TN9710P 10GBase-T/NBASE-T Ethernet Adapter
02:05.0 Ethernet controller: Intel Corporation 82541PI Gigabit Ethernet Controller (rev 05)
04:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 0c)

Tunables
After going through a bunch of posts on ixsystems' forum, like:
https://www.ixsystems.com/community/threads/10-gig-networking-primer.25749/
https://www.ixsystems.com/community/threads/10gbe-performance-iperf-good-data-copy-slow.72239/
and articles, like:
https://fasterdata.es.net/host-tuning/linux/
https://darksideclouds.wordpress.com/2016/10/10/tuning-10gb-nics-highway-to-hell/

I've added following tunables:

Code:

cc_htcp_load=YES
kern.ipc.maxsockbuf=16777216
kern.ipc.somaxconn=1024
net.inet.ip.intr_queue_maxlen=2048
net.inet.tcp.blackhole=1
net.inet.tcp.cc.algorithm=htcp
net.inet.tcp.cc.htcp.adaptive_backoff=1
net.inet.tcp.cc.htcp.rtt_scaling=1
net.inet.tcp.recvbuf_auto=1
net.inet.tcp.recvbuf_inc=131072
net.inet.tcp.recvbuf_max=16777216
net.inet.tcp.recvspace=524288
net.inet.tcp.sendbuf_auto=1
net.inet.tcp.sendbuf_inc=65536
net.inet.tcp.sendbuf_max=16777216
net.inet.tcp.sendspace=262144

Laptop:

Carbon X1 5th gen
OS: Fedora 30
Kernel: 5.2.9-200
CPU: Intel i7-7600U
RAM: 16GB
NIC: QNAP QNA-UC5G1T USB 3.0 to 5GbE adapter (Aquantia AQC11U) https://www.qnap.com/en-uk/product/qna-uc5g1t
I got the NIC driver for Linux from Aquantia's website https://www.aquantia.com/support/driver-download/

Code:

ifconfig enp0s20f0u2
enp0s20f0u2: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 9000
        inet 192.168.3.1  netmask 255.255.255.0  broadcast 192.168.3.255
        inet6 fe80::xxxxxx  prefixlen 64  scopeid 0x20<link>
        ether 24:xxxxx  txqueuelen 1000  (Ethernet)
        RX packets 7208532  bytes 48695803684 (45.3 GiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 3122516  bytes 32464149952 (30.2 GiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

# ethtool enp0s20f0u2
Settings for enp0s20f0u2:
    Supported ports: [ TP ]
    Supported link modes:   100baseT/Full 
                            1000baseT/Full 
                            2500baseT/Full 
                            5000baseT/Full 
    Supported pause frame use: No
    Supports auto-negotiation: Yes
    Supported FEC modes: Not reported
    Advertised link modes:  100baseT/Full 
                            1000baseT/Full 
                            2500baseT/Full 
                            5000baseT/Full 
    Advertised pause frame use: No
    Advertised auto-negotiation: Yes
    Advertised FEC modes: Not reported
    Speed: 5000Mb/s
    Duplex: Full
    Port: Twisted Pair
    PHYAD: 0
    Transceiver: internal
    Auto-negotiation: on
    MDI-X: Unknown
    Supports Wake-on: g
    Wake-on: d
    Current message level: 0x00000007 (7)
                   drv probe link
    Link detected: yes

In terms of Linux tunables I've add following settings to /etc/sysctl.conf:

Code:

net.core.default_qdisc=fq
net.core.netdev_max_backlog=300000
net.core.rmem_max=134217728
net.core.wmem_max=134217728
net.ipv4.tcp_low_latency=1
net.ipv4.tcp_moderate_rcvbuf=1
net.ipv4.tcp_mtu_probing=1
net.ipv4.tcp_no_metrics_save=1
net.ipv4.tcp_rmem=4096 87380 134217728
net.ipv4.tcp_wmem=4096 65536 134217728

Synthetic tests wit iperf:

Case 1: from laptop to FreeNAS

FreeNAS

Code:

iperf -s -w 416K
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size:  416 KByte
------------------------------------------------------------
[  4] local 192.168.3.11 port 5001 connected with 192.168.3.1 port 49118
[ ID] Interval       Transfer     Bandwidth
[  4]  0.0-11.6 sec  5.00 GBytes  3.71 Gbits/sec

Laptop

Code:

iperf -c 192.168.3.11 -n 5G -w 1M
------------------------------------------------------------
Client connecting to 192.168.3.11, TCP port 5001
TCP window size: 2.00 MByte (WARNING: requested 1.00 MByte)
------------------------------------------------------------
[  3] local 192.168.3.1 port 49118 connected with 192.168.3.11 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-11.6 sec  5.00 GBytes  3.71 Gbits/sec

This is the only test where I can fully saturate the link. :)


Case 2: from FreeNAS to Laptop

FreeNAS

Code:

iperf -c 192.168.3.1 -n 5G -w 128K
------------------------------------------------------------
Client connecting to 192.168.3.1, TCP port 5001
TCP window size:  131 KByte (WARNING: requested  128 KByte)
------------------------------------------------------------
[  3] local 192.168.3.11 port 45985 connected with 192.168.3.1 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-21.2 sec  5.00 GBytes  2.03 Gbits/sec

Laptop

Code:

iperf -s -w 64K
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size:  128 KByte (WARNING: requested 64.0 KByte)
------------------------------------------------------------
[  4] local 192.168.3.1 port 5001 connected with 192.168.3.11 port 45985
[ ID] Interval       Transfer     Bandwidth
[  4]  0.0-21.2 sec  5.00 GBytes  2.03 Gbits/sec

Somehow I'm getting half the speed



Real data tests:

6GB file copied from NAS to laptop.
AVG transfer speed shown in Nautilus: ~30MB WTH!


A bunch of small files 5-50MB copied from laptop to NAS.
AVG transfer speed shown in Nautilus: ~70MB transfers.


A bunch of medium size files 80-200MB copied from laptop to NAS.
AVG transfer speed shown in Nautilus: ~100MB


One large 6GB file copied from laptop to NAS.
Initially Nautilus has shown that the AVG tranfer speed was ~290MB but then it dropped to ~130-150MB