Okay, the problem seems to be isolated to a connection passing thru a switch. In my example below, 192.168.1.x is a 10Gb Fiber connection passing thru my Aruba s2500 (sfp+) ports. The 192.168.2.x are a point to point connection. The test is both directions from the Proxmox and Freenas. As you can see, going thru 192.168.1.x network, it chokes at 1/3 to 1/2 speed. Via the 192.168.2.x its full bandwidth.
root@freenas[~]# iperf3 -c 192.168.1.9 -f M
Connecting to host 192.168.1.9, port 5201
[ 5] local 192.168.1.32 port 49231 connected to 192.168.1.9 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 498 MBytes 497 MBytes/sec 0 370 KBytes
[ 5] 1.00-2.00 sec 514 MBytes 514 MBytes/sec 0 491 KBytes
[ 5] 2.00-3.00 sec 519 MBytes 519 MBytes/sec 0 581 KBytes
[ 5] 3.00-4.00 sec 525 MBytes 526 MBytes/sec 0 664 KBytes
[ 5] 4.00-5.00 sec 530 MBytes 530 MBytes/sec 0 738 KBytes
[ 5] 5.00-6.00 sec 534 MBytes 533 MBytes/sec 0 805 KBytes
[ 5] 6.00-7.00 sec 538 MBytes 538 MBytes/sec 0 866 KBytes
[ 5] 7.00-8.00 sec 538 MBytes 537 MBytes/sec 0 919 KBytes
[ 5] 8.00-9.00 sec 536 MBytes 537 MBytes/sec 0 969 KBytes
[ 5] 9.00-10.00 sec 540 MBytes 539 MBytes/sec 0 1013 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 5.15 GBytes 527 MBytes/sec 0 sender
[ 5] 0.00-10.27 sec 5.15 GBytes 513 MBytes/sec receiver
iperf Done.
root@freenas[~]# iperf3 -c 192.168.2.11 -f M
Connecting to host 192.168.2.11, port 5201
[ 5] local 192.168.2.10 port 29804 connected to 192.168.2.11 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 1.15 GBytes 1179 MBytes/sec 0 3.00 MBytes
[ 5] 1.00-2.00 sec 1.15 GBytes 1178 MBytes/sec 0 3.01 MBytes
[ 5] 2.00-3.00 sec 1.15 GBytes 1177 MBytes/sec 0 3.01 MBytes
[ 5] 3.00-4.00 sec 1.15 GBytes 1177 MBytes/sec 0 3.01 MBytes
[ 5] 4.00-5.00 sec 1.15 GBytes 1177 MBytes/sec 0 3.01 MBytes
[ 5] 5.00-6.00 sec 1.15 GBytes 1177 MBytes/sec 0 3.01 MBytes
[ 5] 6.00-7.00 sec 1.15 GBytes 1178 MBytes/sec 0 3.01 MBytes
[ 5] 7.00-8.00 sec 1.15 GBytes 1177 MBytes/sec 0 3.01 MBytes
[ 5] 8.00-9.00 sec 1.15 GBytes 1177 MBytes/sec 0 3.01 MBytes
root@freenas[~]# /mnt/barril/media/testfile bs=4M count=10000
root@freenas[~]# iperf3 -s
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
Accepted connection from 192.168.1.9, port 57854
[ 5] local 192.168.1.32 port 5201 connected to 192.168.1.9 port 57856
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.00 sec 303 MBytes 2.55 Gbits/sec
[ 5] 1.00-2.00 sec 306 MBytes 2.56 Gbits/sec
[ 5] 2.00-3.00 sec 311 MBytes 2.60 Gbits/sec
[ 5] 3.00-4.00 sec 312 MBytes 2.62 Gbits/sec
[ 5] 4.00-5.00 sec 310 MBytes 2.60 Gbits/sec
[ 5] 5.00-6.00 sec 307 MBytes 2.57 Gbits/sec
[ 5] 6.00-7.00 sec 311 MBytes 2.61 Gbits/sec
[ 5] 7.00-8.00 sec 306 MBytes 2.56 Gbits/sec
[ 5] 8.00-9.00 sec 271 MBytes 2.28 Gbits/sec
[ 5] 9.00-10.00 sec 264 MBytes 2.22 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.00 sec 2.93 GBytes 2.52 Gbits/sec receiver
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
Accepted connection from 192.168.2.11, port 42522
[ 5] local 192.168.2.10 port 5201 connected to 192.168.2.11 port 42524
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.00 sec 1.15 GBytes 9.89 Gbits/sec
[ 5] 1.00-2.00 sec 1.15 GBytes 9.90 Gbits/sec
[ 5] 2.00-3.00 sec 1.15 GBytes 9.90 Gbits/sec
[ 5] 3.00-4.00 sec 1.15 GBytes 9.90 Gbits/sec
[ 5] 4.00-5.00 sec 1.15 GBytes 9.90 Gbits/sec
[ 5] 5.00-6.00 sec 1.15 GBytes 9.90 Gbits/sec
[ 5] 6.00-7.00 sec 1.15 GBytes 9.90 Gbits/sec
[ 5] 7.00-8.00 sec 1.15 GBytes 9.90 Gbits/sec
[ 5] 8.00-9.00 sec 1.15 GBytes 9.90 Gbits/sec
[ 5] 9.00-10.00 sec 1.15 GBytes 9.90 Gbits/sec
[ 5] 10.00-10.00 sec 1.19 MBytes 9.80 Gbits/sec
I also see on the Freenas console this message when running the iperf3 test. I suspect that the limiting factor on the 192.168.1.x is the packet/sec. I would love to know how to either eliminate or raise the packet/sec.
Jul 14 16:19:02 freenas kernel: Limiting open port RST response from 320 to 200 packets/sec
Jul 14 16:19:02 freenas kernel: Limiting open port RST response from 320 to 200 packets/sec
Okay I missed setting the MTU on the Freenas side to 9K. Once that was done, I reran my iperf3 tests the issue disappeared.
root@pve:~# iperf3 -c 192.168.1.32 -f M
Connecting to host 192.168.1.32, port 5201
[ 5] local 192.168.1.9 port 57866 connected to 192.168.1.32 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 1.16 GBytes 1183 MBytes/sec 0 1.38 MBytes
[ 5] 1.00-2.00 sec 1.15 GBytes 1180 MBytes/sec 0 1.38 MBytes
[ 5] 2.00-3.00 sec 1.15 GBytes 1180 MBytes/sec 0 1.38 MBytes
[ 5] 3.00-4.00 sec 1.15 GBytes 1181 MBytes/sec 0 1.38 MBytes
[ 5] 4.00-5.00 sec 1.15 GBytes 1180 MBytes/sec 0 1.38 MBytes
[ 5] 5.00-6.00 sec 1.15 GBytes 1180 MBytes/sec 0 1.38 MBytes
[ 5] 6.00-7.00 sec 1.15 GBytes 1180 MBytes/sec 0 1.38 MBytes
[ 5] 7.00-8.00 sec 1.15 GBytes 1180 MBytes/sec 0 1.38 MBytes
[ 5] 8.00-9.00 sec 1.15 GBytes 1180 MBytes/sec 0 1.38 MBytes
[ 5] 9.00-10.00 sec 1.15 GBytes 1180 MBytes/sec 0 1.38 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 11.5 GBytes 1180 MBytes/sec 0 sender
[ 5] 0.00-10.00 sec 11.5 GBytes 1180 MBytes/sec receiver
iperf Done.
root@pve:~# iperf3 -c 192.168.2.10 -f M
Connecting to host 192.168.2.10, port 5201
[ 5] local 192.168.2.11 port 42534 connected to 192.168.2.10 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 1.16 GBytes 1183 MBytes/sec 0 1.29 MBytes
[ 5] 1.00-2.00 sec 1.15 GBytes 1180 MBytes/sec 0 1.29 MBytes
[ 5] 2.00-3.00 sec 1.15 GBytes 1180 MBytes/sec 0 1.29 MBytes
[ 5] 3.00-4.00 sec 1.15 GBytes 1180 MBytes/sec 0 1.29 MBytes
[ 5] 4.00-5.00 sec 1.15 GBytes 1180 MBytes/sec 0 1.29 MBytes
[ 5] 5.00-6.00 sec 1.15 GBytes 1180 MBytes/sec 0 1.29 MBytes
[ 5] 6.00-7.00 sec 1.15 GBytes 1180 MBytes/sec 0 1.29 MBytes
[ 5] 7.00-8.00 sec 1.15 GBytes 1181 MBytes/sec 0 1.29 MBytes
[ 5] 8.00-9.00 sec 1.15 GBytes 1180 MBytes/sec 0 1.29 MBytes
[ 5] 9.00-10.00 sec 1.15 GBytes 1180 MBytes/sec 0 1.29 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 11.5 GBytes 1180 MBytes/sec 0 sender
[ 5] 0.00-10.00 sec 11.5 GBytes 1180 MBytes/sec receiver
iperf Done.
root@pve:~# iperf3 -s
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
Accepted connection from 192.168.1.32, port 14125
[ 5] local 192.168.1.9 port 5201 connected to 192.168.1.32 port 47745
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.00 sec 859 MBytes 7.21 Gbits/sec
[ 5] 1.00-2.00 sec 1.15 GBytes 9.89 Gbits/sec
[ 5] 2.00-3.00 sec 1.15 GBytes 9.90 Gbits/sec
[ 5] 3.00-4.00 sec 1.15 GBytes 9.89 Gbits/sec
[ 5] 4.00-5.00 sec 1.15 GBytes 9.89 Gbits/sec
[ 5] 5.00-6.00 sec 1.15 GBytes 9.90 Gbits/sec
[ 5] 6.00-7.00 sec 1.15 GBytes 9.89 Gbits/sec
[ 5] 7.00-8.00 sec 1.15 GBytes 9.86 Gbits/sec
[ 5] 8.00-9.00 sec 1.15 GBytes 9.89 Gbits/sec
[ 5] 9.00-10.00 sec 1.15 GBytes 9.89 Gbits/sec
[ 5] 10.00-10.27 sec 316 MBytes 9.90 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.27 sec 11.5 GBytes 9.63 Gbits/sec receiver
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
Accepted connection from 192.168.2.10, port 47621
[ 5] local 192.168.2.11 port 5201 connected to 192.168.2.10 port 21998
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.00 sec 865 MBytes 7.25 Gbits/sec
[ 5] 1.00-2.00 sec 1.15 GBytes 9.88 Gbits/sec
[ 5] 2.00-3.00 sec 1.15 GBytes 9.88 Gbits/sec
[ 5] 3.00-4.00 sec 1.15 GBytes 9.87 Gbits/sec
[ 5] 4.00-5.00 sec 1.15 GBytes 9.88 Gbits/sec
[ 5] 5.00-6.00 sec 1.15 GBytes 9.88 Gbits/sec
[ 5] 6.00-7.00 sec 1.15 GBytes 9.87 Gbits/sec
[ 5] 7.00-8.00 sec 1.15 GBytes 9.88 Gbits/sec
[ 5] 8.00-9.00 sec 1.15 GBytes 9.88 Gbits/sec
[ 5] 9.00-10.00 sec 1.15 GBytes 9.87 Gbits/sec
[ 5] 10.00-10.26 sec 311 MBytes 9.88 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.26 sec 11.5 GBytes 9.62 Gbits/sec receiver
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------