10GbE performance issue in one direction

M

mimesot

Dabbler
Joined
Mar 27, 2020
Messages
33
Tony-1971 said:
So the second "direction" is the one with problem.
Click to expand...

Yes, correct.
This is the direction, when data is retrieved from the FreeNAS Server (172.18.2.106)

Tony-1971 said:
The "Retr" column mean congested network or corrupted data (not sure if this is the problem but is something to start understanding why)
Click to expand...

Thanks for pointing this out. I never looked into that. How can I dig into that topic further?

Tony-1971 said:
When you perform
Code:
iperf3 -c 172.18.2.10

on server side there are problems?
Click to expand...

I am not totally sure what you mean. I have run some iperf-tests with 172.18.2.10 and another network client called 172.18.2.20 and there was bidirectional flawless data transport with bidirectional 9,4Gbit/s.

Samuel Tai said:
The problematic direction pretty much confirms my hypothesis of bufferbloat. The transfer rate is lower due to retransmits and TCP congestion window throttling.
Click to expand...

That is a very interesting. I am sorry I didn't pursue your previously mentioned idea of bufferbload because I was thinking you dropped the idea due to your own statement (cited below).
Samuel Tai said:
iperf, so far as I know, bypasses any OS buffers to get true NIC performance.
Click to expand...

So after all this means that iperf can indeed suffer from bufferbloat, when it sends data, right? I will read your attached link and think if i get my head around it. Thank you.
 
Samuel Tai

Samuel Tai

Never underestimate your own stupidity
Moderator
Joined
Apr 24, 2020
Messages
5,399
Iperf bypass OS buffers on the sender. Your problem is bufferbloat on the receiver.
 
M

mimesot

Dabbler
Joined
Mar 27, 2020
Messages
33
Samuel Tai said:
Iperf bypass OS buffers on the sender. Your problem is bufferbloat on the receiver.
Click to expand...
Ah ok.

If the receiver (172.18.2.10) had a problem with bufferbloat, why does it only occur it FreeNAS (172.18.2.106) is the sender?
I have no such problems if I use my other server (172.18.2.20) as the sender and 172.18.2.10 as the receiver.

Can this effect be tied to tho the operating system of the maschines? (172.18.2.20 and 172.18.2.10 are both Linux, whereas 172.18.2.106 is FreeNAS).
Do Linux and Freenas handle the send-reveice-speed negotiation differently? I think this is tomething tied to how TCP works and not to an operating system, right? But can there be any kind of conflict?
 
T

Tony-1971

Contributor
Joined
Oct 1, 2016
Messages
147
Hi,

I mean when you execute (on server with IP 172.18.2.106) the command:
iperf3 -c 172.18.2.10
it means to run iperf3 as client and the server is 172.18.2.10. So in this machine (IP 172.18.2.10) you run:
iperf3 -s
and there is also an output. Did you see the same problem of retransmission?

I usually run iperf3 as server on FreeNAS and then on client I run
Code:
$ iperf3 -c freenas-sm-10gbe
Connecting to host freenas-sm-10gbe, port 5201
[  5] local 192.168.200.10 port 52693 connected to 192.168.200.20 port 5201
[ ID] Interval           Transfer     Bandwidth
[  5]   0.00-1.00   sec   997 MBytes  8.36 Gbits/sec
[  5]   1.00-2.00   sec  1.00 GBytes  8.61 Gbits/sec
[  5]   2.00-3.00   sec  1.02 GBytes  8.75 Gbits/sec
[  5]   3.00-4.00   sec   954 MBytes  8.01 Gbits/sec
[  5]   4.00-5.00   sec  1.04 GBytes  8.91 Gbits/sec
[  5]   5.00-6.00   sec  1008 MBytes  8.46 Gbits/sec
[  5]   6.00-7.00   sec  1004 MBytes  8.43 Gbits/sec
[  5]   7.00-8.00   sec  1007 MBytes  8.45 Gbits/sec
[  5]   8.00-9.00   sec  1021 MBytes  8.55 Gbits/sec
[  5]   9.00-10.00  sec   976 MBytes  8.20 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth
[  5]   0.00-10.00  sec  9.86 GBytes  8.47 Gbits/sec                  sender
[  5]   0.00-10.00  sec  9.86 GBytes  8.47 Gbits/sec                  receiver

iperf Done.

$ iperf3 -c freenas-sm-10gbe -R
Connecting to host freenas-sm-10gbe, port 5201
Reverse mode, remote host freenas-sm-10gbe is sending
[  5] local 192.168.200.10 port 48692 connected to 192.168.200.20 port 5201
[ ID] Interval           Transfer     Bandwidth
[  5]   0.00-1.00   sec   822 MBytes  6.89 Gbits/sec
[  5]   1.00-2.00   sec   904 MBytes  7.58 Gbits/sec
[  5]   2.00-3.00   sec   906 MBytes  7.60 Gbits/sec
[  5]   3.00-4.00   sec   903 MBytes  7.58 Gbits/sec
[  5]   4.00-5.00   sec   904 MBytes  7.58 Gbits/sec
[  5]   5.00-6.00   sec   904 MBytes  7.58 Gbits/sec
[  5]   6.00-7.00   sec   906 MBytes  7.60 Gbits/sec
[  5]   7.00-8.00   sec   886 MBytes  7.43 Gbits/sec
[  5]   8.00-9.00   sec   901 MBytes  7.55 Gbits/sec
[  5]   9.00-10.00  sec   908 MBytes  7.61 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  5]   0.00-10.00  sec  8.74 GBytes  7.50 Gbits/sec    0             sender
[  5]   0.00-10.00  sec  8.73 GBytes  7.50 Gbits/sec                  receiver

iperf Done.


To check your sysctl values you can read https://calomel.org/freebsd_network_tuning.html
There are some good explanation and commands like
Code:
vmstat -i
netstat -Q


Best Regards,
Antonio
 
Last edited:
M

mimesot

Dabbler
Joined
Mar 27, 2020
Messages
33
Tony-1971 said:
I mean when you execute (on server with IP 172.18.2.106) the command:
iperf3 -c 172.18.2.10
it means to run iperf3 as client and the server is 172.18.2.10. So in this machine (IP 172.18.2.10) you run:
iperf3 -s
and there is also an output. Did you see the same problem of retransmission?
Click to expand...

Ah, yes, this is exaclty the problematic direction i was talking about.
And also yes, this results in some retries (3-10 per second) and the iperf displays a congestion window with a size of 60-130kB

Code:
iperf3 -c 172.18.2.10
Connecting to host 172.18.2.10, port 5201
[  5] local 172.18.2.106 port 54950 connected to 172.18.2.10 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   414 MBytes  3.47 Gbits/sec    7   82.7 KBytes
[  5]   1.00-2.00   sec   312 MBytes  2.62 Gbits/sec    6   78.4 KBytes
[  5]   2.00-3.00   sec   390 MBytes  3.27 Gbits/sec    5   85.5 KBytes
[  5]   3.00-4.00   sec   296 MBytes  2.48 Gbits/sec    7   75.5 KBytes
[  5]   4.00-5.00   sec   344 MBytes  2.89 Gbits/sec    5    108 KBytes
[  5]   5.00-6.00   sec   376 MBytes  3.15 Gbits/sec    6   64.1 KBytes
[  5]   6.00-7.00   sec   323 MBytes  2.71 Gbits/sec    7   85.5 KBytes
[  5]   7.00-8.00   sec   294 MBytes  2.46 Gbits/sec   10   55.6 KBytes
[  5]   8.00-9.00   sec   327 MBytes  2.74 Gbits/sec    5   68.4 KBytes
[  5]   9.00-10.00  sec   367 MBytes  3.08 Gbits/sec    3    130 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  3.36 GBytes  2.89 Gbits/sec   61             sender
[  5]   0.00-10.00  sec  3.36 GBytes  2.88 Gbits/sec                  receiver


So, retries occur, when a package ist lost or when the recipient doesnt send an ACK, right?
But 61 retries when sending 3.36GB of data is not a lot, is it?
Can that create such bid dilays, that I only get a third of the possible transmission rate?

Tony-1971 said:
vmstat -i
netstat -Q
Click to expand...

Here is the output of those commands:
Code:
vmstat -i
interrupt                          total       rate
irq4: uart0                           31          0
cpu0:timer                       2819152         18
cpu1:timer                       2364632         15
cpu2:timer                       2368268         15
cpu3:timer                       2294911         14
cpu4:timer                       2445053         15
cpu5:timer                       2190756         14
cpu6:timer                       2368881         15
cpu7:timer                       2301066         14
irq264: mps0                     1067208          7
irq265: nvme0:admin                 2233          0
irq266: nvme0:io0                     12          0
irq267: nvme0:io1                     44          0
irq268: nvme0:io2                     17          0
irq269: nvme0:io3                     10          0
irq270: nvme0:io4                     33          0
irq271: nvme0:io5                     14          0
irq272: nvme0:io6                     48          0
irq273: nvme0:io7                     43          0
irq274: ix0:q0                   2316514         15
irq275: ix0:q1                     82085          1
irq276: ix0:q2                    130908          1
irq277: ix0:q3                    133160          1
irq278: ix0:q4                   4872617         31
irq279: ix0:q5                    314605          2
irq280: ix0:q6                     72620          0
irq281: ix0:q7                   1052585          7
irq282: ix0:link                     599          0
irq292: xhci0                     268520          2
irq293: ahci0                      28023          0
irq300: nvme1:admin                 2233          0
irq301: nvme1:io0                 369976          2
irq302: nvme1:io1                 377649          2
irq303: nvme1:io2                 386780          2
irq304: nvme1:io3                 377965          2
irq305: nvme1:io4                 391501          2
irq306: nvme1:io5                 377647          2
irq307: nvme1:io6                 399616          3
irq308: nvme1:io7                 380024          2
Total                           32558039        205


Code:
netstat -Q
Configuration:
Setting                        Current        Limit
Thread count                         8            8
Default queue limit                256        10240
Dispatch policy                 direct          n/a
Threads bound to CPUs          enabled          n/a

Protocols:
Name   Proto QLimit Policy Dispatch Flags
ip         1   2048   flow  default   ---
igmp       2    256 source  default   ---
rtsock     3   2048 source  default   ---
arp        4    256 source  default   ---
ether      5    256 source   direct   ---
ip6        6    256   flow  default   ---
epair      8   2100    cpu  default   CD-

Workstreams:
WSID CPU   Name     Len WMark   Disp'd  HDisp'd   QDrops   Queued  Handled
   0   0   ip         0     0   186135        0        0        0   186135
   0   0   igmp       0     0       27        0        0        0       27
   0   0   rtsock     0     0        0        0        0        0        0
   0   0   arp        0     0    84152        0        0        0    84152
   0   0   ether      0     0  2325241        0        0        0  2325241
   0   0   ip6        0     0       44        0        0        0       44
   0   0   epair      0     0        0        0        0        0        0
   1   1   ip         0     0    35420        0        0        0    35420
   1   1   igmp       0     0        0        0        0        0        0
   1   1   rtsock     0     0        0        0        0        0        0
   1   1   arp        0     0        0        0        0        0        0
   1   1   ether      0     0    78733        0        0        0    78733
   1   1   ip6        0     0       44        0        0        0       44
   1   1   epair      0     0        0        0        0        0        0
   2   2   ip         0     0    88910        0        0        0    88910
   2   2   igmp       0     0        0        0        0        0        0
   2   2   rtsock     0     0        0        0        0        0        0
   2   2   arp        0     0        0        0        0        0        0
   2   2   ether      0     0   182708        0        0        0   182708
   2   2   ip6        0     0      237        0        0        0      237
   2   2   epair      0     0        0        0        0        0        0
   3   3   ip         0     0    85922        0        0        0    85922
   3   3   igmp       0     0        2        0        0        0        2
   3   3   rtsock     0     0        0        0        0        0        0
   3   3   arp        0     0        0        0        0        0        0
   3   3   ether      0     0   181999        0        0        0   181999
   3   3   ip6        0     0        0        0        0        0        0
   3   3   epair      0     0        0        0        0        0        0
   4   4   ip         0     0  4537047        0        0        0  4537047
   4   4   igmp       0     0        1        0        0        0        1
   4   4   rtsock     0     0        0        0        0        0        0
   4   4   arp        0     0        0        0        0        0        0
   4   4   ether      0     0  9076950        0        0        0  9076950
   4   4   ip6        0     0        0        0        0        0        0
   4   4   epair      0     0        0        0        0        0        0
   5   5   ip         0   468    49623        0        0 11013473 11063096
   5   5   igmp       0     0        3        0        0        0        3
   5   5   rtsock     0     2        0        0        0       17       17
   5   5   arp        0     0        0        0        0        0        0
   5   5   ether      0     0   316462        0        0        0   316462
   5   5   ip6        0     1       44        0        0       24       68
   5   5   epair      0     0        0        0        0        0        0
   6   6   ip         0     5    31033        0        0     1475    32508
   6   6   igmp       0     0        4        0        0        0        4
   6   6   rtsock     0     0        0        0        0        0        0
   6   6   arp        0     0        0        0        0        0        0
   6   6   ether      0     0    67600        0        0        0    67600
   6   6   ip6        0     0      132        0        0        0      132
   6   6   epair      0     0        0        0        0        0        0
   7   7   ip         0     0  1012137        0        0        0  1012137
   7   7   igmp       0     0        1        0        0        0        1
   7   7   rtsock     0     0        0        0        0        0        0
   7   7   arp        0     0        0        0        0        0        0
   7   7   ether      0     0  2026494        0        0        0  2026494
   7   7   ip6        0     0      132        0        0        0      132
   7   7   epair      0     0        0        0        0        0        0
 
T

Tony-1971

Contributor
Joined
Oct 1, 2016
Messages
147
Hi,

They seems OK to me (but is missing iperf3 server side).
But this commands are only examples. If you read the docs (sysctl.conf section) there are other things to check.
And if the problem is on Linux side then you have to check linux documentation.

Best Regards,
Antonio
 
M

mimesot

Dabbler
Joined
Mar 27, 2020
Messages
33
Tony-1971 said:
(but is missing iperf3 server side).
Click to expand...

Aah, you are missing the corresponding output server side. I just did a rerun and have this output:
Serverside (Debian/Proxmox):
Code:
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
Accepted connection from 172.18.2.106, port 58672
[  5] local 172.18.2.10 port 5201 connected to 172.18.2.106 port 53551
[ ID] Interval           Transfer     Bandwidth
[  5]   0.00-1.00   sec   444 MBytes  3.73 Gbits/sec
[  5]   1.00-2.00   sec   291 MBytes  2.44 Gbits/sec
[  5]   2.00-3.00   sec   297 MBytes  2.49 Gbits/sec
[  5]   3.00-4.00   sec   307 MBytes  2.58 Gbits/sec
[  5]   4.00-5.00   sec   312 MBytes  2.62 Gbits/sec
[  5]   5.00-6.00   sec   316 MBytes  2.65 Gbits/sec
[  5]   6.00-7.00   sec   342 MBytes  2.87 Gbits/sec
[  5]   7.00-8.00   sec   310 MBytes  2.60 Gbits/sec
[  5]   8.00-9.00   sec   357 MBytes  3.00 Gbits/sec
[  5]   9.00-10.00  sec   333 MBytes  2.79 Gbits/sec
[  5]  10.00-10.10  sec  26.4 MBytes  2.18 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth
[  5]   0.00-10.10  sec  0.00 Bytes  0.00 bits/sec                  sender
[  5]   0.00-10.10  sec  3.26 GBytes  2.77 Gbits/sec                  receiver


Clientside (FreeNAS):
Code:
iperf3 -c 172.18.2.10
Connecting to host 172.18.2.10, port 5201
[  5] local 172.18.2.106 port 53551 connected to 172.18.2.10 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   475 MBytes  3.99 Gbits/sec   10   79.8 KBytes
[  5]   1.00-2.00   sec   305 MBytes  2.56 Gbits/sec   10    113 KBytes
[  5]   2.00-3.00   sec   282 MBytes  2.37 Gbits/sec    8   81.2 KBytes
[  5]   3.00-4.00   sec   309 MBytes  2.59 Gbits/sec    6   75.5 KBytes
[  5]   4.00-5.00   sec   313 MBytes  2.62 Gbits/sec    7   49.9 KBytes
[  5]   5.00-6.00   sec   324 MBytes  2.72 Gbits/sec    4   98.4 KBytes
[  5]   6.00-7.00   sec   340 MBytes  2.85 Gbits/sec    6   98.4 KBytes
[  5]   7.00-8.00   sec   314 MBytes  2.64 Gbits/sec    7    105 KBytes
[  5]   8.00-9.00   sec   348 MBytes  2.92 Gbits/sec    6   77.0 KBytes
[  5]   9.00-10.00  sec   329 MBytes  2.76 Gbits/sec    5   62.7 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  3.26 GBytes  2.80 Gbits/sec   69             sender
[  5]   0.00-10.00  sec  3.26 GBytes  2.80 Gbits/sec                  receiver

iperf Done.


So it seems Debian does not recognize any problems at all.
Also interesting: They agree on the overall Data transmitted but not on the speed.

Further interesting information: I found a thread on the Proxmox forum where they seem to have the almost exact same problem. https://forum.proxmox.com/threads/unstable-cwnd-from-freenas-to-proxmox-performance-issue.63868/
Perhaps it is really a problem in between FreeeNAS and Debian.
 
M

mimesot

Dabbler
Joined
Mar 27, 2020
Messages
33
I tried something else, because i wanted to know it it is only TCP or UDP too:

Is it even intelligent to test this?

Code:
iperf3 -c 172.18.2.10 -u -b 1000M
Connecting to host 172.18.2.10, port 5201
[  5] local 172.18.2.106 port 51747 connected to 172.18.2.10 port 5201
[ ID] Interval           Transfer     Bitrate         Total Datagrams
[  5]   0.00-1.00   sec   119 MBytes   999 Mbits/sec  85556
...
[  5]   9.00-10.00  sec   119 MBytes  1000 Mbits/sec  85616
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-10.00  sec  1.16 GBytes  1000 Mbits/sec  0.000 ms  0/856103 (0%)  sender
[  5]   0.00-10.00  sec  1.15 GBytes   990 Mbits/sec  0.001 ms  8318/856103 (0.97%)  receiver


Code:
iperf3 -c 172.18.2.10 -u -b 2000M
Connecting to host 172.18.2.10, port 5201
[  5] local 172.18.2.106 port 21006 connected to 172.18.2.10 port 5201
[ ID] Interval           Transfer     Bitrate         Total Datagrams
[  5]   0.00-1.00   sec   238 MBytes  2.00 Gbits/sec  171128
...
[  5]   9.00-10.00  sec   238 MBytes  2.00 Gbits/sec  171233
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-10.00  sec  2.33 GBytes  2.00 Gbits/sec  0.000 ms  0/1712224 (0%)  sender
[  5]   0.00-10.00  sec  1.87 GBytes  1.61 Gbits/sec  0.001 ms  334511/1712224 (20%)  receiver


Code:
iperf3 -c 172.18.2.10 -u -b 3000M
Connecting to host 172.18.2.10, port 5201
[  5] local 172.18.2.106 port 34411 connected to 172.18.2.10 port 5201
[ ID] Interval           Transfer     Bitrate         Total Datagrams
[  5]   0.00-1.00   sec   358 MBytes  3.00 Gbits/sec  256839
...
[  5]   9.00-10.00  sec   358 MBytes  3.00 Gbits/sec  256848
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-10.00  sec  3.49 GBytes  3.00 Gbits/sec  0.000 ms  0/2568480 (0%)  sender
[  5]   0.00-10.00  sec  2.43 GBytes  2.08 Gbits/sec  0.015 ms  783999/2567995 (31%)  receiver


Code:
% iperf3 -c 172.18.2.10 -u -b 4000M
Connecting to host 172.18.2.10, port 5201
[  5] local 172.18.2.106 port 55192 connected to 172.18.2.10 port 5201
[ ID] Interval           Transfer     Bitrate         Total Datagrams
[  5]   0.00-1.00   sec   477 MBytes  4.00 Gbits/sec  342358
...
[  5]   9.00-10.00  sec   477 MBytes  4.00 Gbits/sec  342515
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-10.00  sec  4.66 GBytes  4.00 Gbits/sec  0.000 ms  0/3424600 (0%)  sender
[  5]   0.00-10.00  sec  3.61 GBytes  3.10 Gbits/sec  0.002 ms  768341/3424342 (22%)  receiver


Code:
iperf3 -c 172.18.2.10 -u -b 5000M
Connecting to host 172.18.2.10, port 5201
[  5] local 172.18.2.106 port 22594 connected to 172.18.2.10 port 5201
[ ID] Interval           Transfer     Bitrate         Total Datagrams
[  5]   0.00-1.00   sec   596 MBytes  5.00 Gbits/sec  427881
...
[  5]   9.00-10.00  sec   596 MBytes  5.00 Gbits/sec  428080
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-10.00  sec  5.82 GBytes  5.00 Gbits/sec  0.000 ms  0/4280676 (0%)  sender
[  5]   0.00-10.00  sec  4.42 GBytes  3.80 Gbits/sec  0.002 ms  1028453/4280383 (24%)  receiver



Code:
iperf3 -c 172.18.2.10 -u -b 6000M
Connecting to host 172.18.2.10, port 5201
[  5] local 172.18.2.106 port 38400 connected to 172.18.2.10 port 5201
[ ID] Interval           Transfer     Bitrate         Total Datagrams
[  5]   0.00-1.00   sec   715 MBytes  6.00 Gbits/sec  513596
...
[  5]   9.00-10.00  sec   716 MBytes  6.01 Gbits/sec  514143
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-10.00  sec  6.98 GBytes  6.00 Gbits/sec  0.000 ms  0/5136910 (0%)  sender
[  5]   0.00-10.00  sec  4.89 GBytes  4.20 Gbits/sec  0.003 ms  1537359/5136542 (30%)  receiver


Code:
iperf3 -c 172.18.2.10 -u -b 7000M
Connecting to host 172.18.2.10, port 5201
[  5] local 172.18.2.106 port 35014 connected to 172.18.2.10 port 5201
[ ID] Interval           Transfer     Bitrate         Total Datagrams
[  5]   0.00-1.00   sec   808 MBytes  6.78 Gbits/sec  580176
...
[  5]   9.00-10.00  sec   804 MBytes  6.74 Gbits/sec  577121
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-10.00  sec  7.86 GBytes  6.75 Gbits/sec  0.000 ms  0/5780192 (0%)  sender
[  5]   0.00-10.00  sec  5.37 GBytes  4.61 Gbits/sec  0.004 ms  1828498/5779637 (32%)  receiver


Does the generation of UDP-Packages hit some invisible wall here?
The CPU of some cores temporarily reaches about 60%.

Code:
iperf3 -c 172.18.2.10 -u -b 10000M
Connecting to host 172.18.2.10, port 5201
[  5] local 172.18.2.106 port 55119 connected to 172.18.2.10 port 5201
[ ID] Interval           Transfer     Bitrate         Total Datagrams
[  5]   0.00-1.00   sec   813 MBytes  6.82 Gbits/sec  583841
[  5]   1.00-2.00   sec   795 MBytes  6.67 Gbits/sec  570925
...
[  5]   8.00-9.00   sec   811 MBytes  6.80 Gbits/sec  582143
[  5]   9.00-10.00  sec   801 MBytes  6.72 Gbits/sec  575324
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-10.00  sec  7.89 GBytes  6.77 Gbits/sec  0.000 ms  0/5800223 (0%)  sender
[  5]   0.00-10.00  sec  5.35 GBytes  4.59 Gbits/sec  0.002 ms  1865731/5799604 (32%)  receiver


There seems ti be a distinct max there.
And what does the % actually mean 32% of all data packes are lost? Seriously?
 
T

Tony-1971

Contributor
Joined
Oct 1, 2016
Messages
147
Hi,

So it seems Debian does not recognize any problems at all.
Click to expand...
This is strange. If you put iperf3 server on FreeNAS and use iperf3 as client on Debian (-R to reverse upload direction) the retransmission are shown?

And yes, it seems that 32% of datagrams are lost.

Best Regards,
Antonio
 
Samuel Tai

Samuel Tai

Never underestimate your own stupidity
Moderator
Joined
Apr 24, 2020
Messages
5,399
That many losses suggest bad optics or cables, or lack of offload.
 
M

mimesot

Dabbler
Joined
Mar 27, 2020
Messages
33
Samuel Tai said:
That many losses suggest bad optics or cables, or lack of offload.
Click to expand...

Regarding bad hardware (cables, etc.): When I replace the SSD where FreeNAS is installed on with an SSD where Debian 10 is installed on and rerun these tests I have my 10Gbit/s without any problems. Can I safely assume therefore that hardware is not an issue?

What exactly does a lack of offloaad mean and how can I test it?
 
M

mimesot

Dabbler
Joined
Mar 27, 2020
Messages
33
Hi,
i tried this:
Tony-1971 said:
If you put iperf3 server on FreeNAS and use iperf3 as client on Debian (-R to reverse upload direction) the retransmission are shown?
Click to expand...

And get the following results:
On Debian/Proxmox
Code:
iperf3 -c 172.18.2.106 -R
Connecting to host 172.18.2.106, port 5201
Reverse mode, remote host 172.18.2.106 is sending
[  4] local 172.18.2.10 port 54162 connected to 172.18.2.106 port 5201
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-1.00   sec   675 MBytes  5.66 Gbits/sec
[  4]   1.00-2.00   sec   293 MBytes  2.45 Gbits/sec
[  4]   2.00-3.00   sec   303 MBytes  2.54 Gbits/sec
[  4]   3.00-4.00   sec   291 MBytes  2.44 Gbits/sec
[  4]   4.00-5.00   sec   280 MBytes  2.35 Gbits/sec
[  4]   5.00-6.00   sec   325 MBytes  2.73 Gbits/sec
[  4]   6.00-7.00   sec   298 MBytes  2.50 Gbits/sec
[  4]   7.00-8.00   sec   331 MBytes  2.78 Gbits/sec
[  4]   8.00-9.00   sec   297 MBytes  2.49 Gbits/sec
[  4]   9.00-10.00  sec   297 MBytes  2.49 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-10.00  sec  3.31 GBytes  2.85 Gbits/sec   75             sender
[  4]   0.00-10.00  sec  3.31 GBytes  2.84 Gbits/sec                  receiver


On FreeNAS:
Code:
iperf3 -s
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
Accepted connection from 172.18.2.10, port 54160
[  5] local 172.18.2.106 port 5201 connected to 172.18.2.10 port 54162
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   679 MBytes  5.70 Gbits/sec    9    105 KBytes
[  5]   1.00-2.00   sec   293 MBytes  2.45 Gbits/sec    7   54.2 KBytes
[  5]   2.00-3.00   sec   303 MBytes  2.54 Gbits/sec    6   64.1 KBytes
[  5]   3.00-4.00   sec   291 MBytes  2.44 Gbits/sec    6   62.7 KBytes
[  5]   4.00-5.00   sec   280 MBytes  2.35 Gbits/sec    9   91.2 KBytes
[  5]   5.00-6.00   sec   325 MBytes  2.73 Gbits/sec    8   88.4 KBytes
[  5]   6.00-7.00   sec   298 MBytes  2.50 Gbits/sec    6   94.1 KBytes
[  5]   7.00-8.00   sec   331 MBytes  2.78 Gbits/sec    6   95.5 KBytes
[  5]   8.00-9.00   sec   297 MBytes  2.49 Gbits/sec   12   51.3 KBytes
[  5]   9.00-10.00  sec   297 MBytes  2.49 Gbits/sec    6   88.4 KBytes
[  5]  10.00-10.00  sec  87.7 KBytes  2.67 Gbits/sec    0   88.4 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  3.31 GBytes  2.85 Gbits/sec   75             sender


I hope that helps somehow.

If i have 75 retries with TCP, that seems not like a lot of packages are getting lost, whereas 32% with UDP seems like an insamely huge number. How can that be the case?
 
N

no_connection

Patron
Joined
Dec 15, 2013
Messages
480
Is flow control enabled anywhere?

What do the switch say about checksum/corrupt packets?

mimesot said:
If i have 75 retries with TCP, that seems not like a lot of packages are getting lost, whereas 32% with UDP seems like an insamely huge number. How can that be the case?
Click to expand...
Going off could memory but TCP would not send new data until previous "group" is ackd, and if for example you run into buffer overflow it would "halt" at a dropped packet until it time out and did a retransmit giving the buffer time to clear and then cycle start over again. Did not fact check that theory as it's late but seems reasonable enough.

As far as UDP goes it just spew packets with no regard to weather it will be received or not.


Hmm, is it possible that packets are getting fragmented when going out?
 
M

mimesot

Dabbler
Joined
Mar 27, 2020
Messages
33
Hello,
thank you for your reply!

no_connection said:
Is flow control enabled anywhere?
Click to expand...

You mean things like 802.1p? The packages travel from FreeNAS to a TP-Link Switch to Debian/Proxmox. I didn't set up anything like that on the servers and I just checked the switch. No nothing there.

no_connection said:
What do the switch say about checksum/corrupt packets?

Hmm, is it possible that packets are getting fragmented when going out?
Click to expand...

How can I test this?

Thanks and greetings
mimesot
 
M

mimesot

Dabbler
Joined
Mar 27, 2020
Messages
33
I just ran a Test. Used a Windows 10 with a Intel x550 and connected it to the switch where the servers sit. I got a download from the FreeNAS SMB-file server in the range of about 200MB/s. This means it is most probably not a FreeNAS -> Debian/Proxmox specific issue, right?
 
N

no_connection

Patron
Joined
Dec 15, 2013
Messages
480
I guess you could check with wireshark.

Maybe change MTU up and down and see if that makes a change.
 
J

jims2321

Cadet
Joined
May 23, 2015
Messages
4
Okay, the problem seems to be isolated to a connection passing thru a switch. In my example below, 192.168.1.x is a 10Gb Fiber connection passing thru my Aruba s2500 (sfp+) ports. The 192.168.2.x are a point to point connection. The test is both directions from the Proxmox and Freenas. As you can see, going thru 192.168.1.x network, it chokes at 1/3 to 1/2 speed. Via the 192.168.2.x its full bandwidth.

root@freenas[~]# iperf3 -c 192.168.1.9 -f M
Connecting to host 192.168.1.9, port 5201
[ 5] local 192.168.1.32 port 49231 connected to 192.168.1.9 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 498 MBytes 497 MBytes/sec 0 370 KBytes
[ 5] 1.00-2.00 sec 514 MBytes 514 MBytes/sec 0 491 KBytes
[ 5] 2.00-3.00 sec 519 MBytes 519 MBytes/sec 0 581 KBytes
[ 5] 3.00-4.00 sec 525 MBytes 526 MBytes/sec 0 664 KBytes
[ 5] 4.00-5.00 sec 530 MBytes 530 MBytes/sec 0 738 KBytes
[ 5] 5.00-6.00 sec 534 MBytes 533 MBytes/sec 0 805 KBytes
[ 5] 6.00-7.00 sec 538 MBytes 538 MBytes/sec 0 866 KBytes
[ 5] 7.00-8.00 sec 538 MBytes 537 MBytes/sec 0 919 KBytes
[ 5] 8.00-9.00 sec 536 MBytes 537 MBytes/sec 0 969 KBytes
[ 5] 9.00-10.00 sec 540 MBytes 539 MBytes/sec 0 1013 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 5.15 GBytes 527 MBytes/sec 0 sender
[ 5] 0.00-10.27 sec 5.15 GBytes 513 MBytes/sec receiver

iperf Done.
root@freenas[~]# iperf3 -c 192.168.2.11 -f M
Connecting to host 192.168.2.11, port 5201
[ 5] local 192.168.2.10 port 29804 connected to 192.168.2.11 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 1.15 GBytes 1179 MBytes/sec 0 3.00 MBytes
[ 5] 1.00-2.00 sec 1.15 GBytes 1178 MBytes/sec 0 3.01 MBytes
[ 5] 2.00-3.00 sec 1.15 GBytes 1177 MBytes/sec 0 3.01 MBytes
[ 5] 3.00-4.00 sec 1.15 GBytes 1177 MBytes/sec 0 3.01 MBytes
[ 5] 4.00-5.00 sec 1.15 GBytes 1177 MBytes/sec 0 3.01 MBytes
[ 5] 5.00-6.00 sec 1.15 GBytes 1177 MBytes/sec 0 3.01 MBytes
[ 5] 6.00-7.00 sec 1.15 GBytes 1178 MBytes/sec 0 3.01 MBytes
[ 5] 7.00-8.00 sec 1.15 GBytes 1177 MBytes/sec 0 3.01 MBytes
[ 5] 8.00-9.00 sec 1.15 GBytes 1177 MBytes/sec 0 3.01 MBytes

root@freenas[~]# /mnt/barril/media/testfile bs=4M count=10000
root@freenas[~]# iperf3 -s
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
Accepted connection from 192.168.1.9, port 57854
[ 5] local 192.168.1.32 port 5201 connected to 192.168.1.9 port 57856
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.00 sec 303 MBytes 2.55 Gbits/sec
[ 5] 1.00-2.00 sec 306 MBytes 2.56 Gbits/sec
[ 5] 2.00-3.00 sec 311 MBytes 2.60 Gbits/sec
[ 5] 3.00-4.00 sec 312 MBytes 2.62 Gbits/sec
[ 5] 4.00-5.00 sec 310 MBytes 2.60 Gbits/sec
[ 5] 5.00-6.00 sec 307 MBytes 2.57 Gbits/sec
[ 5] 6.00-7.00 sec 311 MBytes 2.61 Gbits/sec
[ 5] 7.00-8.00 sec 306 MBytes 2.56 Gbits/sec
[ 5] 8.00-9.00 sec 271 MBytes 2.28 Gbits/sec
[ 5] 9.00-10.00 sec 264 MBytes 2.22 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.00 sec 2.93 GBytes 2.52 Gbits/sec receiver
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
Accepted connection from 192.168.2.11, port 42522
[ 5] local 192.168.2.10 port 5201 connected to 192.168.2.11 port 42524
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.00 sec 1.15 GBytes 9.89 Gbits/sec
[ 5] 1.00-2.00 sec 1.15 GBytes 9.90 Gbits/sec
[ 5] 2.00-3.00 sec 1.15 GBytes 9.90 Gbits/sec
[ 5] 3.00-4.00 sec 1.15 GBytes 9.90 Gbits/sec
[ 5] 4.00-5.00 sec 1.15 GBytes 9.90 Gbits/sec
[ 5] 5.00-6.00 sec 1.15 GBytes 9.90 Gbits/sec
[ 5] 6.00-7.00 sec 1.15 GBytes 9.90 Gbits/sec
[ 5] 7.00-8.00 sec 1.15 GBytes 9.90 Gbits/sec
[ 5] 8.00-9.00 sec 1.15 GBytes 9.90 Gbits/sec
[ 5] 9.00-10.00 sec 1.15 GBytes 9.90 Gbits/sec
[ 5] 10.00-10.00 sec 1.19 MBytes 9.80 Gbits/sec

I also see on the Freenas console this message when running the iperf3 test. I suspect that the limiting factor on the 192.168.1.x is the packet/sec. I would love to know how to either eliminate or raise the packet/sec.

Jul 14 16:19:02 freenas kernel: Limiting open port RST response from 320 to 200 packets/sec
Jul 14 16:19:02 freenas kernel: Limiting open port RST response from 320 to 200 packets/sec
 
J

jims2321

Cadet
Joined
May 23, 2015
Messages
4
jims2321 said:
Okay, the problem seems to be isolated to a connection passing thru a switch. In my example below, 192.168.1.x is a 10Gb Fiber connection passing thru my Aruba s2500 (sfp+) ports. The 192.168.2.x are a point to point connection. The test is both directions from the Proxmox and Freenas. As you can see, going thru 192.168.1.x network, it chokes at 1/3 to 1/2 speed. Via the 192.168.2.x its full bandwidth.

root@freenas[~]# iperf3 -c 192.168.1.9 -f M
Connecting to host 192.168.1.9, port 5201
[ 5] local 192.168.1.32 port 49231 connected to 192.168.1.9 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 498 MBytes 497 MBytes/sec 0 370 KBytes
[ 5] 1.00-2.00 sec 514 MBytes 514 MBytes/sec 0 491 KBytes
[ 5] 2.00-3.00 sec 519 MBytes 519 MBytes/sec 0 581 KBytes
[ 5] 3.00-4.00 sec 525 MBytes 526 MBytes/sec 0 664 KBytes
[ 5] 4.00-5.00 sec 530 MBytes 530 MBytes/sec 0 738 KBytes
[ 5] 5.00-6.00 sec 534 MBytes 533 MBytes/sec 0 805 KBytes
[ 5] 6.00-7.00 sec 538 MBytes 538 MBytes/sec 0 866 KBytes
[ 5] 7.00-8.00 sec 538 MBytes 537 MBytes/sec 0 919 KBytes
[ 5] 8.00-9.00 sec 536 MBytes 537 MBytes/sec 0 969 KBytes
[ 5] 9.00-10.00 sec 540 MBytes 539 MBytes/sec 0 1013 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 5.15 GBytes 527 MBytes/sec 0 sender
[ 5] 0.00-10.27 sec 5.15 GBytes 513 MBytes/sec receiver

iperf Done.
root@freenas[~]# iperf3 -c 192.168.2.11 -f M
Connecting to host 192.168.2.11, port 5201
[ 5] local 192.168.2.10 port 29804 connected to 192.168.2.11 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 1.15 GBytes 1179 MBytes/sec 0 3.00 MBytes
[ 5] 1.00-2.00 sec 1.15 GBytes 1178 MBytes/sec 0 3.01 MBytes
[ 5] 2.00-3.00 sec 1.15 GBytes 1177 MBytes/sec 0 3.01 MBytes
[ 5] 3.00-4.00 sec 1.15 GBytes 1177 MBytes/sec 0 3.01 MBytes
[ 5] 4.00-5.00 sec 1.15 GBytes 1177 MBytes/sec 0 3.01 MBytes
[ 5] 5.00-6.00 sec 1.15 GBytes 1177 MBytes/sec 0 3.01 MBytes
[ 5] 6.00-7.00 sec 1.15 GBytes 1178 MBytes/sec 0 3.01 MBytes
[ 5] 7.00-8.00 sec 1.15 GBytes 1177 MBytes/sec 0 3.01 MBytes
[ 5] 8.00-9.00 sec 1.15 GBytes 1177 MBytes/sec 0 3.01 MBytes

root@freenas[~]# /mnt/barril/media/testfile bs=4M count=10000
root@freenas[~]# iperf3 -s
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
Accepted connection from 192.168.1.9, port 57854
[ 5] local 192.168.1.32 port 5201 connected to 192.168.1.9 port 57856
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.00 sec 303 MBytes 2.55 Gbits/sec
[ 5] 1.00-2.00 sec 306 MBytes 2.56 Gbits/sec
[ 5] 2.00-3.00 sec 311 MBytes 2.60 Gbits/sec
[ 5] 3.00-4.00 sec 312 MBytes 2.62 Gbits/sec
[ 5] 4.00-5.00 sec 310 MBytes 2.60 Gbits/sec
[ 5] 5.00-6.00 sec 307 MBytes 2.57 Gbits/sec
[ 5] 6.00-7.00 sec 311 MBytes 2.61 Gbits/sec
[ 5] 7.00-8.00 sec 306 MBytes 2.56 Gbits/sec
[ 5] 8.00-9.00 sec 271 MBytes 2.28 Gbits/sec
[ 5] 9.00-10.00 sec 264 MBytes 2.22 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.00 sec 2.93 GBytes 2.52 Gbits/sec receiver
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
Accepted connection from 192.168.2.11, port 42522
[ 5] local 192.168.2.10 port 5201 connected to 192.168.2.11 port 42524
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.00 sec 1.15 GBytes 9.89 Gbits/sec
[ 5] 1.00-2.00 sec 1.15 GBytes 9.90 Gbits/sec
[ 5] 2.00-3.00 sec 1.15 GBytes 9.90 Gbits/sec
[ 5] 3.00-4.00 sec 1.15 GBytes 9.90 Gbits/sec
[ 5] 4.00-5.00 sec 1.15 GBytes 9.90 Gbits/sec
[ 5] 5.00-6.00 sec 1.15 GBytes 9.90 Gbits/sec
[ 5] 6.00-7.00 sec 1.15 GBytes 9.90 Gbits/sec
[ 5] 7.00-8.00 sec 1.15 GBytes 9.90 Gbits/sec
[ 5] 8.00-9.00 sec 1.15 GBytes 9.90 Gbits/sec
[ 5] 9.00-10.00 sec 1.15 GBytes 9.90 Gbits/sec
[ 5] 10.00-10.00 sec 1.19 MBytes 9.80 Gbits/sec

I also see on the Freenas console this message when running the iperf3 test. I suspect that the limiting factor on the 192.168.1.x is the packet/sec. I would love to know how to either eliminate or raise the packet/sec.

Jul 14 16:19:02 freenas kernel: Limiting open port RST response from 320 to 200 packets/sec
Jul 14 16:19:02 freenas kernel: Limiting open port RST response from 320 to 200 packets/sec
Click to expand...

Okay I missed setting the MTU on the Freenas side to 9K. Once that was done, I reran my iperf3 tests the issue disappeared.
root@pve:~# iperf3 -c 192.168.1.32 -f M
Connecting to host 192.168.1.32, port 5201
[ 5] local 192.168.1.9 port 57866 connected to 192.168.1.32 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 1.16 GBytes 1183 MBytes/sec 0 1.38 MBytes
[ 5] 1.00-2.00 sec 1.15 GBytes 1180 MBytes/sec 0 1.38 MBytes
[ 5] 2.00-3.00 sec 1.15 GBytes 1180 MBytes/sec 0 1.38 MBytes
[ 5] 3.00-4.00 sec 1.15 GBytes 1181 MBytes/sec 0 1.38 MBytes
[ 5] 4.00-5.00 sec 1.15 GBytes 1180 MBytes/sec 0 1.38 MBytes
[ 5] 5.00-6.00 sec 1.15 GBytes 1180 MBytes/sec 0 1.38 MBytes
[ 5] 6.00-7.00 sec 1.15 GBytes 1180 MBytes/sec 0 1.38 MBytes
[ 5] 7.00-8.00 sec 1.15 GBytes 1180 MBytes/sec 0 1.38 MBytes
[ 5] 8.00-9.00 sec 1.15 GBytes 1180 MBytes/sec 0 1.38 MBytes
[ 5] 9.00-10.00 sec 1.15 GBytes 1180 MBytes/sec 0 1.38 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 11.5 GBytes 1180 MBytes/sec 0 sender
[ 5] 0.00-10.00 sec 11.5 GBytes 1180 MBytes/sec receiver

iperf Done.
root@pve:~# iperf3 -c 192.168.2.10 -f M
Connecting to host 192.168.2.10, port 5201
[ 5] local 192.168.2.11 port 42534 connected to 192.168.2.10 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 1.16 GBytes 1183 MBytes/sec 0 1.29 MBytes
[ 5] 1.00-2.00 sec 1.15 GBytes 1180 MBytes/sec 0 1.29 MBytes
[ 5] 2.00-3.00 sec 1.15 GBytes 1180 MBytes/sec 0 1.29 MBytes
[ 5] 3.00-4.00 sec 1.15 GBytes 1180 MBytes/sec 0 1.29 MBytes
[ 5] 4.00-5.00 sec 1.15 GBytes 1180 MBytes/sec 0 1.29 MBytes
[ 5] 5.00-6.00 sec 1.15 GBytes 1180 MBytes/sec 0 1.29 MBytes
[ 5] 6.00-7.00 sec 1.15 GBytes 1180 MBytes/sec 0 1.29 MBytes
[ 5] 7.00-8.00 sec 1.15 GBytes 1181 MBytes/sec 0 1.29 MBytes
[ 5] 8.00-9.00 sec 1.15 GBytes 1180 MBytes/sec 0 1.29 MBytes
[ 5] 9.00-10.00 sec 1.15 GBytes 1180 MBytes/sec 0 1.29 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 11.5 GBytes 1180 MBytes/sec 0 sender
[ 5] 0.00-10.00 sec 11.5 GBytes 1180 MBytes/sec receiver

iperf Done.
root@pve:~# iperf3 -s
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
Accepted connection from 192.168.1.32, port 14125
[ 5] local 192.168.1.9 port 5201 connected to 192.168.1.32 port 47745
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.00 sec 859 MBytes 7.21 Gbits/sec
[ 5] 1.00-2.00 sec 1.15 GBytes 9.89 Gbits/sec
[ 5] 2.00-3.00 sec 1.15 GBytes 9.90 Gbits/sec
[ 5] 3.00-4.00 sec 1.15 GBytes 9.89 Gbits/sec
[ 5] 4.00-5.00 sec 1.15 GBytes 9.89 Gbits/sec
[ 5] 5.00-6.00 sec 1.15 GBytes 9.90 Gbits/sec
[ 5] 6.00-7.00 sec 1.15 GBytes 9.89 Gbits/sec
[ 5] 7.00-8.00 sec 1.15 GBytes 9.86 Gbits/sec
[ 5] 8.00-9.00 sec 1.15 GBytes 9.89 Gbits/sec
[ 5] 9.00-10.00 sec 1.15 GBytes 9.89 Gbits/sec
[ 5] 10.00-10.27 sec 316 MBytes 9.90 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.27 sec 11.5 GBytes 9.63 Gbits/sec receiver
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
Accepted connection from 192.168.2.10, port 47621
[ 5] local 192.168.2.11 port 5201 connected to 192.168.2.10 port 21998
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.00 sec 865 MBytes 7.25 Gbits/sec
[ 5] 1.00-2.00 sec 1.15 GBytes 9.88 Gbits/sec
[ 5] 2.00-3.00 sec 1.15 GBytes 9.88 Gbits/sec
[ 5] 3.00-4.00 sec 1.15 GBytes 9.87 Gbits/sec
[ 5] 4.00-5.00 sec 1.15 GBytes 9.88 Gbits/sec
[ 5] 5.00-6.00 sec 1.15 GBytes 9.88 Gbits/sec
[ 5] 6.00-7.00 sec 1.15 GBytes 9.87 Gbits/sec
[ 5] 7.00-8.00 sec 1.15 GBytes 9.88 Gbits/sec
[ 5] 8.00-9.00 sec 1.15 GBytes 9.88 Gbits/sec
[ 5] 9.00-10.00 sec 1.15 GBytes 9.87 Gbits/sec
[ 5] 10.00-10.26 sec 311 MBytes 9.88 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.26 sec 11.5 GBytes 9.62 Gbits/sec receiver
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
 
M

mimesot

Dabbler
Joined
Mar 27, 2020
Messages
33
mimesot said:
I just ran a Test. Used a Windows 10 with a Intel x550 and connected it to the switch where the servers sit. I got a download from the FreeNAS SMB-file server in the range of about 200MB/s. This means it is most probably not a FreeNAS -> Debian/Proxmox specific issue, right?
Click to expand...

Wrong!
I can even replicate the problem FreeNAS <---> FreeNAS

Hi there!
I had some time and performed another test. This time I created an (in the relevant areas: CPU, Mainboard, NIC) identical FreeNAS device and rerun my test. This new FreeNAS machine currently has no zPools. I just used the default configuration upon install and ran iperf3.

So basically the testing environment looks like this:

FreeNAS (original .106) <---> 10GbE Switch <---> FreeNAS (new .20)

This is when I used the original FreeNAS as the iperf client and the new FreeNAS device as a iperf Server:

Code:
root@freenas[~]# iperf3 -c 172.18.2.20 FreeNAS (new)
Connecting to host 172.18.2.20, port 5201
[  5] local 172.18.2.106 port 11030 connected to 172.18.2.20 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   872 MBytes  7.31 Gbits/sec    0    599 KBytes..
[  5]   9.00-10.00  sec  1.07 GBytes  9.19 Gbits/sec    0   1.76 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  10.5 GBytes  9.00 Gbits/sec    0             sender
[  5]   0.00-10.00  sec  10.5 GBytes  8.99 Gbits/sec                  receiver


Now I reversed the direction.

Code:
root@freenas[~]# iperf3 -R -c 172.18.2.20
Connecting to host 172.18.2.20, port 5201
Reverse mode, remote host 172.18.2.20 is sending
[  5] local 172.18.2.106 port 39003 connected to 172.18.2.20 port 5201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec   136 MBytes  1.14 Gbits/sec
...
[  5]   9.00-10.00  sec  91.6 MBytes   769 Mbits/sec
- - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  1.14 GBytes   980 Mbits/sec  350             sender
[  5]   0.00-10.00  sec  1.14 GBytes   980 Mbits/sec                  receiver
iperf Done.


This means sending orig->new is good and new->orig is bad, right?

Tony-1971 said:
And if the problem is on Linux side then you have to check linux documentation.
Click to expand...
Ok, my conclusion would be that any hypotheses of incompatibilities between operating systems are off the table now. Would you agree?

Samuel Tai said:
Iperf bypass OS buffers on the sender. Your problem is bufferbloat on the receiver.
Click to expand...
Can the default freenas configuration already be set up i a way that it paves the way for bufferbloat?

Thank you and kind regards
mimesot
 
M

mimesot

Dabbler
Joined
Mar 27, 2020
Messages
33
Four computers
  • 172.18.2.106 FreeNAS original
  • 172.18.2.20 FreeNAS new
  • 172.18.2.10 Debian basement
  • 172.18.2.160 Debian workspace

Again tests with iperf3

Viewing the situation from debian.160

debian.160 sends: iperf3 -c 172.18.2.106: 9.35Gbit/s
debian.160 receives: iperf3 -R -c 172.18.2.106: 2.59Gbit/s

debian.160 sends: iperf3 -c 172.18.2.20: 9.37Gbit/s
debian.160 receives: iperf3 -R -c 172.18.2.20: 1.19Gbit/s

debian.160 sends: iperf3 -c 172.18.2.10: 8.54Gbit/s
debian.160 receives: iperf3 -R -c 172.18.2.10: 9.37Gbit/s

Viewing the situation from freenas.106

freenas.106 sends: iperf3 -c 172.18.2.20: 9.37Gbit/s (WHAT? I TRIPLE-CHECKED THIS)
freenas.106 receives: iperf3 -R -c 172.18.2.20: 1.13Gbit/s

freenas.106 sends: iperf3 -c 172.18.2.160: 2,46Gbit/s
freenas.106 receives: iperf3 -R -c 172.18.2.160: 9.26Gbit/s

freenas.106 sends: iperf3 -c 172.18.2.10: 3,59Gbit/s
freenas.106 receives: iperf3 -R -c 172.18.2.10: 9.39Gbit/s


Viewing the situation from freenas.20

freenas.20 sends: iperf3 -c 172.18.2.106: 1.06Gbit/s
freenas.20 receives: iperf3 -R -c 172.106.2.20: 9.17Gbit/s

freenas.20 sends: iperf3 -c 172.18.2.160: 1.14Gbit/s
freenas.20 receives: iperf3 -R -c 172.18.2.160: 9.26Gbit/s

freenas.20 sends: iperf3 -c 172.18.2.10: 1.40Gbit/s
freenas.20 receives: iperf3 -R -c 172.18.2.10: 9.39Gbit/s

Summary
  • freenas.20 is universally good at receiving data and universally bad at sending data
  • freenas.106 is good at receiving from linux devices but bad at receiving from freenas, which can be due to the other freenas being bad at sending
  • freenas.106 is bad at sending data but suddenly comes to live when the partner is the other freenas. WTF???
  • debian devices come along excellently with each other, sending to freenas is working, receiving from freenas is a nightmare
So the most unusual thing about it is the fact that freenas.106 is actually able to send at high rates, but only in one case
Can anybody make some sense of this?
 
You must log in or register to reply here.

Important Announcement for the TrueNAS Community.

The TrueNAS Community has now been moved. This forum will now become READ-ONLY for historical purposes. Please feel free to join us on the new TrueNAS Community Forums.

Related topics on forums.truenas.com for thread: "10GbE performance issue in one direction"

Similar threads

M
Rebuild failure with strange response
Replies
1
Views
1K
Davvo
Davvo
Brailyn
  • Locked
Unable to Resize SSD with camcontrol
Replies
1
Views
3K
Brailyn
Brailyn
Mystik
Problems with LSI SAS 9207-8i after 12.0-U8 -> 13.0-U1.1 upgrade
Replies
8
Views
5K
mardel
M
K
Multiple drive failure at the same time, how to "reattach" them into zfs pool?
Replies
0
Views
2K
Knogle
K
C
  • Locked
Harddrive burn-in questions
2
Replies
20
Views
8K
joeschmuck
joeschmuck
Top