10Gbe on AMD Ryzen slow speeds

[mmbb]

Hi, I'm building my first FreeNAS server and having issues with my 10Gb Ethernet speeds. Let me first explain my setup. I have 3 machines:
192.168.10.1 (server): My future FreeNAS server, running AMD Ryzen 7 1700 on GIGABYTE AX370-Gaming K5 motherboard, FreeNAS-11.3-U2.1
192.168.10.2 (workstation): My main workstation, running AMD Ryzen 7 1700X on ASUS CROSSHAIR VI HERO motherboard, Ubuntu 18.04
192.168.10.4 (test machine): Another machine for testing purposes, running Intel Xeon E3-1230 v5 on ASrock Fatal1ty E3V5 Performance Gaming/OC motherboard, FreeNAS-11.3-U2.1
All machines have Mellanox MCX311A-XCAT cards and are connected through Mikrotik CRS317 switch.

So when I built the server and connected my network, I run an iperf3 test. It surprised me that the speeds from my workstation to server were lower that expected, I got about 6-8Gbps.

Code:

mmbb@Michal:~$ iperf3 -c 192.168.10.1
Connecting to host 192.168.10.1, port 5201
[  4] local 192.168.10.2 port 58660 connected to 192.168.10.1 port 5201
[ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
[  4]   0.00-1.00   sec  1.10 GBytes  9.42 Gbits/sec    0   1.43 MBytes       
[  4]   1.00-2.00   sec  1.09 GBytes  9.40 Gbits/sec    0   1.43 MBytes       
[  4]   2.00-3.00   sec   880 MBytes  7.38 Gbits/sec   95   1.24 MBytes       
[  4]   3.00-4.00   sec   690 MBytes  5.78 Gbits/sec   85   1.04 MBytes       
[  4]   4.00-5.00   sec   685 MBytes  5.75 Gbits/sec   90    813 KBytes       
[  4]   5.00-6.00   sec   679 MBytes  5.69 Gbits/sec    0   1.28 MBytes       
[  4]   6.00-7.00   sec   665 MBytes  5.58 Gbits/sec   84   1.10 MBytes       
[  4]   7.00-8.00   sec   724 MBytes  6.07 Gbits/sec   62    897 KBytes       
[  4]   8.00-9.00   sec   669 MBytes  5.61 Gbits/sec    0   1.33 MBytes       
[  4]   9.00-10.00  sec   669 MBytes  5.62 Gbits/sec   90   1.14 MBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-10.00  sec  7.72 GBytes  6.63 Gbits/sec  506             sender
[  4]   0.00-10.00  sec  7.71 GBytes  6.63 Gbits/sec                  receiver

When I run test in the other direction, from my server to my workstation, speed is OK, about 9.4Gbps.

Code:

mmbb@Michal:~$ iperf3 -c 192.168.10.1 -R
Connecting to host 192.168.10.1, port 5201
Reverse mode, remote host 192.168.10.1 is sending
[  4] local 192.168.10.2 port 58700 connected to 192.168.10.1 port 5201
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-1.00   sec  1.06 GBytes  9.09 Gbits/sec                 
[  4]   1.00-2.00   sec  1.09 GBytes  9.39 Gbits/sec                 
[  4]   2.00-3.00   sec  1.09 GBytes  9.39 Gbits/sec                 
[  4]   3.00-4.00   sec  1.09 GBytes  9.39 Gbits/sec                 
[  4]   4.00-5.00   sec  1.09 GBytes  9.39 Gbits/sec                 
[  4]   5.00-6.00   sec  1.09 GBytes  9.34 Gbits/sec                 
[  4]   6.00-7.00   sec  1.09 GBytes  9.39 Gbits/sec                 
[  4]   7.00-8.00   sec  1.09 GBytes  9.39 Gbits/sec                 
[  4]   8.00-9.00   sec  1.09 GBytes  9.39 Gbits/sec                 
[  4]   9.00-10.00  sec  1.09 GBytes  9.39 Gbits/sec                 
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-10.00  sec  10.9 GBytes  9.36 Gbits/sec   29             sender
[  4]   0.00-10.00  sec  10.9 GBytes  9.36 Gbits/sec                  receiver

Then I tried running Ubuntu on my server and I got 9.4Gbps in both directions, so I thought that there was something wrong with FreeNAS and I tried following various guides on finetuning for 10Gbe, but it didn't help or even made things worse.

Then I had another idea, could it be that there is a compatibility issue with Ryzen CPUs? So I built another machine with Intel CPU, installed FreeNAS and run the same test as on my server before. And I got 9.4Gbps in both directions. So my conclusion is that there is indeed most likely a compatibility issue with Ryzen. I tried searching, but I haven't found anything relevant on this issue. Is there anyone who encountered the same issue before? And if so, have you been able to solve it? I've already spent many hours on this, so it would be great if anyone got any idea how to solve it :)

 

[mmbb]

So, I've done some more testing and it seems it's FreeBSD related and not specifically FreeNAS. What I've done is to install FreeBSD on both server and test machine and run the same test. Iperf between my workstation and my test machine is fine, 9.4Gbps in both direction. But from workstation to server I get about 5-7Gbps and in the opposit direction only about 3Gbps. Now I'm stuck, I don't know how to troubleshoot it further.