40g Mellanox Primer for Freenas 11.1.u5 stable

[rkim43]

hello,

I am trying to maximize the performance of my frenas system at work. here is some info for any suggestions

FreeBSD 11.1-STABLE
System: HP ProLiant DL380p Gen8
CPU: Intel(R) Xeon(R) CPU E5-2680 v2 @ 2.80GHz (2793.32-MHz K8-class CPU)
Build FreeNAS-11.1-U5
Memory 384 GB
Nvme card: Intel Optane DC P4800X Series SSDPED1K375GAQ1 375GB AIC (HHHL) PCI-Express SSD.
ZFS log/cache setup for nvme card: 50GB log/325GB cache
Mellanox ConnectX-3 40GB card with mlx4 driver
Raid setup: 13 raid-z2 with 10 HP MB4000FCWDK 4.00 TB disks


i enabled auto-tune and then added some tunable values that I attached.

I am currently getting throughput values through iperf3 testing

iperf3 -c 10.1.245.2

Connecting to host 10.1.245.2, port 5201

[ 4] local 10.1.245.115 port 55489 connected to 10.1.245.2 port 5201

[ ID] Interval Transfer Bandwidth Retr Cwnd

[ 4] 0.00-1.00 sec 3.39 GBytes 29.1 Gbits/sec 107 1.27 MBytes
[ 4] 1.00-2.00 sec 2.02 GBytes 17.4 Gbits/sec 19 1.33 MBytes
[ 4] 2.00-3.00 sec 2.03 GBytes 17.4 Gbits/sec 13 1.37 MBytes
[ 4] 3.00-4.00 sec 2.03 GBytes 17.4 Gbits/sec 9 1008 KBytes
[ 4] 4.00-5.00 sec 2.04 GBytes 17.5 Gbits/sec 8 1.10 MBytes
[ 4] 5.00-6.00 sec 2.03 GBytes 17.5 Gbits/sec 15 1.13 MBytes
[ 4] 6.00-7.00 sec 2.01 GBytes 17.3 Gbits/sec 8 1.20 MBytes
[ 4] 7.00-8.00 sec 2.03 GBytes 17.4 Gbits/sec 9 1.20 MBytes
[ 4] 8.00-9.00 sec 2.03 GBytes 17.4 Gbits/sec 7 1.29 MBytes
[ 4] 9.00-10.00 sec 2.02 GBytes 17.4 Gbits/sec 9 1.34 MBytes

- - - - - - - - - - - - - - - - - - - - - - - - -

[ ID] Interval Transfer Bandwidth Retr

[ 4] 0.00-10.00 sec 21.6 GBytes 18.6 Gbits/sec 204 sender
[ 4] 0.00-10.00 sec 21.6 GBytes 18.6 Gbits/sec receiver


I want to try and increase transfer and bandwith and reducing the retries. I have seen the 10g primers but this is for a 40g connectx-3 mellanox card.

thx rick kim

 

[rkim43]

sorry here is my tunables attachment

 

[dlavigne]

Have you made progress with this? Does updating to U6 provide an improvement?

 

[Elliott]

Check the CPU usage while running this test. I think you'll find that iperf3 is maxing out one core. Try running more threads, e.g. add -P2 or -P4 to the client command.
Did those tunables make any difference for you?