Register for the iXsystems Community to get an ad-free experience and exclusive discounts in our eBay Store.

SOLVED 10GbE ESXi 6.5 VMXNET 3 performance is poor with iperf3 tests.

Western Digital Drives - The Preferred Drives of FreeNAS and TrueNAS CORE
Status
Not open for further replies.

Brezlord

Member
Joined
Jan 7, 2017
Messages
167
Hi all,

I have been doing some testing with iperf3 and FreeNAS running as a VM in ESXi 6.5 and have found that 10Gbe networking to be poor. I have kept it simple and kept FreeNAS and a CentOS 7 VM on the same host to take any issues with switches and cabling out of the picture.

Everything is set for MTU 1500 and host to host network will hit 9.8GB/s so networking both physical and virtual is working correctly.

The bellow screen shot is CentOS 7 to CentsOS 7 on the same host.

Code:
[root@localhost ~]# iperf3 -c 192.168.10.58 -R
Connecting to host 192.168.10.58, port 5201
Reverse mode, remote host 192.168.10.58 is sending
[  4] local 192.168.10.62 port 59270 connected to 192.168.10.58 port 5201
[ ID] Interval		   Transfer	 Bandwidth
[  4]   0.00-1.00   sec  2.62 GBytes  22.5 Gbits/sec				
[  4]   1.00-2.00   sec  2.29 GBytes  19.6 Gbits/sec				
[  4]   2.00-3.00   sec  2.39 GBytes  20.5 Gbits/sec				
[  4]   3.00-4.00   sec  2.40 GBytes  20.6 Gbits/sec				
[  4]   4.00-5.00   sec  2.21 GBytes  19.0 Gbits/sec				
[  4]   5.00-6.00   sec  2.08 GBytes  17.8 Gbits/sec				
[  4]   6.00-7.00   sec  2.10 GBytes  18.1 Gbits/sec				
[  4]   7.00-8.00   sec  2.23 GBytes  19.2 Gbits/sec				
[  4]   8.00-9.00   sec  2.27 GBytes  19.5 Gbits/sec				
[  4]   9.00-10.00  sec  2.18 GBytes  18.7 Gbits/sec				
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval		   Transfer	 Bandwidth	   Retr
[  4]   0.00-10.00  sec  22.8 GBytes  19.6 Gbits/sec	0			 sender
[  4]   0.00-10.00  sec  22.8 GBytes  19.6 Gbits/sec				  receiver

iperf Done.
[root@localhost ~]#


The next test is FreeNAS to CentOS 7 on the same host.

Code:
root@nas1:~ # iperf3 -c 192.168.10.58 -R
Connecting to host 192.168.10.58, port 5201
Reverse mode, remote host 192.168.10.58 is sending
[  5] local 192.168.10.40 port 60293 connected to 192.168.10.58 port 5201
[ ID] Interval		   Transfer	 Bitrate
[  5]   0.00-1.00   sec   538 MBytes  4.51 Gbits/sec				
[  5]   1.00-2.00   sec   591 MBytes  4.96 Gbits/sec				
[  5]   2.00-3.00   sec   621 MBytes  5.21 Gbits/sec				
[  5]   3.00-4.00   sec   616 MBytes  5.17 Gbits/sec				
[  5]   4.00-5.00   sec   558 MBytes  4.68 Gbits/sec				
[  5]   5.00-6.00   sec   622 MBytes  5.22 Gbits/sec				
[  5]   6.00-7.00   sec   615 MBytes  5.16 Gbits/sec				
[  5]   7.00-8.00   sec   586 MBytes  4.91 Gbits/sec				
[  5]   8.00-9.00   sec   610 MBytes  5.11 Gbits/sec				
[  5]   9.00-10.00  sec   620 MBytes  5.20 Gbits/sec				
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval		   Transfer	 Bitrate		 Retr
[  5]   0.00-10.00  sec  5.84 GBytes  5.02 Gbits/sec  1405			 sender
[  5]   0.00-10.00  sec  5.84 GBytes  5.02 Gbits/sec				  receiver

iperf Done.
root@nas1:~ #


Just to be sure it was not a host issue I migrated all of the above machines to another host and got the same results. I then spun up a napp-it-SAN VM and got the same results and just to be sure I spun up a vanilla FreeBSD 11.1-RELEASE VM and got the same results as the FreeNAS server.

Is this issue to do with FreeBSD and more broadly UNIX systems driver for VMXNET 3. If this is the case is there any way to get 10Gbe out of a FreeNAS VM or is bare metal the only way? Or have I stuffed something up? I'm not sure on what config info you would need so just ask and I will get back to you.

Thanks,
Simon
 
Last edited:

kdragon75

Dedicated Sage
Joined
Aug 7, 2016
Messages
2,458
Try multiple parallel connections and a larger window. If I remember correctly iperf defaults to the minimum pack size of 64 Bytes on a single thread. This is a poor way to test max throughput but will give a feel for PPS.
https://iperf.fr/iperf-doc.php#3doc
 

Brezlord

Member
Joined
Jan 7, 2017
Messages
167
If I run the following from the FreeNAS server to the CentOS server on the same host I get good performance.
Code:
root@nas1:~ # iperf3 -c 192.168.10.58
Connecting to host 192.168.10.58, port 5201
[  5] local 192.168.10.40 port 17074 connected to 192.168.10.58 port 5201
[ ID] Interval		   Transfer	 Bitrate		 Retr  Cwnd
[  5]   0.00-1.00   sec  1.96 GBytes  16.8 Gbits/sec	0	444 KBytes	  
[  5]   1.00-2.00   sec  2.17 GBytes  18.6 Gbits/sec	0	585 KBytes	  
[  5]   2.00-3.00   sec  2.45 GBytes  21.0 Gbits/sec	0	697 KBytes	  
[  5]   3.00-4.00   sec  2.75 GBytes  23.6 Gbits/sec	0	794 KBytes	  
[  5]   4.00-5.00   sec  2.68 GBytes  23.0 Gbits/sec	0	876 KBytes	  
[  5]   5.00-6.00   sec  2.34 GBytes  20.1 Gbits/sec   63	571 KBytes	  
[  5]   6.00-7.00   sec  2.64 GBytes  22.7 Gbits/sec	0	688 KBytes	  
[  5]   7.00-8.00   sec  2.68 GBytes  23.0 Gbits/sec	0	782 KBytes	  
[  5]   8.00-9.00   sec  2.73 GBytes  23.4 Gbits/sec	0	871 KBytes	  
[  5]   9.00-10.00  sec  2.58 GBytes  22.1 Gbits/sec	0	945 KBytes	  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval		   Transfer	 Bitrate		 Retr
[  5]   0.00-10.00  sec  25.0 GBytes  21.4 Gbits/sec   63			 sender
[  5]   0.00-10.00  sec  25.0 GBytes  21.4 Gbits/sec				  receiver

iperf Done.
root@nas1:~ #


But if I run it in the reverse order CentOS 7 to FreeNAS on the same host using the -R switch performance is poor. I for got to mention this in the OP.
Code:
root@nas1:~ # iperf3 -c 192.168.10.58 -R
Connecting to host 192.168.10.58, port 5201
Reverse mode, remote host 192.168.10.58 is sending
[  5] local 192.168.10.40 port 37335 connected to 192.168.10.58 port 5201
[ ID] Interval		   Transfer	 Bitrate
[  5]   0.00-1.00   sec   540 MBytes  4.53 Gbits/sec				
[  5]   1.00-2.00   sec   590 MBytes  4.95 Gbits/sec				
[  5]   2.00-3.00   sec   592 MBytes  4.97 Gbits/sec				
[  5]   3.00-4.00   sec   590 MBytes  4.95 Gbits/sec				
[  5]   4.00-5.00   sec   562 MBytes  4.71 Gbits/sec				
[  5]   5.00-6.00   sec   591 MBytes  4.96 Gbits/sec				
[  5]   6.00-7.00   sec   560 MBytes  4.70 Gbits/sec				
[  5]   7.00-8.00   sec   595 MBytes  4.99 Gbits/sec				
[  5]   8.00-9.00   sec   576 MBytes  4.83 Gbits/sec				
[  5]   9.00-10.00  sec   597 MBytes  5.01 Gbits/sec				
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval		   Transfer	 Bitrate		 Retr
[  5]   0.00-10.00  sec  5.66 GBytes  4.86 Gbits/sec  1202			 sender
[  5]   0.00-10.00  sec  5.66 GBytes  4.86 Gbits/sec				  receiver

iperf Done.
root@nas1:~ #


Now if i run 2 in parallel in revers still not good.
Code:
root@nas1:~ # iperf3 -c 192.168.10.58 -P 2 -R
Connecting to host 192.168.10.58, port 5201
Reverse mode, remote host 192.168.10.58 is sending
[  5] local 192.168.10.40 port 55903 connected to 192.168.10.58 port 5201
[  7] local 192.168.10.40 port 60401 connected to 192.168.10.58 port 5201
[ ID] Interval		   Transfer	 Bitrate
[  5]   0.00-1.00   sec   288 MBytes  2.42 Gbits/sec				
[  7]   0.00-1.00   sec   217 MBytes  1.82 Gbits/sec				
[SUM]   0.00-1.00   sec   505 MBytes  4.23 Gbits/sec				
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   1.00-2.00   sec   238 MBytes  2.00 Gbits/sec				
[  7]   1.00-2.00   sec   328 MBytes  2.75 Gbits/sec				
[SUM]   1.00-2.00   sec   566 MBytes  4.75 Gbits/sec				
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   2.00-3.08   sec   182 MBytes  1.41 Gbits/sec				
[  7]   2.00-3.08   sec   248 MBytes  1.92 Gbits/sec				
[SUM]   2.00-3.08   sec   431 MBytes  3.33 Gbits/sec				
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   3.08-4.00   sec   215 MBytes  1.97 Gbits/sec				
[  7]   3.08-4.00   sec   186 MBytes  1.70 Gbits/sec				
[SUM]   3.08-4.00   sec   401 MBytes  3.67 Gbits/sec				
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   4.00-5.00   sec   300 MBytes  2.51 Gbits/sec				
[  7]   4.00-5.00   sec   255 MBytes  2.14 Gbits/sec				
[SUM]   4.00-5.00   sec   555 MBytes  4.66 Gbits/sec				
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   5.00-6.00   sec   340 MBytes  2.85 Gbits/sec				
[  7]   5.00-6.00   sec   248 MBytes  2.08 Gbits/sec				
[SUM]   5.00-6.00   sec   588 MBytes  4.93 Gbits/sec				
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   6.00-7.00   sec   279 MBytes  2.34 Gbits/sec				
[  7]   6.00-7.00   sec   319 MBytes  2.67 Gbits/sec				
[SUM]   6.00-7.00   sec   598 MBytes  5.02 Gbits/sec				
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   7.00-8.00   sec   250 MBytes  2.10 Gbits/sec				
[  7]   7.00-8.00   sec   324 MBytes  2.71 Gbits/sec				
[SUM]   7.00-8.00   sec   573 MBytes  4.81 Gbits/sec				
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   8.00-9.00   sec   230 MBytes  1.93 Gbits/sec				
[  7]   8.00-9.00   sec   365 MBytes  3.06 Gbits/sec				
[SUM]   8.00-9.00   sec   595 MBytes  4.99 Gbits/sec				
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   9.00-10.00  sec   287 MBytes  2.41 Gbits/sec				
[  7]   9.00-10.00  sec   318 MBytes  2.67 Gbits/sec				
[SUM]   9.00-10.00  sec   605 MBytes  5.08 Gbits/sec				
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval		   Transfer	 Bitrate		 Retr
[  5]   0.00-10.00  sec  2.55 GBytes  2.19 Gbits/sec  2469			 sender
[  5]   0.00-10.00  sec  2.55 GBytes  2.19 Gbits/sec				  receiver
[  7]   0.00-10.00  sec  2.74 GBytes  2.36 Gbits/sec  2179			 sender
[  7]   0.00-10.00  sec  2.74 GBytes  2.36 Gbits/sec				  receiver
[SUM]   0.00-10.00  sec  5.29 GBytes  4.55 Gbits/sec  4648			 sender
[SUM]   0.00-10.00  sec  5.29 GBytes  4.55 Gbits/sec				  receiver

iperf Done.


There seems to be a 5Gbits/s limit when FreeNAS is receiving data.
 
Last edited:

kdragon75

Dedicated Sage
Joined
Aug 7, 2016
Messages
2,458
I see you added the -P 2 option. have you tried a larger window size?

Also it sounds like the centOS version may have different defaults for sending. o_O

Try always setting -P 2 -w 128k and see if the results are more consistent going both directions.
 

Brezlord

Member
Joined
Jan 7, 2017
Messages
167
Thanks for the replies.

I have tested with Ubuntu as well as CentOS 7 and there is no difference. I believe iperf3 defaults to 128K. Test bellow with no improvement. Remember CentOS 7 to CentOS 7 works as expected so by process of elimination the issue is with FreeBSD.

Code:
root@nas1:~ # iperf3 -c 192.168.10.58 -R -P 2 -w 128k
Connecting to host 192.168.10.58, port 5201
Reverse mode, remote host 192.168.10.58 is sending
[  5] local 192.168.10.40 port 14279 connected to 192.168.10.58 port 5201
[  7] local 192.168.10.40 port 14280 connected to 192.168.10.58 port 5201
[ ID] Interval		   Transfer	 Bitrate
[  5]   0.00-1.00   sec   228 MBytes  1.91 Gbits/sec				
[  7]   0.00-1.00   sec   225 MBytes  1.89 Gbits/sec				
[SUM]   0.00-1.00   sec   453 MBytes  3.80 Gbits/sec				
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   1.00-2.00   sec   319 MBytes  2.67 Gbits/sec				
[  7]   1.00-2.00   sec   313 MBytes  2.62 Gbits/sec				
[SUM]   1.00-2.00   sec   631 MBytes  5.30 Gbits/sec				
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   2.00-3.00   sec   308 MBytes  2.58 Gbits/sec				
[  7]   2.00-3.00   sec   300 MBytes  2.51 Gbits/sec				
[SUM]   2.00-3.00   sec   608 MBytes  5.10 Gbits/sec				
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   3.00-4.00   sec   300 MBytes  2.52 Gbits/sec				
[  7]   3.00-4.00   sec   293 MBytes  2.46 Gbits/sec				
[SUM]   3.00-4.00   sec   593 MBytes  4.98 Gbits/sec				
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   4.00-5.00   sec   323 MBytes  2.71 Gbits/sec				
[  7]   4.00-5.00   sec   316 MBytes  2.65 Gbits/sec				
[SUM]   4.00-5.00   sec   639 MBytes  5.36 Gbits/sec				
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   5.00-6.00   sec   317 MBytes  2.66 Gbits/sec				
[  7]   5.00-6.00   sec   312 MBytes  2.61 Gbits/sec				
[SUM]   5.00-6.00   sec   629 MBytes  5.27 Gbits/sec				
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   6.00-7.00   sec   307 MBytes  2.58 Gbits/sec				
[  7]   6.00-7.00   sec   300 MBytes  2.51 Gbits/sec				
[SUM]   6.00-7.00   sec   607 MBytes  5.09 Gbits/sec				
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   7.00-8.00   sec   315 MBytes  2.64 Gbits/sec				
[  7]   7.00-8.00   sec   308 MBytes  2.58 Gbits/sec				
[SUM]   7.00-8.00   sec   623 MBytes  5.22 Gbits/sec				
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   8.00-9.00   sec   299 MBytes  2.51 Gbits/sec				
[  7]   8.00-9.00   sec   293 MBytes  2.46 Gbits/sec				
[SUM]   8.00-9.00   sec   592 MBytes  4.97 Gbits/sec				
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   9.00-10.00  sec   312 MBytes  2.62 Gbits/sec				
[  7]   9.00-10.00  sec   305 MBytes  2.56 Gbits/sec				
[SUM]   9.00-10.00  sec   617 MBytes  5.18 Gbits/sec				
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval		   Transfer	 Bitrate		 Retr
[  5]   0.00-10.00  sec  2.96 GBytes  2.54 Gbits/sec	0			 sender
[  5]   0.00-10.00  sec  2.96 GBytes  2.54 Gbits/sec				  receiver
[  7]   0.00-10.00  sec  2.90 GBytes  2.49 Gbits/sec	0			 sender
[  7]   0.00-10.00  sec  2.90 GBytes  2.49 Gbits/sec				  receiver
[SUM]   0.00-10.00  sec  5.85 GBytes  5.03 Gbits/sec	0			 sender
[SUM]   0.00-10.00  sec  5.85 GBytes  5.03 Gbits/sec				  receiver

iperf Done.
root@nas1:~ #
 
Last edited:

Brezlord

Member
Joined
Jan 7, 2017
Messages
167
I have solved my problem and can confirm that there is nothing wrong with the VMXNET 3 drivers in FreeBSD 11 and by extension FreeNAS 11. I'm running FreeNAS-11.1-U2 as a VM on ESXi 6.5U1 with the HBA passed through to the VM. The server has an Intel X520-DA2 NIC which supports TSO and LRO. If a NIC supports TSO and LRO ESXi will automatically enable these functions, you can find out if they are enabled by following the instructions here Managing Network Resources. I all ready new that the virtual and physical networks where configured correctly because of iperf testing showed expected speeds so the issue had to be with FreeNAS settings. I added the various turnings found in the forums which made no difference. I then went to Network-->Interfaces-->vmx0-->Edit and added tso lro into the options box and hit save. Bam, 10GbE is now working as it should. Just to be clear the bellow results are from a default install of FreeNAS with no turnings and just TSO and LRO enabled. List of available settings that can be put into the NIC options box can be found here.

FreeNAS VM to CentOS 7 VM on the same host
Code:
root@nas1:~ # iperf3 -c 192.168.10.58 -t20 -R
Connecting to host 192.168.10.58, port 5201
Reverse mode, remote host 192.168.10.58 is sending
[  5] local 192.168.10.40 port 19144 connected to 192.168.10.58 port 5201
[ ID] Interval		   Transfer	 Bitrate
[  5]   0.00-1.00   sec  1.76 GBytes  15.1 Gbits/sec				
[  5]   1.00-2.00   sec  1.43 GBytes  12.3 Gbits/sec				
[  5]   2.00-3.00   sec  1.15 GBytes  9.88 Gbits/sec				
[  5]   3.00-4.00   sec  1.52 GBytes  13.0 Gbits/sec				
[  5]   4.00-5.00   sec  1.34 GBytes  11.5 Gbits/sec				
[  5]   5.00-6.00   sec  1.47 GBytes  12.7 Gbits/sec				
[  5]   6.00-7.00   sec  1.66 GBytes  14.3 Gbits/sec				
[  5]   7.00-8.00   sec  1.30 GBytes  11.2 Gbits/sec				
[  5]   8.00-9.00   sec  1.45 GBytes  12.5 Gbits/sec				
[  5]   9.00-10.00  sec  1.59 GBytes  13.7 Gbits/sec				
[  5]  10.00-11.00  sec  1.52 GBytes  13.1 Gbits/sec				
[  5]  11.00-12.00  sec  1.90 GBytes  16.3 Gbits/sec				
[  5]  12.00-13.00  sec  1.92 GBytes  16.5 Gbits/sec				
[  5]  13.00-14.00  sec  1.73 GBytes  14.8 Gbits/sec				
[  5]  14.00-15.00  sec  1.92 GBytes  16.5 Gbits/sec				
[  5]  15.00-16.00  sec  1.73 GBytes  14.9 Gbits/sec				
[  5]  16.00-17.00  sec  1.87 GBytes  16.1 Gbits/sec				
[  5]  17.00-18.00  sec  1.88 GBytes  16.2 Gbits/sec				
[  5]  18.00-19.00  sec  1.92 GBytes  16.5 Gbits/sec				
[  5]  19.00-20.00  sec  1.97 GBytes  16.9 Gbits/sec				
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval		   Transfer	 Bitrate		 Retr
[  5]   0.00-20.00  sec  33.0 GBytes  14.2 Gbits/sec   28			 sender
[  5]   0.00-20.00  sec  33.0 GBytes  14.2 Gbits/sec				  receiver

iperf Done.


FreeNAS VM to CentOS 7 VM on different hosts
Code:
root@nas1:~ # iperf3 -c 192.168.10.58 -t20 -R
Connecting to host 192.168.10.58, port 5201
Reverse mode, remote host 192.168.10.58 is sending
[  5] local 192.168.10.40 port 18660 connected to 192.168.10.58 port 5201
[ ID] Interval		   Transfer	 Bitrate
[  5]   0.00-1.00   sec  1022 MBytes  8.58 Gbits/sec				
[  5]   1.00-2.00   sec  1.02 GBytes  8.73 Gbits/sec				
[  5]   2.00-3.00   sec  1.01 GBytes  8.65 Gbits/sec				
[  5]   3.00-4.00   sec  1.02 GBytes  8.72 Gbits/sec				
[  5]   4.00-5.00   sec  1.01 GBytes  8.71 Gbits/sec				
[  5]   5.00-6.00   sec  1.02 GBytes  8.78 Gbits/sec				
[  5]   6.00-7.00   sec  1.02 GBytes  8.73 Gbits/sec				
[  5]   7.00-8.00   sec  1.03 GBytes  8.83 Gbits/sec				
[  5]   8.00-9.00   sec  1.02 GBytes  8.75 Gbits/sec				
[  5]   9.00-10.00  sec  1.03 GBytes  8.83 Gbits/sec				
[  5]  10.00-11.00  sec  1011 MBytes  8.48 Gbits/sec				
[  5]  11.00-12.00  sec  1.03 GBytes  8.84 Gbits/sec				
[  5]  12.00-13.00  sec  1.03 GBytes  8.86 Gbits/sec				
[  5]  13.00-14.00  sec  1.02 GBytes  8.77 Gbits/sec				
[  5]  14.00-15.00  sec  1.03 GBytes  8.86 Gbits/sec				
[  5]  15.00-16.00  sec  1.02 GBytes  8.79 Gbits/sec				
[  5]  16.00-17.00  sec  1.01 GBytes  8.66 Gbits/sec				
[  5]  17.00-18.00  sec  1.02 GBytes  8.72 Gbits/sec				
[  5]  18.00-19.00  sec  1.02 GBytes  8.75 Gbits/sec				
[  5]  19.00-20.00  sec  1.03 GBytes  8.87 Gbits/sec				
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval		   Transfer	 Bitrate		 Retr
[  5]   0.00-20.00  sec  20.4 GBytes  8.75 Gbits/sec  247			 sender
[  5]   0.00-20.00  sec  20.4 GBytes  8.75 Gbits/sec				  receiver

iperf Done.


Now for disk read/write test to non-synced data set
SMB to non sync data set.PNG


Synced data set with Samsung PM953 MZ1LV480HCHP-000MU 480GB NVMe SLOG
SMB to sync data set NVMe.PNG


I think It's time for and Intel Optane SLOG.

If I make any changes that improve performance I will post them here.

Simon
 
Last edited:
Status
Not open for further replies.
Top