ESXi Vmotion/NFS traffic questions

[Elliot Dierksen]

First the background info. I have 4 servers that are all Cisco C240-M4SX's. Two are TrueNAS-12.0-U8.1 (bare metal), and two are ESXi 7.0.3. The NAS shares its data stores via NFS. The storage network is 40G connections in a Cisco Nexus 3064. The FreeNAS units have Chelsio T580 NIC's on the storage network and the ESXi units have the Cisco MLOM 40G converged NIC's on the storage network.

This may be a dumb question, but I was wondering why when I Vmotion from one FN to the other that the traffic rate on the receiving node is ~=50% of the traffic rate on the sending node. I decided to look at the traffic rates for all 4 of the 40G NIC's during a bulk Vmotion, and that answered some of the questions.

Code:

EBD-N3K# sh interface | b ^Ethernet1/49 | incl ^E|rate|Desc
Ethernet1/49 is up
  Description: FreeNAS
    30 seconds input rate 21166425176 bits/sec, 329890 packets/sec
    30 seconds output rate 121505024 bits/sec, 137763 packets/sec
    input rate 21.17 Gbps, 329.89 Kpps; output rate 121.50 Mbps, 137.76 Kpps
    300 seconds input rate 16490932024 bits/sec, 258887 packets/sec
    300 seconds output rate 98021440 bits/sec, 108077 packets/sec
    input rate 16.49 Gbps, 258.89 Kpps; output rate 98.02 Mbps, 108.08 Kpps
Ethernet1/50 is up
  Description: FreeNAS2
    30 seconds input rate 13627368 bits/sec, 12329 packets/sec
    30 seconds output rate 2384259008 bits/sec, 36927 packets/sec
    input rate 13.63 Mbps, 12.33 Kpps; output rate 2.38 Gbps, 36.93 Kpps
    300 seconds input rate 24082160 bits/sec, 21890 packets/sec
    300 seconds output rate 4496027736 bits/sec, 69378 packets/sec
    input rate 24.08 Mbps, 21.89 Kpps; output rate 4.50 Gbps, 69.38 Kpps
Ethernet1/51 is up
  Description: vm-ucs1 - vmnic3
    30 seconds input rate 2511710224 bits/sec, 174704 packets/sec
    30 seconds output rate 21180008624 bits/sec, 342222 packets/sec
    input rate 2.51 Gbps, 174.70 Kpps; output rate 21.18 Gbps, 342.22 Kpps
    300 seconds input rate 4600094168 bits/sec, 177520 packets/sec
    300 seconds output rate 16516739408 bits/sec, 280870 packets/sec
    input rate 4.60 Gbps, 177.52 Kpps; output rate 16.52 Gbps, 280.87 Kpps
Ethernet1/52 is up
  Description: vm-ucs2 - vmnic3
    30 seconds input rate 736 bits/sec, 0 packets/sec
    30 seconds output rate 1088 bits/sec, 0 packets/sec
    input rate 736 bps, 0 pps; output rate 1.09 Kbps, 0 pps
    300 seconds input rate 176 bits/sec, 0 packets/sec
    300 seconds output rate 536 bits/sec, 0 packets/sec
    input rate 176 bps, 0 pps; output rate 536 bps, 0 pps

I guess it is a 'well duh' thing that the sending unit sends to the ESXi host, which then sends the traffic to the receiving unit. It isn't really a NAS question, but why is the ESXi host not sending at anything close to the rate at which it received traffic from the NAS? Overall all I am pretty freaking happy since my NAS units are sustaining a sending rate of over 16Gbps, but I was wondering if there is anything I can do to make it go faster still.