bhyve vm network performance is very slow

fisherwei

Dabbler
Joined
Oct 28, 2013
Messages
47
TrueNAS 12 U1
CPU e5-2630Lv2
MEM 256G
ETH 10Gbe MTU1500

an ubuntu VM is running in truenas by bhyve, nic is virtio mode, bridged with ix1@host(truenas)

Test 1: VM to Host
truenas bridge0 (ix1 + vnet1) <---> ubuntu enp0s4
iperf server: truenas
iperf client: ubuntu
Code:
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec   572 MBytes   479 Mbits/sec    0             sender
[  5]   0.00-10.00  sec   569 MBytes   477 Mbits/sec                  receiver


Test 2: Windows to Host
windows 1gib <---> c3750x 1g port <---> same c3750x 10g port <---> truenas ix1
iperf server: truenas
iperf client: windows
Code:
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-10.00  sec  1.09 GBytes   937 Mbits/sec                  sender
[  4]   0.00-10.00  sec  1.09 GBytes   937 Mbits/sec                  receiver


Test 3: Windows to VM
windows 1gib <---> c3750x 1g port <---> same c3750x 10g port <---> truenas ix1 -> bridge0 (ix1 + vnet1) <---> ubuntu enp0s4
iperf server: ubuntu
iperf client: windows
Code:
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-10.00  sec   306 MBytes   257 Mbits/sec                  sender
[  4]   0.00-10.00  sec   306 MBytes   257 Mbits/sec                  receiver


Test 4: Jial to Host
truenas bridge0 (ix1 + vnet0.9) <---> jail
iperf server: truenas
iperf client: jail
Code:
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  1.90 GBytes  1.63 Gbits/sec    0             sender
[  5]   0.00-10.00  sec  1.90 GBytes  1.63 Gbits/sec                  receiver
 
Last edited:

Patrick M. Hausen

Hall of Famer
Joined
Nov 25, 2013
Messages
7,740
Have you disabled hardware offload for ix1?
 

Patrick M. Hausen

Hall of Famer
Joined
Nov 25, 2013
Messages
7,740
Yes.
 

fisherwei

Dabbler
Joined
Oct 28, 2013
Messages
47

After disabling, iperf from 250MB/s up to 350MB/s, it is something wrong still :(

Code:
C:\apps\iperf-3.1.3-win64>iperf3.exe -c 10.1.64.2
Connecting to host 10.1.64.2, port 5201
[  4] local 10.1.64.6 port 62767 connected to 10.1.64.2 port 5201
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-1.00   sec  46.9 MBytes   393 Mbits/sec
[  4]   1.00-2.00   sec  43.4 MBytes   364 Mbits/sec
[  4]   2.00-3.00   sec  35.0 MBytes   294 Mbits/sec
[  4]   3.00-4.00   sec  43.9 MBytes   368 Mbits/sec
[  4]   4.00-5.00   sec  35.8 MBytes   300 Mbits/sec
[  4]   5.00-6.00   sec  35.5 MBytes   298 Mbits/sec
[  4]   6.00-7.00   sec  42.4 MBytes   356 Mbits/sec
[  4]   7.00-8.00   sec  71.1 MBytes   596 Mbits/sec
[  4]   8.00-9.00   sec  38.2 MBytes   321 Mbits/sec
[  4]   9.00-10.00  sec  33.4 MBytes   280 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-10.00  sec   426 MBytes   357 Mbits/sec                  sender
[  4]   0.00-10.00  sec   426 MBytes   357 Mbits/sec                  receiver

iperf Done.
 

Mark St.

Dabbler
Joined
Dec 26, 2020
Messages
13
Looks similar.
i have a sequential read performance of about 400MB/s over network.
a win10-virtual-machine with nic virtio and gets between 100-200 MB/S with hardware-offload disabled (enabled is much worse)
 
Top