My TrueNAS seems to use the wrong network card.

roysen

roysen

Dabbler
Joined
Dec 11, 2021
Messages
15
My TrueNas box has two network cards. One Intel X520 with dual-port that is combined into lagg0. It also has an integrated em0 card that I use as a bridge for my VM.
When running iperf3 it seems to use my lagg0 network for incoming connections but when I run it the other way it seems like it uses my bridge or em0. Bridging seems to make so many problems on FreeNas. Does anyone have a solution to how to set this up without losing connections between my host and its VM and at the same time being able to use the full speed of my 10Gb network card?

Iperf3 is started like this on my TrueNas: iperf3 -s -B 192.168.68.5

From my workstation:
Code:
iperf3 -c 192.168.68.5
Connecting to host 192.168.68.5, port 5201
[  4] local 192.168.68.100 port 36358 connected to 192.168.68.5 port 5201
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-1.00   sec  1.02 GBytes  8.79 Gbits/sec
[  4]   1.00-2.00   sec  1.10 GBytes  9.43 Gbits/sec
[  4]   2.00-3.00   sec  1.10 GBytes  9.44 Gbits/sec
[  4]   3.00-4.00   sec  1.10 GBytes  9.45 Gbits/sec
[  4]   4.00-5.00   sec  1.10 GBytes  9.44 Gbits/sec
[  4]   5.00-6.00   sec  1.10 GBytes  9.44 Gbits/sec
[  4]   6.00-7.00   sec  1.10 GBytes  9.46 Gbits/sec
[  4]   7.00-8.00   sec  1.10 GBytes  9.45 Gbits/sec
[  4]   8.00-9.00   sec  1.10 GBytes  9.44 Gbits/sec
[  4]   9.00-10.00  sec  1.10 GBytes  9.45 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-10.00  sec  10.9 GBytes  9.38 Gbits/sec                  sender
[  4]   0.00-10.00  sec  10.9 GBytes  9.38 Gbits/sec                  receiver


Iperf3 in Reverse mode from my workstation:
Code:
iperf3 -c 192.168.68.5 -R
Connecting to host 192.168.68.5, port 5201
Reverse mode, remote host 192.168.68.5 is sending
[  4] local 192.168.68.100 port 37214 connected to 192.168.68.5 port 5201
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-1.00   sec   107 MBytes   899 Mbits/sec
[  4]   1.00-2.00   sec   112 MBytes   941 Mbits/sec
[  4]   2.00-3.00   sec   112 MBytes   942 Mbits/sec
[  4]   3.00-4.00   sec   112 MBytes   942 Mbits/sec
[  4]   4.00-5.00   sec   112 MBytes   940 Mbits/sec
[  4]   5.00-6.00   sec   112 MBytes   942 Mbits/sec
[  4]   6.00-7.00   sec   112 MBytes   942 Mbits/sec
[  4]   7.00-8.00   sec   112 MBytes   942 Mbits/sec
[  4]   8.00-9.00   sec   112 MBytes   942 Mbits/sec
[  4]   9.00-10.00  sec   112 MBytes   942 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-10.00  sec  1.09 GBytes   938 Mbits/sec    0             sender
[  4]   0.00-10.00  sec  1.09 GBytes   938 Mbits/sec                  receiver



ipconfig output:
Code:
ix0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
        description: member of lagg0
        options=e53fbb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
        ether 00:1b:21:a3:94:48
        media: Ethernet autoselect (10Gbase-Twinax <full-duplex,rxpause,txpause>)
        status: active
        nd6 options=9<PERFORMNUD,IFDISABLED>
ix1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
        description: member of lagg0
        options=e53fbb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
        ether 00:1b:21:a3:94:48
        hwaddr 00:1b:21:a3:94:49
        media: Ethernet autoselect (10Gbase-Twinax <full-duplex,rxpause,txpause>)
        status: active
        nd6 options=9<PERFORMNUD,IFDISABLED>
em0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=812098<VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC,VLAN_HWFILTER>
        ether bc:ee:7b:df:f7:a8
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=9<PERFORMNUD,IFDISABLED>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x4
        inet 127.0.0.1 netmask 0xff000000
        groups: lo
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
pflog0: flags=0<> metric 0 mtu 33160
        groups: pflog
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: Bridge for VMs
        ether 02:7a:ae:08:2c:00
        inet 192.168.68.6 netmask 0xffffff00 broadcast 192.168.68.255
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto stp-rstp maxaddr 2000 timeout 1200
        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
        member: vnet0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 7 priority 128 path cost 2000000
        member: em0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 3 priority 128 path cost 55
        groups: bridge
        nd6 options=9<PERFORMNUD,IFDISABLED>
vnet0: flags=8942<BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=80000<LINKSTATE>
        ether fe:a0:98:ff:ff:ff
        hwaddr 58:9c:fc:10:ff:b6
        groups: tap
        media: Ethernet autoselect
        status: active
        nd6 options=1<PERFORMNUD>
        Opened by PID 1505
lagg0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
        description: lagg0
        options=e53fbb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
        ether 00:1b:21:a3:94:48
        inet 192.168.68.5 netmask 0xffffff00 broadcast 192.168.68.255
        laggproto lacp lagghash l2,l3,l4
        laggport: ix0 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
        laggport: ix1 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
        groups: lagg
        media: Ethernet autoselect
        status: active
        nd6 options=9<PERFORMNUD,IFDISABLED>
 
jgreco

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,680
You have a broken networking setup.

See

Resource - Multiple network interfaces on a single subnet

We've had a number of people over the years bring up the topic of multiple network interfaces on a single subnet. Often done by Windows administrators or storage admins, where it works to varying degrees, this is sadly not proper IP networking...
www.truenas.com www.truenas.com

And then go and delete the IP address you have configured on bridge0.

roysen said:
Bridging seems to make so many problems on FreeNas.
Click to expand...

No, it's just that you need to actually configure your networking correctly, and as the complexity increases, the ease of misconfiguration grows exponentially.
 
roysen

roysen

Dabbler
Joined
Dec 11, 2021
Messages
15
jgreco said:
You have a broken networking setup.

See

Resource - Multiple network interfaces on a single subnet

We've had a number of people over the years bring up the topic of multiple network interfaces on a single subnet. Often done by Windows administrators or storage admins, where it works to varying degrees, this is sadly not proper IP networking...
www.truenas.com www.truenas.com

And then go and delete the IP address you have configured on bridge0.



No, it's just that you need to actually configure your networking correctly, and as the complexity increases, the ease of misconfiguration grows exponentially.
Click to expand...
That did of course fix my problem with iperf speed but now my VM will not even start due to the missing IP on my bridge.

CallError​

[EFAULT] VM will not start as VNC Device: 192.168.68.6:6440 device(s) are not available.
Code:
Error: Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/middlewared/main.py", line 138, in call_method
    result = await self.middleware._call(message['method'], serviceobj, methodobj, params, app=self,
  File "/usr/local/lib/python3.9/site-packages/middlewared/main.py", line 1224, in _call
    return await self.run_in_executor(prepared_call.executor, methodobj, *prepared_call.args)
  File "/usr/local/lib/python3.9/site-packages/middlewared/main.py", line 1128, in run_in_executor
    return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs))
  File "/usr/local/lib/python3.9/concurrent/futures/thread.py", line 52, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/local/lib/python3.9/site-packages/middlewared/schema.py", line 979, in nf
    return f(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/middlewared/plugins/vm.py", line 1599, in start
    self.vms[vm['name']].start(vm_data=vm)
  File "/usr/local/lib/python3.9/site-packages/middlewared/plugins/vm.py", line 173, in start
    raise CallError(
middlewared.service_exception.CallError: [EFAULT] VM will not start as VNC Device: 192.168.68.6:6440 device(s) are not available.


I tried to fix that by changing my bridge to be on lagg0 but that didn't work. In my VM I can't see any way to fix it. This problem existed even before I made a lagg device so I guess it is not really connected to the lagg setup. My thought about making the lagg device was to move my VM and bridge over to the 10Gbe network if I got it working.

And as I told "Bridging" is so confusing, and there is so many that got into problems setting it up.

Does anyone have a good solution to my problem?
ifconfig now
Code:
ix0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 9000
        description: member of lagg0
        options=e53bbb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
        ether 00:1b:21:a3:94:48
        media: Ethernet autoselect (10Gbase-Twinax <full-duplex,rxpause,txpause>)
        status: active
        nd6 options=9<PERFORMNUD,IFDISABLED>
ix1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 9000
        description: member of lagg0
        options=e53bbb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
        ether 00:1b:21:a3:94:48
        hwaddr 00:1b:21:a3:94:49
        media: Ethernet autoselect (10Gbase-Twinax <full-duplex,rxpause,txpause>)
        status: active
        nd6 options=9<PERFORMNUD,IFDISABLED>
em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=812098<VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC,VLAN_HWFILTER>
        ether bc:ee:7b:df:f7:a8
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=9<PERFORMNUD,IFDISABLED>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x4
        inet 127.0.0.1 netmask 0xff000000
        groups: lo
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
pflog0: flags=0<> metric 0 mtu 33160
        groups: pflog
lagg0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 9000
        description: lagg0
        options=e53bbb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
        ether 00:1b:21:a3:94:48
        inet 192.168.68.5 netmask 0xffffff00 broadcast 192.168.68.255
        laggproto lacp lagghash l2,l3,l4
        laggport: ix0 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
        laggport: ix1 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
        groups: lagg
        media: Ethernet autoselect
        status: active
        nd6 options=9<PERFORMNUD,IFDISABLED>
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: Bridge for VMs
        ether 02:7a:ae:08:2c:00
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto stp-rstp maxaddr 2000 timeout 1200
        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
        member: lagg0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 6 priority 128 path cost 2000
        groups: bridge
        nd6 options=9<PERFORMNUD,IFDISABLED>
 
Patrick M. Hausen

Patrick M. Hausen

Hall of Famer
Joined
Nov 25, 2013
Messages
7,776
Remove the IP address from the bridge. Adjust the configuration of the VNC device of your VM to bind to 0.0.0.0. Use the IP address of your lagg0 to connect to VNC - the VM itself will still be connected to the bridge interface.

Nothing confusing about that, just network 101 ...

A bridge is a layer 2 device. A virtual switch if you want to call it that.
 
You must log in or register to reply here.

Similar threads

F
Slow pool read performance
Replies
0
Views
664
fulder
F
fisherwei
bhyve vm network performance is very slow
Replies
5
Views
2K
Mark St.
M
S
Slow network performance from Mac Pro w/ Thunderbolt 10GbE
Replies
0
Views
1K
skyforest
S
HolyK
SOLVED Disable hardware offload (bandwidth still meh...)
Replies
1
Views
5K
Patrick M. Hausen
Patrick M. Hausen
Marcelo Ruiz
SOLVED Extremely Bad Networking Performance TrueNas 12-Core Freenas Mini
Replies
5
Views
2K
Marcelo Ruiz
Marcelo Ruiz
Top