Bhyve VM and lagg

[rafaelagp]

Hello, I'm very new to the BSD world, but I'm pretty familiar with Linux.
I'm trying to configure a TrueNAS Core setup to replace my OMV 4 based server.

I have the five ethernet ports in the server set as a roundrobin lagg interface and I'm trying to install Debian in a VM.
The problem I'm having is that no matter if I use virtio or the other option for NIC, the VM gets no network over the lagg interface. If I remove a port from the lagg interface and bind the VM's NIC to it then it networks fine.

Does Bhyve not support lagg interfaces or am I doing something wrong?

Thanks in advance.

 

[Patrick M. Hausen]

It does. You will probably have to create a bridge interface with the lagg as the only member, then assign that bridge to the virtio device of your VM.

 

[rafaelagp]

Could not make it work creating a bridge interface with lagg as the only member.
Could you explain the bridge configuration in more detail, Patrick?
Any other ideas?

Thanks again.

 

[Patrick M. Hausen]

I am using a VLAN on top of the lagg, but fundamentally it should be the same. Just skip the VLAN and make the lagg0 a member of the bridge like in these screenshots:

Then in your VMs NIC config pick bridge0:


And make sure to disable HW acceleration for the physical NICs. And if you have an IP address configuration on the lagg, move that to the bridge once you got everything else working.

 

[rafaelagp]

make the lagg0 a member of the bridge like in these screenshots

Well, I got that much right :p

And make sure to disable HW acceleration for the physical NICs.

How do I do that?

if you have an IP address configuration on the lagg, move that to the bridge once you got everything else working.

Ah! That might be it, I'll try it out.

 

[Patrick M. Hausen]

How do I do that?

Network - Interfaces - edit Interface - activate "Disable HW acceleration" checkbox - save.

Ah! That might be it, I'll try it out.

Probably not. VMs should be able to communicate fine. That point is about the connection of the TrueNAS itself.

 

[rafaelagp]

Network - Interfaces - edit Interface - activate "Disable HW acceleration" checkbox - save.

That would be the same as the Disable Hardware Offloading checkbox, right? At least that's what it's labeled in my TrueNAS Core version. I've set that for all the physical interfaces. No change to the VM networking status.

Probably not. VMs should be able to communicate fine. That point is about the connection of the TrueNAS itself.

I had the lagg interface set up with a static ip and TrueNAS was networking fine but the VM had no network. Same is true with the bridge set up, TrueNAS networks as well but not the VM.

So, still no dice. Any more ideas to try?

 

[Patrick M. Hausen]

That would be the same as the Disable Hardware Offloading checkbox, right?

Yes, that's what I meant.

Any more ideas to try?

Output of ifconfig -a on the NAS. Screenshot of your VMs network interface configuration. Output or screenshot of the network configuration inside the VM, please.

 

[rafaelagp]

Output of ifconfig -a on the NAS.

Code:

igb0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=c103ba<TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWFILTER,TXCSUM_IPV6>
        ether a0:36:9f:a8:e2:50
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=9<PERFORMNUD,IFDISABLED>
igb1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=c103ba<TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWFILTER,TXCSUM_IPV6>
        ether a0:36:9f:a8:e2:50
        hwaddr a0:36:9f:a8:e2:51
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=9<PERFORMNUD,IFDISABLED>
igb2: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=c103ba<TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWFILTER,TXCSUM_IPV6>
        ether a0:36:9f:a8:e2:50
        hwaddr a0:36:9f:a8:e2:52
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=9<PERFORMNUD,IFDISABLED>
igb3: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=c103ba<TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWFILTER,TXCSUM_IPV6>
        ether a0:36:9f:a8:e2:50
        hwaddr a0:36:9f:a8:e2:53
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=9<PERFORMNUD,IFDISABLED>
re0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=82098<VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC,LINKSTATE>
        ether 0a:e0:af:ad:02:bf
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=9<PERFORMNUD,IFDISABLED>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x6
        inet 127.0.0.1 netmask 0xff000000
        groups: lo
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
pflog0: flags=0<> metric 0 mtu 33160
        groups: pflog
lagg0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: lan
        options=c103ba<TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWFILTER,TXCSUM_IPV6>
        ether a0:36:9f:a8:e2:50
        laggproto roundrobin lagghash l2,l3,l4
        laggport: igb0 flags=4<ACTIVE>
        laggport: igb1 flags=4<ACTIVE>
        laggport: igb2 flags=4<ACTIVE>
        laggport: igb3 flags=4<ACTIVE>
        groups: lagg
        media: Ethernet autoselect
        status: active
        nd6 options=9<PERFORMNUD,IFDISABLED>
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        ether 02:fd:d1:fe:04:00
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto stp-rstp maxaddr 2000 timeout 1200
        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
        member: vnet0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 10 priority 128 path cost 2000000
        member: re0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 5 priority 128 path cost 20000
        groups: bridge
        nd6 options=1<PERFORMNUD>
bridge1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        ether 02:fd:d1:fe:04:01
        inet 192.168.1.10 netmask 0xffffff00 broadcast 192.168.1.255
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto stp-rstp maxaddr 2000 timeout 1200
        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
        member: lagg0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 8 priority 128 path cost 5000
        groups: bridge
        nd6 options=9<PERFORMNUD,IFDISABLED>
vnet0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=80000<LINKSTATE>
        ether fe:a0:98:41:95:f6
        hwaddr 58:9c:fc:10:85:57
        groups: tap
        media: Ethernet autoselect
        status: active
        nd6 options=1<PERFORMNUD>
        Opened by PID 3948

Screenshot of your VMs network interface configuration.

Output or screenshot of the network configuration inside the VM, please.

I've also tried configuring a static ip and gateway, no dice either.

 

[Patrick M. Hausen]

Was the VM booted when you did the ifconfig on the NAS? I guess not.
Is the interface in the guest Linux really enp0s6? Try ifconfig -a in the guest. If you e.g. removed the CDROM device after installation, PCI based device names tend to get juggled around.

 

[rafaelagp]

Was the VM booted when you did the ifconfig on the NAS? I guess not.

I think it wasn't, but here's another ifconfig -a with it running:

Code:

root@thoth[~]# ifconfig -a
igb0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=8100b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWFILTER>
        ether a0:36:9f:a8:e2:50
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=9<PERFORMNUD,IFDISABLED>
igb1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=8100b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWFILTER>
        ether a0:36:9f:a8:e2:50
        hwaddr a0:36:9f:a8:e2:51
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=9<PERFORMNUD,IFDISABLED>
igb2: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=8100b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWFILTER>
        ether a0:36:9f:a8:e2:50
        hwaddr a0:36:9f:a8:e2:52
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=9<PERFORMNUD,IFDISABLED>
igb3: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=8100b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWFILTER>
        ether a0:36:9f:a8:e2:50
        hwaddr a0:36:9f:a8:e2:53
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=9<PERFORMNUD,IFDISABLED>
re0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=8209a<TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC,LINKSTATE>
        ether 0a:e0:af:ad:02:bf
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=9<PERFORMNUD,IFDISABLED>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x6
        inet 127.0.0.1 netmask 0xff000000
        groups: lo
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
pflog0: flags=0<> metric 0 mtu 33160
        groups: pflog
lagg0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: lan
        options=8100b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWFILTER>
        ether a0:36:9f:a8:e2:50
        laggproto roundrobin lagghash l2,l3,l4
        laggport: igb0 flags=4<ACTIVE>
        laggport: igb1 flags=4<ACTIVE>
        laggport: igb2 flags=4<ACTIVE>
        laggport: igb3 flags=4<ACTIVE>
        groups: lagg
        media: Ethernet autoselect
        status: active
        nd6 options=9<PERFORMNUD,IFDISABLED>
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        ether 02:fd:d1:fe:04:00
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto stp-rstp maxaddr 2000 timeout 1200
        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
        member: re0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 5 priority 128 path cost 20000
        groups: bridge
        nd6 options=1<PERFORMNUD>
bridge1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        ether 02:fd:d1:fe:04:01
        inet 192.168.1.10 netmask 0xffffff00 broadcast 192.168.1.255
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto stp-rstp maxaddr 2000 timeout 1200
        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
        member: vnet0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 10 priority 128 path cost 2000000
        member: lagg0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 8 priority 128 path cost 5000
        groups: bridge
        nd6 options=9<PERFORMNUD,IFDISABLED>
vnet0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=80000<LINKSTATE>
        ether fe:a0:98:41:95:f6
        hwaddr 58:9c:fc:10:85:57
        groups: tap
        media: Ethernet autoselect
        status: active
        nd6 options=1<PERFORMNUD>
        Opened by PID 13196

Is the interface in the guest Linux really enp0s6? Try ifconfig -a in the guest. If you e.g. removed the CDROM device after installation, PCI based device names tend to get juggled around.

Yes, I'm aware that the NIC interface changes with device changes in the VM but it is the correct interface: