[RESOLVED] 11.2-U5 bhyve vlans & Alpine guest with multiple NICs

[0xD0NK3Y]

Hello,

Host: FreeNAS-11.2-U5 (Build Date: Jun 24, 2019 18:41)
Guest: Alpine Linux vanilla 4.16.53

I have come across some weirdness when adding multiple NICs to bhyve guests. Both interfaces are VLANs which are otherwise working.

Works = can ping router

First I add NIC1 and assign it to INT1

I start the guest and NIC1 works

I then add NIC2 to INT2

Now NIC1 no longer works, NIC2 works

If I then destroy the bridge for NIC2, NIC1 now works but NIC2 does not.

NIC2 on its own works fine.

The routing table in the guest is:

Code:

Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
0.0.0.0         10.3.2.1        0.0.0.0         UG        0 0          0 eth0
10.3.2.0        0.0.0.0         255.255.255.0   U         0 0          0 eth0
172.17.0.0      0.0.0.0         255.255.0.0     U         0 0          0 docker0
172.18.0.0      0.0.0.0         255.255.0.0     U         0 0          0 docker_gwbridge
192.168.50.0    0.0.0.0         255.255.255.0   U         0 0          0 eth1

Which seems normal to me.

When running tcpdump initially the bridge for NIC1 shows unanswered ARP requests. After I kill the bridge for NIC2, the bridge for NIC1 shows traffic flowing as normal.

In fact while testing it seemed whenever there were two bridges one would not function. Killing the functional bridge made the non-functional bridge work. I managed to reverse the problem, NIC2 coming to life when the bridge for NIC1 was killed.

Can anyone help me understand what might be happening here?

 

[0xD0NK3Y]

Well predictably I was able to finally figure out what was wrong shortly after posting

I am using vlans. I wanted the guest to have access to one vlan(vlan2) and the default network. My default network is vlan1.

When using just one NIC inside the guest assigned to the physical interface it worked

Adding a second NIC inside the guest assigned to a vlan interface meant that that the physical interface no longer worked

Assigning the first NIC to a vlan interface instead of the physical interface resolved the issue

In short: If you're using vlans with bhyve then all interfaces must be vlans.

This wasn't obvious to me. Hope this helps someone.