VM network routing (FreeNAS 11.0-U1)

[sophacles]

Hi all,

tl;dr - it appears my main ethernet interface in FreeNAS is not being added to the bridge for VMs

I'm encountering a strange issue with VMs. I can create a VM and boot it just fine. I set it up to have the default NIC and VNC devices, and add a hard drive and a CD-ROM drive (using an ubuntu iso).

When I start the VM I can connect over VNC and am in the Ubuntu installer. When I get to Ubuntu's network setup DHCP fails. Adding network settings manually seems to work fine on the VM side (route looks correct, interface has the right ip and so on). This is where the problem starts...

Poking around the FreeNAS console, it seems that my ethernet card is not being added to the bridge:

Code:

[sophacles@freenas ~]$ ifconfig -v
igb0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
		options=6403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
		ether d0:50:99:0a:69:d3
		inet 192.168.1.50 netmask 0xffffff00 broadcast 192.168.1.255
		inet6 fe80::d250:99ff:fe0a:69d3%igb0 prefixlen 64 scopeid 0x1
		inet6 2602:306:3148:f4c0:d250:99ff:fe0a:69d3 prefixlen 64 autoconf
		nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>
		media: Ethernet autoselect (1000baseT <full-duplex>)
		status: active
igb1: flags=8c02<BROADCAST,OACTIVE,SIMPLEX,MULTICAST> metric 0 mtu 1500
		options=6403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
		ether d0:50:99:0a:69:d4
		nd6 options=9<PERFORMNUD,IFDISABLED>
		media: Ethernet autoselect (1000baseT <full-duplex>)
		status: active
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
		options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
		inet6 ::1 prefixlen 128
		inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3
		inet 127.0.0.1 netmask 0xff000000
		nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
		groups: lo
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
		ether 02:51:c2:35:cb:00
		nd6 options=1<PERFORMNUD>
		groups: bridge
		id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
		maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
		root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
		member: tap0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
				ifmaxaddr 0 port 4 priority 128 path cost 2000000
tap0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
		options=80000<LINKSTATE>
		ether 00:bd:78:11:10:00
		nd6 options=1<PERFORMNUD>
		media: Ethernet autoselect
		status: active
		groups: tap
		Opened by PID 6486

When I ping from inside the VM with manual network settings, I can see the pings on tap0:

Code:

[root@freenas /mnt/main_storage/userhomes/sophacles]# tcpdump -i tap0
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on tap0, link-type EN10MB (Ethernet), capture size 262144 bytes
15:22:02.572670 ARP, Request who-has 192.168.1.254 tell 192.168.1.51, length 46
15:22:03.575409 ARP, Request who-has 192.168.1.254 tell 192.168.1.51, length 46
15:22:04.571337 ARP, Request who-has 192.168.1.254 tell 192.168.1.51, length 46
15:22:12.589657 ARP, Request who-has 192.168.1.254 tell 192.168.1.51, length 46
15:22:13.588467 ARP, Request who-has 192.168.1.254 tell 192.168.1.51, length 46
15:22:14.589009 ARP, Request who-has 192.168.1.254 tell 192.168.1.51, length 46
15:22:15.591765 ARP, Request who-has 192.168.1.254 tell 192.168.1.51, length 46
15:22:16.601531 ARP, Request who-has 192.168.1.254 tell 192.168.1.51, length 46
15:22:17.605681 ARP, Request who-has 192.168.1.254 tell 192.168.1.51, length 46
15:22:18.611642 ARP, Request who-has 192.168.1.254 tell 192.168.1.51, length 46

Please note the igb1 interface is manually set up in the GUI to have a static IP setting.

I have tried:

* deleting and readding the VM. Deleting and rebooting the VM then adding (and several other combinations of those steps).
* Using the virtio and e1000 adapter types.
* adding the igb1 interface to the bridge via 'ifconfig bridge0 addm igb1'

I'm not sure what else I should do to make this work, or what the problem even is. If more information would be helpful I'll be glad to provide it.

 

[skywise_ca]

I can see two things wrong right off the bat.
First, you mentioned you set a static IP on igb1. It's not there. (on the interface)
Second, the bit you're missing on the bridge is the real interface (whichever it is)
...
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
ether 02:fd:7f:fd:97:00
nd6 options=9<PERFORMNUD,IFDISABLED>
groups: bridge
id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
member: epair1a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 6 priority 128 path cost 2000
member: epair0a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 5 priority 128 path cost 2000
member: igb0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 1 priority 128 path cost 20000
...
(I've got jails running at the moment, not VMs but generally same mech going on here)
You need an interface with access to the network on the bridge. (in your case, only igb0)
In the GUI, what interfaces do you show? (Network/interfaces) Both igb0 and igb1?

These are the first places I'd look.
If you want to just try a thing, add igb0 to the bridge manually. If that works, then you can work on figuring out why freenas isn't adding it by itself.

 

[sophacles]

Oh wow, good catch - thanks!

I both managed to get confused about the interfaces and set up the default route wrong. Now that have my network settings fixed up now. I manually added igb0 to the bridge and the VMs work fine.

Tomorrow night I'll try to reboot and see if the bridge gets created right on a fresh boot with the right network settings.