No Networking in RancherOS VM

Status
Not open for further replies.

brando56894

Wizard
Joined
Feb 15, 2014
Messages
1,537
I haven't really messed around with the virtualization in FreeNAS since it's not really useful (yet) for my needs. I decided to give the RancherOS VM a try and I have it all setup, to the point where I actually install the UI, but my network connection stops at my FreeNAS VM (I have it virtualized in ESXi). I'm currently running FreeNAS-11-MASTER-201801090514

Initially I had no network connection at all, because my NIC in RancherOS wasn't assigned an IP, so I assigned one manually, along with a default route for my LAN. Once I did that I was able to ping the FreeNAS IP and vice versa, but I can't go beyond FreeNAS.

Code:
[rancher@rancher ~]$ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1
	link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
	inet 127.0.0.1/8 scope host lo
	   valid_lft forever preferred_lft forever
	inet6 ::1/128 scope host
	   valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
	link/ether 00:a0:98:48:1a:03 brd ff:ff:ff:ff:ff:ff
	inet 192.168.1.5/24 scope global eth0
	   valid_lft forever preferred_lft forever
	inet6 fe80::2a0:98ff:fe48:1a03/64 scope link
	   valid_lft forever preferred_lft forever
3: docker-sys: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000
	link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff
	inet 172.18.42.2/16 brd 172.18.255.255 scope global docker-sys
	   valid_lft forever preferred_lft forever
	inet6 fe80::90d5:d9ff:fe6f:369f/64 scope link
	   valid_lft forever preferred_lft forever
5: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default
	link/ether 02:42:b0:e8:18:e7 brd ff:ff:ff:ff:ff:ff
	inet 172.17.0.1/16 scope global docker0
	   valid_lft forever preferred_lft forever

[rancher@rancher ~]$ ip route show
default via 192.168.1.1 dev eth0
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown
172.18.0.0/16 dev docker-sys proto kernel scope link src 172.18.42.2 linkdown
192.168.1.0/24 dev eth0 proto kernel scope link src 192.168.1.5

root@freenas:~ # ifconfig
em0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
		options=98<VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM>
		ether 00:0c:29:1c:92:a8
		hwaddr 00:0c:29:1c:92:a8
		inet 192.168.1.6 netmask 0xffffff00 broadcast 192.168.1.255
		nd6 options=9<PERFORMNUD,IFDISABLED>
		media: Ethernet autoselect (1000baseT <full-duplex>)
		status: active
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
		options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
		inet6 ::1 prefixlen 128
		inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2
		inet 127.0.0.1 netmask 0xff000000
		nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
		groups: lo
tap0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
		options=80000<LINKSTATE>
		ether 00:bd:67:bb:ff:00
		hwaddr 00:bd:67:bb:ff:00
		nd6 options=1<PERFORMNUD>
		media: Ethernet autoselect
		status: active
		groups: tap
		Opened by PID 5329
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
		ether 02:b5:21:16:71:00
		nd6 options=1<PERFORMNUD>
		groups: bridge
		id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
		maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
		root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
		member: em0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
				ifmaxaddr 0 port 1 priority 128 path cost 20000
		member: tap0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
				ifmaxaddr 0 port 3 priority 128 path cost 2000000


What's the issue?
 

mwp

Cadet
Joined
Mar 18, 2015
Messages
2
I'm struggling with the same issue. Weirdly, I had connectivity on the first couple boots, then I lost it. I'm also frustrated that I can't attach the host's LAGG interface to the guest's NIC (I'm using a VLAN for the time being). And I think it's weird that RancherOS apparently doesn't use a paravirtualized kernel.
 

KrisBee

Wizard
Joined
Mar 20, 2017
Messages
1,288
Can't speak for ESXI , but rancheros basic networking functions without problem when FreeNAS is virtualised under qemu/KVM.

Code:
rancher@rancher ~]$ sudo dmesg | grep virtual
[	0.000000] CPU MTRRs all blank - virtualized system.
[	0.000000] Booting paravirtualized kernel on bare hardware
[   11.212660] VMware vmxnet3 virtual NIC driver - version 1.4.a.0-k-NAPI
[rancher@rancher ~]$ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1
	link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
	inet 127.0.0.1/8 scope host lo
	   valid_lft forever preferred_lft forever
	inet6 ::1/128 scope host 
	   valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
	link/ether 00:a0:98:27:e9:08 brd ff:ff:ff:ff:ff:ff
	inet 192.168.0.133/24 brd 192.168.0.255 scope global eth0
	   valid_lft forever preferred_lft forever
3: docker-sys: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000
	link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff
	inet 172.18.42.2/16 brd 172.18.255.255 scope global docker-sys
	   valid_lft forever preferred_lft forever
	inet6 fe80::446a:10ff:fe20:166b/64 scope link 
	   valid_lft forever preferred_lft forever
5: docker0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
	link/ether 02:42:de:12:73:12 brd ff:ff:ff:ff:ff:ff
	inet 172.17.0.1/16 scope global docker0
	   valid_lft forever preferred_lft forever
	inet6 fe80::42:deff:fe12:7312/64 scope link 
	   valid_lft forever preferred_lft forever
7: vethf8f74fb@if6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master docker0 state UP group default 
	link/ether 76:00:eb:21:bc:64 brd ff:ff:ff:ff:ff:ff link-netnsid 0
	inet6 fe80::7400:ebff:fe21:bc64/64 scope link 
	   valid_lft forever preferred_lft forever
[rancher@rancher ~]$


This test FreeNAS VM was created with CPU set to "host-passthrough" on a debian host set up for nested-virtualisation, whch might explain why rancheros sees this as "Booting paravirtualized kernel on bare hardware". NIC type used was e1000 on FreeNAS and of course in rancheros. Rancheros cloud-config sets a static IP.

What do the rancheros boot logs show ( /var/log/boot/netconf.log
, etc.)?

As you are using ESXI, why rely on nested virtualisation? Why not set up a separate rancher VM directly under ESXI?

P.S. On rancheros instance running under VirtualBox you see:

Code:
[root@rancher ~]# dmesg | grep virtual
[	0.000000] CPU MTRRs all blank - virtualized system.
[	0.000000] Booting paravirtualized kernel on KVM
[	0.274846] KVM setup paravirtual spinlock
[	6.746733] VMware vmxnet3 virtual NIC driver - version 1.4.a.0-k-NAPI
[root@rancher ~]#


Can't check on a FreeNAS host at the moment, but I think we can say rancheros has a kernel which is "virtualisation aware".
 

mwp

Cadet
Joined
Mar 18, 2015
Messages
2
What do the rancheros boot logs show ( /var/log/boot/netconf.log, etc.)?

Not sure. My FreeNAS machine is disassembled right now (trying to fix iDRAC and exploring virtualization) but I'll bring it back up this weekend and see what if anything is in the guest logs.

As you are using ESXI, why rely on nested virtualisation? Why not set up a separate rancher VM directly under ESXI?

In my particular case, I'm currently running FreeNAS on bare metal.

Can't check on a FreeNAS host at the moment, but I think we can say rancheros has a kernel which is "virtualisation aware".

Sure, so why can't we use VirtIO devices? Maybe it's a limitation of bhyve?
 

brando56894

Wizard
Joined
Feb 15, 2014
Messages
1,537
What do the rancheros boot logs show ( /var/log/boot/netconf.log
, etc.)?

Here's the /var/log/boot/netconf.log
Code:
[rancher@rancher ~]$ cat /var/log/boot/netconf.log
time="2018-01-23T14:15:59Z" level=debug msg="START: [netconf] in /"
time="2018-01-23T14:15:59Z" level=info msg="Apply Network Config"
time="2018-01-23T14:15:59Z" level=debug msg="Config: &netconf.NetworkConfig{PreCmds:[]string(nil), DNS:netconf.DNSConfig{Nameservers:[]string(nil), Search:[]string(nil)}, Interfaces:map[string]netconf.InterfaceConfig{"eth*":netconf.InterfaceConfig{Match:"", DHCP:true, DHCPArgs:"", Address:"", Addresses:[]string(nil), IPV4LL:false, Gateway:"", GatewayIpv6:"", MTU:0, Bridge:"", Bond:"", BondOpts:map[string]string(nil), PostUp:[]string(nil), PreUp:[]string(nil), Vlans:""}, "lo":netconf.InterfaceConfig{Match:"", DHCP:false, DHCPArgs:"", Address:"", Addresses:[]string{"127.0.0.1/8", "::1/128"}, IPV4LL:false, Gateway:"", GatewayIpv6:"", MTU:0, Bridge:"", Bond:"", BondOpts:map[string]string(nil), PostUp:[]string(nil), PreUp:[]string(nil), Vlans:""}}, PostCmds:[]string(nil), HTTPProxy:"", HTTPSProxy:"", NoProxy:""}"
time="2018-01-23T14:15:59Z" level=debug msg="runCmds(on ): []"
time="2018-01-23T14:15:59Z" level=debug msg="applyOuter(%!V(bool=false), false)"
time="2018-01-23T14:15:59Z" level=debug msg="Config(lo): netconf.InterfaceConfig{Match:"lo", DHCP:false, DHCPArgs:"", Address:"", Addresses:[]string{"127.0.0.1/8", "::1/128"}, IPV4LL:false, Gateway:"", GatewayIpv6:"", MTU:0, Bridge:"", Bond:"", BondOpts:map[string]string(nil), PostUp:[]string(nil), PreUp:[]string(nil), Vlans:""}"
time="2018-01-23T14:15:59Z" level=debug msg="runCmds(on lo): []"
time="2018-01-23T14:15:59Z" level=info msg="Applying 127.0.0.1/8 to lo"
time="2018-01-23T14:15:59Z" level=info msg="Applying ::1/128 to lo"
time="2018-01-23T14:15:59Z" level=debug msg="runCmds(on lo): []"
time="2018-01-23T14:15:59Z" level=debug msg="applyOuter(%!V(bool=false), false)"
time="2018-01-23T14:15:59Z" level=debug msg="Config(eth0): netconf.InterfaceConfig{Match:"eth*", DHCP:true, DHCPArgs:"", Address:"", Addresses:[]string(nil), IPV4LL:false, Gateway:"", GatewayIpv6:"", MTU:0, Bridge:"", Bond:"", BondOpts:map[string]string(nil), PostUp:[]string(nil), PreUp:[]string(nil), Vlans:""}"
time="2018-01-23T14:15:59Z" level=debug msg="runCmds(on eth0): []"
time="2018-01-23T14:15:59Z" level=debug msg="runCmds(on eth0): []"
time="2018-01-23T14:15:59Z" level=info msg="Running DHCP on eth0: dhcpcd -MA4 -e force_hostname=true -w --debug eth0"
time="2018-01-23T14:16:29Z" level=info msg="Checking to see if DNS was set by DHCP"
time="2018-01-23T14:16:29Z" level=info msg="dns testing lo"
time="2018-01-23T14:16:29Z" level=error msg="exit status 1"
time="2018-01-23T14:16:29Z" level=debug msg="getDhcpLease lo: "
time="2018-01-23T14:16:29Z" level=debug msg="line: []"
time="2018-01-23T14:16:29Z" level=info msg="dns testing eth0"
time="2018-01-23T14:16:29Z" level=error msg="exit status 1"
time="2018-01-23T14:16:29Z" level=debug msg="getDhcpLease eth0: "
time="2018-01-23T14:16:29Z" level=debug msg="line: []"
time="2018-01-23T14:16:29Z" level=debug msg="runCmds(on ): []"
time="2018-01-23T14:16:29Z" level=info msg="Writing default resolv.conf - no user setting, and no DHCP setting"
time="2018-01-23T14:16:29Z" level=debug msg="Resolve.conf == [nameserver 8.8.8.8
nameserver 8.8.4.4
], %!s(<nil>)"
time="2018-01-23T14:16:29Z" level=info msg="Apply Network Config SyncHostname"
time="2018-01-23T14:16:29Z" level=info msg="Restart syslog"


dmesg shows that it loads the driver successfully.

Code:
[rancher@rancher ~]$ dmesg|grep e1000
[   13.345454] e1000: Intel(R) PRO/1000 Network Driver - version 7.3.21-k8-NAPI
[   13.347005] e1000: Copyright (c) 1999-2006 Intel Corporation.
[   13.876549] e1000 0000:00:03.0 eth0: (PCI:33MHz:32-bit) 00:a0:98:6b:b2:9b
[   13.878136] e1000 0000:00:03.0 eth0: Intel(R) PRO/1000 Network Connection
[   14.056947] e1000: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None


So IDK what's going on, I do remember having issues with nested virtualization in Bhyve in FreeNAS 10, but I have different hardware now, and if this was a general issue I'm sure the devs would pick up on it since they use ESXi for their testing.

As you are using ESXI, why rely on nested virtualisation? Why not set up a separate rancher VM directly under ESXI?

That doesn't help find and resolve bugs in FreeNAS, now does it? ;)
 

KrisBee

Wizard
Joined
Mar 20, 2017
Messages
1,288
You’re expecting ESXI nested virtualisation to just work with FreeNAS Nightly, but does it?

Is this dhcp failure in your rancheros VM an isolated problem, or is it present in all/other VMs created within your virtualised FreeNAS? If it’s a network problem in your virtualised set-up, is it internal or external to FreeNAS/bhyve?

When you create a VM in virtualised FreeNAS do any of the following make a difference?

1. Using virtio for the NIC type
2. Assigning a MAC address to the NIC and using dhcp reservation
3. In the case of rancheros using a static ip in cloud-config.
 

mmeasel

Dabbler
Joined
Dec 4, 2018
Messages
11
Old thread, but I have the same issue and I'm running 11.2-Release. I've tried many variations and none work. I can get into rancher as the OP describes, but no packets out. I even tried assigning an IP to the bridge and I can ping it, but traffic stops there. TCP dump shows traffic moving on the tap, but the other side ( in my case vmx1 ) ...nada.
 
Status
Not open for further replies.
Top