Network gone from VM after reboot/upgrade?

[tobiasbp]

Hello...

I have been running a VM (pfSense) on FreeNAS 11.1 for some months with out problems. A few days ago, I upgraded from 11.1-RELEASE to 11.1-U1. When the FreeNAS machine came back up again, I no longer had network connectivity to/from the VM. I tried to reboot to FreeNAS-RELEASE, but the VM still had no network.

From the VM, I can no longer ping machines on the network(s), and I can no longer ping the VM from other machines, including the FreeNAS host.

In the VM, all connections are via VLANs on the vtnet0 NIC. There is no untagged traffic from/to the VM. I have checked the switch, and all relevant are untagged on the relevant port.

On the FreeNAS side, the VM has a single NIC (igb2) set up as VirtIO.

I'm not sure how to troubleshoot this. Any ideas much appreciated.


The vtnet0 interface as seen in the VM (Can't seem to insert image in post).

Network configuration on the FreeNAS side:

Code:

igb2: flags=8d02<BROADCAST,PROMISC,OACTIVE,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=2400b9<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWTSO,RXCSUM_IPV6>
	ether 00:1b:21:8c:bc:f2
	hwaddr 00:1b:21:8c:bc:f2
	nd6 options=9<PERFORMNUD,IFDISABLED>
	media: Ethernet autoselect (1000baseT <full-duplex>)
	status: active
...
...
bridge1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	ether 02:fc:e4:2e:92:01
	nd6 options=1<PERFORMNUD>
	groups: bridge
	id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
	maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
	root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
	member: tap0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
			ifmaxaddr 0 port 14 priority 128 path cost 2000000
	member: igb2 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
			ifmaxaddr 0 port 5 priority 128 path cost 20000
...
...
tap0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=80000<LINKSTATE>
	ether 00:bd:a3:ee:68:00
	hwaddr 00:bd:a3:ee:68:00
	nd6 options=1<PERFORMNUD>
	media: Ethernet autoselect
	status: active
	groups: tap
	Opened by PID 44876
...
...
root@kisaragi:~ # ps aux | grep 44876
root	   44876   15.2  0.0 1113348 122584  -  I	16:36	  4:37.75 bhyve: 1_pfSense (bhyve)

Regards,
Tobias

 

[dlavigne]

Were you able to resolve this? If not, is it still an issue on 11.1-U2?

 

[tobiasbp]

Were you able to resolve this? If not, is it still an issue on 11.1-U2?

No, I have not been able to resolve the issue.

I have not yet updated to 11.1-U2. Are there any changes in U2 which you feel could be relevant to this issue?

 

[dlavigne]

This one might: https://redmine.ixsystems.com/issues/28383.

 

[tobiasbp]

This one might: https://redmine.ixsystems.com/issues/28383.

So, in that case, I should be able to "Choosing Edit and clicking OK to save the NIC fixes this, and then the VM can start.".

Correct?

FYI. My VM starts up, but the network is not working.

Is there a way I can confirm that my issue is what is described in that bug?

 

[dlavigne]

Hard to tell without updating. If the update does not fix it, it is worth creating another bug report. If you end up making one, post the issue number here.

 

[Stefan Krieger]

My problem with the network being unreachable from the VM, despite it being allocated an IP address, is still there in 11.1-U2. The bug https://redmine.ixsystems.com/issues/28383 and the associated "Choosing Edit and clicking OK to save the NIC fixes this, and then the VM can start." doesn't help. Starting the VM was never a problem for me.

So I'm still stuck. I'm assuming this is a bug. I'll try again on the next update (or I should create a new VM from scratch). No time right now to play around....

 

[tobiasbp]

My problem with the network being unreachable from the VM, despite it being allocated an IP address, is still there in 11.1-U2. The bug https://redmine.ixsystems.com/issues/28383 and the associated "Choosing Edit and clicking OK to save the NIC fixes this, and then the VM can start." doesn't help. Starting the VM was never a problem for me.

Just to clear things up: Did you ever have working network in your VM? I did, but I lost if after an upgrade to 11.1-U1. Reverting to 11.1-RELEASE did not bring the network back.

 

[Stefan Krieger]

Just to clear things up: Did you ever have working network in your VM? I did, but I lost if after an upgrade to 11.1-U1. Reverting to 11.1-RELEASE did not bring the network back.

No, this is a new VM, networking never worked for it. I did have working VMs on my network during the Corral period, can't recall if I did have networking at any other time.

 

[tobiasbp]

No, this is a new VM, networking never worked for it. I did have working VMs on my network during the Corral period, can't recall if I did have networking at any other time.

OK, so your issue sounds to me like "Networking on VM do not work at all".

Is that correct? Which version of FreeNAS what the first you experienced this issue on?

 

[Stefan Krieger]

OK, so your issue sounds to me like "Networking on VM do not work at all".

Is that correct? Which version of FreeNAS what the first you experienced this issue on?

I created several VMs on 11.1-RELEASE, none had functional networking. The odd thing is that the router assigns them IPs, then can't see them as active.

 

[tobiasbp]

Looking at this again (After having upgraded to 11.1-U2 with out fixing the problem), I realized igb2 was not up (As seen in my own documentation earlier in this thread). Take a look at the flags. bridge1 is UP, but igb2 is not. Doh!

Code:

...
igb2: flags=8d02<BROADCAST,PROMISC,OACTIVE,SIMPLEX,MULTICAST>
...
bridge1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST>
...

I brought up the igb2 by choosing the NIC in the GUI, press edit and save with no change to the configuration. The NIC is only used by the VM, but set up in the GUI with no configuration.

So, is the NIC not being up after a reboot/upgrade a bug? It seems to me like the NIC should be up.

Any thoughts?

 

[tobiasbp]

Were you able to resolve this? If not, is it still an issue on 11.1-U2?

Upgrading to 11.1-U2 did not resolve the problem. I did find a solution though, as described in the thread.

 

[tobiasbp]

It looks as if I just need to add up to the options field in the configuration of my NIC as described on this issue.

The issue of the NIC not being up, is also an issue when using VLANs. The need for using the option up, is in the documentation.

 

[wblock]

It looks as if I just need to add up to the options field in the configuration of my NIC as described on this issue.

Is that a separate NIC from the main one being used for the FreeNAS GUI? If nothing on the host is using it, that would make sense.

This would be a different issue from the bug, just that the status of an interface used by a jail should be checked when a jail is started, and brought up if it is not already up.

 

[tobiasbp]

Is that a separate NIC from the main one being used for the FreeNAS GUI? If nothing on the host is using it, that would make sense.

Yes, the NIC in question is only used by the VM.

This would be a different issue from the bug, just that the status of an interface used by a jail should be checked when a jail is started, and brought up if it is not already up.

It's not a jail. It's a VM in FreeNAS 11.1-U2.

So, the issues seems to be: "A NIC used only by a virtual machine is not brought up on boot".

 

[tobiasbp]

This is still a problem in FreeNAS-11.1-U4 (Just had a power loss)

I don't know if the issue is covered by this closed bug, but my issue persists.

I have not found an oppertune time to update this server to the latest FreeNAS.