SOLVED After upgrade to 11.1, network issues in iohyve, also after fallback to 9.10, SOLVED by moving VMs

[tslw]

Yesterday I upgraded to FreeNAS 11.1 (from 9.10u6).
Most things work fine, except the network in iohyve. Jails on the same bridge work well.
Even though some iohyve VM network functions seem to work - the VM has an IP address, the route is good, ping works - I cannot transfer data in and out the VM. Most obvious is that when I make an ftp connection, I can log in, change directory but not transfer data.
Because 5 iohyve VMs do not work anymore I suspect the bridge is the issue.
Even after a rollback to 9.10 is does not work anymore.
Who can help with finding the issue?

Should the bridge have an IP address?
Do you advise to try to fix it in 9.10 or 11.1 (e.g. using the new VM's in 11.1).

Bridge configuration

Code:

bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
		description: iohyve-bridge
		ether 02:77:13:69:62:00
		nd6 options=1<PERFORMNUD>
		id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
		maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
		root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
		member: epair6a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
				ifmaxaddr 0 port 16 priority 128 path cost 2000
		member: epair5a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
				ifmaxaddr 0 port 15 priority 128 path cost 2000
		member: epair4a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
				ifmaxaddr 0 port 14 priority 128 path cost 2000
		member: epair3a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
				ifmaxaddr 0 port 13 priority 128 path cost 2000
		member: epair2a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
				ifmaxaddr 0 port 12 priority 128 path cost 2000
		member: epair1a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
				ifmaxaddr 0 port 11 priority 128 path cost 2000
		member: epair0a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
				ifmaxaddr 0 port 10 priority 128 path cost 2000
		member: tap5 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
				ifmaxaddr 0 port 9 priority 128 path cost 2000000
		member: tap10 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
				ifmaxaddr 0 port 8 priority 128 path cost 2000000
		member: tap12 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
				ifmaxaddr 0 port 7 priority 128 path cost 2000000
		member: tap9 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
				ifmaxaddr 0 port 6 priority 128 path cost 2000000
		member: tap7 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
				ifmaxaddr 0 port 5 priority 128 path cost 2000000
		member: ix0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
				ifmaxaddr 0 port 1 priority 128 path cost 2000

Configuration of one of the iohyve VMs (related to tap7 in the bridge)

Code:

eth0	  Link encap:Ethernet  HWaddr 00:a0:98:32:81:d8
		  inet addr:192.168.178.219  Bcast:192.168.178.255  Mask:255.255.255.0
		  inet6 addr: fe80::2a0:98ff:fe32:81d8/64 Scope:Link
		  UP BROADCAST RUNNING MULTICAST  MTU:1464  Metric:1
		  RX packets:168455 errors:0 dropped:472 overruns:0 frame:0
		  TX packets:37403 errors:0 dropped:0 overruns:0 carrier:0
		  collisions:0 txqueuelen:1000
		  RX bytes:23746094 (22.6 MiB)  TX bytes:2503402 (2.3 MiB)

 

[tslw]

When I set the iohyve VM to a dynamic IP address, I can see on the bridge interface that a request for an IP address is send and a reply is received on the bridge. However the reply with the IP address is not received in the iohyve VM.

Who has a working setup and can you share the most relevant parts?
Should have one or two bridges for the jails and iohyve VMs?

 

[Patrick M. Hausen]

Does iohyve have a NAT option? Ping and FTP control connection working but no data exchange looks like there might be something weird in this regard.

Patrick

 

[tslw]

I do not see any NAT options in iohyve (only FreeBSD options).
I see in https://github.com/pr1ntf/iohyve/issues/64 a reference to NAT but I cannot see how I can use that.

I have started FreeNAS version 11.1 and created a VM (using the GUI).
The VM I created connected to the network immediately. It created a new tap interface to the bridge.
I am currently comparing the network settings between the iohyve VM and the Freenas 11 VM....

An alternative route is to start the iohyve VMs as a Freenas 11 VM but I do not yet see how that works (I see you can

 

[Patrick M. Hausen]

You should be able to create a new FreeNAS VM and simply point a new virtual disk device to the ZVOL created with iohyve.

Patrick

 

[tslw]

hi Patrick,

Thanks for the tip.
I am looking into mounting the zvols in a new VM. However, the zvols do not have a EPS partition, so it does not boot (I get an EFi boot shell).
The description I found to fix this is quite complex so I am creating new virtual machines.
The problem is that I cannot easily copy the configuration from the old to the new VM.

I noticed one strange thing this morning. I started an iohyve VM that I did not use since before upgrading to Freenas 11.1 and it worked immediately. I still don't understand why some VMs do not work anymore since the upgrade to Freenas 11.1.

 

[Patrick M. Hausen]

Add a second disk - more precisely a small-ish first one - and make your proper ZVOL the second disk. Put rEFInd on that first additional disk ...

Patrick

 

[tslw]

Thanks again Patrick.
While I was reading the link to rEFInd (it looks interesting but I could not get it to work properly) it made me think of two similar ideas:
1. I can add the old disk (zvol) to the VM and just copy the files from the old system.
2. I can also convert the MBR to GPT and make the disk bootable through EFI.

I tried both. Number 1 works fine (although I had some issues getting all the files in the proper location).
Number almost works. A copy of the old zvol (a made a snapshot to do some testing), now has an EFi partition.
I used this description: https://blog.getreu.net/projects/legacy-to-uefi-boot/
In the beginning it looks complicated but a lot in link is about encrypted disks.
I was able to make all the changes to the zvol, however it does not start yet (boots in grub) so i have some more research to do....

Edit: it works now. I somehow removed grub.cfg