SOLVED Bhyve: FreeNAS doesn`t bridge connections on reboot

[siconic]

I have been using Freenas for a few years now, and love a lot of the new features, especially bhyve. I was able to take down all of my other VM servers (3 of them), and consolidate everything into one server because of it.

My problem is however, and the one thing I cant seem to figure out, how do I get the tap interfaces to bridge automatically?

When the server reboots, the tap interfaces show up in ifconfig, and are reporting "up", but are not in my "bridge0" group. bridge0 is created by FreeNAS for the Jails. I have to manually add the bhyve (iohyve) tap interfaces with "ifconfig addm tap0..." and then stop and start each iohyve machine to obtain an IP, and then everything works like it should.

I have tried everything I can think of. I modified the /etc/rc.conf by adding ifconfig_bridge0="addm tap0...", I tried adding tunables, I tried this:

1. /conf/base/etc/sysctl.conf
   net.link.tap.up_on_open=1
2. /boot/loader.conf
   vmm_load="YES"
   nmdm_load="YES"
   if_bridge_load="YES"
   if_tap_load="YES"
3. /conf/base/etc/rc.conf
   cloned_interfaces="tap0"
   ifconfig_bridge0="addm tap0"

etc. No method I have tried works.

Any ideas?

I did consider a script, but I don't really want to do that if there is a better or simpler way. I suspect it has something to do with the timing of the bridge interface being created by freenas for the jails and if I could add the tap interfaces AFTER the bridge interface is created then it would work. But I am not sure how to do that in FreeBSD, or at what point FreeNAS creates the bridge interface. Maybe create the bridge interface from rc.conf?

 

[m0nkey_]

I have tried everything I can think of. I modified the /etc/rc.conf by adding ifconfig_bridge0="addm tap0...", I tried adding tunables,

Why are you modifying files in the base OS? Are you running Bhyve VMs directly or making use of iohyve?

By the screenshot you provided, you've not been using the included iohyve script to create and run your virtual machines. Iohyve will do that all for you.

I also have a basic how-to guide on iohyve: https://forums.freenas.org/index.php?resources/iohyve-set-up-and-basic-usage.9/.

 

[siconic]

I am modifying the files in the base OS because they overwrite the files in the standard FreeBSD location. IE, whatever is in the /conf/base/etc/rc.conf location will overwrite any changes made to the file in /etc/rc.conf. Since I wanted to make persistent changes, that is why.

That aside, yes I used the iohyve scripts installed in the FreeNAS distribution. Here is an example of what I used:

iohyve create kippo 40G
iohyve set kippo loader=grub-bhyve os=debian ram=4G cpu=8
iohyve install kippo <name of ISO>
iohyve console kippo

That is ~ what I used to do my setup. As far as I know, that is the iohyve scripts you speak of. Let me know if there is something I could have done different.

P.S. I did watch your video a few months ago, along with a few other sources to figure out iohyve... I did just notice that I left out of the tunables iohyve_flags the net=em0. Not sure if that makes a difference, and if it does, I would like to know the true function of that flag.

Thanks!

 

[m0nkey_]

P.S. I did watch your video a few months ago, along with a few other sources to figure out iohyve... I did just notice that I left out of the tunables iohyve_flags the net=em0. Not sure if that makes a difference, and if it does, I would like to know the true function of that flag.

The iohyve_flags allow you to set the bridge interface (net=) at boot and tell the iohyve script to load kernel modules (kmod=). The tuneables are covered in the iohyve readme on GitHub. It will make a difference.

You may also want to check out the FreeNAS tunables section of their handbook so you can add iohyve_enable="YES" and iohyve_flags="kmod=1 net=[iface]" thus setting up the kernel modules and iohyve networking at boot time on your FreeNAS install.

 

[siconic]

Solved: I was missing flag net=bce0. So for any who may be having issues, here is the flag you need in system -> tunables:

iohyve_flags kmod=1 net=<Main Interface Name>

My freenas instance used bce0 for the jails bridge. So, iohyve_flags kmod=1 net=bce0

At first, I tried net=em0 and then had 2 bridges, bridge0 and bridge1. bridge0 is where all the tap devices were, but had no internet connection, but that was a good start since my other interfaces still had internet. I then realized "duh, my interaces have different names!" Changed the interface name to bce0, and voila! On reboot all my interfaces are bridged to a single bridge, bridge0.

 

[bmp]

Solved: I was missing flag net=bce0. So for any who may be having issues, here is the flag you need in system -> tunables:

iohyve_flags kmod=1 net=<Main Interface Name>

My freenas instance used bce0 for the jails bridge. So, iohyve_flags kmod=1 net=bce0

At first, I tried net=em0 and then had 2 bridges, bridge0 and bridge1. bridge0 is where all the tap devices were, but had no internet connection, but that was a good start since my other interfaces still had internet. I then realized "duh, my interaces have different names!" Changed the interface name to bce0, and voila! On reboot all my interfaces are bridged to a single bridge, bridge0.

Hey Man, I am trying to do something similar to add a NIC to a bridge that will survive a reboot, but forgive my inexperience, have been unable to follow exactly how to setup this in tunables. have any pointers? Mine is more simple I need to add Nic X to bridge0. I have been using an Ifconfig bridge0 addm nicx to get it to work but it doesn't survive a reboot. and when I use a pre\post int section of startup it doesn't work at all. ANy pointers?