Tunables bridges and taps not working after upgrade to 11.3

[asimov-solensan]

Hello,

I use a lot tunables to create bridges and tap interfaces in my freenas. This worked fine for a couple of years but after upgrading to 11.3U5 from 11.2U8 the commands in tunables are simply ignored.

I'm attaching a picture of the commands, they are pretty simple, just some bridges and members attached to it. But they don't appear created after the upgrade.

Code:

root@lavochkin:~ # ifconfig | grep tap113
root@lavochkin:~ # ifconfig | grep bridge113
root@lavochkin:~ # ifconfig | grep tap114
root@lavochkin:~ # ifconfig | grep bridge114
root@lavochkin:~ # etc ...

If anyone wonders I need this configuration to have my network segmented also for jails and virtual machines. So essentially I need to have those bridges and taps created before jails and virtual machines start. And tunables has worked well in 11.1 and 11.2

Did the code for this changed in 11.3? Feature deprecated? Alternative solution?

Thanks in advance for any help.

 

[asimov-solensan]

Small update. Searching more information on tunables I was able to confirm that configuration is indeed saved in rc.conf.freenas.

Code:

cloned_interfaces="tap113 tap114 tap115 tap116 tap117 tap169 tap1000 bridge113 bridge114 bridge115 bridge116 bridge117 bridge169 bridge1000"
ifconfig_bridge1000="addm tap1000 up"
ifconfig_bridge113="addm tap113 up"
ifconfig_bridge114="addm tap114 up"
ifconfig_bridge115="addm tap115 up"
ifconfig_bridge116="addm tap116 up"
ifconfig_bridge117="addm tap117 up"
ifconfig_bridge169="addm tap169 up"
root@lavochkin:~ #

But can't find any error in the logs. It seems that those lines are just ignored.

 

[ikke]

This is about the same I'm experiencing. I only have one bridge for jails, but the bridge just doesn't get created. I was thinking if something in the predessor lines in the file breaks rc init, and the lines at the end won't get applied? It's odd as there really aren't any log entries.

This is some serious regression in freenas. I've not gotten any answers for ,my postings about it, I wonder if ticket to Jira could help get it fixed? Or even workaround to be instructed.

This breaks important jail functionality for me.

 

[ikke]

My earlier post: https://www.ixsystems.com/community/threads/jails-bridge0-not-coming-up-at-boot.86474/ which I forgot about, and reposted about the subject couple of days ago again https://www.ixsystems.com/community/threads/bridge-does-not-get-build-at-boot.87932/

 

[Patrick M. Hausen]

I just replied in that earlier post - why create bridge interfaces with tuneables at all?

bridge does not get build at boot

I have suffered this for long now, perhaps years. I would like to finally get it solved. Why does the network bridge for iocage jails not start at boot? I am running Version: FreeNAS-11.3-U5. I have iocage jails set to use bridge0 as net device. On every boot, the device is down. I have...

www.ixsystems.com

 

[asimov-solensan]

Indeed that seems to be the correct solution whatever happens with tunables. I guess this is new to 11.3 (if it was in 11.2 then I'm just idiot).

Will try later and confirm if it works. Right now I'm just using a handmade script at boot.

 

[ikke]

I verify, using "network -> interfaces -> add -> bridge" works perfectly. I never noticed the UI has such option. Great!

 

[asimov-solensan]

@Patrick M. Hausen Sadly this does not work for me. I'm going to simplify the problem, I need to have a jail and a VM on the same network segment, but NOT attached to a physical interface.

In tunables I created the bridge and and tap interface attached to the bridge. Then a jail would be linked to the bridge and the VM to the tap interface. This was with 11.2.

The approach on 11.3 is almost perfect because it allows you to create the bridge, have an IP address on the bridge itself and then link jails and VMs to that bridge. And I remark almost perfect because UI forces you to add a physical member to the bridge. What's the point? You can create a bridge without members in command line.

And as I said I need VMs and jails on the same bridge but not attached to a physical interface. Or at least have tap interfaces recognized in the network section on UI, that I can use as "dummy" interfaces.

Maybe I should open a new thread with that particular question.

PS:
Right now I have got a script that creates all I need but and everything already configured works. Also works for jails because you write the bridge name. Doesn't work for VMs because it only lists bridges created on the UI, therefore right now I cannot create new VMs.

 

[Patrick M. Hausen]

Nope. You create a bridge without a physical interface as member. Then attach the jail and the VM both to the bridge.
Tap (VM) and epair (jails) interfaces are created automatically and have been for quite a while. While it is true that there was a time without a UI for the bridge interfaces, I never had to create tap or epair manually.

 

[asimov-solensan]

You create a bridge without a physical interface as member

But how do I do this? The UI forces me to add a member, and only lists physical interfaces. The field is required, that's the problem.

 

[Patrick M. Hausen]

Oh - sorry. This is of course not necessary and an oversight in the UI. I sometimes confuse our data centre server setup and FreeNAS. We run lots of jails with private networks without problems - not on FreeNAS but stock FreeBSD.

If you have an unused physical port you could use that and simply not connect anything. Or use a dummy VLAN interface that does not carry traffic. Just an idea.

If it finally works for you with the "standard" bridge setup, could you file a bug ticket, please? Bridges without physical members should be allowed.

 

[asimov-solensan]

Or use a dummy VLAN interface that does not carry traffic. Just an idea.

That's why I wanted to have these dummy tap interface created at the beginning. If there was a way to create them at boot and recognized as physical that would workaround the problem. I need like 4 of these internal bridges, so not enough physical interfaces.

Let me think about VLAN interfaces, maybe it suits my configuration.

Will open a bug ticket anyhow. The ability to open a bridge without members to have internal networks between VMs/jails is a must on any hipervisor.

 

[Patrick M. Hausen]

Why I recommended against tap: these are the ones dynamically created for the VMs. So I suspect something gets messed up, here. Let's see how using VLANs goes ...

 

[asimov-solensan]

VLANs also force me to attach them to a physical interface. It is not a solution for my use case.

 

[Patrick M. Hausen]

Not a single physical interface to sacrifice for N VLANs?

 

[asimov-solensan]

I'm stupid, I didn't thought I can create all vlans with the same parent interface.

That may work but I need to rethink the whole network. In fact I used bridges to split VLANs but then I have got a pfsense virtualized in bhyve, then, a openvswitch and a trunk between them, and another trunk to an external interface, and is all a long story.

I'm pretty sure I can keep the configuration and even delete the openvswitch. Let's see how it goes.