Double interface not work in Jails-Plugins

[gianfelicevincenzo]

I tried all possible combinations but it doesn't work. I have some plugins that must work on both network interfaces that I have in Truenas (the main ones: re0 and re1). How is it possible to enable both for a specific plugin?

 

[Volts]

Which interface is the main TrueNAS box using?

What's the goal of using multiple interfaces?

 

[gianfelicevincenzo]

I have two interfaces: re0 (My Personal LAN Switch) re1 (Router with internet). I have some plugins that I want to use on both interfaces.

 

[Patrick M. Hausen]

Roughly you need to:

• reboot your NAS with all jails turned off
• create an interface of type bridge named bridge0 with re0 as a member interface (Network > Interfaces > Add)
• create an interface of type bridge named bridge1 with re1 as a member interface
• test and save
• disable hardware offloading for both your physical interfaces
• if there is an IP address on re0 and/or re1, delete it and move it to the respective bridge interface instead
• test and save

Then for your jails configure:

• vnet_default_interface: none
• interfaces: vnet0:bridge0,vnet1:bridge1
• configure IP addresses for both interfaces

Done.

HTH,
Patrick

 

[gianfelicevincenzo]

disabling offloading I've read may reduce performance. Why disable it?

 

[Patrick M. Hausen]

Because it only speeds up TCP connections terminating in the hosts IP stack. But it interferes with packets passing through the host, e.g. when using the host as a bridge or a router.

It won't speed up anything ending in your jails, anyway.

And lastly it's a micro optimization not really worth the trouble unless you have a big highly loaded machine serving thousands of web pages or similar simultaneously. Not having hardware offloading for TCP simply means the main CPU needs to do the processing. Any contemporary reasonable CPU can easily saturate a Gigabit/s. 5-10% increase in CPU usage under load, if noticeable at all.

HTH,
Patrick

 

[gianfelicevincenzo]

I understand. So, is this the only solution? After all, even packets passing "through" always end up on physical interfaces (and I thought offloading might still help...or "not interfere" anyway...at least)

 

[ChrisRJ]

re0 and re1 indicate that we are talking about Realtek NICs. If so, there may issues caused by that in addition to other aspects.

 

[Patrick M. Hausen]

Offloading does interfere and packets destined for a jail don't end on a physical interface but only pass through one. They end in the jail's virtual epair interface and the jail's private IP stack. VNET is network virtualisation.

 

[Volts]

Offload is silently disabled behind the scenes when there's a bridge anyway.

In the screenshot there appears to be also be a VLAN interface. Is that correct? If so, adjust @Patrick M. Hausen's instructions slightly - vlan1 should be the member of the bridge, not re0.

But I ask again about the goal. What's the reason for having multiple interfaces?

If it's an attempt to optimize things, I would encourage you to start more simply. Multihoming adds a bunch of complexity and some risk.

 

[Volts]

You might enjoy this presentation from the Netflix guys about how they get absolutely *bonkers* performance out of FreeBSD hosts. In their scenario LRO is meaningful (timestamp 20:00). He explains what these functions do, why they help. And also some of the problems they can cause. The good news is you aren't Netflix!

The “other” FreeBSD optimizations used by Netflix to serve video at 800Gb/s from a single server :: FreeBSD Presentations and Papers

papers.freebsd.org

 

[Patrick M. Hausen]

Offload is silently disabled behind the scenes when there's a bridge anyway.

Since when? Kristof's refactoring of if_bridge(4)?

 

[Volts]

The behavior has changed over time, but it was introduced before the big refactor. I noticed it a long time ago when comparing the output of ifconfig before and after adding member interfaces to a bridge.

LRO is simply disabled. The other capabilities can be re-enabled if all bridge members support them - which if_epair doesn’t, obviously.

Editing, because if_bridge(4) documents this nicely:

The TOE, TSO, TXCSUM and TXCSUM6 capabilities on all interfaces added to the bridge are disabled if any of the interfaces doesn't support/enable them. The LRO capability is always disabled. All the capabilities are restored when the interface is removed from bridge.

freebsd-src/sys/net/if_bridge.c at main · freebsd/freebsd-src

The FreeBSD src tree publish-only repository. Experimenting with 'simple' pull requests.... - freebsd/freebsd-src

github.com

Code:

/*
 * List of capabilities to possibly mask on the member interface.
 */
#define    BRIDGE_IFCAPS_MASK        (IFCAP_TOE|IFCAP_TSO|IFCAP_TXCSUM)

/*
 * List of capabilities to strip
 */
#define    BRIDGE_IFCAPS_STRIP        IFCAP_LRO

 

[Patrick M. Hausen]

Thanks! Then for some reason this does not always work reliably or did not at some point in time. I have a datacentre full of bridged jails and extensive documentation how to configure each type of phys. interface to disable everything that would lead to unexplainable drops of connectivity for the jails.

Hope I can make it to next year's EuroBSDCon or a dev summit nearby. Way past time to chat with Kristof over a beer again.

 

[Volts]

That tickles some cold neurons for me ... some hardware+drivers renegotiate the link when capabilities are changed ... I've had switches that got fussy when they thought that was happening "too frequently". Or maybe it was a particular motherboard, I don't recall. For those it was very helpful to set the interface capabilities before bringing it up, and to avoid multiple rapid transitions when joining the bridge.

It's interesting looking at the small ongoing changes to if_bridge.c. Kinda what I love about FreeBSD. Fine grit, keep polishing.